外文資料--Evaluation of an Authentic Examination System (AES).pdf

EvaluationofanAuthenticExaminationSystem(AES)forProgrammingCoursesTorbjrnJonsson,PouriaLoghmaniandSiminNadjm-TehraniDepartmentofComputerandInformationScienceLinkpingUniversity,Swedentorjo,poulo,siminida.liu.seAbstractThispaperdescribesourexperiencewithanauthenticexaminationsystemforprogrammingcourses.Webrieflydescribethearchitectureofthesystem,andpresentresultsofevaluatingthesysteminrealexaminationsituations.Someofthefactorsstudiedindetailaretheon-lineinteractionsbetweenthestudentsandexaminers,theresponsetimesandtheireffectsonthepressureexperiencedbystudent,theacceptanceofthemethodamongthestudents,andwhethertheexaminationformisgender-neutral.IntroductionAsexperiencedteachersinprogrammingcourseswehavenoticedthedrawbacksinthetraditionalexaminationformusedinprogrammingcourses.Thestudentslearntoprogramvialaboratoryexercises,butthefinalevaluationoftheirabilitiesandthegradingoftheexaminationareinaformthatusespaperandpeninsteadofcomputers.Consideringthatthestudentwillneverusethismodeforproducingaprogramthroughtheprofessionallife,weconsiderthistobenotasuitablemethod.AttheDepartmentofComputerScienceatLinkpingUniversity12fundamentalprogrammingcoursesforapproximately1000studentsindifferenteducationalprogramsaretaughtannually.Thispaperdealswithanewpedagogicalviewintheseprogrammingcourses,whichcanbeappliedtoanyprogramminglanguage,typeofstudentandeducationalprogram.Theideaisbasedonextensivestudiesarounddifferentexaminationforms,whereindividualgrading,efficientandusefulfeedbackandtheauthenticityoftheexaminationformareusedasbasiccriteriaforthechoiceofexaminationmethod.Webelievethatthechoiceofmethodtogetherwiththeaddedefficiencyintheassessmentprocessimprovesthequalityofourstudyprogrammes.Inparticular,webelievethatitwillchangetheexaminationprocessfromasummativetoanormativeassessmentoccasion1.Foranumberofyearswehaveexperimentedwithtestingthestudentsviacomputer-aidedexaminationsinsomepilotcoursesanauthenticexaminationformforthistypeofcourse.However,thisexaminationformhasnotbecomemorewidespreadduetoinsufficientsupportforthecomputerenvironmentnecessaryforthiskindofexamination.Duringthepastyearanewauthenticexaminationsystem(AES)hasbeendeveloped,whereallthestudentsandtheexaminingteachersareconnectedtothesamesystem.Theprocess,includingcommunicationandgrading,issupportedbythisenvironment.Inthispaperwedescribetheexaminationsystemandourinitialevaluationsofthissysteminanumberofrelativelylargeexaminationsessions.ThecoursesinquestioncoveredprogramminginAdaandweretakenbyfirstandsecondyearstudents.DuringthepastyearwehaveevaluatedtheAES.Theinstrumentsusedfortheevaluationconsistedofquestionnairesfilledby231studentsoveraperiodof3monthsand4examinations.Thepaperisorganisedasfollows.Insection1wedescribewhythetypeofexaminationweproposeisthemostappropriateforprogrammingcoursesandcomparetosomerelatedsystems.Section2includesabrieftechnicaldescriptionoftheexaminationsystems,includingitsarchitecturaldesign.Insection3wedescribehowthecomputersystem,thatmanagestheexaminationprocesson-line,hastobeaugmentedbyrulessetupineachparticularcourse.Section4coversourevaluationmethodsandisfollowedbyevaluationresultsinsection5.Section6concludesthepaper.1ExaminationformsEveryexaminationmethodhasspecificcharacteristicsthatmakeitmoreorlessappropriatetoaparticularcoursesetting.HkanOswaldssonstudiedtherangeofpossibleexaminationformsforatypicalprogrammingcoursepriortothedevelopmentofthecurrentexaminationsysteminourdepartment5.Whileseveralmodesofexaminationcanbeconsideredaseffectivemeansforenhancedlearning(e.g.homeassignments,oralexaminationsfollowingadesignassignment,etc),therearenotmanyexaminationtypesthatcombinetheneedforasummativeassessment,withadequatefeedbacktoinducelearning.Combinedwiththelargenumberofstudentsthatwearecurrentlyteaching,designofanidealexaminationsettingisatrulychallengingtask.TheworkbyDawson-Howeisanearlyattempttobringcomputersupportintotheprocessofprogrammingassignmentevaluationandadministration2.Theneedforautomatedexaminationsystemshasbecomemorepertinentduringthelate90swiththeadventofdistanceandlifelonglearning.Forexample,attheOpenUniversityinUKtherehavebeenattemptstoexchangestudentassignments,andtheir(subsequent)correctionandassessmentbyexaminersviaMSWorddocuments8.However,theavailablereports(e.g.theworkbyPriceandPetre)concentrateontheeaseofadministrationforcourseassignmentandgrading,ratherthanthepedagogicalfeedbackinanon-lineauthenticexamination.Inrecentyearsseveralauthorsreportonautomaticassessmentsystems,mostlyconcentratingonpresentationofthetechnicalaspectsofthesystemandtheresultsofthestudentsintermsofgrading4,5,7,8.Whilewesharetheaspirationoftheseresearchteamsandconductsimilarstudies,ourfocushasbeenontheformalevaluationofhowthestudentsperceivedtheexaminationenvironment.Inadditionwehavestudiedhowtheywereaffectedbyfactorsspecifictoauthenticexaminations,howthesystemperformanceandtheexaminerson-linebehaviouraffectstheperceivedloadonthestudent,andothersuchaspects.2TechnicaldescriptionoftheAESAEShasbeendevelopedusingtheJ2EEplatform.Thisrepresentsasinglestandardforimplementinganddeployingcomplexenterpriseapplications.Havingbeendesignedthroughanopenprocess,J2EEmeetsawiderangeofenterpriseapplicationrequirements,includingdistribution-specificmechanismssuchasmessagingsystem,scalabilityandmodularity.TheclientsarebasedontheModel-View-Controller(MVC)applicationarchitecture,whichseparatesthreedistinctformsoffunctionalitywithintheapplication:TheModelrepresentsthestructureofthedataintheapplication,aswellasapplication-specificoperationondata.TheViewaccessesdatafromthemodelandspecifieshowthatdatashouldbepresented.ViewsintheAESconsistofstand-aloneapplicationsthatprovideviewfunctionality.TheControllertranslatesuseractionsonthemodelandselectstheappropriateviewbasedonuserpreferences.TheAESisdesignedasasetoflooselycoupledmodules,whicharetightlycoupledinternally.Groupingfunctionalityintomodulesprovidesintegrationbetweenclassesthatcooperate,yetdecouplesclassesthatrefertoeachotheroccasionally.Modulardesignsupportsthedesigngoalthatsoftwarewillbereusable.Eachmodulehasaninterfacethatdefinesthemodulesfunctionalrequirementsandprovidesaplacewherelatercomponentsmaybeintegrated.TheAESincludesmodulesfor:StudentaccountsTeacheraccountsExamsExaminationProcessingMessagingStatisticsTheAESdesignisdividedintomultipletiers:theClienttier,theMiddletier(consistingofoneormoresub-tiers),andtheBackendtier(seefigure2.1).Partitioningthedesignintotiersallowsustochoosetheappropriatetechnologyforagivensituation.Multipletechnologiescanevenbeusedtoprovidethesameserviceindifferentsituations.Forexample,HTMLpages,JSPpages,andstand-aloneapplicationscanallbeusedintheclienttier.Eachofthethreetiersplaysaspecificroleinthedesign.TheClienttierisresponsibleforpresentingdatatotheuser,interactingwiththeuser,andcommunicatingwiththeothertiersofthesystem.InthiscasetheClienttieristheonlypartofthesystemvisibletotheuser.TheAESClienttierconsistsmainlyofastand-aloneapplicationthatcommunicateswiththeothertiersthroughwell-definedinterfaces.Amessage-orientedapproachbasedonJMS(JavaMessagingSystem)hasbeenchosentotakecareofthecommunicationbetweentheClienttierandtheMiddletier.TheMiddletierisresponsibleforanyprocessinginvolvingEnterpriseJavaBeans.EnterpriseJavaBeansaresoftwarecomponentsthatextendserverstoperformapplicationspecificfunctionality.TheinterfacebetweenthesecomponentsandtheircontainersisdefinedintheEnterpriseJavaBeansspecification.ThecontainersprovideservicestotheEnterpriseJavaBeansinstancestheycontain,suchascontrollingtransactions,managingsecurity,threadorotherresourcepooling,andhandlingpersistence,amongotherhigh-levelsystemtasks.TheBackendtieristhesysteminformationinfrastructure.Thistierincludesoneormorerelationaldatabasemanagementsystemsandpotentiallyotherinformationassetsthatcouldbeuseful,e.g.thecentraluniversitycourseresultsadministrationsystem(LADOK).TheEIStieralsoenforcessecurityandoffersscalability.TheBackendtierprovidesalayerofsoftwarethatmapsexistingdataandMiddleTierJNDIJavaNamingandDirectoryInterfaceBackendTierClientTierEJBContainerEnterpriseBeansMessageDrivenBeansJMSJavaMessagingServiceClientStandaloneSwingapplicationRDBMSFigure2.1:TheAESdesign.applicationresourcesintothedesignofAESinanimplementation-neutralway.Thesystemisseparatedintofivedifferentfunctionallayers,eachwithitsownresponsibilitiesanditsownAPI.Theselayersarephysicallysplitacrossthethreedifferenttiers.Thepersistencelayer,forexample,providesthemechanismsnecessarytopermanentlysaveobjectstate.ItprovidesbasicCRUD(create,read,update,delete)servicesandalsodealswiththeobject-to-relationalmappingissues.Thisleadstoamoreflexibleandmaintainablesystem,e.g.layerscanbechangedwithnoeffectonotherlayers,aslongastheAPIremainsconstant.3Examinationset-upTheexaminationsystemisonlyonepartoftheexaminationprocess.Thesecondpartistheset-up(therules)wehaveforthestudents.Wehavetriedafewset-upsoveranumberofyears(usingaprototypeforthesystemfor5-6years).3.1Thefirstset-upThefirstversionallowedthestudentstowritetheprogramsusingacomputerinsteadofwritingonpaper.Wefoundthismethodtobeanimprovementbecausewedidnothavetoread“illegible”textsandthesubmittedsolutionscouldbetestedafterwards.Gradeswerebasedonthenumberofcorrectlysolvedexercises.Aproblemwiththisset-upwasthatallthegradingstillhadtobedoneaftertheexamwasfinished.Mostofthestudentswaitedtosendinthesolutionsuntilthelastminuteoftheexam.3.2Thesecondset-upOurintentionwastohaveanexaminationwherethestudentsshouldhavearesponsefromtheexaminer(s)withinafewminutesandwheregradesweregiventothestudentswhentheylefttheexam.Wealsointendedtoprovidethestudentwiththepossibilityofgettingaresponseforeachexercisewithinafewminutes,sotheycouldcorrectanearlycorrectsolution.Thesecondset-up(whichweusetoday)isbasedonbothnumberofcorrectlysolvedexercisesandtheamountoftimetakentosolvethem.Anumberofdeadlinesaregiven.Ifthestudentwantsahighgradehe/shehastosolveanumberofexerciseswithinapre-specifiedtimelimit.Thecurrentexaminationprocessfollowsafewsteps:1.Thestudentsendsanexaminationrequestforanexercisetotheexaminer(s).2.Theexaminerscanreturnoneofthefollowingresults.Passed-thesolutioniscorrect.Incomplete-thesolutionhaserrors,andmustbecorrected.Itspossibletomakeanewattemptlater.Fail-thesolutionisincorrectandthestudentisnotallowedtocontinuetoworkonthisexercise.3.Everyexaminationattemptandtheresultwillcontributetothefinalexamgrade,andthestudentisinformedofhis/hercurrentgrade.Ifthestudentsubmitsanewexaminationrequestonanadditionalexercisehe/shecanreachahighergrade.ThisexaminationprocessisbuiltintoourcurrentAES,buttherules(timelimitsetc.)canbechangedforseparatecourses.Thismakesthesystemflexible.TimelimitsandgradingInthecoursesthissystemwastestedtherewerethreeexercisesineachexamandtherequirementsfordifferentgradeswere:Forthegrade5(excellent)thestudentmustcomplete:o3exercisescorrectin3hoursoro2exercisescorrectin2hoursForthegrade4(verygood)thestudentmustcomplete:o2exercisescorrectin3hoursoro1exercisecorrectin1.5hoursForthegrade3(passed)thestudentmustcomplete:o1exercisecorrectin4hoursTheaboveset-uptogetherwiththeAESsupportgivesustheopportunitytogradethestudentsduringtheexam.Studentswhohavesolvedanexerciseareinformedofthegradetheyhavereached.Iftheyaresatisfiedwiththatgradetheycanleavetheexam(manystudentsleaveafteronetotwohourswhentheyhavegrade4or5).StudentquestionsInanordinarycomputer-aidedexam,anumberofquestionsaresubmittedbythestudents,wheretheanswercaneitherbeclassifiedaspersonalorasinterestingforallstudents.Theexaminercandecideifhe/shewillsendtheanswertothewholegroupofstudentsorjusttoaspecificstudent.Thenumberofquestionsseemstoberelativelyconstantduringtheexam(approximately2-5questionsper5minutes).Mostquestionsaresentinduringthebeginningoftheexam,whichcanbeexplainedbythefactthatthestudentsaskaboutspecificthingspertainingtotheexercisesandthattherearemorestudentsinthebeginningoftheexam.Submission/approvalattemptsInanordinarycomputer-aidedexamwehavealargenumberofexaminationrequestsfromthestudents.Aswecanseeinfigure3.1wehavearelativelyhighfrequencyintheperiodfrom30minutesto3hours.Afterthat,mostofthestudentsleave(theycantgetahighergradethan3afterthattime).Aroundthedeadlineswecanseethattheexaminationattemptsappearmoreoften,butnotsignificantlymoreoften.Still,theincreaseofexaminationrequestsleadstomoreworkfortheexaminers.Thiscanresultinanincreaseintheresponsetime(waitingtimeforthestudent).4EvaluationmethodsThedevelopmentofthecurrentsystemstartedinsummer2001andcontinuedthroughwinter2001/2002.Whenwebegantestingthissystemwewantedasatestexampleacoursewithalargenumberofstudents.Oneofourintroductorycoursesinprogramminghasaround270studentseachyear,sothatwasourfirstchoice.Approximately180ofthesestudentsareIndustrialManagementEngineeringstudentsandtherestareTechnicalBiologystudents.Ourstatisticsarebasedontheirfirstexaminationinthiscourse,whichtookplaceinMarch2002.Wealsousedaretakeexaminthiscoursetodoanewstudywithanewsetofquestions.ThisevaluationwasdoneinMay2002.Inthesetwostudies,studentsfilledinquestionnairesdirectlyaftertheexam.Thefinalquestionnairehadtwoparts.Thefirstpartwasmainlyquestionswhereanswersareinfreetextformat.Thesecondpartincludedquestionswithscaledanswers(gradeontofive,disagree-agree,worse-better).Thefirstpartwasusedinthreeevaluations.Themoreextensivequestionnairewithtwopartswasusedonlyforthelastevaluation(i.e.forthetwolastexams).Theappendixshowsthefinalquestionnaire.Bothtypesofquestionnaireswereanonymousandthequestionnaireswerefilledinafterthegradingwasdonefortheexams.Thestudentshadalreadyreceivedtheirgradeswhentheyfilledinthequestionnaires.Webelievethatthisprovidesameasureofobjectivityonthestudentside.WealsousedthelogfilesfromtheAESfortheexamstogetstatisticaltrendsaboutgrades,gender,responsetimesforquestionsrespectivelyapprovalattemptsamongothers(seesection5).5EvaluationresultsUnfortunatelyalmostallstudentshadnopreviousexperiencewithpaperbasedprogrammingexaminations,sotherepliescouldnotbeusedforcomparisonswiththatexaminationform.However,weusedtheresponsetostudyotherquestionsindetail(speciallythepartrelatedtothetime/stressfactor).First,howoftenthestudentssentinarequest(questionsorapprovalattempts),andhowlongthetimeforaresponsewas?Secondly,howwellwastheexaminationsystemacceptedbythestudents?Athirdquestionwasacomparisonbygradesbetweenthegenders.Theresponserateofthequestionnaireswasquitegood.Wehadfourexamsduringtheevaluationperiodwiththefollowingresponserates:Exam1:76answersof112students(67.8%)Exam2:87answersof105students(82.8%)Exam3:50answersof66students(75.7%)Exam4:18answersof22students(81.8%)Thefirstthreequestionnairesweredoneatthefirstexaminationoccasionforthestudentsandthefourthonewasdoneinaretakeexaminationwhereallthestudentswerestudentswithnogradefr