簡(jiǎn)潔大方標(biāo)準(zhǔn)規(guī)定模板課件

2011年11月2011年11月1MonitoringandEvaluatingODAThispresentationwillcoverWhymonitor?Whyevaluate?MonitoringintheWorldBank,othermulti-lateraldevelopmentbanks,bilateraldonorsEvaluationintheWorldBank,othermulti-laterals,bilateraldonorsMonitoringandEvaluatingODAT2Whatismonitoring?Definitionofmonitoring:acontinuousprocessthatusessystematiccollectionofdataaboutprogressinanon-goingdevelopmentproject,forusebymanagersandother

stake-holdersWhatismonitoring?Definitio3Whymonitor?Toimproveimplementationofon-goingprojects;mustprovidepracticalinformationToassesswhetherdevelopmentobjectiveslikelytobemetToidentifysystematicissuesandrisksacrossprojects,bycountry,bysector,andoverallToprovideinputforexpostevaluationWhymonitor?Toimproveimpleme4WhatisEvaluation?Definitionof(expost)evaluation:assessmentofacompletedproject,program,orpolicy,itsdesign,implementation,andresults.WhatisEvaluation?Definition5Whyevaluateandwhocares?Why?Tworeasons:TolearnfromexperienceToprovideaccountabilityWhouses(orshoulduse)evaluations:BoardsofDirectors(MDBs)andMinistersandDirectorsGeneralforoversight;ODAmanagersforpoliciesandstrategies;ODAstaffforprogramandprojectdesigns,waysofdoingbusiness;Clientcountries,externalpartners,andthepublicforpolicies,aligningprograms,researchWhyevaluateandwhocares?Why6M&EincontextWorldBankcommits/disburses$20-23billion/year;Approves~280loansandcredits/year;activeloanportfolio:~1,400in>120countriesBankalsoprovidesgrantsandothernon-lendingservices(diagnosticwork)Bankhastotaladministrativebudgetof$2billionMostotherMDBscommit$1–6billion/yearBilateralODA:$200million(Greece,Luxembourg,NewZealand,Portugal)-$25billion/year(USA)Bilateralagencieshaveadministrativebudgetsof$11million(Luxembourg)to>$700million(Japan)M&EincontextWorldBankcom7MonitoringintheWorldBankEveryon-goinglendingoperationismonitoredbyvisitsfromBankstaffabouttwiceayear;Implementationstatusandresults(ISR)reportissuedtwiceeachyear:ProgressonimplementationProgressinmeetingdevelopmentobjectivesProblemprojectsidentifiedPortfolioatriskreports–separatemethodology–updatedmonthlyOtherqualitycontrolactivitiesincludequalityatentryreviews;qualityofsupervisionMonitoringintheWorldBankEv8Evaluation:thetopicsSelfevaluationIndependentevaluation:whyitmattersTools:projects;sectors;countries;etc.Coverage:howmanyevaluationsResources/budgetMainchallengeforevaluationDisclosureOtherEvaluationIssuesEvaluation:thetopicsSelfev9Self-EvaluationintheBankSelfevaluationintheBank:Everyprojecthasanimplementationcompletionreport,donebystaffofresponsibleunit(about280/year)Everynewcountryassistancestrategynowhascompletionreportcoveringpreviouscountryassistancestrategy,donebystaffofresponsibleunit(about20/year)Self-EvaluationintheBankSel10Self-evaluationinotherdonorsMulti-lateralinstitutions:almostallMDBshaverequirementforself-evaluation,althoughinsomeMDBs(AsianDB,EBRD,EIB),theevaluationdepartmentisinvolvedBilateraldonors:somerequireforprojectsovergivenvalue,othersdon’t;SDC(Swiss):externalevaluatorsdoself-evaluation;GTZ(German)hasinnovativeeVAL(electronic)system.Self-evaluationinotherdonor11IndependentevaluationWhatisindependence?Theextenttowhichtheevaluationunit:ReportsdirectlytoaBoardorothergoverningunit;Hasstaff,budget,andfunctionsindependentfrommanagement;notinvolvedwithmanagementofODABehavesindependentlybyissuingobjective,impartial,andbalancedreportsIndependentevaluationWhatis12IndependentevaluationIntheWorldBank,independentevaluation,since1973,haslookedlikethis:BoardofDirectorsInstitutionalIntegrityPresidentPaulWolfowitzIndependentEvaluationGroupInternalAuditingSVP&ChiefEconMDOperationsCFOSVP&GenCounselInspectionPanelMDOperationsIndependentevaluationIntheW13IndependentevaluationInbilateraldonors,independentevaluationhasevolvedtolooklikethis:MinistryofForeignAffairs

EvaluationUnitDirectorGeneralAidagencyOperationalunitOperationalunitOperationalunitIndependentevaluationInbilat14IndependentevaluationWhyindependencematters:Allowsforobjective,unbiasedevaluationsEvaluationfindingsarenottailoredtomanagement’swishesIndependencelendscredibilitytofindingsIndependencedifficulttoachieveandmaintainIndependentevaluationWhyinde15EvaluationTools:WorldBankProjectevaluationsImplementationCompletionReport(ICR)reviewsProjectperformanceassessmentreviews(PPARs)CountryevaluationsCASCompletionReport(CASCR)ReviewsCountryassistanceevaluations(CAEs)Sector,thematic,globalevaluationsProcessevaluationsCorporateevaluations(ARDE,AROE)EvaluationTools:WorldBankP16EvaluationTools:interactionsSynergiesamongdifferentevaluationtoolscanprovidevaluableinputGlobal(Globalprograms)EvaluationtoolsintheWorldBankCorporate(ARDE)Sector(finance,education)Thematic(gender,decentralization)Process(PRSP)CountryAssistanceEvaluation

CASCRReview

CASCRReviewICRReview(validationofself-evaluation)ProjectPerformanceAssessmentReviewCountryAssistanceEvaluationICRReview(validationofself-evaluation)ProjectPerformanceAssessmentReviewProjectlevelCountrylevelSector,thematic,policylevelGlobalandcorporatelevelEvaluationTools:interactions17EvaluationCoverage:WorldBankEvaluationCoverage:WorldBa18EvaluationcoverageInmostMDBs,fullrangeofevaluations:project,country,sector,thematicevaluationsInmostbilaterals,fewprojectevaluations;somecountrylevel;mostfocusonthemesorprogramsMostbilateralsproduces3-10evaluations/yearDFIDproduces20-25reports/yr;PotentialweaknessofcountryandthematicevaluationswithoutunderpinningsofprojectevaluationstouseasinputEvaluationcoverageInmostMDB19Resourcesforevaluation:MDBsResourcesvary,butaspercentoftotaladministrativebudget,fairlysimilar:

Resourcesforevaluation:MDB20Resourcesforevaluation:bilateralsBilateraldonoragenciesaremorelikelytohireexternalconsultantsforvirtuallyallevaluations(UK,Norway,Denmark,Switzerland,Australia)Basedonnumbersfromafewbilaterals(DFID,NORAD,SDC),evaluationbudgetis~2%oftotaladministrativebudget;figuresof0.5–2%havebeencitedbyothers.Smallerdonors(Greece,NewZealand)arestillsettinguptheirevaluationunitsResourcesforevaluation:bila21MainchallengeforevaluationHowtomakeevaluationsusefulMostmanagers,stafffocusedonfuture,notonpastEvaluationmust/shouldbe:TimelyOperationallyrelevant:avoidgeneralizations,platitudesEasilyaccessible:avoidlongreportsConstructive:goodpracticeshouldberewardedWidelydisseminatedanddisclosedMainchallengeforevaluationH22DisclosureofevaluationreportsWorldBank:publicdisclosureofall:PPARsCAEsSector,thematic,globalevaluationsProcessevaluationsCorporateevaluationsEBRD,ADB:selectivepublicdisclosure:evaluationsinvolveprivatesectorBilaterals:majoritydisclose,butvaries:UK,Norway–fulldisclosure;Germany–selectivedisclosureDisclosureofevaluationrepor23DisclosureForWorldBank:reportsandinformationonevaluationtools,approaches,onwebsite:andscrolldowntoright-hadsideofhomepage,or:/iegForotherdonors:checktheirwebsitesDisclosureForWorldBank:rep24OtherissuesinevaluationEvaluationapproachUseofperformanceindicatorsSourcesofinformationRatingsProcessesPartnershipsOtherchallengesOtherissuesinevaluationEval25EvaluationApproachWithincontextofresults-basedmanagement,Bankusesobjectives-basedapproach,focusonoutcomesandimpactBankloans;grants;diagnosticwork;dialoguePhysicaloutputs;privatization;newlaws;agencycreated;trainingcompleted;sub-loansdisbursed.Moreefficientoperations;betterfiscalmanagement;employment

creation;betteraccesstoservices;InputsOutputsExogenousfactorsExogenousfactorsOutcomeImpactLonger-termobjective,likepovertyreduction,bettereducational,healthoutcomesEvaluationApproachWithincont26EvaluationApproachBankassesses:Outcome(relevance,efficacy,efficiency)Useofperformanceindicatorsforoutputs,outcomes,impactRisktoDevelopmentBankandBorrowerPerformanceEvaluationApproachBankassess27SourcesofinformationforevaluationBaselinedatainprojectappraisaldocuments(sometimes)MonitoringreportsshouldhaveinformationonperformanceindicatorsDatabasesforprojectarea;sector;countryatbeginningofproject,endofprojectOthercountry-levelindeces(IMFfinancialstatistics,DoingBusinesssurveys;WorldBankIndicatorsofcorruptionsurveys;UNDPhumandevelopmentindicators)SurveysInterviews,focusgroups Sourcesofinformationforeva28UseofRatingsWorldBankusesratingsonprojectoutcomesandoutcomesofcountryassistance.Noratingsonsector,thematic,otherevaluationsOtherMDBsuseratingsforprojectoutcomes;notforcountryassistanceBilaterals:someuseratings,onlyforprojects;somedonotuseanyratingsUseofRatingsWorldBankuses29EvaluationApproach:ratingsProjectRatingsinWorldBank:Outcomes:6pointscale(HS-HU)RiskstoDevelopment:4pointscale(H-N)BankandBorrowerperformance:4pointscale(HStoHU)ProsandconsofratingPro:providesdataandaccountabilityCon:takesfocusawayfromlessonlearningandrichnessofevaluationEvaluationApproach:ratingsP30ProcessesIndependencedoesnotmeanisolationMDBsandbilateraldonorsincorporatecommentsfromoperationalmanagersandstaffandfromstakeholdersinclientcountriesintoevaluationreports:Bank’sinitiatingpapersforcountry,thematic,globalevaluationssenttomanagementforcommentandtoBoardofDirectorsforinformation;IMF’sinitiatingpaperforallevaluationspostedonIMFwebsiteandpubliccommentinvited;wideconsultationsthroughoutprocess.ProcessesIndependencedoesnot31PartnershipsinevaluationCountryandsectorevaluationssometimesinvolvepartnershipswithotherMDBsand/orbilateraldonors(jointorparallelevaluations)Opportunistic,notsystematic,partnershipsProsandconsofevaluationpartnershipsPros:widercoverage,otherviewpointsCons:morecostlyandmoretimerequired;maynotagreeonoutcomes;mayaffecttimelinessoffinalreportPartnershipsinevaluationCoun32OtherchallengesDatadifficulttoobtain,ordifferentsourceshaveconflictingdataOftendifficultorimpossibletohaveacontrolgroup(especiallyinsectorandcountryevaluations)Beneficiarysurveyscanbecostly,lowresponserateProblemofattribution(cause-effect)difficultorimpossibletoestablishOtherchallengesDatadifficult332011年11月2011年11月34MonitoringandEvaluatingODAThispresentationwillcoverWhymonitor?Whyevaluate?MonitoringintheWorldBank,othermulti-lateraldevelopmentbanks,bilateraldonorsEvaluationintheWorldBank,othermulti-laterals,bilateraldonorsMonitoringandEvaluatingODAT35Whatismonitoring?Definitionofmonitoring:acontinuousprocessthatusessystematiccollectionofdataaboutprogressinanon-goingdevelopmentproject,forusebymanagersandother

stake-holdersWhatismonitoring?Definitio36Whymonitor?Toimproveimplementationofon-goingprojects;mustprovidepracticalinformationToassesswhetherdevelopmentobjectiveslikelytobemetToidentifysystematicissuesandrisksacrossprojects,bycountry,bysector,andoverallToprovideinputforexpostevaluationWhymonitor?Toimproveimpleme37WhatisEvaluation?Definitionof(expost)evaluation:assessmentofacompletedproject,program,orpolicy,itsdesign,implementation,andresults.WhatisEvaluation?Definition38Whyevaluateandwhocares?Why?Tworeasons:TolearnfromexperienceToprovideaccountabilityWhouses(orshoulduse)evaluations:BoardsofDirectors(MDBs)andMinistersandDirectorsGeneralforoversight;ODAmanagersforpoliciesandstrategies;ODAstaffforprogramandprojectdesigns,waysofdoingbusiness;Clientcountries,externalpartners,andthepublicforpolicies,aligningprograms,researchWhyevaluateandwhocares?Why39M&EincontextWorldBankcommits/disburses$20-23billion/year;Approves~280loansandcredits/year;activeloanportfolio:~1,400in>120countriesBankalsoprovidesgrantsandothernon-lendingservices(diagnosticwork)Bankhastotaladministrativebudgetof$2billionMostotherMDBscommit$1–6billion/yearBilateralODA:$200million(Greece,Luxembourg,NewZealand,Portugal)-$25billion/year(USA)Bilateralagencieshaveadministrativebudgetsof$11million(Luxembourg)to>$700million(Japan)M&EincontextWorldBankcom40MonitoringintheWorldBankEveryon-goinglendingoperationismonitoredbyvisitsfromBankstaffabouttwiceayear;Implementationstatusandresults(ISR)reportissuedtwiceeachyear:ProgressonimplementationProgressinmeetingdevelopmentobjectivesProblemprojectsidentifiedPortfolioatriskreports–separatemethodology–updatedmonthlyOtherqualitycontrolactivitiesincludequalityatentryreviews;qualityofsupervisionMonitoringintheWorldBankEv41Evaluation:thetopicsSelfevaluationIndependentevaluation:whyitmattersTools:projects;sectors;countries;etc.Coverage:howmanyevaluationsResources/budgetMainchallengeforevaluationDisclosureOtherEvaluationIssuesEvaluation:thetopicsSelfev42Self-EvaluationintheBankSelfevaluationintheBank:Everyprojecthasanimplementationcompletionreport,donebystaffofresponsibleunit(about280/year)Everynewcountryassistancestrategynowhascompletionreportcoveringpreviouscountryassistancestrategy,donebystaffofresponsibleunit(about20/year)Self-EvaluationintheBankSel43Self-evaluationinotherdonorsMulti-lateralinstitutions:almostallMDBshaverequirementforself-evaluation,althoughinsomeMDBs(AsianDB,EBRD,EIB),theevaluationdepartmentisinvolvedBilateraldonors:somerequireforprojectsovergivenvalue,othersdon’t;SDC(Swiss):externalevaluatorsdoself-evaluation;GTZ(German)hasinnovativeeVAL(electronic)system.Self-evaluationinotherdonor44IndependentevaluationWhatisindependence?Theextenttowhichtheevaluationunit:ReportsdirectlytoaBoardorothergoverningunit;Hasstaff,budget,andfunctionsindependentfrommanagement;notinvolvedwithmanagementofODABehavesindependentlybyissuingobjective,impartial,andbalancedreportsIndependentevaluationWhatis45IndependentevaluationIntheWorldBank,independentevaluation,since1973,haslookedlikethis:BoardofDirectorsInstitutionalIntegrityPresidentPaulWolfowitzIndependentEvaluationGroupInternalAuditingSVP&ChiefEconMDOperationsCFOSVP&GenCounselInspectionPanelMDOperationsIndependentevaluationIntheW46IndependentevaluationInbilateraldonors,independentevaluationhasevolvedtolooklikethis:MinistryofForeignAffairs

EvaluationUnitDirectorGeneralAidagencyOperationalunitOperationalunitOperationalunitIndependentevaluationInbilat47IndependentevaluationWhyindependencematters:Allowsforobjective,unbiasedevaluationsEvaluationfindingsarenottailoredtomanagement’swishesIndependencelendscredibilitytofindingsIndependencedifficulttoachieveandmaintainIndependentevaluationWhyinde48EvaluationTools:WorldBankProjectevaluationsImplementationCompletionReport(ICR)reviewsProjectperformanceassessmentreviews(PPARs)CountryevaluationsCASCompletionReport(CASCR)ReviewsCountryassistanceevaluations(CAEs)Sector,thematic,globalevaluationsProcessevaluationsCorporateevaluations(ARDE,AROE)EvaluationTools:WorldBankP49EvaluationTools:interactionsSynergiesamongdifferentevaluationtoolscanprovidevaluableinputGlobal(Globalprograms)EvaluationtoolsintheWorldBankCorporate(ARDE)Sector(finance,education)Thematic(gender,decentralization)Process(PRSP)CountryAssistanceEvaluation

CASCRReview

CASCRReviewICRReview(validationofself-evaluation)ProjectPerformanceAssessmentReviewCountryAssistanceEvaluationICRReview(validationofself-evaluation)ProjectPerformanceAssessmentReviewProjectlevelCountrylevelSector,thematic,policylevelGlobalandcorporatelevelEvaluationTools:interactions50EvaluationCoverage:WorldBankEvaluationCoverage:WorldBa51EvaluationcoverageInmostMDBs,fullrangeofevaluations:project,country,sector,thematicevaluationsInmostbilaterals,fewprojectevaluations;somecountrylevel;mostfocusonthemesorprogramsMostbilateralsproduces3-10evaluations/yearDFIDproduces20-25reports/yr;PotentialweaknessofcountryandthematicevaluationswithoutunderpinningsofprojectevaluationstouseasinputEvaluationcoverageInmostMDB52Resourcesforevaluation:MDBsResourcesvary,butaspercentoftotaladministrativebudget,fairlysimilar:

Resourcesforevaluation:MDB53Resourcesforevaluation:bilateralsBilateraldonoragenciesaremorelikelytohireexternalconsultantsforvirtuallyallevaluations(UK,Norway,Denmark,Switzerland,Australia)Basedonnumbersfromafewbilaterals(DFID,NORAD,SDC),evaluationbudgetis~2%oftotaladministrativebudget;figuresof0.5–2%havebeencitedbyothers.Smallerdonors(Greece,NewZealand)arestillsettinguptheirevaluationunitsResourcesforevaluation:bila54MainchallengeforevaluationHowtomakeevaluationsusefulMostmanagers,stafffocusedonfuture,notonpastEvaluationmust/shouldbe:TimelyOperationallyrelevant:avoidgeneralizations,platitudesEasilyaccessible:avoidlongreportsConstructive:goodpracticeshouldberewardedWidelydisseminatedanddisclosedMainchallengeforevaluationH55DisclosureofevaluationreportsWorldBank:publicdisclosureofall:PPARsCAEsSector,thematic,globalevaluationsProcessevaluationsCorporateevaluationsEBRD,ADB:selectivepublicdisclosure:evaluationsinvolveprivatesectorBilaterals:majoritydisclose,butvaries:UK,Norway–fulldisclosure;Germany–selectivedisclosureDisclosureofevaluationrepor56DisclosureForWorldBank:reportsandinformationonevaluationtools,approaches,onwebsite:andscrolldowntoright-hadsideofhomepage,or:/iegForotherdonors:checktheirwebsitesDisclosureForWorldBank:rep57OtherissuesinevaluationEvaluationapproachUseofperformanceindicatorsSourcesofinformationRatingsProcessesPartnershipsOtherchallengesOtherissuesinevaluationEval58EvaluationApproachWithincontextofresults-basedmanagement,Bankusesobjectives-basedapproach,focusonoutcomesandimpactBankloans;grants;diagnosticwork;dialoguePhysicaloutputs;privatization;newlaws;agencycreated;trainingcompleted;sub-loansdisbursed.Moreefficientoperations;betterfiscalmanagement;employment

creation;betteraccesstoservices;InputsOutputsExogenousfactorsExogenousfactorsOutcomeImpactLonger-termobjective,likepovertyreduction,bettereducational,healthoutcomesEvaluationApproachWithincont59EvaluationApproachBankassesses:Outcome(relevance,efficacy,efficiency)Useofperformanceindicatorsforoutputs,outcomes,impactRisktoDevelopmentBankandBorrowerPerformanceEvaluationApproachBankassess60SourcesofinformationforevaluationBaselinedatainprojectappraisaldocuments(sometimes)MonitoringreportsshouldhaveinformationonperformanceindicatorsDatabasesforproject