ApilotstudyofvalueaddedanalysisforBeijingseniorsecondaryscnool

1、a pilot study of value added analysis for beijing senior secondary scnool abstract: data from seven beijing senior secondary schools was analyzed using multilevel analysis technique for school effectiveness in terms of value added measure and its implication for school administrationsimilar trends i

2、n our study were found with the international studiesstudents prior attainment can explain a large part of outcome variance，the model including students prior attainment，school context factors，student individual variables is optimal modelschool effects differ in term of subject，student group categor

3、ized by gender and other individual factorsthis strongly show the multi facet characteristics of the value added measure，and suggest a multiple，dynamic value added measure is essential for school evaluation we find different school effects in term of subject，this means value added measure is a usefu

4、l tool for school evaluation，but for implication in beijing educational practice，it need further researchthe research also provided an example about how to make good use of exam score for education administration and other educational practice key words: multilevel analysis technique; value added me

5、asure; exam score; school administrationintroduction there have been many studies about school effectiveness measurement based on value added analysis using multilevel statistics technique(thomas et al，l994，1996，1997，1999，sammons l997，1999，yang 1999，goldstein l996，1 997)generally, these study topics

6、 cover the definition of school effectiveness and improvement，the valueadded method and the application of multilevel analysis，and so on，mainly on the analysis of student academic attainmenthere is a brief review of these studiesschool effectiveness and valued added method as response to colemans re

7、port(coleman et al，l966)，which referred to the equality of educational opportunity and to jencks book(jencks et al., l972), which was on the inequality： a reassessment of the effect of family and schooling in america，and response to the publication of league table in uk(goldstein l996)，many research

8、es on schooling effect on student attainment emerged in us，uk，and other areasalthough the definition of school effectiveness have some discrepancies for different researchers(cheng 1996)，some characteristics of effective school such as professional leadership，maximization of learning time，purposeful

9、 teaching，high expectation，positive reinforcement，et al，are accepted by most people(sammons l 999)generally，measure of school effectiveness involves choosing an outcome and then studying average difference among schools after adjusting for relevant factors such as the intake achievements and some st

10、udent background characteristicsalthough the unadjusted results are informative to some extent for education evaluation，just an exam result does not show a schools progress completelyfurthermorea single statistic may not be an adequate summary of schools effect on students progressschool may boost t

11、he progress of different types of student at different rate in different departmentssubjects for fairer and more accurate way to measure and report school performance， the valued added method is accepted by many education researcher and practitioner now it is seen as both less problematic in terms o

12、f how acceptable principles are to policymaker and school senior managers，and less questioned even in the most sceptical of staff roomthis method has became the mainstream of british school effectiveness studies now there are various of defining value added and these encompass both qualitative judgm

13、ent and quantitative measures value added approaches also differ in the balance of emphasis placed on evaluating student outcomes directly，or indirectly, via the quality of the teaching and learning process.all definitions of value added have the common aim of assessing the quality and extent of a s

14、chools effectiveness in promoting student achievement. (sammons et al1997, p.24)thomas and collegues (1998)define the value added as an indication of the extent to which any given school has fostered the progress of all students in a range of subject during a particular time period or over particula

15、r years in comparison with the effects of other schools in the same samplethis definition is not only focusing on the academy progress，also on the students attitudes to learning and other important outcomes. however, this definition refers specially to academic attainment level of attainment in comp

16、arison with similar students in other schools for calculating the value added component，accurate baseline information about student's prior attainment is very important，so the value added methods can compares outcome after adjusting for varying intake achievementthe concept of valued added is no

17、t only an indicator of school effectiveness， also a tool for school self evaluationthe value added a school contributed to individual student has the following purpose(thomas l998)： it offers a fairer and more meaningful way of presenting school examination result； it is a tool that can provide both

18、 detailed and summary data that a school can analysis as part of its selfevaluation； it can be used to examine trends in value added performance over time. in relation to school improvement initiatives； it provide performance measures that can be contrasted against other types of data available in s

19、chools such as measures of pupils affective or vocational outcomes or information about the views of key groups obtained using teacher，parent and pupil questionnaires； it can provide additional guidance in monitoring and target setting for individual students and specific groups of studentssome resu

20、lts from previously studies comparison between school value added for overall attainment and subjects show schools vary differently across different area of academic attainmentthomas(1 996)found correlation between value added scores for english and mathematics is 0.46other research(thomas l995)foun

21、d these correlation between different subjects(english，english literature，math，science，french and history)range from 0.20 to 0.72，this means many schools have mixed results for different departmentsonly a few has the consistent results for different subjectsthese findings suggest strongly the need t

22、o look at schools value added performance in detail not only across different academic subjects as significant subject differences may be masked by overall attainment in some school studies show， on average， some different levels of academic performanceprogress between boys and girlsyangs(1999)resea

23、rch show girls perform better than boys in ks1 reading，ks1 writing，but worse in mathematics and sciences；thomas(1996)find girls perform better in total score and english languagegoldsteins(1996)research also shows a small different effect for boys in comparison to girlsstudents age seem show some ef

24、fect on theirs attainmenttwo studies show old students make more progress in their studies(thomas et al，l996，1997)，0n average，the oldest l5+pupils f with birthdays in septemberoctober)attained approximately 2 gcse points more than the youngest l5+pupils(with birthdays in julyaugust)(thomas l996，p12)

25、 studies on pupils social - economic status show students entitled with free meal，which is a comprehensive indicator of family economic class，seem perform worse than those without free meal on average(yang et、al，l999，thomas s，et al，l997，1999)these findings could be usefully interpreted as meaning th

26、at most schools have different levels of effectiveness for pupils from more or less economically advantaged backgroundpupils with different ethnic background also show different gains in academic attainment (nuttall et al., l989，thomas l996，sammons l999). in comparison to the white group，other ethni

27、c groups obtained significantly high scores，given the attainment on entry to school have been taken into account(thomas et al.，1996p12)but some studies do not find evidence of significant difference in progress in some subject for some ethnic group sammons l999)these need further study and highlight

28、 again the importance of detailed analysis in exploring the affecting factor on students improvement in short， student's prior attainment has the largest impact on their later attainment，the adjustment using gender or socialeconomic status are small by comparison. nuttalls(1989)study suggest var

29、iability in high ability pupils between schools is much larger than that of low ability pupils，thomas(1993)found correlation between value added score for highest and lowest ability student range from 0.73 to 0.76 for different subjectthe recently studies using finely graded prior attainment measure

30、s in both inner city areas (goldstein et al.，1993) and in county leas (thomas，et al，l996) have also established significant differential secondary school effectsthese evidence point that some school are not equally effective in promoting the attainment of all pupils with different prior attainment，i

31、rrespective of their previous strengths or weaknesses studies on stability and consistencies of school effectiveness show school effects are most stable for total gcse performance score but the department effects are less stable over timethe correlation between school effect across time for performa

32、nce in different academic subjects is from 0.38 to 0.92 (thomas et al，l997)，and only a very low percentage of school can maintain the same position，says，always effective or ineffective，for total score and subject score through 3 yearsthese suggest that we should look results or trends over several y

33、ears in school evaluation practicestudies about school background variables effects on students attainment gained show some school level variable have little effect on childrens progressyangs(1999) research shows the coefficient for variable of percentage of students entitled with free meal in model

34、s fixed part is just-0.007given other variable controlled，some studies find when prior attainment data are available no school context factors are significant and the fit of the model is substantially improved(thomas et al1996) there are some research on school effectiveness and school improvement a

35、rea in china(cheng 1996，wang 1998)，but most of them focus on the rational thought，few research based on evidence analysis is foundthis pilot study aims to give a tentative analysis for some data from several beijing senior secondary schools，using multilevel analysis technique，then compare the result

36、 with the previously studies in term of international viewbeing supported by british council，for the feedback purpose to school，this study will explore the implication of value added measurement method in education practice in beijing areathe analysis of valued added measuer data the data come from

37、several senior secondary schools in beijing city，china，which refer to the students who left senior secondary school in l998originally there are 7 schools data available， but since one school did not provide its students prior attainment information，data from this school is only included in the begin

38、ning analysisthe other 6 school data are included in the final multilevel analysisthe case having missing value can not be included in the analysis automatically when multilevel model are fitted，so，only 617 case can enter the final analysis in our study，contrast with the original l051 case in the sa

39、mple，that means the sample size is a bit smalldescriptive analysis is available in other paper the variables in sample database refer to each students age，gender，ethnicity，parents occupation，students type，major，and their different attainment of different subjects in different time，et alsince differe

40、nt students from different district have different subject prior attainment，only three core subjects，chines，math，english and the total scores for national exam are included in the current analysisstudents are classified to low band (bottom 25)middle band(middle 50)and high band(upper 25)for analysis

41、 according to the sum of their chinese，math and english score when leaving junior secondary schoolschool background variable includes percentage of low band students，ratio of teacherpupil，school type，which is a term formerly called and not used now as a crude indicator of comprehensive school input

42、and teaching levelno other background variables available this time. age，ratio of teacherpupil and attainment scores are normalized (mean 0，standard deviation l)for the analysis purpose(goldstein et al，l999) analysis method advanced multilevel modeling is used as the method of analysis (goldstein，19

43、95，1999)the advantage of multilevel modeling is not only that it can capitalize on the hierarchical structure of the data，also that this technique can be used to look at potentially interesting differences，such as those between the performance of different student type having taken into account of t

44、heir prior attainmentthe analysis can provide an estimate of residual(value added score)for each school after established every variables impacta measure of residual plot with 95% confidence interval for each school can be attached for comparing graphically(goldstein l995) four measures of students

45、performance in national exam as output variable are analysed：the total score，the chinese score，the english score，the math score，their corresponding subjects scores in leaving junior secondary school are fitted as prior attainmentsince there are only 6 school，which means number of 2nd level is insuff

46、icient，there are many computing problems in our multilevel analysis，so all variables entering the model are only set in fixed parts except constant variablevariables contributing no significant change of goodnessfit to the model or having no significant fixed part coefficient will not be included in

47、 most equation for most casedifferent models for different academic performance are fitted firstly，and then a multivariate model for total score and math is fittedthe models are as follow： 1intercept only model：this is the basic model that provides basic analysis and comparing baseline for other ana

48、lysisonly the constant variable is included in the equation and set random at both school and student level 2prior attainment only model：only single prior attainment or combined prior attainment is fitted in the model，all prior attainment is set only in the fixed part 3student background only model：

49、only student background variable is fitted in the model，all are set only in the fixed part 4school background only model：only school background variable is fitted in the model，all are set only in the fixed part 5prior attainment + student background model：the combined prior attainment + student back

50、ground variables are fitted in the model, all are set only in the fixed part 6prior attainment + school background model：the combined prior attainment + school background variables are fitted in the modelall are set only in the fixed part 7full variable model：all variables are fitted in the modelall

51、 are set only in the fixed part the new mlwin(vision l.1)software is used for our studyone feature of this new vision software is to provide detailed simulation result for the analysis (goldstein et al，l999)the simulation technique can make accurate inferences on the basis of simulated parameter est

52、imate, this is useful to have methods for producing accurate interval estimates with small samplesafter each model was fitted，three simulation method are used：gibbs sampling(mlwin default option)，metropolis hasting sampling(univariate mh for fixed and random effects parameter is set，the other option

53、 is mlwin default)，bootstrap estimation(mlwin default option)as said in the mlwin manualall simulation methods are not used for model exploration，just for obtaining unbiased estimates and accurate interval estimates at the final stages of analysisit is need to try different simulation methods to ens

54、ure stable conclusion (goldstein et al，l999)in our analysis，results from three simulation method show some kind consistency for fixed part parameters，so only results from mh methods are reported(see appendix l821)together with mlwin default igls analysis resultssince we only have six school data ava

55、ilable, the analysis show school level parameter having large standard error, which means the sampling error is largethe simulation result based on these school level parameters with large sampling error is uncertainty for further analysisso the comparison and analysis are based on igls result. the

56、finding descriptive analyses of each variable show students intake academic attainments except chinese subject are different in different school. after the descriptive analysis, each variable mentioned above is fitted for the model separatelyethnic，major and ratio of teacherpupil do not show signifi

57、cant effect on model goodness - fit change，so these variables arent included in the later multilevel analysisstudents type，parents job have many missing valuethey are not included in the later multilevel analysis for making good use of the prior attainment informationschool type have some effect for

58、 model goodnessfit change，but when included with other variable，there will no significant effect，even cause computing error probably from the small sample sizewe do not include it in the later model toofinally only student s academic score，age，gender and percentage of low band student are included in the last multilevel model setting the multilevel analysis results are shown in table l to table 4(p128p131)for total score(table l)，without including prior attainment in the model，both pupil background and school backgroun