Test Scaling nd ValueAdded Measurement WCER 測試結垢與附加值的測量wcer

1、test scaling and value-added measurementdale ballouvanderbilt universityapril, 2008 va assessment requires that student achievement be measured on an interval scale: 1 unit of achievement represents the same amount of learning at all points on the scale. scales that do not have this property: number

2、 rightpercentile ranksnce (normal curve equivalents)irt “true scores” scales that may have this propertyirt ability trait (“scale score”)item response theory models one-parameter logistic modelpij = 1 + exp(-d(i-j)-1, pij is the probability examinee i answers item j correctlyi is examinee i abilityj

3、 is item j difficulty two- and three-parameter logistic irt modelspij = 1 + exp(-j(i-j)-1pij = cj + (1-cj)1 + exp(-j(i-j)-1j is an item discrimination parametercj is a guessing parameterirt isoprobability contours (1-parameter model)linear, parallel isoprobability curves are the basis for the claim

4、that ability is measured on an interval scale. the increase in difficulty from item p to item q offsets the increase in ability from examinee a to b, from b to c, and from c to d. in this respect, ab = bc = cd, etc. moreover, the same relations hold for any pair of items. do achievement test data co

5、nform to this model? pij and isoprobability contours arent given. data are typically binary responses. testable hypotheses can be derived from this structure, but power is low. the model doesnt fit the data when guessing affects pij . or when difficulty and ability are multidimensional. “data” are s

6、elected to conform to the model ability may be too narrowly defined.implications: it seems unwise to take claims that ability is measured on an interval scale at face value. we should look at the scales.ctb/mcgraw-hill ctbs math (1981)grademean gain from previous gradestandard deviation2186773674443

7、33552324614207623ctb/mcgraw-hill terra nova, mean gain from previous grade (mississippi, 2001)gradereadinglanguage artsmath323.830.247.3421.520.424.7515.517.819.0610.56.921.779.29.910.7812.410.917.0northwest evaluation association, fall, 2005, readinggrademean gain from previous gradestandard deviat

8、ion314.715.649.515.056.914.664.814.874.014.883.514.8northwest evaluation association, fall, 2005, mathgrademean gain from previous gradestandard deviation314.012.1410.912.858.513.966.415.075.815.084.816.8appearance of scale compression declining between-grade gains constant or declining variance of

9、scoreswhy? in irt, the same increase in ability is required to raise the probability of a correct answer from .2 to .9, regardless of the difficulty of the test item. do we believe this?to raise the probability of a correct response from 2/7 to 1, who must learn the most math?student awhat makes us

10、think of a circle?a.blockb.penc. doord. a football fielde.bicycle wheelstudent busing the pythagoreantheorem, a2 + b2 = c2, when a = 9 and b = 12, then c = ?a.8b.21c. 15d. 21e.225responses conference participants a: 11 b: 26 equal: 7 indeterminate: 30 faculty and graduate students, peabody college a

11、: 13 b: 37 equal: 15 indeterminate: 33implications bad idea to construct single developmental scale spanning multiple grades even within a single grade, broad range of items required to avoid floor and ceiling effects. scale compression affects gains of high-achievers vis-vis low achievers within a

12、grade.what to do? use the scale anyway, on the assumption that value added estimates are robust to all but “grotesque transformations” of .test of this hypothesis: rescaled math scores to equate between-grade gains (sample of 19 counties, southern state, 2005-06)original scalerelative to students at

13、 the 10th percentile, growth by students at the:grade25th percentilemedian75th percentile90th percentile2 to 3.971.061.03.953 to 41.101.031.161.344 to 5.961.151.351.235 to 61.646 to 71.211.391.431.627 to 61.13transformed scalerelative to students at the 10th percentile, growt

14、h by students at the:grade25th percentilemedian75th percentile90th percentile2 to 32.635.066.837.903 to 41.692.102.933.954 to 51.301.942.742.875 to 62.123.484.254.916 to 71.602.292.793.507 to 81.511.892.172.35what to do? (cont.) transform to a more acceptable scale =g() and treat as an interval scal

15、e.example: normalizing by mean gain among examinees with same initial score.problem: this doesnt produce an interval scale. what to do? (cont.) map to something we can measure on an interval (or even ratio) scaleexamples: inputs, future earningswhat to do? (cont.) ordinal analysishow it works: teach

16、er a has n students. other teachers in a comparison group have m students. there are nm pairwise comparisons. each comparison that favors teacher a counts +1 for a. each comparison that favors the comparison group counts -1 for a. sum and divide by number of pairwise comparisons. yields an estimate

17、of the probability that a randomly selected student of a outperforms a randomly selected student in the comparison group, minus the probability of the reverse. example of a statistic of concordance/discordance. somers d statistic. can control for covariates by conducting pairwise comparisons within

18、groups defined on the basis of a confounding factor (e.g., prior achievement).illustration sample of fifth grade mathematics teachers in large southern city. two measures of value-addedregression model with 5th grade scores regressed on 4th grade scores, with dummy variable for teacher (fixed effect)somers d, with students grouped by deciles of prior achievementresults hypothesis that teachers are ranked the same by both methods rejected (p=.008) maximum discrepancy in ranks = 229 (of 237 teachers in all)