Validation methodology from HKMA CA-G-4

1、validation methodology from hkma ca-g-4 1. validation of rating discriminatory power： ?cumulative accuracy profile (“cap”) and itssummary index, the accuracy ratio (“ar”);?receiver operating characteristic (“roc”) and itssummary indices, the roc measure and the pietraindex; ?bayesian error rate (“be

2、r”); ?conditional entropy, kullback-leibler distance, andconditional information entropy ratio (“cier”); ?information value (“iv”); ?kendalls and somers d (for shadow ratings); ?brier score (“bs”); (this is similar to rmse which we will do in validation) ?divergence. not applicable since we will fol

3、low basels rankingand our validation is not on discriminatory power. 2. validation of pd calibration ?binomial test with assumption of independentdefault events; ?binomial test with assumption of non-zero defaultcorrelation; ?chi-square test. we will do this validation on the calibration, for the ac

4、curacy instead of conservative. 3. validation of lgd estimates ?comparisons between internal lgd estimatesand relevant external data sources this may not be applicable since currently in hk and prc region, no bank has fully completed the ifrs9 modelling work, including hsbc. for bond, the external l

5、gd has been used in ifrs9 model already. ?comparisons between realised lgd of newdefaulted facilities and their lgd estimates we will do this. 4. validation of ead estimates ?back-test their internal ead estimates against therealised ead of the new defaulted facilities we will do this for some segme

6、nts which basel ead will be modified in ifrs9 ?where available, ais should compare their internalestimates with external benchmarks. this may not be applicable since currently in hk and prc region, no bank has fully completed the ifrs9 modelling work, including hsbc. ?compare the estimated aggregate

7、 ead amount forthe subject facility type with the realised aggregateead amount for that facility type we will do this for some segments which basel ead will be modified in ifrs9 5. benchmarking ?hkma will expect ais to obtain theirbenchmarks from third parties, provided that relevantexternal benchma

8、rks for a specific portfolio areavailable. when external benchmarks are not used,despite being available, the hkma will expect ais toprovide valid justifications and demonstrate that theyhave other compensating measures (prehensive back-testing at a frequency higher thanrequired, such as quarterly,

9、with sufficient defaultobservations to ensure the reliability of the back-testingresults) to ensure the accuracy of their rating systems.the hkma will not accept cost implications as the solejustification for not using external benchmarks. for external benchmark, it may not be applicable since curre

10、ntly in hk and prc region, no bank has fully completed the ifrs9 modelling work, including hsbc. and thats why we suggest using a higher frequency data to do the back testing. ?where a relevant external benchmark is not available(e.g. pd of sme and retail exposures, lgd and ead),an ai should develop

11、 an internal benchmark. forexample, to benchmark against a model-based ratingsystem, an ai might employ internal rating reviewers tore-rate a sample of credits on an expert-judgementbasis. internal benchmark will be useful for irb since we could ask rating reviewers to do the judgmental rating for s

12、ome sampling. however our ifrs9 methodology is based on basel rating, this is not applicable for ifrs9 validation. and normally we do not build new models with other methodology since it is not comparable for the model performance under different methodologies. ?the hkma willnormally expect ais to u

13、se in validating their ratingsystems and internal estimates: ?comparison of internal estimates with benchmarkswith respect to a common or similar set ofborrowers/facilities; ?comparison of internal ratings and migrationmatrices with the ratings and migration matrices ofthird parties such as rating a

14、gencies or data pools; ?comparison of internal ratings with external expertjudgements, for example, where a portfolio has notexperienced recent losses but historical experiencesuggests that the risk of loss is greater than zero; ?comparison of internal ratings or estimates withmarket-based proxies f

15、or credit quality, such asequity prices, bond spreads, or premiums for creditderivatives; ?analysis of the rating characteristics of similarlyrated exposures; and ?comparison of the average rating output for theportfolio as a whole with actual experience for theportfolio rather than focusing on esti

16、mates forindividual borrowers/facilities. as hkma required, if we choose external benchmark, we need to assess the quality in adequately representing the riskcharacteristics of the portfolio under consideration, including definition of default, rating criteria, data quality, frequency of rating upda

17、tes and assessment horizon.this will be challenging to us since it is not easy to get all these information for peers.also currently in hk and prc region, no bank has fully completed the ifrs9 modelling work, including hsbc. 6. the other information that hkma mentioned in ca-g-4: validation methodol

18、ogy mentioned in terminology: ?“k-fold cross validation” means a kind of testemploying resamplingtechniques. the data set isdivided into k subsets. each time, one of the ksubsets is used as the validation data set and theother k-1 subsets are put together to form thedevelopment data set. by repeatin

19、g theprocedures k times, the targeted test statisticacross all k trials is then computed; ?“bootstrapping” means a resampling technique withreplacement of the data sampled, aiming togenerate information on the distribution of theunderlying data set; ?“in-sample validation” means validation of a ratingsystem employing observations that have beenused for developing the rating system; ?“out-of-sample