2024網(wǎng)絡安全領域可解釋人工智能綜合調(diào)查白皮書（英文版）

WhitepaperonACOMPREHENSIVESURVEYONEXPLAINABLEAIINCYBERSECURITYDOMAINPreparedbyMsNEswariDevi,DrNSubramanian,DrNSaratChandraBabuSocietyforElectronicTransactionsandSecurity(SETS)UnderOfficeofthePrincipalScientificAdvisertotheGovernmentofIndiaMGRKnowledgeCity,CITCampus,Taramani,Chennai–600113SETS|XAIinCybersecurityDomainSETS|XAIinCybersecurityDomainUnderstandingamodel'sdecisionsalsoaddressesfairnessissuesandaidsindebuggingthemodel.TheresearchonXAIbeganintheyear2004,withthesignificantbreakthroughsstartingaround2014,whenDARPAannounceditsXAIproject.AccordingtoDARPA[3],thegoalofXAIisto‘‘producemoreexplainablemodels,whilemaintainingahighleveloflearningperformance(predictionaccuracy);andenablehumanuserstounderstand,appropriatelytrust,andeffectivelymanagetheemerginggenerationofartificiallyintelligentpartners’’.ObjectivesandScopeThisapproachpaperfocusesontherelatedworksthatarebeingcarriedouton“ExplainableArtificialIntelligence(XAI)”forcybersecurityatbothinternationalandnationallevels.OrganizationofthePaperTheapproachpaperisstructuredasfollows:theFundamentalsofXAIsectionprovidesanoverviewofExplainableAIconcepts,techniques,andtheimportanceofinterpretabilityinAImodels.TheXAIinCybersecuritysectionexplorestheapplicationsandbenefitsofXAIspecificallyincybersecuritytasks.TheReviewofRelatedWorksectionsummarizesexistingresearchinthefield,highlightingmethodologiesandfindingsrelatedtoXAIincybersecuritybringinginviewthenationalandinternationalefforts.Subsequently,theChallengesofXAIinCybersecuritysectiondiscussesthelimitations,andpotentialadversarialthreatsassociatedwithimplementingXAIincybersecuritysystems.Finally,theapproachpaperconcludeswithinsightsintoFutureDirections,suggestingresearchdirections,andissuesinXAIforcybersecurity.XAIinCybersecurityRoleofAIinCybersecurityArtificialintelligencesignificantlyenhancescybersecuritybyimprovingthreatdetection,prevention,andresponsecapabilities.AccordingtoaNASSCOMreportfromFebruary2024,India’sAImarketisgrowingataCAGRof25-35%andisprojectedtoreacharound$17billionby2027[4].AIsystemsanalyzevastamountsofdatatoidentifyanomalies,detectmalware,andpredictpotentialcyberthreats.Theyautomatethreatintelligencegatheringandincidentresponse,enablingrapidandeffectivecountermeasures.AIalsooptimizesscanningandpatchingprocesses,enhancingvulnerabilitymanagementandstrengtheninguserbehavioranalyticstodetectinsiderthreatsandfraud.Furthermore,integratingAIintocybersecurityframeworksnotonlystrengthensdefensemechanismsbutalsoenhancestheabilitytoforeseeandcounteractsophisticatedcyberattacks,ensuringamorerobustandresilientdigitalenvironment.NecessityofXAIinCybersecurityExplainableAIhasbecomeessentialforsecurityasitenhancestransparency,trust,andaccountability.OneofthemainreasonsXAIiscrucialincybersecurityisitsabilitytofostertransparencyandtrustinAIsystems'decision-makingprocesses.IthelpscybersecurityprofessionalsunderstandwhyanAImodelflaggedaparticularactivityasmaliciousorbenign.BymakingAIsystems'workingstransparent,XAIallowsforongoingrefinementandimprovement.XAIprovidesinsightsintothemodel'sdecision-makingprocess,enablingtheidentificationandcorrectionofbiases.Thisensuresthatcybersecuritymeasuresarefairandunbiased,maintainingtheintegrityofsecurityprotocols.Statisticsshowthatglobalattacksincreasedby28%inthethirdquarterof2022comparedtothesameperiodin2021[5].Theuseofexplainablemethodsfromtheperspectivesofthreecybersecuritystakeholdersare1)designers,2)modelusers,and3)adversaries.Theirworkthoroughlyexaminesvarioustraditionalandsecurity-specificexplanationmethodsandexploresintriguingresearchdirections[6].GautamSrivastava[7]focusedontheapplicationofXAIforcybersecurityinspecifictechnologyverticalssuchassmarthealthcare,smartbanking,smartagriculture,smartcities,smartgovernance,smarttransportation,Industry4.0,and5Gandbeyond.AshortsurveyonXAIforcybersecuritylistsseveralXAItoolkitsandlibrariesthatsupporttheimplementationofexplainability[8,9].Anexhaustiveliteraturereviewwith244references,highlightsvariouscybersecurityapplicationsusingdeeplearningtechniquessuchasintrusiondetectionsystems,malwaredetection,phishingandspamdetection,botnetdetection,frauddetection,zero-dayvulnerabilities,digitalforensics,andcryptojacking.Theirsurveyalsoexaminesthecurrentuseofexplainabilityinthesemethods,identifiespromisingworksandchallenges,andprovidesfutureresearchdirections.Itemphasizestheneedformoreformalism,theimportanceofhuman-in-the-loopevaluations,andtheimpactofadversarialattacksonXAI[10].TheX_SPAMapproachcombinesthemachinelearningtechniqueRandomForestwiththedeeplearningtechniqueLSTMtodetectspam,usingtheExplainableArtificialIntelligencetechniqueLIMEtoenhancetrustworthinessbyexplainingclassificationdecisions[11].AsurveyonclassifiesXAIapplicationsincybersecurityintothreegroupssuchasdefensiveapplicationsagainstcyber-attacks,potentialsforcybersecurityinvariousindustries,andcyberadversarialthreatstargetingXAIapplicationsalongwithdefenseapproaches.TheyalsohighlightchallengesinimplementingXAIforcybersecurityandstresstheimportanceofstandardizedexplainabilityevaluationmetrics[12].InfocusingonexplainableAI(XAI)applicationsincybersecurity,especiallyregardingfairness,integrity,privacy,androbustness,it'sclearthatreal-worldscenariosareoftenoverlooked.Additionally,currentcountermeasurestodefendXAImethodsarelimited[13].TheapplicationofXAIinCyberThreatIntelligence(CTI)isacrossthreemajorthemes:phishinganalytics,attackvectoranalysis,andcyber-defensedevelopment[14].Anotherstudy[15]discussesthestrengthsandconcernsofexistingmethodsinapplicationslikesecurityloganalysis,presentingapipelinetodesigninterpretableandprivacy-preservingsystemmonitoringtools.XAIisbecomingincreasinglyessentialincybersecurityapplications,asalackofexplainabilityunderminestrustinAIpredictions.Beyondexplainability,accuracyandperformancemustalsobeguaranteedinAImodelsusedforcybersecurity.Trainingdatasetsplayacrucialroleinanymachinelearningapplication.XAIhelpsdetecthighlyimbalanceddatasetsandcorrectbiases,therebyimprovingsystemrobustness.TheconceptofExplainableSecurity(XSec)[16]addressesthesecurityofXAIsystems,providingathoroughreviewofhowtosecurethem.SideChannelAnalysis(SCA)extractsthesecretinformationfromthecryptographicdevicesbyanalysingthephysicalemissionssuchaspowerconsumption,electromagneticleakages,ortiminginformation.AIalgorithmsplayasignificantroleinenhancingprofilingSCA.AIalgorithmsareusedforrecognizingpatternsinlargedatasetsandusedtoidentifysubtlecorrelationsbetweenside-channelemissionsandthesecretinformation[17,18].InAIassistedSCA,thefeatures/PointsofInterest(PoI)contributefortheretrievalofthesecretinformation.Explainabilityonthefeatures/PointsofInterest(PoI)whichcontributetothemostfordecisionmakingisexplained[19].ThishelpsinthevalidationofadesignagainstSCAbyidentifyingthepotentialvulnerableleakagepointsandimplementingsuitablecountermeasures.AninterpretableneuralnetworkknownasTruthTableDeepConvolutionalNeuralNetwork(TT-DCNN)whichhasinterpretableinnerstructureisusedtoperformSCA.TheinteroperabilityisachievedbyconvertingtheNeuralNetwork(NN)intoSATequationstounderstandwhattheNNmodelhaslearnt[20].Maskingandhidingarethecountermeasuresusedtoprotectthesecretinformationattheimplementationlevel.ButAIalgorithmsareeffectiveindefeatingthesecountermeasures.ExDL-SCA(explainabilityofdeeplearning-basedSCA)methodologyisusedinSCAtounderstandtheeffectofsuchcountermeasuresagainstAIassistedSCA[21].Thishelpsthedevelopersinevaluatingthesecurityoftheimplementedcountermeasures.Also,XAIplaysanimportantroleinhardwaretrojandetection[81].Figure1depictstheapplicationofXAIinvariousdomainsofcybersecurity.PhishingDetectionBotnetDetectionRansomeware&MalwareDetectionPhishingDetectionBotnetDetectionRansomeware&MalwareDetectionHardwareSecurityHardwareSecurityIntrusionDetectionSystemExplainableAIinSecurityForallstakeholdersWhogivesandreceivestheexplanationonsecurity?Whatisexplained?Whenistheexplanationonsecuritygiven?Whyistheexplainablesecurityneeded?Howtoexplainsecurity?SpamDetectionAnomalyDetectionDomainGenerationAlgorithmDetectionFigure1:ApplicationofExplainableAIinCybersecurityDomainGenerationAlgorithmDetectionReviewofRelatedWorkInternationalEffortsIn2017,JohnLaunchbury,DirectorofDARPA'sInformationInnovationOffice(I2O),discussedthe"threewavesofAI"todemystifythetechnology.Thefirstwave,"handcraftedknowledge,"involvesencodingdomain-specificknowledgeintorulesthatcomputersfollow.Thesecondwave,"statisticallearning,"usesstatisticalmodelstrainedonspecificdata.Thethirdwave,"contextualadaptation,"featuressystemsthatcanunderstandandexplainthereasoningbehindtheirdecisions[22].DARPA’sXAIprogramfocusedontwomainchallenges:(1)solvingmachinelearningproblemstoclassifyeventsofinterestinheterogeneous,multimediadata,and(2)developingmachinelearningmethodstoconstructdecisionpoliciesforautonomoussystemstoperformavarietyofsimulatedmissions[23].In2020,IBMlaunched"ThePolicyLab,"aplatformdesignedtodevelopAIpoliciesandrecommendations,providingpolicymakerswithavisionandpracticalguidancetoharnesstheadvantagesofinnovationwhileensuringtrustinarapidlychangingtechnologylandscape[24].In2022,theIBMInstituteforBusinessValuepublishedastudyonAIethicsinaction.ItstatesthatbuildingtrustworthyAIisperceivedasastrategicdifferentiatorandthatorganizationsarebeginningtoimplementAIethicsmechanisms.Thestudysuggestsaddressingdimensionssuchasprivacy,robustness,fairness,explainability,transparency,andotherrelevantprinciplestoestablishagovernanceapproachforethicalAIimplementation[25].Alsoin2020,Googlereleasedawhitepapertitled"AIExplainability,"atechnicalreferenceaccompanyingGoogleCloud'sAIExplanationsproduct.ThepaperaimstoleverageAIExplanationstosimplifymodeldevelopmentandexplainthemodel’sbehaviortokeystakeholders.The"GlobalAIActionAlliance"project,partoftheWorldEconomicForum’sShapingtheFutureofTechnologyGovernance:ArtificialIntelligenceandMachineLearningPlatform,aimstoharnessthetransformativepotentialofAIbyacceleratingtheadoptionoftrusted,transparent,andinclusiveAIsystemsglobally.By2026,Gartnerexpectsorganizationstoachievea50%improvementintheirAImodelsintermsofadoption,businessgoals,anduseracceptancebyoperationalizingAItransparency,trust,andsecurity[26].GartnernotesthattechnologyserviceprovidersareincreasinglyusingexplainableAIintheirmodels,especiallyinsecurityandregulatedindustrieslikehealthcareandfinancialservices[26,27]."AIprivacy,security,and/orriskmanagementstartswithAIexplainability,whichisarequiredbaseline."-AvivahLitan,vicepresidentanddistinguishedanalystatGartnerIn2019,TheRoyalSocietypublished"ExplainableAI:thebasicsPolicybriefing,"summarizingthechallengesandconsiderationsfordevelopersandpolicymakerswhenimplementingexplainableAIsystems.ItemphasizesthattheexplanationforanAImodelvariesbasedonitsapplicationandprovidesaviewonhowXAIcanbeimplementedwithoutaffectingsystemperformance,includingaccuracy,interpretability,andprivacy[28].InSeptember2022,theNISTpublishedadraftontheAIRiskManagementFramework(AIRMF)aimingatthedevelopmentandimplementationoftrustworthyAI,withexplainabilityasoneofthecharacteristicstoconsiderincomprehensiveapproachesforidentifyingandmanagingAIsystemrisks[29].InMay2022,IBMprovidedinsightsintooverallAIadoptionworldwide,thebarriersandchallengeshinderingAIfromreachingitspotential,andtheusecases,industries,andcountrieswhereAIismostlikelytothrivethroughthe"IBMGlobalAIAdoptionIndex2022"[30].ThestatisticsinFigure2revealthatmostorganizationshavenottakenessentialstepstowardtrustworthyAI.Specifically,61%oftheorganizationssurveyedhavenotmadesignificanteffortstoexplainAI-powereddecisions.SecurityprofessionalsarethefourthlargestgroupofAIusersatorganizations,representing26%.Additionally,29%oforganizationsuseAIforsecurityandthreatdetection.Figure2:Statisticsoforganizationsthathaven'ttakenkeystepstowardstrustworthyAIasperIBMGlobalAIAdoptionindex[30]Additionally,84%ofITprofessionalsacknowledgetheimportanceoftrustinAIandbelievethatexplainingAIdecisionsiscrucialfortheirbusiness.Currently,17%ofITprofessionalsaremorelikelytoreportthattheirbusinessvaluesAIexplainabilitycomparedtothoseonlyexploringAI.Furthermore,companiesfaceseveralbarriersindevelopingtrustworthyandexplainableAI.Notably,63%ofcompaniesstruggleduetoalackofskillsandtrainingtodevelopandmanagetrustworthyAI.ITprofessionalsingovernmentandhealthcaresectors,whoarecurrentlyexploringordeployingAI,aremorelikelytoidentifybarrierstoexplainabilityandtrustcomparedtothoseinotherindustries.NationalEffortsAsoneofthefastest-growingeconomiesglobally,IndiahasakeeninterestintheAIrevolutionreshapingtheworld.RecognizingAI'spotentialtodriveeconomictransformationandtheneedforIndiatostrategicallypositionitselfinthisshift,thegovernmenthasinitiatedthedevelopmentofanationalAIstrategy.Atthenationallevel,NITIAayog,MeitY(MinistryofElectronics&InformationTechnology),MinistryofCommerce&Industry,andtheOfficeofthePrincipalScientificAdvisertotheGovernmentofIndiahaveformedvarioustaskforcescomprisingdomainexpertstocraftthisstrategy.Thesetaskforceshaveproducednational-leveldocumentsfocusingonArtificialIntelligence.InJune2018,NITIAayogpublishedadiscussionpapertitled‘NationalStrategyforArtificialIntelligence#AIForAll’[31].Thisdocumentaimstoguideresearchanddevelopmentinnewandemergingtechnologies,withaspecificfocusonleveragingAIforsocialandinclusivegrowthinIndia.Thepaperalsoemphasizestheimportanceof‘explainability’inAI,addressingthecommonissuewhereAIsolutionsoftenfunctionasblackboxes,withlittleunderstandingoftheprocessesinvolvedbeyondinputandoutputdata.InFebruary2018,theMinistryofElectronics&IT(MeitY)constitutedfourcommitteestopromoteAIanddevelopaPolicyFramework.ThesecommitteesaretaskedwithunderstandingregulatoryandtechnicalchallengesassociatedwithAIandidentifyingdomainswhereAIimplementationcanbebeneficial[32].InMarch2018,ataskforcecreatedbytheMinistryofCommerceandIndustryreleasedareporton"TheArtificialIntelligence(AI),"whichfocusedontheFourthIndustrialRevolutionanditseconomicimplications,particularlyconcerningAI.[33].InJuly2020,theOfficeofthePrincipalScientificAdviserthroughtheSocietyforElectronicTransactionsandSecurity(SETS)releasedataskforcereporttitled“CybSec4AI,”highlightingcutting-edgeresearchattheinterplayofArtificialIntelligenceandCybersecurity.ThisreportunderscorestheimportanceofsecurityforAImodels[34].InOctober2020,theMinistryofElectronicsandIT(MeitY)andNITIAayogorganizedRAISE2020-'ResponsibleAIforSocialEmpowerment2020,'amajorvirtualsummitonArtificialIntelligence(AI)[35].ThesummithighlightedExplainableAIascrucialforfosteringtrustintechnologyandemphasizedtheimportanceofunderstandingAIdecisionsbetterforitswidespreadadoption.DiscussionsalsocoveredtheimplicationsoflackingexplainabilityinAIandproposedtechnologicalandpolicysafeguardstomitigatetheseissues.ExplainableAIisakeycomponentwithinthebroaderscopeofresponsibleAI,withperiodicexpertreportsbeingpublishedonResponsibleAI(RAI)[36].In2020,theNationale-GovernanceDivision,MinistryofElectronics&IT,GovernmentofIndia,launchedtheinnovationchallenge'YUVAi-YouthforUnnatiandVikaswithAI'programunderResponsibleAIforYouth,anationalinitiativeforschoolstudents.Theprogramaimstoenhancedigitalreadinessamongthenextgeneration,continuingtheinclusiveandcollaborativeAIskillingefforts[37].Buildingonthisinitiative,IntelIndiajoinedin2022andlaunched'ResponsibleAIforYouth2022'[38].InFebruary2021,NITIAayogpublished'ResponsibleAI#AIFORALL,'thefirstpartofanapproachdocumentforIndiathatproposesprinciplesforresponsibleAImanagement.Thisdocumentexploresethicalconsiderations,bothsystemicandsocial,anddelvesintolegalandregulatoryapproachesforgoverningAIsystems.Transparencyishighlightedasacrucialprincipletoensurefair,honest,impartialdeploymentandaccountability.Additionally,thedocumentdiscussesdevelopmentsinXAI[39].本報告來源于三個皮匠報告站（）,由用戶Id:879887下載,文檔Id:618526,下載日期:2025-03-20InAugust2021,NITIAayogcontinueditseffortswiththesecondpartoftheapproachdocumentonResponsibleAI,focusingonoperationalizingtheseprinciples[40].Later,inNovember2022,NITIAayogpublishedthethirddiscussionpaper,titled'ResponsibleAIforAll-AdoptingtheFramework:AUseCaseApproachonFacialRecognitionTechnology.'ThispaperexplorestheapplicationofResponsibleAIprinciplesinthecontextoffacialrecognitiontechnology(FRT)withinIndia,aimingtoestablishaframeworkforitssafeandresponsibledevelopmentanddeployment.ThepaperalsosuggestsprinciplesfordeveloperstobuildexplainableFRTsystems,ensuringclearexplanationsofthesystem'sdecision-makingprocesses[41].OnOctober11,2022,theNationalAssociationofSoftwareandServicesCompanies(NASSCOM)disclosedthe"ResponsibleAIHubandResourceKit"incollaborationwithMicrosoft,TataConsultancyServices(TCS),IBMResearch,Deloitte,andFractalAnalytics.ThisinitiativeaimstopromoteresponsibleAIpracticesandfacilitateitswidespreadadoption.Theresourcekitincludestoolsandguidancethatareapplicableacrosssectors,helpingbusinessesconfidentlyscale-upAItechnologieswhileensuringusersafetyandtrust.OneofthekeyfocusesduringthelauncheventistoaddressthechallengesinXAI[42,43].In2022,NASSCOM,inpartnershipwithMicrosoft,Capgemini,Ernst&Young,andEXL,introducedtheNASSCOMAIAdoptionIndexspecifictoIndia.ThiscomprehensiveframeworkassessesthematurityofAIadoptionnationwideandacrossdifferentsectors,providingacompositescore.DespiteAIinvestmentsinIndiaconstitutingasmallpercentage(~1.5%)ofglobalspending,thecountryhasmadesignificantstridesinthistransformativetechnology.Tofullycapitalizeonthispotential,adoptingbestpracticesisessentialforconvertingtechnologicaladvancementsintotangiblenationalvalue[44].InJune2020,IndiabecameafoundingmemberoftheOrganizationforEconomicCooperationandDevelopment’s(OECD)"GlobalPartnershipforArtificialIntelligence(GPAI)".ThisinitiativeaimstoguidetheresponsibledevelopmentanduseofAIglobally,emphasizinghumanrights,inclusion,diversity,innovation,andeconomicgrowth[45,46].InDecember2022,Googleannounceda$1milliongrantinvestmenttoIIT-MadrastoestablishIndia'sfirstmultidisciplinarycenterforresponsibleAI,incollaborationwithNITIAayog[47].TheGlobalPartnershiponArtificialIntelligence(GPAI)summitheldinNewDelhiinDecember2023broughttogetherexpertsfromscience,industry,civilsociety,governments,internationalorganizations,andacademia.ThesummitaimedtofosterinternationalcooperationonAI-relatedpriorities.IntheUnionBudget2023-2024,theestablishmentofthreeCentresofExcellence(COEs)inArtificialIntelligenceisannouncedwiththevisionto"MakeAIinIndiaandMakeAIworkforIndia".SETS|XAIinCybersecurityDomainSETS|XAIinCybersecurityDomainSETS|XAIinCybersecurityDomainNational-leveleffortshaveproducedseveralreportsonAIhighlightingitspotentialwithintheframeworkofDigitalIndia.TheNITIAayogreportfocusesonAIapplicationsacrossdiversesectorssuchashealthcare,agriculture,education,andfinance.Meanwhile,theCybSec4AIreport,initiatedbytheSocietyforElectronicTransactionsandSecurity(SETS)undertheOfficeofthePrincipalScientificAdvisertotheGovernmentofIndia,aimstofosterself-relianceindevelopingAI-basedtoolsandsystemsforcybersecurity.ItemphasizesthecreationofsecureAIsolutionsandsystems.Currently,therearenoprecisenationalstandards,policies,orframeworksforExplainableAI(XAI)inthecybersecuritydomain,despiteongoingresearchinthisarea.Implementingsuchstandardswoulddemystifytheworkingsofcomplexdatamodels,instillingtrustinAIdecisionsandpromotingthedevelopmentofresponsibleAImodels.ExplainableAIisacrucialaspectofResponsibleAI(RAI),whichencompassesbroaderprinciplessuchasfairness,unbiasedness,transparency,privacy,security,reliability,safety,compliance,protection,andthereinforcementofpositivevalues.Whiletestingsystemswithethicalhackingisbeneficial,maliciousattacksaimedatdisruptingsystemsareundesirable.XAImethodologiesusedincybersecurityThissectionprovidesasurveyofExplainableAI(XAI)applicationsinthecybersecuritydomain,coveringnetworksecurity,anomalydetection,andmicroarchitecturalattacks.IntrusionDetectionSystems(IDSs)havebecomecrucialtoolsincomputernetworkstoensureasecurenetworkenvironment.Inrecentyears,IDSshaveutilizedvariousclassificationalgorithmssuchasDecisionTrees,SupportVectorMachines(SVM),K-NearestNeighbors(KNN),NaiveBayesclassifier,DeepNeuralNetworks(DNN),ConvolutionalNeuralNetworks(CNN),RecurrentNeuralNetworks(RNN),Autoencoders,andotherstodetectintrusions.Itisessentialtoprovideexplanationsforthepredictionsmadeaboutdetectedintrusions,whichaidsinunderstandingthespecificcharacteristicsofdifferenttypesofattacks.Atwo-stagemodelforperformingintrusiondetectionisimplementedin[48].InthefirststageXGBoost(eXtremeGradientBoost)modelisalongwiththeSHAPexplanatoryframeworktoprovideexplanationfortheresultsofthesupervisedlearningmodelinthefirststage.Inthesecondstagetheexplanationsobtainedfromstage1wereusedtotraintheautoencoderin-ordertomakethemodelachievegoodresultsagainstzero-dayattacksorunseenattacks.ThereasonforthemodelbeingmisclassifiedbyIntrusionDetectionSystems(IDS)isexplainedin[49].Theseexplanationsforthecauseofmisclassificationhelpsindecidingthestepstofollowagainstfutureattacks.Theadversarialmachinelearningtechniqueisusedtogenerateexplanationsforincorrectestimationsmadebythetrainedclassifier.Theadversarialapproachinvolvesmodifyingthemisclassifiedsamplesuntilthemodelassignsthecorrectclass.Thedifferencebetweenthemodifiedsamplesandtherealsamplesareusedtoillustratethepredominantfeaturesthatleadtomisclassification.Inthecontextofcybersecurity,thefalsepositiveoutputofthedata-drivenmodelwouldleadtobreachandcompromiseofthewholesystem.Consideringitinview,inordertoachieveaccuracyandreliabilityinasystem,amodelknownasHybridOracle-ExplainerIntrusionDetectionSystemisproposed.Theproposedmodelusestwoseparatemodulestodeliverhumaninterpretableanswersaboutsystemdecisions,atthesametimeachievinghighestpossibleaccuracy[50].ThelocalexplanationsofSHAPframeworkisusedtoprovidedetailontheimpactofeachfeaturehelpfulformakingdecisionsforIntrusionDetectionSystem(IDS).Theglobalexplanationsextracttheimportantfeaturesandexploretherelationshipsbetweenthevaluesoffeaturesandspecifictypesofattacks[51].XAItechniquesareusedinimprovingthedetectionanddefenseofscaffoldingattacksinthecontextofnetworksecurity[52].Anonlineandofflinefeedbackmechanismswhichprovidetheuserwiththemostrelevantinputfeaturesthatimpactsthedecisionofthelearntmodelisproposedin[53].DeeplearningtechniquesisusedforsecurityapplicationssuchasMalwareclassification,binaryreverseengineering,whichlacktransparencyindecisionmaking.MostofthemodelslikeLIMEassumethatthedecisionboundaryislocallylinear.However,inmostofthecomplexsecurityproblems,thisledtoinaccurateexplanation[54].Amodelknownas‘‘LocalExplanationMethodusingNonlinearApproximation’’(LEMNA)isintroduced,whichincludesthemixtureregressionmodel,usedtoconsiderthelinearandnon-lineardecisionboundariestoimprovethelocalexplanationfidelityforcomplexsecurityapplications.TheFusedLassotechniqueisintegratedtocapturefeaturedependency.LEMNAmodelproposedisshowntohelptoestablishTRUSTbyexplaininghowtheclassifiermadecorrectdecisions.LEMNAisusedtoexplaintheoutputofananomaly-basedIDS[55].Byconsideringvitalfeaturesthatcontributetotheprediction,networkaccesscontrolpoliciesarederived.Thevitalfeaturesareselectedbasedonthescoresderivedindicatingthecontributionofeachfeaturetothepredictionagainstthegiveninput.Thedecision-tree-basedautoencodermodelisproposedin[56],todetectanomaliesandprovidesexplanationtothepredictionmadebythemodelbycalculatingthecorrelationsamongdifferentattributevalues.AframeworkforanomalydetectionusingDeepNeuralNetworks(DNN)isproposedin[57].Layer-wiseRelevancePropagation(LRP)isusedtodecomposetheDNNcompositefunctiontocalculatetheinputfeaturerelevantscorethatinturnindicatesthecontributionofeachfeaturetodetecttheanomaly.Thesystemisdesignedinsuchawaytoprovideconfidencescoreforthepredictionmadeandalsotextualdescriptionofthedetectedanomaly.Asthephishingattacksaregrowingeveryyear,theexplainableframeworkssuchasLIMEandEBMareusedtoclassifyURLsasphishingorlegitimatealongwiththeappropriateexplanations[58].LIMEandSaliencymapsareusedtoexplainthedecisionmadebytheAImodelonmicroarchitecture-basedwebsitefingerprintingattacks[59].