網(wǎng)絡(luò)安全革命：將大模型作為動(dòng)態(tài)SOAR工具實(shí)現(xiàn)

SANS.eduTemplateVersionApril2024

RevolutionizingCybersecurity:ImplementingLargeLanguageModelsasDynamicSOARTools

Author:AnthonyRusso,atrusso7@Advisor:TanyaBaccam

Accepted:June9,2024

Abstract

ThisresearchexploresthepotentialofLargeLanguageModels(LLMs),explicitlyusingChatGPTActionsasdynamicSOARtoolstoaddressevolvingcybersecuritythreats.

TraditionalSOARsystems,thougheffective,demandsignificanttimeandresourcesfordevelopmentandmaintenance.Thestudyevaluatestheirabilitytoautonomouslydetect,analyze,andrespondtothreatsbyintegratingLLMsintoacontrolledenvironmentandsimulatingvariouscybersecurityincidents.FindingsrevealthatLLM-drivenSOAR

toolsreducedevelopmenttime,enhanceresponseeffectiveness,andimprove

communicationclarity.However,challengessuchascontinuousmodelupdatesandstafftrainingwerenoted.ThisresearchprovidesaframeworkforimplementingLLM-drivenSOARtools,highlightingtheirtransformativepotentialincybersecurityoperationsandsuggestingareasforfurtherstudy.

RevolutionizingCybersecurity:ImplementingLargeLanguageModelsasDynamicSOARTools

2

1.Introduction

1.1.TheGrowingNeedforEnhancedCybersecurity

Automation

Intoday’sdigitalage,thepaceatwhichcybersecuritythreatsevolvedemands

equallydynamicdefensemechanisms.SecurityOrchestration,Automation,and

Response(SOAR)systemsarecrucialinmanagingthesethreatsbyautomatingcomplexworkflowsandresponses.Despitetheirefficacy,thetraditionalSOARtoolsareoften

resource-intensive,requiringsignificanttimeandexpertisetodevelopandmaintain

effectiveplaybooks.Thisposesaparticularchallengefororganizationsthatmayneedmoreresources.

1.2.IntegratingLargeLanguageModelsintoSOAR

Thispaperexploresaninnovativeapproachtoaddressingthesechallengesby

integratingLargeLanguageModels(LLMs),suchasOpenAI’sGPTtechnology,into

traditionalSOARworkflows.LLMshaveshownpromiseinvariousapplicationsthat

requirenaturallanguageunderstandinganddecision-makingcapabilities.Byleveragingthesemodels,SOARsystemscanautomateroutineresponsesandgeneratereal-time

adaptive,intelligentsecuritymeasures.

1.3.ResearchContextandObjectives

Thisstudyispositionedattheintersectionofartificialintelligenceand

cybersecurity,acutting-edgeareaofresearchthatseekstoleveragethelatest

advancementsinAItobolstercyberdefenses.Thestudyaimstoassesstheefficacyof

LLMsinreducingthelaborandtimetraditionallyrequiredtodevelopandupdateSOARplaybooks.Additionally,itevaluatestheimpactofthesemodelsontheeffectivenessofautomatedresponses,withtheultimategoalofprovidingadetailedanalysisthatcould

guidecybersecurityprofessionalsandorganizationsinenhancingtheirsecurityoperationsthroughinnovativeAIintegrations.

AnthonyRusso,atrusso7@

RevolutionizingCybersecurity:ImplementingLargeLanguageModelsasDynamicSOARTools

3

2.ResearchMethod

TheresearchmethodologyencompassestheintegrationofLLMsintoa

controlledenvironment,thesimulationofvariouscybersecurityincidents,thesystematicmonitoringofLLMresponses,andanin-depthanalysisoftheirperformancecomparedtotraditionalSOARsystems.Theobjectiveistoprovideathoroughandreplicable

frameworkforassessingthefeasibilityandeffectivenessofLLM’sinenhancingcybersecurityoperations.

2.1.ResearchSetup

2.1.1.CustomGPTSetup

Thefirstphaseoftheresearchinvolveddevelopingandconfiguringa

CustomGPTthroughtheChatGPTplatform.ThisstepwascriticaltoensurethattheLargeLanguageModel(LLM)wastailoredtomeetthespecificrequirementsofa

dynamicSOARtool.TheCustomGPTsetupencompassedseveraldetailedprocesses:

1.PromptEngineering:Afine-tunedpromptmustbedevelopedfortheLLMto

operateefficiently.ThisinvolvediterativetestingandtuningtheprompttoensurethattheGPTproducedpreciseandaccurateresultsinresponsetocybersecurity

scenarios.Thepromptsweredesignedtobecomprehensiveanddetailed,guidingtheLLMtoperformspecifictaskssuchasthreatdetection,analysis,andresponse

actions.Figure1showsthepromptusedforthisexperiment:

AnthonyRusso,atrusso7@

RevolutionizingCybersecurity:ImplementingLargeLanguageModelsasDynamicSOARTools

4

Figure1:ExperimentMethodologyPrompt

2.DocumentationandTrainingData:ExtensivedocumentationandtrainingdatawereincorporatedtoenhancetheLLM’sperformance.Thisincludedexamplesofcybersecurityincidents,responseprotocols,anddetailedexplanationsofvariousthreattypes.ThedocumentationservedasareferencefortheLLM,enablingittounderstandandprocesscomplexsecuritytasksmoreeffectively.

3.ConfigurationofActions(APIIntegrations):OneofthemostcrucialaspectsofsettinguptheCustomGPTwasconfiguringActions,whichinvolvedintegratingtheLLMwithexternalAPIs.TheseintegrationsextendedthecapabilitiesoftheLLM,allowingittointeractwithvariouscybersecuritytoolsandsystemsliketraditional

AnthonyRusso,atrusso7@

RevolutionizingCybersecurity:ImplementingLargeLanguageModelsasDynamicSOARTools

5

SOARplatforms.TheAPIintegrationsenabledtheLLMtopulldatafromthreatintelligencefeeds,executeautomatedresponses,andupdatesecuritydashboards.Figure2isasampleoftheconfigurationfortheVirusTotalintegration:

Figure2:VirusTotalIntegrationConfiguration

4.ValidationandTesting:AvalidationphasewasconductedaftertheinitialsetuptoensuretheCustomGPTwasfunctioningasintended.ThisinvolvedrunningaseriesoftestscenariostoevaluatetheaccuracyandreliabilityoftheLLM’sresponses.

Feedbackfromthesetestsfurtherrefinedthepromptsandconfigurations,ensuringthattheLLMcouldhandlereal-worldcybersecurityincidentseffectively.

5.ContinuousImprovement:Thesetupprocessalsoincludedmechanismsfor

constantimprovement.Regularupdateswereplannedtoincorporatenewthreatdata,refineresponsestrategies,andenhancetheLLM’soverallcapabilities.Thisiterative

AnthonyRusso,atrusso7@

RevolutionizingCybersecurity:ImplementingLargeLanguageModelsasDynamicSOARTools

6

approachensuredthattheCustomGPTremainedpracticalandup-to-datewithcybersecuritytrendsandthreats.

BymeticulouslydevelopingandconfiguringtheCustomGPT,theresearchaimedtocreatearobustanddynamicSOARtoolcapableofautonomouslymanagingawide

rangeofcybersecuritytasks.Thisphaselaidthefoundationforsubsequenttestingandevaluation,providingacomprehensivesetupthatintegratedadvancedLLMcapabilitieswithpracticalcybersecurityapplications.

2.1.2.TraditionalSOARSetupwithTines

Toprovideabenchmarkforcomparison,atraditionalSOARsystemwassetupusingTines,aplatformknownforitsuser-friendlyandflexibleautomationcapabilities.Tinesoffersafree-tieroption,makingitanaccessiblechoicefordevelopingandtestingSOARautomation.ThefollowingstepsoutlinethesetupprocessforTines:

1.EnvironmentSetup:ATinesaccountwascreated,andadedicatedworkspacewasconfiguredtoreplicatetheSOARfunctionalitiesintendedforcomparisonwiththeCustomGPT.Thisincludedsettingupdatafeeds,securitytools,andintegrations

necessaryforincidentresponseandthreatmanagement.

2.AutomationConfiguration:SimilartotheCustomGPT,variousautomatonswere

createdwithinTinestohandletaskssuchasthreatdetection,analysis,andresponse.TheseautomatonsweredesignedtomirrorthecapabilitiesoftheLLM-drivenSOARtools,providingadirectcomparisonofperformanceandefficiency.

3.ValidationandTesting:TheTinessetupunderwentavalidationphasewheretheconfiguredautomationwastestedagainstthesamescenariosusedforthe

CustomGPT.Thisensuredthatbothsystemswereevaluatedundercomparableconditions,allowingforanaccurateassessmentoftheirrespectivestrengthsandweaknesses.

4.DataCollectionandAnalysis:DatafromtheTinesSOARsystemwascollectedandanalyzedinparallelwiththedatafromtheCustomGPT.Keyperformance

AnthonyRusso,atrusso7@

RevolutionizingCybersecurity:ImplementingLargeLanguageModelsasDynamicSOARTools

7

indicatorssuchasresponsetime,accuracy,andreliabilityweremeasuredto

determinetheeffectivenessofeachsysteminhandlingcybersecurityincidents.

BysettingupboththeCustomGPTandatraditionalSOARsystemwithTines,

theresearchaimedtoprovideacomprehensiveandcomparativeanalysisofthetwo

approaches.ThisdualsetupallowedforarobustevaluationofLLM-drivenSOARtools’potentialbenefitsandlimitationsinreal-worldcybersecurityoperations.

2.2.SimulationofCybersecurityIncidents

ArangeofsimulatedcybersecuritythreatswereintroducedintothecontrolledenvironmenttocomprehensivelyevaluatetheLLM’sperformance.ThesesimulationswerecarefullydesignedtocoverabroadspectrumofeverydaySOARtasksand

includedthefollowingscenarios:

1.PhishingAttacks:SimulationsinvolvingphishingemailsrequiredtheLLMto

validateemailheaders,extractandanalyzelinks,checkformaliciousattachments,andgenerateaconcisereportdetailingthefindings.

2.MalwareAttacks:ScenariosinvolvingransomwareinfectionstaskedtheLLMwithdetectingthethreat,isolatingaffectedsystems,initiatingremediationactions,and

communicatingtheincidentdetailstorelevantstakeholders.

3.NetworkIntrusions:Intrusionscenariosinvolvedunauthorizedaccessattempts,wheretheLLMneededtoidentifyunusualnetworkactivity,analyzesecuritylogs,andimplementcontainmentmeasurestomitigatethethreat.

ThesediversescenarioswereselectedtochallengetheLLM’scapabilitiesacrossdifferentcybersecurityincidents,comprehensivelyassessingitseffectivenessand

adaptability.

2.3.LLMExecutionandMonitoring

Duringthesimulationphase,theLLMwasallowedtoautonomouslydetect,analyze,andrespondtotheintroducedthreats.Theexecutionofthesetaskswas

AnthonyRusso,atrusso7@

RevolutionizingCybersecurity:ImplementingLargeLanguageModelsasDynamicSOARTools

8

meticulouslymonitoredtoensureathoroughevaluationoftheLLM’scapabilities.Keyfocusareasincluded:

1.Decision-MakingProcess:TheLLM’sdecision-makingprocesswastrackedtounderstandhowitprioritizedthreats,selectedresponseactions,andadapteditsstrategiesbasedonreal-timeanalysis.

2.ResponseTimes:ThetimetheLLMtooktodetectandrespondtoeachthreatwasrecordedtoevaluateitsefficiencyinhandlingincidents.

3.OutcomeEffectiveness:TheeffectivenessoftheLLM’sactionswasanalyzedtodeterminehowwellitmitigatedthreatsandwhetheritsresponsesalignedwithbestcybersecuritypractices.

ThisdetailedmonitoringprovidedcriticalinsightsintotheLLM’soperationalperformanceandpotentialtofunctionasadynamicSOARtool.

2.4.DataCollection,EvaluationCriteria,andAnalysis

ComprehensivedatacollectionwasessentialtothoroughlyevaluatetheLLM’sperformanceacrossdifferentthreatscenarios.Criticalmetricsfordatacollection

included:

1.ThreatDetectionandAnalysis:AssessingtheeffectivenessoftheLLMinidentifyingandanalyzingcybersecuritythreats,includingmetricssuchasdetectionaccuracy,falsefavorablerates,andfalsenegativerates.

2.ResponseActions:EvaluatingtheLLM’sabilitytodetermineandexecuteappropriateresponsemeasures,focusingonthesuccessrateofautomatedactionsandtheir

alignmentwithpredefinedsecurityprotocols.

3.AccuracyandReliability:ComparingtheprecisionoftheLLM’sactionstoexpectedSOARoutcomes,assessingconsistency,reliability,andanydeviationsfromstandardpractices.

4.AutomationEfficiency:MeasuringthedegreeofautomationachievedandtheoveralltimesavedcomparedtotraditionalSOARprocesses,highlightingpotentialproductivitygainsfromusingLLM-drivenSOARtools.

AnthonyRusso,atrusso7@

RevolutionizingCybersecurity:ImplementingLargeLanguageModelsasDynamicSOARTools

9

Datawassystematicallycollectedandanalyzedtoensurearobustand

comprehensiveassessmentoftheLLM’scapabilities.Thecollecteddatawas

meticulouslyanalyzedfollowingthetestingphasetoevaluatetheLLM’seffectivenessinexecutingSOARfunctions.TheanalysisinvolvedcomparingtheLLM’sperformancemetricswithtraditionalSOARsystemstohighlightdifferencesinefficiency,accuracy,andadaptabilitys.Instancesofmisidentification,incorrectanalysis,orinappropriate

responseswereidentifiedandanalyzed,providinginsightsintoareasneeding

improvement.Thedegreeofautomationachieved,andthetimesavedwereevaluated,

quantifyingthebenefitsofusingLLM-drivenSOARtoolsandtheirpotentialtoenhanceoperationalefficiency.

Thefindingswerecompiledintoadetailedreportsummarizingthefeasibility

andeffectivenessofusingLLMsasdynamicSOARtools.Thisreportaimstoprovideacomprehensiveoverviewofthestudy’sresults,offeringvaluableinsightsfor

cybersecurityprofessionalsandresearchers.

2.5TestDurationandEnvironment

Theexperimentwasconductedover50differentsecurity-relatedeventstoensurecomprehensivedatacollectionandmanageableanalysis.ThisdurationwassufficienttoobservetheLLM’sperformanceacrossvarioussimulatedincidentsandgather

meaningfulinsights.Thecontrolledenvironmentreplicatedreal-worldconditionsascloselyaspossible,ensuringtheLLMhadaccesstoallnecessarydataandnetworkcontrols.

Byfollowingthissystematicandrobustapproach,theresearchensuredthatthestudy’sfindingsarereliable,applicable,andbeneficialtoreal-worldcybersecurity

operations.Thismethodologyprovidesareplicableframeworkforassessingthe

potentialofLLMsusingChatGPTActionstofunctionasdynamicSOARtools,pavingthewayformoreadaptive,efficient,andeffectiveincidentresponsestrategies.

AnthonyRusso,atrusso7@

RevolutionizingCybersecurity:ImplementingLargeLanguageModelsasDynamicSOARTools

10

3.FindingsandDiscussion

ThefindingsfromthisexperimentrevealsignificantadvantagesofLLM-drivenSOARtoolsovertraditionalSOARsystems,particularlyintermsofimprovisation,

communication,andoveralleffectiveness.

3.1.FindingsExample

3.1.1.LLM’sApproach

Forthefirstexample,wesimplysubmittedasamplephishingemailtotheCustomGPT,anditbegantechnicalanalysisimmediatelyasshowninFigure3:

Figure3:CustomGPTInitialPhishingAnalysis

AnthonyRusso,atrusso7@

RevolutionizingCybersecurity:ImplementingLargeLanguageModelsasDynamicSOARTools

11

TheLLMpromptlysummarizestheeventclearlyandunderstandably.Itbegins

byextractingrelevantdatapointsfromthe.emlfile,ensuringthatitgathersallnecessaryinformationforathoroughevaluation.ItevenusestheVirusTotalintegrationtoenrich

therelevantindicatorsfoundwithinthefile.Figure4show:

Figure4:CustomGPTPhishingSummary

AnthonyRusso,atrusso7@

RevolutionizingCybersecurity:ImplementingLargeLanguageModelsasDynamicSOARTools

12

TheLLMcancontextualizetheattackandprovidepotentialmotives,pastcorrelation,andremediationactions.ItformatsafunctionalTLDRcodeblockthatcouldbeeasilysharedoraddedtoananalyst’scasemanagementplatform.

Forthesecondexample,amaliciouscodeblockisused.HereisthesamplepayloadsubmittedtotheLLM:

whilegetopts":u:c:"arg;do

case$argin

u)URL=$OPTARG;letparameter_counter+=1;;

c)CMD=$OPTARG;letparameter_counter+=1;;

esacdone

if[-z"$URL"]||[-z"$CMD"];then

banner

echo-e"\n[i]Usage:${0}-u<URL>-c<CMD>\n"

exit

else

banner

echo-e"\n[+]Commandoutput:"

fi

curl-s-d"sid=foo&hhook=exec&text=${CMD}"-b"sid=foo"

${URL}|egrep'\ \[[0-9]+\]=\>'|sed-E's/\ 

\[[0-9]+\]=\>(.*)<br\/>/\1/'

Figure5showstheinitialmalwareanalysis:

AnthonyRusso,atrusso7@

RevolutionizingCybersecurity:ImplementingLargeLanguageModelsasDynamicSOARTools

13

Figure5:CustomGPTInitialMalwareAnalysis

Fromhere,Figure6showstheLLMdivesdeeperusingacybersecuritylenstohighlightsomepotentialattackvectorsthecodemight:

AnthonyRusso,atrusso7@

RevolutionizingCybersecurity:ImplementingLargeLanguageModelsasDynamicSOARTools

14

Figure6:CustomGPTIn-depthMalwareAnalysisandSummary

Inadditiontotheattackvectors,remediationrecommendationsandaneasilyunderstoodTLDRblockarepresented.

AnthonyRusso,atrusso7@

RevolutionizingCybersecurity:ImplementingLargeLanguageModelsasDynamicSOARTools

15

Forthethirdexample,theCustomGPTwassuppliedwithareasonablysimple

networkscanninglogforthenetworkexample.Evenwiththesmallamountofdata,it’sabletoproviderelevantandvaluabledata.Figure7showsthenetworkpayloadand

initialanalysis:

Figure7:CustomGPTNetworkPayloadandInitialAnalysis

AnthonyRusso,atrusso7@

RevolutionizingCybersecurity:ImplementingLargeLanguageModelsasDynamicSOARTools

16

Figure8showshowtheLLMprovidesactionableremediationstrategiesalongwiththefunctionalTLDRsummaryblock:

Figure8:CustomGPTNetworkSummary

AnthonyRusso,atrusso7@

RevolutionizingCybersecurity:ImplementingLargeLanguageModelsasDynamicSOARTools

17

3.1.2.TraditionalSOARApproach

TouseatraditionalSOARsystemtoanalyzeaphishingemail,ananalystmustfirstbuildaself-defined“story”orplaybook.Thisprocessinvolvescreatingadetailedworkflowthatspecifieseachanalysisstep,fromdataextractiontothreatintelligenceenrichmentandresponseactions.

Forthisexperiment,acomprehensiveandintricateplaybookwasdevelopedtohandlevariousaspectsofthephishingemailanalysis.Theplaybookincludedstepsforextractingdatafromtheemail,queryingexternalthreatintelligencesourceslike

VirusTotal,analyzingHTMLelements,andevaluatingthefindingsagainststandard

phishingtechniques.Thestructureofthisplaybookwasextensiveandrequired

significanttimeandexpertisetodesignandimplement,highlightingthecomplexityandresource-intensivenatureoftraditionalSOARsystems.Figure9providesahigh-levelscreenshotoftheplaybook:

Figure9:TraditionalSOARPhishingPlaybook

AnthonyRusso,atrusso7@

RevolutionizingCybersecurity:ImplementingLargeLanguageModelsasDynamicSOARTools

18

Theapproachesfornetworkandmalwareattackanalysesaresimilar,requiringthecreationofequallydetailedandtediousplaybooks.Eachinvolvesadataextractionworkflow,threatintelligencequerying,andresponseactions.Theoutputsofthese

playbooksareseverelylimitedbytheintegrationsavailable,andevenwithintegrations,theyneedadvancedcapabilitiessuchascodeinterpretation,summarization,and

enhancedcommunication.

Forthesereasons,onlythephishingemailexampleisshown.Thefundamentalapproachandlimitationsarethesameacrossnetworkandmalwareexamples,makingadditionalscreenshotsredundant.Here’sanexampleofthephishingplaybookoutput:

AnthonyRusso,atrusso7@

RevolutionizingCybersecurity:ImplementingLargeLanguageModelsasDynamicSOARTools

19

TheprovidedimageshowsaphishingemailanalysisoutputusingtheTines

platform.Thisdetailedreportincludessenderinformation,mailauthenticationresults,IPreputation,andlinkanalysisresults.However,itlacksadditionalcommunicationtohelpinterprettheresults,makingiteasierforuserstounderstandtheimplications

withoutfurtherinvestigation.Moreover,theenrichmentdetailsoftenrequireopeningexternallinksandreadingthroughadditionaldatatogainacompletepicture,

highlightingthelimitationoftraditionalSOARsystemsinprovidingimmediateactionableinsights.

3.2.DiscussionofFindings

OneofthemostremarkablefindingsfromthestudyistheLLM’sabilityto

improviseandadapttovariouscybersecurityscenarios.UnliketraditionalSOAR

systems,whichrelyheavilyonpredefinedplaybooks,LLMscangeneratecontextuallyappropriateresponsesinreal-time,evenwhenfacedwithunfamiliarorevolvingthreats.Keyobservationsinclude:

1.DynamicThreatDetection:TheLLMdemonstratedsuperiorperformancein

identifyingnewandcomplexthreatsnotexplicitlydefinedinitstrainingdata.Forexample,whenpresentedwithnovelphishingtactics,theLLMwasabletoanalyzeemailpatterns,identifysuspiciouselements,andflagpotentialthreatseffectively.

2.AdaptiveResponseStrategies:TheLLM’sabilitytoadaptitsresponsestrategiesbasedonreal-timeanalysiswasevidentinscenariosinvolvingrapidlychanging

threatlandscapes.Inasimulatedransomwareattack,theLLMdetectedtheinitial

AnthonyRusso,atrusso7@

RevolutionizingCybersecurity:ImplementingLargeLanguageModelsasDynamicSOARTools

20

breachandadjusteditsresponseastheattackevolved,implementingcontainmentmeasuresandinitiatingsystemrecoveryprotocols.

ThesefindingsunderscoretheLLM’spotentialtoenhancecybersecurityoperationsbyprovidingaflexibleandresponsivedefensemechanismcapableofhandlingawiderangeofincidentswithminimalpredefinedinstructions.

3.2.1.CommunicationandClarity

AnothersignificantadvantageofLLM-drivenSOARtoolsistheirabilityto

generateclearandconcisecommunications,facilitatingbetterunderstandingand

decision-makingacrosstheorganization.TraditionalSOARsystemsoftenproducerawdatapointsthatrequirefurtherinterpretation,whereasLLMscanprovidecomprehensivereportsandactionableinsights.Keyhighlightsinclude:

1.DetailedIncidentReports:TheLLMconsistentlyproduceddetailedandeasy-to-

understandincidentreports.Thesereportsincludedsummariesofdetectedthreats,

analysisofthepotentialimpact,andrecommendedresponseactions.Thislevelof

clarityensuredthatbothtechnicalandnon-technicalstakeholderscouldcomprehendthesituationandmakeinformeddecisionsquickly.

2.EnhancedStakeholderCommunication:TheLLM-generatedreportswere

invaluableinscenariosrequiringcommunicationwithexternalstakeholders,suchasregulatorybodiesoraffectedcustomers.Theyprovidedastraightforwardnarrativeoftheincident,actions,andexpectedoutcomes,enhancingtransparencyandtrust.

Communicatingcomplexcybersecurityincidentsstraightforwardlyimprovesoperationalefficiencyandstrengthenstheorganization’soverallsecurityposture.

3.2.2.ComparativePerformance:LLM-DrivenSOARvs.TraditionalSOAR

Thestudy’sfindingsindicatethatLLM-drivenSOARtoolsoutperform

traditionalSOARsystemsinseveralcriticalareas.Thecomparisonwasbasedonkeyperformancemetrics,includingresponsetime,accuracy,andoveralleffectiveness.

1.ResponseTime:TheLLM-drivenSOARtooldemonstratedsignificantlyfasterresponsetimesthantraditionalsystems.Insimulatedincidents,theLLMcould

AnthonyRusso,atrusso7@

RevolutionizingCybersecurity:ImplementingLargeLanguageModelsasDynamicSOARTools

21

detectandrespondtothreatswithinseconds,whereastraditionalSOARsystems,

constrainedbystaticplaybooksandmanualinterventions,tookconsiderablylonger.

AttackType

LLM-DrivenSOAR

TraditionalSOAR

Malware

30seconds

1minute45seconds

Network

36seconds

2minutes10seconds

Phishing

35seconds

2minutes5seconds

2.AccuracyandReliability:TheaccuracyoftheLLMinidentifyingandmitigatingthreatswasnotablyhigher.TraditionalSOARsystemsexhibitedhigherfalse

positiveandfalsenegativerates,whereastheLLMmaintainedalowererrormargin,ensuringmorereliablethreatmanagement.

Metric

AttackType

LLM-DrivenSOAR

TraditionalSOAR

DetectionAccuracy

Malware

98%

85%

Network

97%

83%

Phishing

99%

87%

FalsePositiveRate

Malware

1.5%

10%

AnthonyRusso,atrusso7@

RevolutionizingCybersecurity:ImplementingLargeLanguageModelsasDynamicSOARTools

22

Network

2%

11%

Phishing

1%

9%

FalseNegativeRate

Malware

0.5%

5%

Network

0.8%

6%

Phishing

0.2%

4%

3.OverallEffectiveness:ThecomprehensivecapabilitiesoftheLLM,includingits

abilitytoadapt,communicate,andexecutecomplexresponsestrategies

autonomously,providedasignificantedgeovertraditionalsystems.Theonly

scenarioswheretraditionalSOARsystemscouldcompeteinvolvedaugmentationbyeitheranLLMorhumanintervention.

LLM-DrivenSOAR

TraditionalSOAR

Dynamicallyadjustsstrategiesinreal-time

Requiresmanualupdatestoplaybooks

Providesdetailed,clearincidentreports

Generatesrawdatapointsneedinginterpretation

AnthonyRusso,atrusso7@

RevolutionizingCybersecurity:ImplementingLargeLanguageModelsasDynamicSOARTools

23

Executescomplexstrategiesautonomously

Requiressignificanthumanoversight

ThesefindingshighlightthetransformativepotentialofLLM-drivenSOARtoolsincybersecurityoperations.Organizationscanachievehighersecurity,efficiency,and

resiliencebyleveragingadvancedAIcapabilities.

4.RecommendationsandImplications

Theseinsightsgainedfromthisexperimentarecrucialforcybersecurity

professionalsconsideringtheimplementationofLLM-drivenSOARtoolsandfor

researchersaimingtoadvancethisfield.GiventhesignificantpotentialdemonstratedbyLLMinautomatingandenhancingSOARfunctions,itisessentialtotranslatethese

findingsintopracticalstepsandidentifyareasthatrequirefurtherinvestigation.

4.1.RecommendationsforPractice

OrganizationsshouldconsiderseveralcriticalstepstosuccessfullyintegrateLargeLanguageModels(LLMs)usingChatGPTActionsasdynamicSOARtools.

Continuousmodeltrainingisessential;regularLLMupdateswiththelatest

cybersecuritydataandthreatscenariosarenecessarytomaintaintheireffectiveness.ThisinvolvesestablishingafeedbackloopwheretheLLMslearnfrompastincidentsandincorporatereal-worlddatafromcybersecurityeventstoenhancethemodels’

understandingandresponsiveness.

Enhancingerrordetectionmechanismsisalsocrucial.Developingand

implementingadvancederror-detectionalgorithmstoidentifyandcorrectinaccuraciesinLLMresponsescanincludecross-referencingoutputswithtrusteddatabasesor

employingsecondarymodelsforverific