大模型工具學(xué)習(xí)權(quán)威綜述 Tool Learning with Foundation Models

arXivv[cs.CL]17AprarXivv[cs.CL]17Apr2023YujiaQinShengdingHuYankaiLinWeizeChenNingDing,GanquCui111,ZheniZeng1,YufeiHuang1,ChaojunXiao1,ChiHan3,YiRenFung3,YushengSu1,HuadongWang1,ChengQian1,RunchuTian1,KunlunZhu8,ShihaoLiang8,XingyuShen1,BokaiXu1,ZhenZhang1,YiningYe1,BowenLi1,ZiweiTang5,JingYi1,YuzhangZhu1,ZhenningDai1,LanYan1,XinCong1,JasonPhang4,ChengYang5,TongshuangWu6,HengJi3,ZhiyuanLiu1*,MaosongSun1*qyj20@AbstractHumanspossessanextraordinaryabilitytocreateandutilizetools,allowingthemtofulptinsurentedropriatetoolsWealsodiscusshowtotrainmodelsforimprovedtoolusecapabilitiesandfacilitatethegeneralizationintoollearning.Consideringthelackofaya1/OpenBMB/BMTools2Contents1Introduction42Background6 ive 2.3ParadigmShift 8 ing ToolorientedLearning 123ToolLearning14 3.2.2PlanningwithReasoning 17elsforImprovedToolLearning LearningfromFeedback 21 4ApplicationandExperiment24 4.2Experiments 275Discussion28 lMakerAIsEvolutionaryRole dgeConictsinToolLearning s 6Conclusion36Contributions36ACaseStudy51A.13DModels 513A.2Stock 53A.3MakingSlides 54A.4MovieHunter 59A.5SearchEngine 60A.6ChemicalsMining 61A.7CookingAssistant 62A.8AIPainting 63vigatingKnowledgeGraphs A.10ALFWorld 65ACalculator 67ather AOnlineShopping 68A.14Map 69 A.16Translation 72A.17Wikipedia 734+otenign+SEearchnoisMuofdfeiDl+ngRobCookiIwouldrecommend:model.sliceswithyogurt.+otenign+SEearchnoisMuofdfeiDl+ngRobCookiIwouldrecommend:model.sliceswithyogurt.1Introductioninhumanactivity.Sincethedawnofcivilization,toolshavebeenintegraltotheveryessenceofouromerydbettedusingshallowstatisticalmodelsoritysikeurrentlyspecializedtoolscanbePleasePleaserecommendsomebooksonpersonalfinanceandinvesting.PleasePleasemakeabananayogurtforme.PleasePleasedrawawatercolorstyleEiffelTowerpainting.2/Blendthebanana1/2/BlendthebananaStableDiffusion2.SimpleWealth1.The2.SimpleWealth3.…HereisawatercolorHereisawatercolorgeneratedbydiffusion5asanetalketalcyframeworkfortooluseinAIsystems(§2.1),followedbyacategorizationoftoolsfromtheperspectiveoftheuserinterface(§2.2).ThenwereviewtheAIparadigmshiftbroughtbyfoundationmodelsandhighlightthehensiveliteraturereviewforexistingexplorationintoollearningwhichisdividedintotwongftoollsvelyinallywediscussotherimportantresearchtopicsforapplyingourgeneralframeworktorealworldsllearningcon?ictsintoollearning,whichcanleadtoinaccurateandunreliablemodelpredictions.Weidentifytwowe62Backgroundussing2.1CognitiveOriginsofToolUsedtxtensionsofhumanbeingsjustasancientghtingmpleondtheanteriorsupramarginalgyrusactivationofobservingtooluseistypicalofhumansubjects,ofwhichrelatedtotheoriginsofcumulativetechnologicalevolution(e.g.,theimprovementintheef?ciencyandolApartfromtoolsinthephysicalworld,wecanalsoturntomoreabstracttoolbehavior.Takecognitivetools(Heyes,2018)asanexample:itreferstoanauxiliarytoolforthinkingandhelpslearnersachieven72.2ToolCategorization:AUser-InterfacePerspectiveThegrowingnumberandcomplexityoftoolsinourworldmakeitincreasinglyimportanttounderstandbenetsandpotentialforgrowthInthispaperweareparticularlyinterestedinthosetoolsthatcanbetools.ednglaninteractiveinterfaceievisualngcurvefornontechnicalusers.Fromthisviewpoint,toollearningwithfoundationmodelssharethesameprimarygoal,reusuallyattheyake8(b)GUI-basedToolsGraphicalUserInterface(GUI)(c)Program-basedToolsProgrammingInterface(a)(b)GUI-basedToolsGraphicalUserInterface(GUI)(c)Program-basedToolsProgrammingInterface(a)PhysicalInteraction-basedToolsPhysicalToolsObservationSoftwares/SDKDeveloperAgentRealworldKnowledgeGraphDatabaseObservationVisualoperationtoprogrammingoperationPhysicalworldtovirtualworldUserPhotoshopWebPhotoshopWebsndofelxiblyexecutingtoolsofdifferenttypesInthispaperwee2.3ParadigmShiftnNenkovaMcKeownAlthoughtrainedonmassivecorpora,fromwhichgenerallinguisticabilityandworldknowledgearelearned.Thisropriatetextualpromptscanyieldthedesiredoutputfornewureandnaturallanguageasthemediumtouniformlyperformvarious9ancepplicationsandschedulingmeetingsmodelsserveastranslatorsmakingcomplextasksmoreaccessibletoindividualswithoutspecializedlfromtheuseoftoollearningintherealrceof2.4ComplementaryRolesofSpecializedToolsandFoundationModelssfortoollearning:(1)MitigationforMemorization.Althoughfoundationmodelshavedemonstratedangeverytharelativelyshortcontextlyhatarecictasks,suchasWolfram3forscienti?ccalculation,throughtheutilizationoftailoredalgorithms.Insteadetoolstoationssuchashealthcareornancewhereexplainabilityiscritical2/en-us/microsoft-3653/istanttoadversarialattacksOverallincorporatingtoolsintotheolsMakingandReasoningAbilitiesFoundationmodelsaretrainedonvastamountsofdata,enablingthemtoacquireworldknowledgeacrossawiderangeofdomains.Ifproperlysteered,suchknowledgecannizesse2.5LiteratureReviewofToolLearning(i.e.,AIfortool).2.5.1Tool-augmentedLearningnspecicknowledgeandimprovetheirgenerationqualityResearchinthisareahasoraugmentationisthetextretrievertoolwhichdevelopsfromtheearlysparseretrieverSparckJonesRobertsonetal,1995)totheknowledgeretriever.Forinstance,kNN-LM(Khandelwaletal.,2020)combinesaPLMandak-nearestPLMinInstructionIAnswerTwitterTrendingis1.HotInstructionIAnswerTwitterTrendingis1.HotWeather2.NBAPLAYOFFS3.…Tweet“ILOVEPIZZA”Tool1.Openbrowser2.LoginTwitter3.Createatweet(a)Tool-AugmentedLearningTellTellTwitterTrendingFounFoundationResult1.HotWeather2.NBAPLAYOFFS3.…(b)Tool-OrientedLearningPlanning1.Openbrowser.2.LoginTwitter.3.Createatweet.InstructionundationInstructionundationeinratethenalanswerTheintuitionisthatinmOtherToolsApartfromthetextorimageretrieverresearchershaveexploredemcalculatortoperformbasicarithmeticteachPLMsbetterutilizeatool.Toolformer(Schicketal.,2023)extendstheideaofParisietal.(2022)translationsystem,Wikipediasearchingtool,andcalendar.HereweprovideanexemplarygenerationheetalbGaoetal.(2022)proposetoaugmentPLMswithPythoninterpreters.Speci?cally,givenTable1:Representativeworksoftool-augmentedlearning.Foreachwork,wespecifythetoolusedforName&RefAugmentationAugmentationMethodwaletallTOOLFORMERSchicketal23)MIND’SEYE(Liuetal.,2022)SHOWYOURWORK(Nyeetal.,2021)2.5.2Tool-orientedLearningedloitingfoundationmodelsvastEmbodiedRoboticLearning.Themostrepresentativeapplicationoftool-orientedlearningisroboticsendonsAsthecenterpieceforplanningandreasoning,PLMsarelimitedtoprocessingtextualinputs.Fortoolsmultimodalcapabilitieswithout?ne-tuning.Incontrast,othersexploredbuildingmultimodalfoundationAutomationforOtherTools.Besidesroboticlearning,tool-orientedlearninghasbeenappliedtootherscenarios,including(1)websearchautomation:WebGPT(Nakanoetal.,2021)interactswithasearchdrecordingimportantinformationToachievethistherchgivenhumaninstructions;(3)dialogue-basedimagedrawingandediting:toenableunderstandingandnctexistingonKimetalproposetopromptlargelanguagewithtoollearningGivenatoolexecutioncouldsolvethetask.Bothstreamsshareacommongoal,i.e.,toleveragethestrengthsofspecializedtoolsandfoundationmodelsfortargettasks(§2.4).Whileresearchintoollearninghas4/en-us/bing/apis/bing-web-search-api5https://huggingface.co6/Torantulino/Auto-GPT3ToolLearningtddiscuss3.1ComponentsofToolLearningplexhother3.1.1UnderstandingtheComponentsAPIerwithsanningristhenpassedtothecontrollertoassistitsdecision-making.Byobservingthisfeedback,thecontrolleriple3.1.2ConnectingtheComponentstenatingftandettoformxtormodeledwithcomplexneuralPerceivernedbackPerceivernedbackntrollerningFouningModelantructionHuHumancutionnmentedbackFigure4:Illustrationofthetoollearningframework,wherewedisplaythehumanuserandfourcorespC(at)=p9C(at|xt,Ht,q),(1)canp9C(at|xt,Ht,q)=p9C(at|Ti,xt,Ht,q)×p9C(Ti|xt,Ht,q),(2)JieJennarytongistondanactionsequenceatthatultimatelyfulllsthetask3.2TheGeneralProcedure:FromIntenttoPlan3.2.1UnderstandingIntentandToolsspectssunderstandingcapabilities,challengesstillexistinreal-worldtoollearningscenarios:(1)Understandingllengeisdealingwiththeinherentvaguenessandambiguityintheuseryngthemodelsgeneralizetomorediverseuserinstructions.Astheintentspaceistheoreticallyin?nite,itisalmostimpracticalforurposengofvelusageidesconcretetooluseZeroZero-shotPrompting:Hereweprovideatool(API)"forecast_weather(city:str,N:int)",whichcouldforecasttheweatheraboutacityonaspeci?cdate(afterNdaysfromtoday).ThereturnedryoutoanswerQuestion:"What’sthetemperatureinShanghaitomorrow?"returnforecast_weather("Shanghai",1)["temperature"]foriinrange(2):ifforecast_weather("London",i+1)["precipitation"]>0:returnTruereturnFalseitelApotentialsolutionistoaddanintermediatestageoftoolselection,which?rstretrievesasmallsetoftaremostsuitableforthetaskathandAnothersolutionisnetuningwhichoptimizesmodelseedforincorporatingtooldenitionsintheinputwhichshrinkstheinputlengthandacceleratesmodel3.2.2PlanningwithReasoningingptoacertainsize(Weietal.,2022b).Inparticular,foundationmodelswithtensorhundredsofbillionsofietalbElicitingReasoninginFoundationModels.DespitetheextensivestudyoftheconceptofreasoninginWasonKelleythenotionofreasoningasappliedtofoundationIntrospectiveIntrospectiveReasoningControllerFoundationModelEnvControllerFoundationModelEnv&HumanMulti-IterationExtrospectiveReasoningToolSetControllerFoundationModelEnv&HumanToolControllerFoundationModelEnv&HumanToolSetestheillustrationforsimplicity.gThevanillafew-shotpromptlearning(Brownetal.,2020),wherebymodelsareprovidedwithaprompte(CoT)prompting.Unlikevanillafew-shotpromptlearning,CoTadditionallyinsertsthereasoningtracerequiredtoderivethe?nalanswerforeachexampleintheprompt.Inthisway,CoTpromptsmodelstoiderangeoftasksincludingarithmeticreasoningessfultiveTheformerinvolvesgeneratingastaticplanoftoolusewithoutgtsIntrospectiveReasoningThiskindofreasoningdirectlygeneratesmulti-stepplansfortoolusewithoutramsforvisionmodelsAlthoughVisualChatGPThastheformofiterativereasoningineachintermediatestep,bleediateexecutionresultsAmorerationalapproachtoplanningistakingtheenvironmentEinto