AIGC時(shí)代的ChatGPT全面綜述

arXiv:2304.06488v1[cs.CY]4Apr2023

OneSmallStepforGenerativeAI,OneGiantLeapforAGI:ACompleteSurveyonChatGPTinAIGCEra

CHAONINGZHANG,KyungHeeUniversity,SouthKorea

CHENSHUANGZHANG,KAIST,SouthKorea

CHENGHAOLI,KAIST,SouthKorea

YUQIAO,KyungHeeUniversity,SouthKorea

SHENGZHENG,BeijingInstituteofTechnology,China

SUMITKUMARDAM,KyungHeeUniversity,SouthKorea

MENGCHUNZHANG,KAIST,SouthKorea

JUNGUKKIM,KyungHeeUniversity,SouthKorea

SEONGTAEKIM,KyungHeeUniversity,SouthKorea

JINWOOCHOI,KyungHeeUniversity,SouthKorea

GYEONG-MOONPARK,KyungHeeUniversity,SouthKorea

SUNG-HOBAE,KyungHeeUniversity,SouthKorea

LIK-HANGLEE,HongKongPolytechnicUniversity,HongKongSAR(China)

PANHUI,HongKongUniversityofScienceandTechnology(Guangzhou),China

INSOKWEON,KAIST,SouthKorea

CHOONGSEONHONG,KyungHeeUniversity,SouthKorea

OpenAIhasrecentlyreleasedGPT-4(a.k.a.ChatGPTplus),whichisdemonstratedtobeonesmallstepforgenerativeAI(GAI),butonegiantleapforartificialgeneralintelligence(AGI).SinceitsofficialreleaseinNovember2022,ChatGPThasquicklyattractednumeroususerswithextensivemediacoverage.SuchunprecedentedattentionhasalsomotivatednumerousresearcherstoinvestigateChatGPT

fromvariousaspects.AccordingtoGooglescholar,therearemorethan500articleswithChatGPTintheirtitlesormentioningitintheirabstracts.Consideringthis,areviewisurgentlyneeded,andourworkfillsthisgap.Overall,thisworkisthefirsttosurveyChatGPTwithacomprehensivereviewofitsunderlyingtechnology,applications,andchallenges.Moreover,wepresentanoutlookon

Authors’addresses:ChaoningZhang,KyungHeeUniversity,SouthKorea,chaoningzhang1990@;ChenshuangZhang,KAIST,SouthKorea,zcs15@kaist.ac.kr;ChenghaoLi,KAIST,SouthKorea,lc;YuQiao,KyungHeeUniversity,SouthKorea,qiaoyu@khu.ac.kr;ShengZheng,BeijingInstituteofTechnology,China,zszhx2021@;SumitKumarDam,KyungHeeUniversity,SouthKorea,skd160205@khu.ac.kr;MengchunZhang,KAIST,SouthKorea,zhangmengchun527@;JungUkKim,KyungHeeUniversity,SouthKorea,ju.kim@khu.ac.kr;SeongTaeKim,KyungHeeUniversity,SouthKorea,st.kim@khu.ac.kr;JinwooChoi,KyungHeeUniversity,SouthKorea,jinwoochoi@khu.ac.kr;Gyeong-MoonPark,KyungHeeUniversity,SouthKorea,gmpark@khu.ac.kr;Sung-HoBae,KyungHeeUniversity,SouthKorea,shbae@khu.ac.kr;Lik-HangLee,HongKongPolytechnicUniversity,HongKongSAR(China),lik-hang.lee@.hk;PanHui,HongKongUniversityofScienceandTechnology(Guangzhou),China,panhui@ust.hk;InSoKweon,KAIST,SouthKorea,iskweon77@kaist.ac.kr;ChoongSeonHong,KyungHeeUniversity,SouthKorea,cshong@khu.ac.kr.

Permissiontomakedigitalorhardcopiesofallorpartofthisworkforpersonalorclassroomuseisgrantedwithoutfeeprovidedthatcopiesarenotmadeordistributedforprofitorcommercialadvantageandthatcopiesbearthisnoticeandthefullcitationonthefirstpage.Copyrightsforcomponents

ofthisworkownedbyothersthanACMmustbehonored.Abstractingwithcreditispermitted.Tocopyotherwise,orrepublish,topostonserversortoredistributetolists,requirespriorspecificpermissionand/orafee.Requestpermissionsfrompermissions@.

?2022AssociationforComputingMachinery.

ManuscriptsubmittedtoACM

ManuscriptsubmittedtoACM1

2Zhangetal.

howChatGPTmightevolvetorealizegeneral-purposeAIGC(a.k.a.AI-generatedcontent),whichwillbeasignificantmilestoneforthedevelopmentofAGI.

CCSConcepts:?Computingmethodologies→Computervisiontasks;Naturallanguagegeneration;Machinelearningapproaches.AdditionalKeyWordsandPhrases:Survey,ChatGPT,GPT-4,GenerativeAI,AGI,ArtificialGeneralIntelligence,AIGC

ACMReferenceFormat:

ChaoningZhang,ChenshuangZhang,ChenghaoLi,YuQiao,ShengZheng,SumitKumarDam,MengchunZhang,JungUkKim,SeongTaeKim,JinwooChoi,Gyeong-MoonPark,Sung-HoBae,Lik-HangLee,PanHui,InSoKweon,andChoongSeonHong.2022.

OneSmallStepforGenerativeAI,OneGiantLeapforAGI:ACompleteSurveyonChatGPTinAIGCEra.1,1(April2022),

29

pages.

.org/XXXXXXX.XXXXXXX

https://doi

Contents

Abstract

1

Contents

2

1Introduction

2

2OverviewofChatGPT

4

2.1OpenAI

4

2.2Capabilities

5

3TechnologybehindChatGPT

6

3.1Twocoretechniques

6

3.2Technologypath

7

4ApplicationsofChatGPT

10

4.1Scientificwriting

10

4.2Educationfield

13

4.3Medicalfield

14

4.4Otherfields

15

5Challenges

16

5.1Technicallimitations

16

5.2Misusecases

17

5.3Ethicalconcerns

18

5.4Regulationpolicy

19

6Outlook:TowardsAGI

20

6.1Technologyaspect

20

6.2Beyondtechnology

21

7Conclusion

22

References

22

1INTRODUCTION

ThepastfewyearshavewitnessedtheadventofnumerousgenerativeAI(AIGC,a.k.a.AI-generatedcontent)tools[

73

,

135

,

141

],suggestingAIhasenteredaneweraofcreatinginsteadofpurelyunderstandingcontent.Foracomplete

ManuscriptsubmittedtoACM

Fig.1.Structureoverviewofthissurvey.

OneSmallStepforGenerativeAI,OneGiantLeapforAGI:ACompleteSurveyonChatGPTinAIGCEra3

ManuscriptsubmittedtoACM

4Zhangetal.

surveyongenerativeAI(AIGC),thereaderscanreferto[

214

].AmongthoseAIGCtools,ChatGPT,whichwasreleasedinNovember2022,hascaughtunprecedentedattention.Itattractednumeroususers,andthenumberofactivemonthlyuserssurpassed100millionwithinonlytwomonths,breakingtheusergrowthrecordofothersocialproducts[

118

].ChatGPTwasdevelopedbyOpenAI,whichstartedasanon-profitresearchlaboratory,withamissionofbuildingsafeandbeneficialartificialgeneralintelligence(AGI).AfterannouncingGPT-3in2020,OpenAIhasgraduallybeenrecognizedasaworld-leadingAIlab.Veryrecently,IthasreleasedGPT-4,whichcanbeseenasonesmallstepforgenerativeAI,butonegiantstepforAGI.

Duetoitsimpressivecapabilitiesonlanguageunderstanding,numerousnewsarticlesprovideextensivecoverageandintroduction,tonameafew,BBCScienceFocus[

69

],BBCNews[

39

],CNNBusiness[

79

],BloombergNews[

157

].

Google’smanagementhasissueda“codered"overthethreatofChatGPT,suggestingthatChatGPTposedasignificantdangertothecompany,especiallytoitssearchservice.ThisdangerseemsmoredifficulttoignoreafterMicrosoftadoptedChatGPTintheirBingsearchservice.ThestockpricechangealsoreflectsthebeliefthatChatGPTmighthelpBingcompetewithGooglesearch.SuchunprecedentedattentiononChatGPThasalsomotivatednumerousresearcherstoinvestigatethisintriguingAIGCtoolfromvariousaspects[

149

,

163

].Accordingtoourliteraturereview

ongooglescholar,nofewerthan500articlesincludeChatGPTintheirtitlesormentionthisviraltermintheirabstract.ItischallengingforreaderstograsptheprogressofChatGPTwithoutacompletesurvey.OurcomprehensivereviewprovidesafirstlookintoChatGPTinatimelymanner.

Sincethetopicofthissurveycanberegardedasacommercialtool,wefirstpresentabackgroundonthecompany,i.e.OpenAI,whichdevelopedChatGPT.Moreover,thissurveyalsopresentsadetaileddiscussionofthecapabilitiesofChatGPT.Followingthebackgroundintroduction,thisworksummarizesthetechnologybehindChatGPT.Specifically,

weintroduceitstwocoretechniques:Transformerarchitectureandautoregressivepertaining,basedonwhichwepresentthetechnologypathofthelargelanguagemodelGPTfromv1tov4[

18

,

122

,

136

,

137

].Accordingly,wehighlighttheprominentapplicationsandtherelatedchallenges,suchastechnicallimitations,misuse,ethicsandregulation.Finally,weconcludethissurveybyprovidinganoutlookonhowChatGPTmightevolveinthefuture

towardsgeneral-purposeAIGCforrealizingtheultimategoalofAGI.AstructuredoverviewofourworkisshowninFigure

1

.

2OVERVIEWOFCHATGPT

First,weprovideabackgroundofChatGPTandthecorrespondingorganization,i.e.,OpenAI,whichaimstobuildartificialgeneralintelligence(AGI).ItisexpectedthatAGIcansolvehuman-levelproblemsandbeyond,onthepremiseofbuildingsafe,trustworthysystemsthatarebeneficialtooursociety.

2.1OpenAI

OpenAIisaresearchlaboratorymadeupofagroupofresearchersandengineerscommittedtothecommissionofbuildingsafeandbeneficialAGI[

50

].ItwasfoundedonDecember11,2015,byagroupofhigh-profiletechexecutives,

includingTeslaCEOElonMusk,SpaceXPresidentGwynneShotwell,LinkedInco-founderReidHoffman,andventurecapitalistsPeterThielandSamAltman[

78

].Inthissubsection,wewilltalkabouttheearlydaysofOpenAI,howitbecameafor-profitorganization,anditscontributionstothefieldofAI.

Inthebeginning,OpenAIisanon-profitorganization[

24

],anditsresearchiscenteredondeeplearningandrein-forcementlearning,naturallanguageprocessing,robotics,andmore.Thecompanyquicklyestablishedareputationforitscutting-edgeresearchafterpublishingseveralinfluentialpapers[

123

]anddevelopingsomeofthemostsophisticated

ManuscriptsubmittedtoACM

OneSmallStepforGenerativeAI,OneGiantLeapforAGI:ACompleteSurveyonChatGPTinAIGCEra5

AImodels.However,tocreateAItechnologiesthatcouldbringinmoney,OpenAIwasreorganizedasafor-profitcompanyin2019[

31

].Despitethis,thecompanykeepsdevelopingethicalandsecureAIalongsidecreatingcommercialapplicationsforitstechnology.Additionally,OpenAIhasworkedwithseveraltoptechfirms,includingMicrosoft,Amazon,andIBM.Microsoftrevealedanewmultiyear,multibillion-dollarventurewithOpenAIearlierthisyear[

21

].

ThoughMicrosoftdidnotgiveaprecisesumofinvestment,SemaforclaimedthatMicrosoftwasindiscussionstospendupto$10billion[

101

].AccordingtotheWallStreetJournal,OpenAIisworthroughly$29billion[

13

].

Fig.2.OpenAIproductstimeline.

Fromlargelanguagemodelstoopen-sourcesoftware,OpenAIhassignificantlyadvancedthefieldofAI.Tobeginwith,OpenAIhasdevelopedsomeofthemostpotentlanguagemodelstodate,includingGPT-3[

95

],whichhasgainedwidespreadpraiseforitsabilitytoproducecohesiveandrealistictextinnumerouscontexts.OpenAIalsocarriesoutresearchinreinforcementlearning,abranchofartificialintelligencethataimstotrainrobotstobasetheirchoicesonrewardsandpunishments.ProximalPolicyOptimization(PPO)[

71

],SoftActor-Critic(SAC)[

189

],andTrustArea

PolicyOptimization(TRPO)[

181

]arejustafewofthereinforcementlearningalgorithmsthatOpenAIhascreatedsofar.Thesealgorithmshavebeenemployedtotrainagentsforvarioustasks,includingplayinggamesandcontrollingrobots.OpenAIhascreatedmanysoftwaretoolsuptothispointtoassistwithitsresearchendeavors,includingtheOpenAIGym[

76

],atoolsetforcreatingandcontrastingreinforcementlearningalgorithms.Intermsofhardware,OpenAIhasinvestedinseveralhigh-performanceprocessingsystems,includingtheDGX-1andDGX-2systemsfromNVIDIA[

150

].ThesesystemswerecreatedwithdeeplearninginmindandarecapableofofferingtheprocessingpowerneededtobuildsophisticatedAImodels.ExceptforChatGPT,otherpopulartoolsdevelopedbyOpenAIincludeDALL-E[

141

]

andWhisper[

135

],Codex[

25

].AsummarizationoftheOpenAIproductpipelineisshowninFigure

2

.

2.2Capabilities

ChatGPTusesinteractiveformstoprovidedetailedandhuman-likeresponsestoquestionsraisedbyusers[

1

].ChatGPTiscapableofproducinghigh-qualitytextoutputsbasedonthepromptinputtext.GPT-4-basedChatGPTpluscanadditionallytakeimagesastheinput.Exceptforthebasicroleofachatbot,ChatGPTcansuccessfullyhandlevarioustext-to-texttasks,suchastextsummarization[

45

],textcompletion,textclassification[

86

],sentiment[

221

]analysis[

112

],paraphrasing[

104

],translation[

35

],etc.

ChatGPThasbecomeapowerfulcompetitorinsearchengines.Asmentionedinourintroductorysection,Google,whichsuppliesthemostexcellentsearchengineintheworld,considersChatGPTasachallengetoitsmonopoly[

188

].

ManuscriptsubmittedtoACM

6Zhangetal.

Notably,MicrosofthasintegratedChatGPTintoitsBingsearchengine,allowinguserstoreceivemorecreativereplies[

174

].WeseeanobviousdistinctionbetweensearchenginesandChatGPT.Thatis,searchenginesassistusersinfindingtheinformationtheywant,whileChatGPTdevelopsrepliesinatwo-wayconversation,providinguserswithabetterexperience.

Othercompaniesaredevelopingsimilarchatbotproducts,suchasLamMDAfromGoogleandBlenderBotfromMeta.

UnlikeChatGPT,theLaMDA,developedbyGooglein2021,activelyparticipatesinconversationswithusers,resultinginracist,sexist,andotherformsofbiasinoutputtext[

119

].BlenderBotisMeta’schatbot,andthefeedbackfromusersisrelativelydullbecausethedeveloperhassettighterconstraintsonitsoutputmaterial[

130

].ChatGPTappearsto

havebalancedthehuman-likeoutputandbiastosomelevel,allowingformoreexcitingresponses.Significantly,inadditiontobeingmoreefficientandhavingahighermaximumtokenlimitthanvanillaChatGPT,ChatGPTpoweredbyGPT-4cancreatemultipledialectlanguagesandemotionalreactions,aswellasreduceundesirableresults,therebydecreasingbias[

169

].Itisnotedin[

96

]thatthemodelingcapacityofChatGPTcanbefurtherimprovedbyusingmulti-tasklearningandenhancingthequalityoftrainingdata.

3TECHNOLOGYBEHINDCHATGPT

3.1Twocoretechniques

Backbonearchitecture:Transformer.BeforetheadventofTransformer[

182

],RNNwasadominantbackbonearchitectureforlanguageunderstanding,andattentionwasfoundtobeacriticalcomponentofthemodelperformance.

Incontrasttopriorworksthatonlyuseattentionasasupportivecomponent,theGoogleteammadeaclaimintheirworktitle:“AttentionisAllYouNeed"[

182

]claimedthatsinceGooglereleasedapaper,namely“AttentionisAllYouNeed"[

182

]in2017,researchanduseoftheTransformerbackbonestructurehasexperiencedexplosivegrowthinthedeeplearningcommunity.Therefore,wepresentasummaryofhowtheTransformerworks,withafocusonitscorecomponentcalledself-attention.

Theunderlyingprincipleofself-attentionpositsthatgivenaninputtext,themechanismiscapableofallocatingdistinctweightstoindividualwords,therebyfacilitatingthecaptureofdependenciesandcontextualrelationshipswithinthesequence.Eachelementwithinthesequencepossessesitsuniquerepresentation.Tocalculatetherelationshipofeachelementtootherswithinthesequence,onecomputestheQ(query),K(key),andV(value)matricesoftheinputsequence.Thesematricesarederivedfromthelineartransformationsoftheinputsequence.Typically,thequerymatrixcorrespondstothecurrentelement,thekeymatrixrepresentsotherelements,andthevaluematrixencapsulatesinformationtobeaggregated.Theassociationweightbetweenthecurrentelementandotherelementsisdeterminedbycalculatingthesimilaritybetweenthequeryandkeymatrices.Thisisgenerallyachievedthroughadotproductoperation.Subsequently,thesimilarityisnormalizedtoensurethatthesumofallassociationsequals1,whichiscommonlyexecutedviathesoftmaxfunction.Thenormalizedweightsarethenappliedtothecorrespondingvalues,followedbytheaggregationoftheseweightedvalues.Thisprocessresultsinanovelrepresentationthatencompassestheassociationinformationbetweenthecurrentwordandotherwordsinthetext.Theaforementionedprocesscanbeformallyexpressedasfollows:

Attention(Q,K,V)=Softmax()V.(1)

Transformertechniqueshavebecomeanessentialfoundationfortherecentdevelopmentoflargelanguagemodels,suchasBERT[

41

]andGPT[

18

,

122

,

136

,

137

]seriesarealsomodelsbasedonTransformertechniques.Thereisalsoa

ManuscriptsubmittedtoACM

OneSmallStepforGenerativeAI,OneGiantLeapforAGI:ACompleteSurveyonChatGPTinAIGCEra7

lineofworksextendingTransformerfromlanguagetovisuals,i.e.,computervision[

42

,

63

,

100

],whichsuggeststhat

TransformerhasbecomeaunifiedbackbonearchitectureforbothNLPandcomputervision.

Generativepretraining:Autoregressive.Formodelpertaining[

64

,

212

,

216

–

218

],therearemultiplepopulargenerativemodelingmethods,includingenergy-basedmodels[

56

,

159

,

160

,

186

],variationalautoencoder[

5

,

84

,

124

],GAN[

17

,

54

,

198

],diffusionmodel[

20

,

33

,

213

,

215

,

220

],etc.Here,wemainlysummarizeautoregressivemodelingmethods[

11

,

90

,

90

,

177

,

178

]astheyarethefoundationofGPTmodels[

18

,

122

,

136

,

137

].

Autoregressivemodelsconstituteaprominentapproachforhandlingtimeseriesdatainstatisticalanalysis.Thesemodelsspecifythattheoutputvariableislinearlydependentonitsprecedingvalues.Inthecontextoflanguagemodeling[

18

,

122

,

136

,

137

],autoregressivemodelspredictthesubsequentwordgiventhepreviousword,orthelastprobablewordgiventhefollowingwords.Themodelslearnajointdistributionofsequencedata,employingprevioustimestepsasinputstoforecasteachvariableinthesequence.Theautoregressivemodelpositsthatthejointdistribution

pe(x)canbefactorizedintoaproductofconditionaldistributions,asdemonstratedbelow:

pe(x)=pe(x1)pe(x2|x1) pe(xn|x1,x2,...,xn?1).(

2)

Whilebothrelyonprevioustimesteps,autoregressivemodelsdivergefromrecurrentneuralnetwork(RNN)

architecturesinthesensethattheformerutilizesprevioustimestepsasinputinsteadofthehiddenstatefoundinRNNs.Inessence,autoregressivemodelscanbeconceptualizedasafeed-forwardnetworkthatincorporatesallprecedingtime-stepvariablesasinputs.

Earlyworksmodeleddiscretedataemployingdistinctfunctionstoestimatetheconditionaldistribution,suchaslogisticregressioninFullyVisibleSigmoidBeliefNetwork(FVSBN)[

51

]andonehiddenlayerneuralnetworksinNeuralAutoregressiveDistributionEstimation(NADE)[

90

].Subsequentresearchexpandedtomodelcontinuousvariables[

177

,

178

].Autoregressivemethodshavebeenextensivelyappliedtootherfieldswithrepresentativeworks:PixelCNN[

180

]andPixelCNN++[

153

]),audiogeneration(WaveNet[

179

]).

3.2Technologypath

ThedevelopmentofChatGPTisbasedonaseriesofGPTmodels,whichconstituteasubstantialachievementforthefieldofNLP.AnoverviewofthisdevelopmentissummarizedinFigure

6

.Inthefollowing,wesummarizethekeycomponentsofGPTaswellasthemajorchangesintheupdatedGPTs.

Table1.ComparisonbetweenGPTandBERT.

Category

Description

Similarities

Backbone

BothGPTandBERTuseattention-basedTransformer.

LearningParadigm

BothG