AIGC-也混淆了-AI：調(diào)查和解釋大型視覺語言模型中合成圖像引起的幻覺

AIGCsConfuseAIToo:InvestigatingandExplainingSyntheticImage-inducedHallucinationsinLargeVision-LanguageModelsYifeiGao1,JiaqiWang1,ZhiyuLin1,JitaoSang1,2BeijingJiaotongUniversity,ChinaPengChengLab,Shenzhen518066,China{yifeigao,jiaqiwang,zyllin,jtsang}@AbstractTheevolutionofArtificialIntelligenceGeneratedCon-tents(AIGCs)isadvancingtowardshigherquality.ThegrowinginteractionswithAIGCspresentanewchallengetothedata-drivenAIcommunity:WhileAI-generatedcon-tentshaveplayedacrucialroleinawiderangeofAImod-els,thepotentialhiddenriskstheyintroducehavenotbeenthoroughlyexamined.Beyondhuman-orientedforgeryde-tection,AI-generatedcontentposespotentialissuesforAImodelsoriginallydesignedtoprocessnaturaldata.Inthisstudy,weunderscoretheexacerbatedhallucinationphe-nomenainLargeVision-LanguageModels(LVLMs)causedbyAI-syntheticimages.Remarkably,ourfindingsshedlightonaconsistentAIGChallucinationbias:theobjecthallu-cinationsinducedbysyntheticimagesarecharacterizedbyagreaterquantityandamoreuniformpositiondistribution,eventhesesyntheticimagesdonotmanifestunrealisticoradditionalrelevantvisualfeaturescomparedtonaturalim-ages.Moreover,ourinvestigationsonQ-formerandLinearprojectorrevealthatsyntheticimagesmaypresenttokende-viationsaftervisualprojection,therebyamplifyingthehal-lucinationbias.1.IntroductionWiththerapidevolutionofgenerativemodeltechniques,ArtificialIntelligenceGeneratedContents(AIGCs)haveusheredinaneweraofprosperity[21,26].AIGCsarenolongermereoutputsofgenerativemodels;rather,theyen-compasstheinformationgeneratedduringhuman-modelormodel-modelinteractions[3].ThisresultinalargeamountofsyntheticcontentrapidlyfloodingintotheInternet,andpeoplemayhaveinteractedwithsyntheticcontentuncon-sciously.Nevertheless,thepervasiveAI-generatedcontentmaygiverisetoseveralchallenges.Thefirstchallengetogainwidespreadattentionisforgerydetection[20].ThisfieldFigure1.Hallucinationexamplesonbothsynthetic(right)andnat-uralimages(left),wherethehighlightedfontsindicatethehallu-cinatedcontent.Evaluationresultsacrossvariousvision-languagetasks,suchassemanticdescriptionsandfactualjudgments,con-sistentlyillustratetheexistenceofasyntheticimage-inducedhal-lucinationbias.aimstoassisthumansindistinguishingbetweennaturalandsyntheticcontentandisconsideredacrucialaspectinAIsafety.Particularly,arecentstudy[17]hasindicatedthattherecognitionofsyntheticimageshasresultedinanapproxi-mate40%humanerrorrate,solidifyingthefactthathumansareeasilyconfusedbyAIGCs.Anotherunder-examinedchallengeistheimpactofAI-generatedcontentonAImod-elsthemselves.Assyntheticdataplaysamorecommonroleinthetrainingandreasoningprocess[28],thehiddenrisksofAIGCstotheAImodelsareurgentyetlargelyun-explored.TakingFigure1asanexample,syntheticimageswithidenticalsemanticsaremorelikelytoinducehallucina-tionsinLVLMsthannaturalimages.Thisislargelybecausethesemodels’trainingdata,architecture,andtrainingpro-cessesareinherentlydesignedfornaturaldata.Applyingmodelstrainedonnaturaldatatosyntheticdatasetswithoutcomplicationscouldleadtounexpectedoutcomes.WearemotivatedtoexplorethetopicsofhowsyntheticdatamayimpactAImodels.Inparticular,thisstudyfo-cusesonthehallucinationissuesthatsyntheticimagesmaycauseinLargeVision-LanguageModels(LVLMs).Beforeaddressingthecoreresearchquestion,wefirstestablishasyntheticimage-involvedhallucinationevaluationenviron-mentforLVLMs.ManycurrentgenerativemodelsadopttheText-to-Imagesynthesisapproach[30,21].However,thegenerationprocessleadstotwoprimaryissues:1)se-manticdistortion[31],wheresyntheticimageslackauthen-ticity(e.g.,thefingerproblem[17])and;2)semanticam-biguity[18],wheresyntheticimageslackconsistencyandstruggletorespondtotextprompts.Giventheabsenceofanavailablesyntheticimagedatasetforhallucinationeval-uation,mitigatingtheimpactoftheaforementionedissuesisnecessary.Tothisend,weintroduceaSemanticsTrans-lation(ST)method,whichbeginswithanaturalimageandemploys1)captiongenerationandrevision,and2)seman-ticfilteringstrategiestocontroltheauthenticityandconsis-tencyofthesyntheticimage,ensuringthattheevaluationisnotaffectedbythequalityofthesyntheticimage.Wetranslatetwowidelyusedhallucinationevaluationdatasets:POPE[12]andAMBER[22],anddelveintothehallucinationsinducedbysyntheticimages.Wealsocom-parethemwiththecorrespondingnaturalimages.Surpris-ingly,ourfindingsindicatethatLVLMshaveabiastowardssyntheticimages,asshowninFigure1.Werefertothisphenomenonassyntheticimage-inducedhallucinationbias(shortenashallucinationbias).Ourfurtherexperimentsre-vealthatthehallucinationbiasmainlyexhibits1)agreaterquantityand2)amoreuniformpositiondistributionofhal-lucinatedcontent.Particularly,thesephenomenaarecor-roboratedacrossdifferentLVLMsandevaluationdatasets.Inotherwords,theseLVLMsappeartoadoptsomeinherentnon-semanticshortcutsinsyntheticimages,whichleadtoacontinuousimpactontheextrapolationprocess.Then,wearecommittedtofurtherstudyinghowsyn-theticimagesconfuseLVLMs.DrawinginspirationfromthevisualprojectionprocessofLVLMs,weexaminetwoprevalentvisualprojectionmodules:Q-formerandLinear.Specifically,ourinvestigationsshedlightonthefactthat1)turningofftheQ-formerprojectionor2)deepeningthelayersoftheLinearprojectioncan1)effectivelymitigatethetokendeviationofsyntheticimagesand2)narrowthesyntheticimage-inducedhallucinationbias.Thatistosay,LVLMstunedinthiswaymaygeneratelesshallucinatedcontentinresponsetosyntheticimages.Ourcorecontributionsareasfollows:(1)InthecontextoftherapiddevelopmentofAIGC,weexploretheimpactofsyntheticimagesonthehallucinationproblemofLVLMsforthefirsttime.Toachievethis,weintroduceasemanticstranslationmethodtoestablishasyntheticimage-involvedhallucinationevaluationenvironment.(2)Extensiveexper-imentsuncoverthesyntheticimage-inducedhallucinationbiasofLVLMs,mainlymanifestingas(i)agreaterquan-tity,and(ii)amoreuniformpositiondistribution.(3)Weprovideanin-depthanalysisonthesyntheticimage-inducedhallucinationbiasfromtheperspectiveofvisual-textalign-ment.Experimentalresultsrevealthatthedesignofthepro-jectionmodulemaycausethetokendeviationofsyntheticimages,thuscontributingtothehallucinationbias.2.RelatedWorksAIGCanditsChallenges:RecentadvancesinArtificialIntelligenceGeneratedContents(AIGCs)haveprofoundlytransformedtheapproachtocontentsgeneration,andof-ferednumerousbenefitsforhumansinvariousaspectsoflifeandwork[25].Forinstance,generativemodelssuchasStableDiffusion[18]orChatGPT-4[1]cangeneratehigh-qualityimageortextinformationbyadheringtothetextualdescriptionspromptedbyusers.However,asAIGCsareprogressivelyintroducedintotheonlineworldandappliedtosociety,it’sessentialtotakenoteofcertainpotentialhid-denriskstheymayintroduce.Thefirstchallengetogainwidespreadattentionisfakedetection,whichhasbeenseenastheboundaryofAIsafety[20].Anotherprevailingchal-lengestemsfromtheinsufficientexaminationontheimpactofAI-generatedcontentonAImodelsthemselves.Specif-ically,certainresearcheshaveidentifiedmultifacetedrisksassociatedwithAIGCs,includingissuesrelatedtosecurity,bias,andprivacybreaches[3,8,24,4].Furthermore,re-centstudies[6,27]havedemonstratedthatsyntheticdatacanintroducesourcebiasinbothtextandcross-modalre-trievalforwebsearch,leadingtoanelevatedrankingofAI-generatedcontent(AIGCs).Therefore,withinthecontextofthewidespreadapplicationofAIGC,itiscrucialtothor-oughlyinvestigatethepotentialrisksitmaypose.HallucinationsofLVLMs:LargeVisionLanguageMod-els[2,15,29,34](LVLMs)areregardedasanaturalexten-sionofLargeLanguageModels[32](LLMs).Throughrichvisual-textinstructions-tuning,LVLMshavedemonstratedremarkableprogressintacklingcomplexmulti-modaltasks,suchasVisualGroundingandVisualQuestionAnswering(VQA)[2,15,29,34].However,LVLMsarealsoplaguedbythehallucinationproblem[16,22,12],inwhichthemodeleither1)depictsinaccurateobjectsor2)entirelyfab-ricatescontentfromassociatedimages.Thesephenomenarepresentamajorbottleneckintheirdeploymentandthuslimittheirpracticalityinmanyscenarios[33,5].Recentresearchhasdelvedintohallucinationproblemsfromtheperspectiveofevaluationandmitigation[16].Intermsofevaluation,POPE[12]definesthehallucinationproblemasabinaryclassificationtask,aimingtoexplorethemodel’sperceptualabilitywithrespecttospecificobjectspresentintheimage.Meanwhile,AMBER[22]hascontributedtothemostcomprehensivehallucinationevaluationdatasetFigure2.Thepipelineofsemanticstranslationmethod.Ontheleftside,weintroducecaptiongenerationandrevisionmethodtosynthesizeacorrectdescriptionofthegivennaturalimage.Redrepresentstheredundantorincorrectinformationwithintheinitialcaption.Ontherightside,weutilizeimagesynthesisandfilteringstrategytosamplethefinalsyntheticimage,ensuringastrictcorrespondencetotherevisedcaptionandtheinputnaturalimage.highlightedrepresentstheredundantobjectinimagesynthesisprocess.Thefinalsyntheticimagesatisfiesthecriteriaofauthenticityandconsistency.byextendingthebinaryevaluationapproachofPOPEandintroducinggenerativetaskevaluation.Themitigationofhallucinationgenerallyinvolvesthreeaspects:(1)Pre-processing,commonlyachievedbyprovidinghigh-qualityimage-textpairsandwell-designedinstruction-tuning[13];(2)In-processing,wherethemodelisenhancedbystrength-eningvisual-textualfeaturelearning[9];(3)Post-hocpro-cessing,knownforitssuperiorscalability,usuallyalleviatesmodelhallucinationsinthedecodingstage[10].3.SemanticsTranslationWefocusonensuringaccuratehallucinationevaluationonsyntheticimages,withthepreconditionofexcludingthequalityinfluence.Specifically,asyntheticimageshouldmeet:1)authenticity,wheresyntheticsemanticsshouldalignwithhumancognition;and2)consistency,wherethesyntheticimageshouldaccuratelyrespondtothetextprompt.Inthissection,weintroduceasemanticstranslationmethodtosynthesizehigh-qualityimages.AsshowninFig-ure2,thesynthesisprocessofsemanticstranslationmethodisconstrainedunderanaturalimagesupervisionthroughthefollowingtwosteps:1)CaptionGenerationandRevi-sion,transformingvisualsemanticsintodetailedtextualse-mantics(Sec.3.1);and2)ImageSynthesisandFiltering,involving(i)imageover-samplingand(ii)imagefilteringbasedonsimilarity(Sec.3.2).3.1.CaptionGenerationandRevisionInthissubsection,wetranslatethevisualsemanticsofthegivennaturalimageintothetextualsemantics.AsshowninFigure2(left),weemployGPT-4V(ision)tocap-turethekeysemantics.Tomaintainaccuratetextualseman-Figure3.Thecomparisonofobjectpositionsbeforeandafterthecaptionrevision.TakingStableDiffusionasanexample,wheretheacceptedcharacterlimitis77,thedistributionofkeysemanticsintherevisedcaptiongenerallysatisfiesthecharacterlimits.tics,werevisetheextractedinformationthroughGPT-3.5.CaptionGeneration:Inordertoensurethatthegenera-tivemodelreceivestextpromptscloselyalignedwiththesemanticsofthegivennaturalimage,weemployGPT-4Vtoobtaincoarse-grainedcaptions.However,twonotableissuespersistinthegeneratedcaptions:1)redundantormissinginformation,wherethegeneratedsemanticsdonotexistorfailtoincludetheobjectthatshouldbepresentedintheimage;and2)excessivecaptionlength,wherethegeneratedcaptionsoftenexceedthewordlimitacceptedbymostgenerativemodels.Thisresultsinthelossofsemanticinformationbeyondthewordlimit,therebydisruptingtheconsistencyofsemantics.CaptionRevision:Tomitigatetheaforementionedissues,thegeneratedcaptionarerevisedbyGPT-3.5.Specifically,weprovidemanualannotationstoassistGPT-3.5incom-prehendingandrevisingtheexistence,quantityandthere-lationsemanticsinthescene.Detailedinstructionisavail-ableintheAppendix.AsshowninFigure3,thelengthoftherevisedcaptionisgenerallyinlinewiththewordlimitsetbythegenerativemodel,ensuringthatallkeysemanticinformationcaneffectivelypromptthegenerativemodel.3.2.ImageSynthesisandFilteringGiventherevisedcaption,weutilizeStable-Diffusionv1.5tosynthesizeasetofcandidateimages.AsshowninFigure2(right),weapplyafilteringstrategytothecandi-dateset,resultinginthefinalsyntheticimagewiththemostauthenticandconsistentsemantics.ImageSynthesis:Therevisedcaptionsareemployedasin-putpromptsforimagesynthesis.Wecanmoreeasilysam-plethesyntheticimagethataresimilartonaturalimagebyincreasingthesamplingtimes.Therefore,weadoptanover-samplingstrategybyconductingmultiplegenerationswithdifferentrandomseedstoobtainasetofcandidateimages.ImageFiltering:Thefilteringprocessincludestwostages:(1)Ensuringauthenticsemanticsinsyntheticimages,withafocusonavoiding(i)thedepictionofobjectsnotexist-inginnaturalimagesor(ii)introducingobjectsthatcontra-dicthumancognition.Toachievethis,weinitiallyextractobjectsusingautomatedsegmentationtools[35].Subse-quently,weeliminateimagesdisplayinganexcessorab-senceofobjectsintheirannotationresultswhencomparedtothecorrespondingnaturalimage.(2)Maintainingconsis-tentsemanticswithnaturalimages,withafocusonthesim-ilaritybetweensyntheticandnaturalimages.Specifically,wefilterthecandidatesetfromtwodimensions:(i)Imageperceptualsimilarity,referredtoastheperceptualsystem’sunderstandingofthesimilaritybetweentwoimages(e.g.,high-levelsemanticsintermsofattributesandrelationsofobjects).WeuseDreamSim[7],whichbettercorrespondstohumanperception,tomeasuretheperceptualsimilaritybetweensyntheticandnaturalimages.(ii)Imagesemanticfaithfulness,referredtoasthealignmentwithtextualan-notations(e.g.,existence-levelsemantics).Specifically,wecomputethecosinesimilaritybetweenthesyntheticimageandtextualannotationsthroughtheCLIP[19]model.De-tailedsettingsareavailableintheAppendix.Finally,weselecttheimageswithlowerDreamSimscoresandhigherCLIPscoresasthefinalsyntheticimages.4.HallucinationsonSyntheticImagesAfterexcludingtheinfluenceofsyntheticimagequalityissuesonhallucinationevaluation,thissectiondelvesintothesyntheticimage-inducedhallucination.Tomoreintu-itivelyreflecttheimpactofsyntheticimages,wealsoreportthehallucinationresultsonthecorrespondingnaturalim-agesandconductafaircomparisoninthecontextofcon-sistentsemantics.Specifically,wefirstintroducetheevalu-ationdatasetsandmetricstoensureacomprehensiveexam-ination(Sec.4.1).Subsequently,wequantifythesyntheticimage-inducedhallucinationsacrossdifferentLVLMsanddatasetsfromperspectiveofhallucinationquantityandpo-sitiondistribution(Sec.4.2).Finally,wefurtherinvesti-gatetheeffectsonhallucinationbiasthroughexperimentsinvolving1)prompttemplates,and2)generationtempera-tures(Sec.4.3).4.1.ExperimentSetupDataset:Weselectedtwowidelyuseddatasets,POPE[12]andAMBER[22],asbenchmarksforhallucinationevalua-tion.Syntheticimagescorrespondingtothesedatasetsareobtainedthroughsemanticstranslationmethod.POPEfo-cusesonexistence-typehallucination,comprising500im-ageswithcorresponding9000annotations.AMBERoffersarichersettingwithdiversedatasetscaling,reasoningtasks,andhallucinationtypes.Specifically,AMBER1)includes1004imageswithcorresponding15200annotations;2)as-sessesbothgenerativeanddiscriminativetasksreasoningand;3)encompassesthreetypesofmodelhallucination,in-cludingexistence,attribute,andrelation.MetricsonGenerativeTaskReasoning:WefollowthesettingsinAMBER,whereCHAIRandCoverareusedtoevaluatethehallucination.CHAIRmeasuresthefrequencyofhallucinatedobjectsintheresponses,whiletheCoverreferstoasthecoverageofobjectsoccurringinnaturalim-ages.Generally,anidealresponseshouldmaintainalowhallucinationlevelwithoutsacrificingresponsequalitytoomuch,whichmeansalowerCHAIRandahigherCover.MetricsonDiscriminativeTaskReasoning:Thehalluci-nationevaluationforthediscriminativetaskisusuallyde-finedasabinaryclassification.Consideringtheimbalanceddistributionofyesandnoanswersinthequestionannota-tions,wereferredtoPOPEandadoptedvariousmetrics,includingAccuracy,Precision,Recall,andF1-score.Ad-ditionally,wereportthe’Yes’ratioinPOPEtorevealtheconfidencebehaviorofLVLMs.ModeltobeEvaluated:Weconducthallucinationeval-uationonthecurrentmainstreamLVLMs,includingMiniGPT4-13B[34],LLaVA-v1-7B[15],LLaVA-v1.5-7B[14],mPLUG-Owl-7B[29]andQwen-VL-13B[2].4.2.OverallEvaluationResultsonSyntheticImage-inducedHallucinationObservationonHallucinationQuantity:Table1andTa-ble2presenttheevaluationresultsofthemainstreamopen-sourceLVLMsonbothAMBERandPOPEdatasets.Aconsistentobservationemergesthathallucinationsinducedbysyntheticimagesaregenerallymorepronouncedthanthoseobservedinnaturalimages.Additionally,wehavethefollowingnoteworthyobservations:(1)LVLMsareeas-ilyconfusedbysyntheticimagesingenerativetasksreason-ing.Thismanifestsparticularlyini)anincreasedfrequencyTable1.TheoverallevaluationresultsofPOPEonbothsyntheticandnaturalimages.?indicatesthehallucinationgapbetweennaturalandsyntheticimages.Weuse?representthesyntheticimage-inducedhallucinationbias.ModelImageRandomPopularAdversarialAccuracyF1AccuracyF1AccuracyF1MiniGPT-4(13B)Natural70.0072.3858.6062.5067.3164.7063.4368.2165.03Synthetic66.7071.2756.7058.4366.6364.5757.8766.1964.60?3.301.111.904.070.680.135.562.020.43mPLUG-Owl(7B)Natural60.2070.9987.2053.2367.3993.4353.5067.7594.17Synthetic58.9070.0487.1752.4367.4153.4352.8767.2593.93?1.300.950.030.80-0.0240.000.630.500.24LLaVA-v1(7B)Natural62.4772.5586.7355.5369.0293.5353.3368.0295.93Synthetic60.1071.3883.4353.2068.0592.4752.4067.6693.70?2.371.173.302.330.971.060.930.362.23LLaVA-v1.5(7B)Natural90.0090.1251.2086.4087.0154.6779.7081.7461.17Synthetic84.3783.5745.1781.3781.0148.1074.7375.8454.60?5.636.556.035.036.006.574.975.906.57QWen-VL(13B)Natural86.0784.1838.0784.9083.2740.2382.7381.2942.27Synthetic78.3773.3331.1076.8372.2333.4374.9770.7035.43?7.7010.856.978.0711.046.807.7610.596.84Table2.TheoverallevaluationresultsofAMBERonbothsyntheticandnaturalimages.Weconsideronegenerativeandthreediscrimina-tivetasks,includingtheunderstandingonexistence,attributeandrelationsemanticsofobjects.ModelEXISTENCEAMBER??mPLUG-Owl????ofhallucinatedobjectsandii)thelimitedcoverageofob-jectsoccurringintheimage.Moreover,giventheconsis-tencyconstraintonglobalsemanticsbetweennaturalandsyntheticimages,itiscounter-intuitivefortheLVLMstogeneratemorehallucinationsinresponsetosyntheticim-ages.ThisobservationsuggeststhatLVLMsmaycapturesomenon-semanticshortcutsbeyondthecapabilitiesofthehumanvisionsystem.(2)Theprimarysourceofsyntheticimage-inducedhallucinationsindiscriminativetasksrea-soningstemsfromtheattributesemantics.Table2providesadetailedcomparisonamongthreediscriminativetasks,re-vealingthatsyntheticimagesinducehigherhallucinationFigure4.Hallucinationstatisticsondifferentdiscriminativetasksreasoningwithineachpairofsyntheticandnaturalimage.Discriminativetaskconsiderreasoningonattribute,existenceandrelationsemantics,separately.Wehighlightthattheattributesemanticcontainstheaction,numberandstateinformationoftheannotatedobjects,separately.Figure5.Therelativepositiondistributionofhallucinatedobjectsbetweensyntheticandnaturalimages.resultsinattributesemantics.Ourfurtheranalysisincludesacomparisonofhallucinationnumberswithineachpairofsyntheticandnaturalimagesacrossdiverseattributese-mantics,involvingnumber,action,andstatesemanticsofobjects.AsshowninFigure4,syntheticimagesinducehigherhallucinationsacrossallthreeattributes.(3)Thesyn-theticimage-inducedmodelreasoningbehaviorappearstobeunder-confident.AsshowninTable1,asurprisingob-servationemergesthatthe’Yes’ratioinLVLMsreasoningaboutsyntheticimagesismuchlowerthanthatofnaturalimages.Thisimpliesthatthenon-semanticshortcutsinsyn-theticimagesweakentheconfidenceofreasoningprocess.Inotherwords,syntheticimagesaremorelikelytoinducethemodeltosay’No’.However,ontheAMBERresultofexistencetaskrea-soning,Qwen-VLexhibitshigheraccuracyonsyntheticim-ages,yieldinginconsistentbehavior.Weattributethisdis-paritytothenatureofexistence-typehallucinationannota-tionsintheAMBERdataset,whichconsistentirelyofcoun-terexample(i.e.,questionsthatconsiderobjectsnotpresentintheimage).TheresultsinPOPEindicatethatQwen-VLhasthelowest’Yes’ratioamongallevaluationmodels,sug-gestingalowconfidenceinreasoningdiscriminativetasks.GiventhisdiscrepancytakesadvantageofAMBER’sanno-tationtosomeextent,itdoesnotimpacttheoverallfindings.Table3.Hallucinationevaluationongenerativetaskunderdif-ferenttemplates,wherebrief-descanddetailed-descrefertoas”Generateabrief/detailedcaptionoftheimage”,separately.Redindicatesamoreseverehallucinationbias.ModelImagebrief-descdetailed-descCHAIR(↓)Cover(↑)CHAIR(↓)Cover(↑)MiniGPT-4(13B)Natural5.3032.2058.00Synthetic7.7028.5048.50?-2.403.70-3.409.50mPLUG-Owl(7B)Natural9.7039.7020.6048.60Synthetic33.2021.0044.30?-2.806.50-0.404.30LLaVA-v1.5(7B)Natural2.8036.306.2049.80Synthetic6.7032.1043.10?-3.904.20-4.106.70Qwen-VL(13B)Natural6.1030.606.3046.30Synthetic21.9032.20?-6.708.70-8.5014.10ObservationonHallucinationPositionDistribution:WemainlyfocusongenerativetaskreasoningandperformKernelDensityEstimation(KDE)examiningtherelativepositiondistributiononbothsyntheticandnaturalimage-inducedhallucinatedobject.AsshowninFigure5,weob-servethatinLVLMs’responsestonaturalimages,hallu-cinatedobjectstendtoappearmoreatthefrontofthere-sponse,correspondingtothepeakofthedensitycurve(i,e.,bluedistribution)locatedatthebeginningoftherelativepo-sition.Incontrast,inresponsestosyntheticimages,hallu-cinatedobjectsarerelativelyuniformlydistributedacrossvariouslocations(i.e.,purpledistribution).Thisobserva-tiondirectlyindicatesthatLVLMsusuallygeneratemore”securitycontents”attheendoftheresponse.Incontrast,syntheticimagesexhibitacontinuousimpactontheextrap-olationprocessofLVLMs,thusresultinginhigherhalluci-nationresults.4.3.AblationStudyPreviousanalyseshavedemonstratedthatthehallucina-tionsinducedbysyntheticimagesdifferfromthoseofnat-uralimages,characterizedbyagreaterquantityandamoreuniformdistribution.WedefinetheabovephenomenonasFigure6.(a)Quantityevolutionofhallucinatedobjectswhenre-sponsetobothsyntheticandnaturalimages.(b)TheevolutiontrendofthehallucinationquantitywithrelativeresponselengthonMiniGPT-4-13B.thesyntheticimage-inducedhallucinationbias.Inthissub-section,wefurtherinvestigatetheeffectsonhallucinationbias,specificallyfromtheperspectivesofprompttemplatesandgenerationtemperature.ObservationonPromptTemplates:Forgenerativetask,AMBERusesthemostconciseandcommonlyusedgener-ativeprompt,”Describethisimage”toobtaindescriptionsofimagesfromLVLMs.DrawinginspirationfromPOPE’sdesign,weopt”Generateabrief/detailedcaptionoftheim-age”,separately,andexploretheinfluenceofprompttem-platesonthehallucinationbias.Table3presentstheeval-uationresultsofdifferentLVLMsundertwoprompttem-plates.Anoticeablehallucinationbiaspersistsforsyntheticimages,regardlessofwhethertheprompttemplateisde-signedforobtainingdetailedorbriefdescriptions.Intriguingly,weobservethatthelong-textgenerationprocessappearstoamplifythehallucinationbias,indicat-ingthatthetrendofincreasingthequantityofhallucinatedcontentinsyntheticimagessurpassesthatinnaturalimages.Buildinguponthisfinding,weseekinsightsintothequan-tityevolutionofhallucinatedobjectsinthetwotypesofim-ages.AshypothesizedinFigure6(a),thehallucinationbiasinpositiondistributionrevealsthatsyntheticimagesexertacontinuousimpactontheextrapolationprocess,leadingtotheongoinggenerationofhallucinations.Incontrast,natu-ralimagestypicallyachievethesaturationofhallucinationsbeforesyntheticimages,thusamplifyingthehallucinationbias.Nevertheless,theextrapolationlengthistypicallypredeterminedbeforethereasoning(e.g.,’maxnewtoken’generallydoesnotexceed512),imposinganupperboundonthehallucinationsquantityforbothtwotypesofimages.ResultsonMiniGPT-4confirmourhypothesis,asshowninFigure6(b).ObservationonTemperature:Thetemperatureservesasacrucialhyper-parameterincontrollingtherandomnessandcreativityofthetextgenerationprocessinLVLMs.Wein-vest