可控圖像生成綜述 Controllable Generation with Text-to-Image Diffusion Models A Survey

NumberofpapersNumberofpapersreviewoftheliteratureoncontradvancementsinthisdo(DDPMs)andwidelyusedT2Iresearchinthisarea,orgpleaserefertoourcuratedrepositoryat/PRIV-Creation/Awesome-Controllable-T2I-Diffusion-Models. 1INTRODUCTIONIFFUSIONmodels,representingIFFUSIONmodels,representingaparadigmshiftinthevisualgeneration,havedramaticallyoutperformedtraditionalframeworkslikeGenerativeAdversarialNet-works(GANs)[1]–[8].AsparameterizedMarkovchains,diffusionmodelsexhibitaremarkableabilitytotransformrandomnoiseintointricateimages,progressingsequentiafromnoisetohigh-fidelityvisualrepresentations.Withtheadvancementoftechnology,diffusionmodelshavedemon-stratedimmensepotentialinimagegenerationandrelateddownstreamtasks.Asthequalityofimagerygeneratedbythesemodelssimplyenhancingimageresolutionorrealism;itinvolvesmeticulouslyaligningthegeneratedoutputwiththeuser’sspecificandnuancedrequirementsaswellastheircreativeaspirations.Fueledbytheadventofextensivemulti-modaltext-imagedatasets[9]–[17]anddevelopmentofguidancemechanism[18]–[21],text-to-image(T2I)diffusionmodelshaveemergedasacornerstoneinthecontrollablevisualofgeneratingrealistic,high-qualityimagesthataccuratelyreflectthedescriptionsprovidedinnaturallanguage.Whiletext-basedconditionshavebeeninstrumentalinpropellingthefieldofcontrollablegenerationforward,theyinherentlylackthecapabilitytofullysatisfyalluserrequirements.Thislimitationisparticularlyevidentinscenarioswhereconditions,suchasthedepictionofanT2I（e.g.,LDM）Half-YearIntervals(2021-2024)(a)Half-yearlypapercount.(b)Schematicdiagramofcontrollablegeneration.Fig.1:Anoverviewofconditionalgeneratdiffusionmodel.(a)WeplotthenumberofpapersoncontrollablegenerationbasedonT2Idiffusionmodels,im-plyingthatitisincreasingrapidlyafterpowerfulgeneratorsarereleased.(b)WepresentaschematicillustrationofcontrollablegenerationusingtheT2Idiffusionmodel,wherenovelconditionsbeyondtextareintroducedtosteertheunseenpersonoradistinctartstyle,arenoteffectivelyconveyablethroughtextpromptsalone.Thesescenariosthenuancesandcomplexitiesofaredifficulttoencapsulateintextform.RecognizingthisintegratingnovelconditionsthatextendbeyondthecooftextualdescriptionsintoT2Idiffusionmodels.ThispivotTheseadvancementshaveledtotheexplorationofdiverseconditions,therebyenrichingthespectrumofpossibilitiesforconditionalgenerationandaddressingthemoreintricateandnuanceddemandsofusersinvariousapplications.TherearenumeroussurveyarticlesexploringtheAI-generatedcontent(AIGC)domain,includingdiffusionmodelmulti-modalimagesynthesisandediting[30],visualdiffu-sionmodel[31]–[34],andtext-to-3Dapplications[35].How-ever,theyoftenprovideonlyacursorybriefofcontrollingtext-to-imagediffusionmodelsorpredominantlyfocusonalternativemodalities.Thislackofin-depthanalysisoftheintegrationandimpactofnovelconditionsinT2Imodelshighlightsacriticalareaforfutureresearchandexploration. ingboththeoreticalfoundationsandpracticalapplications.Initially,weprovideaconciseoverviewofthebackgroundofT2Idiffusionmodelsanddelveintothetheoreticalunderpinningsofthesemethods,elucidatinghownovelconditionsareintegratedintoT2Idiffusionmodels.Thisandfacilitatesadeeperunderstandingofthefield.Subse-quently,weofferathoroughoverviewofpreviousstudies,highlightingtheiruniquecontributionsanddistinguishingfeatures.Additionally,weexplorethevariedapplicationsofthesemethods,showcasingtheirpracticalutilityandimpactindiversecontextsaInsummary,ourcontributionsare:?Weintroduceawell-structuredtaxonomyofcontrollablegenerationmethodsfromtheconditionperspective,sheddinglightontheinherentchallengesandcompltiesinthisstudyarea.?Weconductanin-depthanalysisoftwocoretheoreticalintoT2Idiffusionmodels:conditionalscorepredictionandcondition-guidedscoreestimation,providinganu-atagranularlevel.?Ourreviewiscomprehensive,coveringawiderangeofconditionalgenerationstudiesaccordingtoourproposedtaxonomy.Wemeticulouslyunderscorethesalientfeaturesanddistinctivecharacteristicsofeachmethod.?WeshowcasethediverseapplicationsofconditionalgenerationusingT2Idiffusionmodelsacrossvariousgenerativetasks,demonstratingitsemfundamentalandinfluentialaspectintheAIGCera.Therestofthispaperisorganizedasfollows.Section2providesabriefintroductiontodenoisingdiffusionproba-bilisticmodels(DDPMs),demonstratesthewidelyusedtext-to-imagediffusionmodels,andpresentsawell-structuredtaxonomy.InSection3,weanalyzethecontrollingmecha-imagediffusionmodels.InSection4approachesforcontrollingthetext-to-imagediffusionmodelaccordingtoourproposedtaxonomy.Finally,section7demonstratestheapplicationsofcontrollabletext-to-imagegeneration.2PRELIMINARIESDenoisingDiffusionProbanovelclassofgenerativemodelsthatoperateontheprincipleofreversediffusion.ThesemodelsareformulatedasparameterizedMarkovchainsthatsynthesizeimagesbygraduallyconvertingnoiseintostructureddatathrougha?ForwardProcess.Thediffusionprocessbeginswiththedatadistributionx0～q(x0)andaddsgaussiannoiseincrementallyoverTtimesteps.Ateachstept,thenoisedbyatransitionkernel:Tq(x1:T|x0):=Ⅱq(xt|xt?1),(1)t=1q(xt|xt?1)=Ⅵ(xt;‘1?βtxt?1,βtI),whereβtarevariancehyperparametersofthenoise.themodel’sobjectiveistoprogressivelydenoisethedata,therebyapproximatingthereverseoftheMaprocessbeginsfromthenoisevectorxTandtransitionstowardstheoriginaldatadistributionq(x0).Thegenerativemodelparameterizesthereversetransitionpθ(xt?1|xt)asanormaldistribution:pθ(xt?1|xt)=Ⅵ(xt?1;μθ(xt,t),Σθ(xt,t))(3)wheredeepneuralnetworks,ofteninstantiatedbyarchi-tectureslikeUNet,parameterizethemeanμθ(xt,t)andvarianceΣθ(xt,t).TheUNettakesthenoiseddataxtandtimesteptasinputsandoutputstheparametersofthenormaldistribution,therebypredictingthenoise?θthatthemodelneedstoreversethediffusionprocesxT～p(xT)andthensuccessivelysamplefromthelearnedtransitionkernelsxt?1～pθ(xt?1|xt)untilwereacht=1,completingthereversediffusionprocess.Inthissection,wespotlightseveralpivotalandwidelyutilizedtext-to-imagefoundationalmodels.Detailedinfor-mationregardingthesemodelsissystematicallycompiled?GLIDE[21].Togenerateimagesalignedwithfree-formtextprompts,GLIDEintuitivelyreplacetheclasslabelinclass-conditioneddiffusionmodels(i.e.ADM[18])withtext,formalizingthefirsttext-to-imagediffusionmodel.Theauthorsexploretwodifferentguidancefortext-conditioning.Forclassifierguidance,GLIDEtrainsaCLIPmodelinnoisyimagespacetoprovideCLIPguidance.Following[20],GLIDEadditionallyinvestigatescla(CFG)forcomparison,whichyieldsmorepreferableresultsinbothimagephoto-realismandtextualalignmeevaluatorsandischosenatext-to-imagegeneration.Fortextcondition,GLIDEfirsttransformstheinputtextcintoatokensequenceviaatrainabletransformer[36].Subsequently,theyreplacetheclassembeddingwiththepooledtextfeaturesandfurtherconcatenatetheprojectedsequencetattentioncontextateachattentionlayerindiffusionmodel.GLIDEtrainsthediffusionmtopredictp(xt?1|xt,c)andgenerateimageswithCFG.?Imagen[24].FollowingGLIDE,Imagenadoptsclassifier-freeguidance(CFG)fortext-to-imagegeneration.Insteadoftrainingatask-specifiedtextencoderfromscratchinGLIDE,Imagenleveragesapre-trainedandfrozenlargelanguagemodel(LLM)asitstextencoder,aimingtoreduceanalysisofvariousLLMs,includingthosetrainedonimage-BERT[40],T5[37]).Theirfindingssuggestthatincreasingthescaleoflanguagemodelsmoreeffecthefidelityofsamplesandthecongruencebetweenimageandtext,comparedtotheenlargementofimagediffusionmodels.Furthermore,Imagen’sexplorationintodifferentrepresentationsofimagesfromcont[39],DALL·E2,alsoknownasunCLIP,trainsagenerativediffusiondecodertoinverttheCLIPimageencoder.Thegeneratingprocessconsistsofthefollowingsteps.First,givenanimagecaptionyanditstextembeddingzt,apriorp(zi|zt)bridgesthegapbetweenCLIPtextandtheimagelatentspace,whereziistheimageembedding.Second,aSpecifically,thedecoderisadiffusionmodelmodifiedfromthearchitectureofGLIDE,wheretheCLIPembeddingisprojectedandaddedtotheexistingtime-stepembedding.Thepriorcanbeoptimizedusingeitheranautoregressiveapproachoradiffusionmodel,withthelatterdemonstratingsuperiorperformance.?LatentDiffusionModel(LDM)[23].Toenablediffusionmodeltrainingandinferenceonlimitedcomputationalandflexibility,LDMappliesthedenoisingprocessinthelatentspaceofpre-trainedautoencoders.Specifically,theautoencoderEmapsimagesx∈Dxintoaspatiallatentspacez=E(x).Todevelopaconditionalimagegenerator,LDMenhancestheunderlyingUNetwiththecross-attentionmechanismtoeffectivelymodeltheconditionaldistributionp(zt?1|zt,c),wherecistheconditionalinput,suchastextpromptsandsegmentationmasks.Intherealmoftext-to-imagegeneration,theauthorsemploytheLAION-400Mdatasettotraina1.45billionparametertext-to-imageLDMmodel,capableofproducing[40]wasutilizedasthetextencoder.?StableDiffusion(SD).BuiltupontheLatentDiffu-sionModel(LDM)framework,StabilityAIdevelopedandlaunchedseveralseriesoftext-to-imagediffusionmodels,termedStableDiffusion.SDdemonstratesunparalleledcapabilitiesintext-to-imagegeneration,andwithitsmodelsbeingopen-sourced,ithasgainedwidespreadusagewithinthecommunity.Thetaskofconditionalgenerationutilizingtext-to-diffusiontheconditionperspective,wedividethistaskintothreesub-tasks(refertoFigure2).Mostworksstudyhowtogeneration,andsketch-to-imagegeneration.Torevealthemechanicaltheoryandfeaturesoftheseapproaches,wefurthercategorizethemaccordingtotheirconditiontypes.Theprimarychallengeinthistaskliesinhowtoenablepretrainedtext-to-image(T2I)diffusionmodelstolearntowithtextualconditionswhileensuringtheimagesproducedareofhighquality.Additionally,somemethodsinvestigatehowtogenerateimagesusingmultipleconditions,suchasgivenacharacter’sidentityandpose.Themainchallengeinthesetasksistheintegrationofmultipleconditions,necessitatingthecapabilitytoexpressseveralconditionssimultaneouslyinthegeneratedresults.Furthermore,someworksattempttodevelopacondition-agnosticgenerationapproachthatcanutilizetheseconditi3HOWTOCONTROLTEXT-TO-IMAGEDIFFUSIONMODELSWITHNOVELCONDITIONSInthissection,wepresentthecontrollingmechanismofdif-[188],wecansettheaμθ(xt,t)inEquation.3as:μθ(xt,t)=xt?1tsθ(xt,t)(4)wheresθ(xt,t)isaneuralnetworkthatlearnstopredictthescorefunction?xtlogpt(x).InDDPM,wehave:?xtlogpt(x)=??where?～N(0,I)isthetβt,andμθ(xt,t)=gaussiannoiseusedinforwardt:=u=0αs.Then,Equation.4(xtt(xt,t))(6)where(xt,t)predicts?.Inconditionalgeneration(cdenotescondition),thescorefunctionisextendedwithaposteriorprobabilityterm?xtlogpt(c|x)andbecomes?xtlog(pt(x)p(x|c))(wrepresentsahyper-parametertocontrolconditioninten-sity),following[18],[20].ToemployaneuralnetworkforTABLE1:Collectionofprimaryandusedtext-toencoder’sparameters(defaultrefersonlytoUNet).f:downsamplingfactorofautoencoderinlatent-CLIP:opensourceimplementationofCLIP.*:trainfromscratch.ModelModelPublicationParam.ResolutionfTextEncoderTrainingDatasetOpenSourcePixelSpaceDiffusionModelsGLIDE[21]Imagen[24]DALL·E2[38]ICML2022NeurIPS2022arXiv2022256210242---plainTransformer*[36]T5-XXL[37]CLIP*[39]&Diffusionprior*DALL·E[22]>LAION-400M[16]CLIP[39]&DALL·E[22]???LatentSpaceDiffusionModelsLDM[23]SDv1.x[23]SDv2.x[23]SDXL[25]ICLR2024903M860M865M256251225122/7682102428888BERT-tokenizer[40]CLIP-ViT-L/14[39]CLIP-ViT-H/14[39]CLIP-ViT/G&CLIP-ViT/L[39]LAION-400M[16]LAION-2B[17]LAION-5B[17]internaldataset????conditionalgeneration,classifier-freeguidance(CFG)[20]transformsitto:=Δxtlogpt(從)+山Δxtlog)bytrainingamodelEθ(從t,·,t),whichpredicttheformerviaEθ(從t,φ,t)andthelatterviaEθ(從t,c,t).ExistingT2IdiffusionmodelstrainEθ(從t,·,t)byrandomlydroppingthetextprompt,andthedenoisingprocesswithCFGisasfollows:t,ctext,t)=(1?山)Eθ(從t,φ,t)+山Eθ(從t,ctext,t)(8)t,ctext,t)isusedinEquation.4forconditionalsyn-Hence,thekeytocontrollingtext-to-imagemod-elswithnovelconditionscnovelistomodelscoreΔxtlogpt(從|ctext,cnovel).Following[18],[32],therearetwotypesofmechanisms,i.e.,conditionalscorepredictionandconditioned-guidedscoreestimation,whichweillustratebelow.WhileT2IdiffusionmodelsleverageEθ(從t,ctext,t)topredictΔxtlogpt(從|ctext),afundamentalandpowerfulwayforsteeringdiffusionmodelsicnovelintoEθ(從t,ctext,t),constructinga(從t,ctext,cnovel,t)tostraightforwardlypredictΔxtlogpt(從|ctext,cnovel).Then,thedenoisingprocesswithCFGofconditionalscorepredic-tionmethodsisasfollows:t,ctext,ccond,t)(9)t,ctext,ccondt,ctext,ccond,t)(9)Wehereillustrateseveralmainstreamwaystoattaint,ctext,cnovel,t).proachesemploysanadditionalencoderEtoencodenovelconditionalscorepredictionprocessisasfollows:t,ctext,cnovel,t)=Eθ*(從t,ctext,E(cnovel),t)(10)whereEandθ*aretrainable.TheschematicillustrationisshowninFigure3a.basedmethodstypicallyfocusonadaptingtoaspecificcondition,ofteninscenarioswithlimiteddata,suchassingleorfew-shotexamples.Thesemethodsachieveconditionalpredictionbytransformingeitherthetextconditionctextorthemodelparametersθintoaformspecifictothegivent,ctext,cnovel,t)=Eθ*(從t,cext,t)(11)whereconditioninformationismemorizedinctextandθ.?Training-freeConditionalScorePrediction.Whiletheabovetechniquesrequireatrainingprocess,somemethoaredesignedinatraining-freemanner(refertoFigure3c).TheyintroduceconditionstocontrolthegenerationdirectlythroughtheintrinsicabilityofthestructureofUNet,suchasmodulatingthecross-attentionmaptocontrolthelayout[135],[142]orintroducingfeaturesofthereferenceimageinself-attentiontocontrolthestyle[101].UnlikeconditionalscorepredictionapproachespredictingΔxtlogpt(從|ctext,cnovel),condition-guidedestimationap-proachesaredesignedtogainΔxtlogpt(cnovel|從t)withoutneedofCFG,whichgenerallytrainanadditionalmodelwithparametersΨtopredicttheconditionfromlatentorinternalΔxtlogpt(cnovel|從)viabackpropagation,asillustratedinFigure4.Andthedenoisingprocessnowreads:t,ctext,cnovel,t)=(從t,ctext,t)+TΔxtlogpφ(cnovewhereTisahyper-parametertoadjustthet,ctext,t)istheoriginalscorepredictionoftext-conditioneddiffusionmodelswithCFG.ControllableGenerationwithText-to-ImageDiffusionModelsPersonalizationSpatialControlGenerationwithspecificcondition(§4)AdvancedText-ConditionedIn-Context(§4.4)Brain-Guided(§4.5)Sound-Guided(§4.6)JointTraining(§5.1)ContinualLearning(§5.2)GenerationwithmultipleControllableGenerationwithText-to-ImageDiffusionModelsPersonalizationSpatialControlGenerationwithspecificcondition(§4)AdvancedText-ConditionedIn-Context(§4.4)Brain-Guided(§4.5)Sound-Guided(§4.6)JointTraining(§5.1)ContinualLearning(§5.2)Generationwithmultipleconditions(§5)Attention-basedIntegration(§5.4)GuidanceComposition(§5ConditionalScorePrediUniversalControllableGeneration(§6)Condition-GuidedScoreEstimation(§6.2)(a)(a)Subject-Driven:TextualInversion[41],Dreambooth[42],Re-ImaCustomDiffusion[45],DVAR[46],E4T[47],ELITE[48],UMM-Diffusion[49],XTI[50],SVDiff[51],ANOVA[52],SuTI[53],Jiaetal.[54],InstantBootPACGen[65],Araretal.[66],Subject-Diffusion[67],LyCORIS[68]Heetal.[71],KCLoss[72],MATTE[73],Lego[74],CaHiFiTuner[78],VideoBooth[79],CAFE[(b)Person-Driven:FaHyperDreamBooth[87],PhotoVerse[27],MagiCapture[88],Face-diffuser[89],W+Adapter[90],RetriBooru[91],FaceStudio[92],ViscoPortraitBooth[97](c)Style-Driven:StyleDrop[98],StyleCrafter[99],ArtAdapter[100],StyleAligned[101],SAG(d)Interaction-Driven:Reversion[103],AnimateDiff[104],MotionDirector[105],LAMP[106],SAVE[107],Materzynskaetal.[108],DreaMoving[109],MotionCrafter[110],InteractDiffusion[111](e)Image-Driven:unCLIP[112],VersatileDiffusion[113],Prompt-FreeDiffusion[114],Uni-ControlNet[115],IP-Adapter[116],ViscoNet[93],ContextDiffusion[117],FreeControl[118](f)Distribution-Driven:Caoetal.[119],DreamDistribution[120]eDiff-IeDiff-I[26],LGP[121],SpaText[122],GLIGEN[123],UnivMCM[126],FreeDoM[127],FLIS[128],LayoutDiffusion[129],HumanSD[1Giambietal.[84],GeoDiffusion[132],Attention-Refocusing[133],ZestGuide[134],CADenseDiffusion[137],JointNet[138],HyperHuman[139],Region&Boundary[140],EOCNet[141],MATTE[73],LoCo[142],AnyLens[143],LRDiff[144],LooseControInteractDiffusion[111],FreeControl[118],LocalControl[146],SCStructureStructureDiffusion[149],Attend-and-Excite[150],GlueGen[151],Rich-texTailoredVisions[154],ParaDiffusion[155],PEA-Diffusion[156]PromptPromptDiffusion[157],iPromptDiff[158]Mind-VisMind-Vis[159],Takagietal.[160],Brain-Diffuser[161],MindDiffuser[162],Nietal.[163],DreamDiffusion[164],BrainVis[165]LiuLiuetal.[167],GlyphDraw[168],TextDiffuser[169],GlyphControl[170],AnyText[171],TextDiffuser-2[172],UDiffText[173],DiffC-C-LoRA[177],L2DM[178],STAMINA[179]CustomCustomDiffusion[45],Cones[180],Mix-of-Show[181],ZipLoRA[182],OrthogonalAdaptation[183]DiffBlenderDiffBlender[186],EmuFig.2:TaxonomyofControllableGeneration.Fromtheconditionperspective,wecategorizecontrollablegenerationapproachesintothreesub-tasks,includinggenerationwithspecificconditions,generationwithmultipleconditions,anduniversalcontrollablegeneration.4CONTROLLABLETEXT-TO-IMAGEGENERATIONWITHSPECIFICCONDITIONSBuildinguponthefoundationoftext-to-imagediffusionmodels,introducingnovelconditionstosteerthegenerativeprocessrepresentsacomplexandmultifacetedtask.Inthefollowingchapters,wereviewtheexistingmethodsofprovidingacomprehensivecritiqueoftheirmethodologies.Personalizationtaskaimstocaptureandutilizeconceptsasgenerativeconditions,whicharenoteasilydescribablethroughtext,fromexemplarimagesforcontrollablegen-eration.Inthissection,weprovideanoverviewofthesepersonalizedconditions,categorizingthemtoofferaclearerunderstandingoftheirdiverseapplicationsandfunctional-ities.WeillustratetheresultsofpersonalizationinFigureInthissection,weprovideadetailedoverviewofsubject-drivengenerationmethods.Thesubject-drivengenerationtask(alsoknownassubject-centricpersonalization)isde-signedtoproducevisualcontentthatretainsthesubjectsofprovidedsamples.Inpractice,manysubject-drivengen-erationmethodsarenotconfinedtoconditionsspecificctextpφ(cnovel|xt)xt,tctextctextpφ(cnovel|xt)xt,tctextcnovelxt,tctextEConditionEncoderTextEncoderEθ(xt,ctext,E(cnovel),t)(a)Model-basedConditionalScorePrediction.xt,tEθ(xtxt,tTextEncoder(b)Tuning-basedConditionalScorePovelxt,tctextTextEncoderEθ(xt,ctext,cnovel,t)(c)Training-freeConditionalScorePrediction.ConditionPredictorEθxt,ctext,t+y?xlogpφ(cnovel|xt)TextEncodertosubjecttypes;theyoftendemonstrateamoreuniversalcapability.Thus,manyoftheapproachesdiscussedinthischaptercanbeextendedtoawiderrangeofcustomizedtasks.Insummarizingtheseworks,weadoptabroaderperspectivetoshowcasetheirgeneralapplicabilityasmuchaspossible,aimingtofacilitateabetterunderstandingoftheircontributionsandroles.AccordingtothecontrollingmechanismmentionedinSection3,sinceallofthesemethodsuseconditionapredictiontointroducetheconditions,wecategorythembytheirpipelines:tuning-basedmethods,whichadaptmodelparametersorembeddingstocatertospecificcondi-tions;model-basedmethods,employingencoderstoextractpersonalizedconditionsandfeedingthemintodiffusionmodels;andtraining-freemethods,whichleverageexternalreferencestosteerthegenerativeprocesswithouttheneedoftraining.yeteffectivewaytograspconceptsfromprovidedsamplesinvolvesselectivelytuningasubsetofparameterstorecon-Asthebasicinputfortext-to-imagediffusionmodels,textplaysacrucialroleinadaptingthesemodelstospecificuserneeds.TextualInversion(TI)[41]adoptsaninnovativeapproachbyembeddinguser-providedconceptsintonew’words’withinthetextembeddingspace.Thismethodexpandsthetokenizer’sdictionaryandoptimizesadditionalBooth[42]followsasimilarpathbututilizeslow-frequencywords(i.e.,sks)torepresentconceptsandadditionallyupdatestheparametersoftheUNetwithaclass-specificpriorpreservationlosstoenhancethediversityofgeneratedoutputs.ThestraightforwardandadaptableframeworksofTIandDreamBoothestablishthemasfoundationalmodelsfornumeroussubsequenttuning-basedmethods.Further-more,CustomDiffusion[45]analyzesweightdeviationsduringthefine-tuningprocessanddiscoversthepivotalroleofcross-attentionlayerparameters,particularlykeyandvalueprojections(i.e.,WkandWv).Thisinsightleadstoaextratexttokensandregularizationlossforfine-tuning.embeddingspace,particularlybyconsideringthedistinctionofeachUNetlayer[50],[61].Theyapplydistincttextembeddingsacrossvariouslayers.Incontrast,CatVersion[75]divergesfromthefocusontextembeddingsandtheUNet’sparametersandadvocatesfortuningconcatenatedembeddingswithinthefeature-densespaceofthetextinlearningthenuancesbetweenapersonalizedconceptanditsbaseclass,contributingtothepreservationofpriorknowledgewithinthemodel.Inaddition,parameter-efficienttu[194]playsapivotalroleinpersonalizationmethods[63].whichoptsfortheadapter[191],andrevealsthatplacingadapterssubsequenttothecross-attentionblockenhancesperformancesignificantly.TofacilitatethecomprehensiveapplicationandevaluationofPEFTinthefine-tuningofdif-methods,includingbutnotlimitedtoLoRA[192],LoHa,andDyLoRA[193].LyCORISfurtherintroducesadetailedframeworkforthesystematicanalysisandassessmentofthesePEFTtechniques,significantlyadvancingthefieldofdiffusionmodelpersonalization.Moreover,acriticalchallengeintherealmofperson-alizationisthedisentanglementofspecificconceptsfromtheprovidedsamples.Numerousstudies[59],[62],[65],[74],[81]haveidentifiedacommonissuewhereextraneousinformationbecomesintertwinedwiththeintendedconceptduringthecustomizationprocess,suchasinadvertentlylearningthecontextsurroundingimagesinsubject-driven1./KohakuBlueleaf/LyCORISMixSoundofCarHornandDogBarkingMixSoundofCarHornandDogBarking(a)PersonalizationInteractionPersonImageDistributionInteractionPersonImageMaskEdgeDepthMaskEdgeDepth(c)In-ContextGeneration(d)Paragraph-to-ImageGenerationAAclose-uppictureofpeopleandscenery.Thesubjectisamiddle-agedman.Amaningrayclothingisstandingonarockbythesea.Heiswearingablackhat.Themanhashishandsinsertedintothepocketsofthegrayclothing.Thebackgroundisthevastoceanandsky,withafewwhitecloudsinthesky.(f)Sound-GuidedGeneration(e)(f)Sound-GuidedGenerationcuriouslyexaminedthestreetsignthatreads‘bioethanol’