Python 中的機(jī)器學(xué)習(xí)

MACHINELEARNINGINPYTHON

(PART4):

DIFFUSIONMODELSINPYTORCH

LUKESHENEMAN

GENERATIVEARTIFICIALINTELLIGENCE

TexttoImage(StableDiffusion)

TexttoVideo

GenerativeAI:Learnalatentrepresentationofthedistributionofourcomplextrainingdataandthensamplefromit

TrainingData

DeepLearning

Diffusion,etc.

Transformers,etc.

DIFFUSIONMODELS

CONDITIONINGIMAGEGENERATION

Providenaturallanguagetextpromptsto

guidereversediffusionprocess

Text-to-ImageDiffusionModelsareboth:

ImageGenerationModels

LanguageModels

StableDiffusionArchitecture

OVERVIEW

RecapfromParts1-3

MachineLearningBasics

NeuralNetworks

Tensors

ConvolutionalNeuralNetworks(CNNs)

GPUsandCUDA

PyTorch

WhyusePyTorch?

ImplementingaDiffusionModelinPython

TrainandTestourDiffusionModel

REVIEWOFBASICS

Machinelearningisadata-drivenmethodforcreatingmodelsforprediction,optimization,classification,generation,andmore

Pythonandscikit-learn

MNIST

ArtificialNeuralNetworks(ANNs)

MNIST

NEURALNETWORKBASICS

WeightsandBiases

FULLY-CONNECTEDNEURALNETWORKS

Imagesaretensors!

FEATUREHIERARCHIES

Weneedimagefilterstohelpusextractfeatures

EXAMPLE: SOBELFILTER

Sobelkernels=

CONVOLUTIONALNEURALNETWORK

GPUvs.CPU

“Moore’sLawforCPUsisDead”

WHYGPUSEXACTLY?

CNNsareallaboutmatrixandvectoroperations(multiplication,addition)

GPUscanperformparallelmultiplicationandadditionstepspereachclockcycle.

FrameworksmakeGPUsEasy

DIFFUSIONMODELS

FORWARDDIFFUSION

Definehowmanytimestepswillbeused(commontousehundredsormore)

EstablishanoiseschedulewhichdescribestherateatwhichGaussiannoiseisadded

Linear

Cosine

T=0 T=1 T=2 T=3 …

T=n

Iused100timesteps. LargermodelslikeStableDiffusionusethousandsofsmallersteps.Iusedacosinenoiseschedule.

TIMESTEPENCODING

30

30

+ =

RGBImage Integertimestep 4-ChannelRGB+Timestep

Iencodetimestepasanotherbandintheimageinpixel-space

U-NetArchitecture

4

U-NetDenoiser

3

RGB+T RGB

TrainingourNeuralNetwork

PossiblelossfunctionsforourU-Net

loss1=MSE(predt,original)loss2=MSE(predt,noisyt-1)loss3=predt-noisyt

OtherHyperparameters:

Epochs=100

Timesteps=100BatchSize=1250Optimizer=AdamLearningRate=0.001

CoreTrainingLoop

schedule=cosine_schedule(TIMESTEPS)foreachEpoch:

foreachBatchb:

foreachTimestept:

img=add_gaussian_noise(img,schedule(t))predicted=UNet(img)

loss=loss_function(img,predicted)backward_propagationandoptimization

CELEBFACESATTRIBUTES(CELEBA)DATASET

202,599numberoffaceimagesofvariouscelebrities

10,177uniqueidentities,butnamesofidentitiesarenotgiven

40binaryattributeannotationsperimage

5landmarklocations

Images”inthewild”orCropped/Aligned

SOMEPRELIMINARYOUTPUT

Ohno!

UseseparateAImodelforupsampling

64

64

512

SRResNet

512

/twtygqyy/pytorch-SRResNet

MyModel

Mightnotbeterrific,but…

Itwastrainedononly5000imagesforafewhoursonasingleRTX4090GPU

StableDiffusionwastrainedon600millioncaptionedimages

Took256NVIDIAA100GPUsonAmazonWebServicesatotalof150,000GPU-hoursAtacostof$600,000

StableDiffusion

ConditioningreverseDiffusiononTextprompts

PRE-PROCESSINGCELEBADATASET

Readfirst5000annotationsintoPANDASdataframe(easy!)

Foreachimage,gettheheadingnamesforpositiveattributes

Convertheadingnamesintoatextprompt:

e.g.“Photoofperson<attribute_x>,<attribute_y>,<attribute_z>,…”

e.g.“Photoofpersonbushyeyebrows,beard,mouthslightlyopen,wearinghat.”

Cropthelargestsquarefromtheimage,thenresizeto64x64x3numpyarray

UseOpenAICLIPmodeltofindtheimageembeddingsandtextembeddingsforeveryimage/promptpair.

Createa5000elementPythonlistof4-tuples:

(filename,64x64xRGBimage_array,image_embedding,prompt_embedding)

Picklelisttoafilewecanquickyloadintomemorywhenwetrainourmodel!

OPENAICLIPMODEL (CONTRASTIVELANGUAGE–IMAGEPRE-TRAINING)

Opensource/weightsmulti-modalAImodeltrainedonimage,captionpairs

Sharedembeddingspace!

Usetransformermodel(GPT-2)tocreatetokenembeddingsfromtext

Usevisiontransformer(VIT)tocreatetokenembeddingsfromimages

CLIPExamples/research/clip

USINGCLIPISTRIVIAL

/openai/CLIP

Zero-shotclassifications!ConditioningGenerativeAI(DALL-E)

GeneratingcaptionsforimagesorvideoImagesimilaritysearch

ContentModerationObjectTracking

CLIPusesvectorswith512dimensionsGPT3(Davinci)uses12888dimensions

Vectorembeddingscapturethedeepersemanticcontextofawordortextchunk…orimage…oranything.

Thesemanticsofanobjectaredefinedbyitsmulti-dimensionalandmulti-scaleco-occurrenceandrelationshipswithotherobjectsinthetrainingdata

Semanticvectorembeddingsarelearnedfromvastamountsofdata.

400,000,000(image,text)pairs

CLIPwastrainedon256largeGPUsfor2weeks.

SizeofEmbeddingVector

Onewaytotrainamulti-modalembeddinglayer

“AcuteWelshCorgidog.”

Learningyoursemanticembeddingsf