Dragonfly圖像分發(fā)技術(shù)概述

1、Dragonfly圖像分發(fā)技術(shù)概述Dragonfly - Make Image Distribution Efficiently and SafelyAgenda030102IntroStatusOCI V204NydusPART ONEIntro01What &WhyP2P based, high reliable, efficient, image distribution infrastructureBottleneck in pulling images at large scale156.525110.306196.928285.896434.864663.83276157638.8

2、47639.736794.587871.9.7480.6980.2381.9183.05584.285.3457006005004003002001000Concurrency15102050100200300500800Average Cost(seconds)Image Distribution Average Cost Time(Native VS Dragonfly)NativeDragonflyP2P MechanismPull image from node proxySend pulling requests to SuperNodeCache image from Regist

3、ry if non-existReply to node peers which have blocksTransport blocks among all peersFinish whole pulling when all blocks downloadedHow to UseDocker Registry Mirrors-registry-mirror=https:/Containerd CRI Registry Mirrorsplugins.cri.registry.mirrors plugins.cri.registry.mirrors.docker.ioendpoint = htt

4、ps:/registry-1.docker.io plugins.cri.registry.mirrors.test.secure-registry.ioendpoint = https:/HostIP1:Port1 plugins.cri.registry.mirrors.test.insecure-registry.ioendpoint = http:/HostIP2:Port2Add redirection in your registryDeploy SuperNodeDeploy dfclient on every clientConfig Container Engine Daem

5、onPull images with DragonflyPART TWOStatus02EcologyCommunity SituationMilestonePART THREEOCI V203Container Image BackgroundContainer image is a lightweight, standalone, executable package of software including everything needed to run an application, e.g. code, system tools and librariesa3192f12d6c6

6、974kBcc91ded4e8bc3.97MB5a45e2e1049e12MB713a1caa34d73.77MBimage (R/O)Container Rootfs (R/W layer)/index.json/blobs/sha256/a3192f./blobs/sha256/708bc6./blobs/sha256/fce289/blobs/sha256/5a45e2/blobs/sha256/f9d5demanifest jsonconfig jsontar layersWhy need OCIv2 format specThe OCI organization is now dis

7、cussing the new OCIv2 format, which focuses on these issues in OCIv1 formatLazyloadWhile only a small fraction of image might be used by application, we waste a lot of disk space, image pulling time, network bandwidth and cpu cycles for the entire image downloading & decompressing procedureDeduplica

8、tionMetadata or small part of modified data will lead to a total file copy in the upper layer, while deleted files still downloaded in the lower layers.SecurityAfter decompressed, image files are not verifiable, and the targz lacks of signature verification, vulnerable to man-in-the-middle attacks.

9、How to verify data in the lazyload mode is still a problem.Community Prior WorkP2P based image distribution: Dragonfly, Kraken etc.Leveraging and rely on storage level snapshot capability:e.g. Slacker, Ceph rbd solutionsLayered userspace filesystem: CRFS, but it needs a new stargz file format, which

10、 is not a standardLeveragin file level blob and local file cache: NTT-filegrain, CERN-vmfs, umoci, but these images package too many blobs, and umoci also rely on local filesystem reflink capabilityAll these solutions lacks of data chunk level on-demand loading(except CRFS), data chunk in-flight ver

11、ification, or real digital signature preventing from man-in-the-middle attacks.PART FOURNydus04Nydus: The Dragonfly Image ServiceWe introduces Nydus, a new container image storage and distribution solution, which targets on current image problems and highly meets the ongoing OCIv2 image format requirementNydus Image FormatNydus BenchmarkFuture WorkOpen source as a reference implementation for incoming OCIv2 image specificationContribute to a new dragonfly p2p distribution ser