電商平臺(tái)的原生遷移技術(shù)實(shí)踐

電商平臺(tái)的原生遷移技術(shù)實(shí)踐SomeLessons

WeLearnedfromMovingE-businessGianttoCloud

NativeAlibaba’sJourneytoCloud

Native2011:

containerizeLXCOnepoolper

BU2015:dockerize

+smart

schedulingDockerIn-housescheduling2017-18:k8swithbasic

orchestration“Rich

Container”O(jiān)nePod

onecontainerK8S

API2019:cloud

nativeContainerdPod

withsidecarsFullK8S

stackExplore(K8sAPIcentric)Cloud

Native(Standard+Open)Containerize(container

centric)Current

StateAlibababegantomoveitse-businessplatformtocloudStandingontheshoulderofopen

source:KubernetesOperator

FrameworkCNI,CSI,CRI,DevicePlugin

…PrometheusContainerdrunC+

KataContainersDevOpsframeworkfrom

ACKACK=AlibabaContainerService

forKubernetesandmuchmore

…ArchitectureEliminate“Rich

Container”Before

2018JavaPID1processis

SystemdALLinONEcontainer("RichContainer"),independent

upgradeapp,sshd,log,monitoring,cache,VIP,DNS,proxy,agent,start/stopscripts

…Traditionaloperating

workflowStartcontainer->SSHintocontainer->Startthe

appLogfiles&userdataaredistributedeverywhereinthecontainerIn-houseorchestration&scheduling

systemEliminate“Rich

Container”Declarative:TheBiggestBenefitisUpgradeofOperating

ModelImperative:Create3more

containersDeletecontainernamedxxxUpgradethiscontainerwithnew

imageOnenodehasbroken,weshoulddeletethosecontainersonitandcreatetheminother

placesIneed5replicasformy

applicationThisapplicationshouldupdatetoanewversionAutomation

!Stability!Efficiency!As

ResultResourceutilizationCo-locationbyunified

schedulerApp

managementCloudNativeAppMgmtStorage

strategyDatainpersistentvolumeCooperatewithdevelopersalloverthe

worldStandard+Open+KeepupdatingwithupstreamK8s1.10->1.12->1.14

…CloudNativeApp

MgmtCloudNativeApp

MgmtK8s

作業(yè)管理挑戰(zhàn)巨大WorkloadMgmtinWeb-scaleis

ChallengingapplicationmigrationcomplicatedscenehelpPaaSmoveto

cloudnativecontainertransformationstabilityguaranteeWhatwearemovingtoCloudNative:almostonehundred

sitesmorethanahundredthousandapplicationsnearlyonemillion

containersBeforeAlibabaSinglesDaysale,we

may:CreatemorethanahundredthousandofPodsoverthousandsofapplicationsNodesofallthesePodswillbecalculatedbyaoff-linebatchschedulerbeforePod

creation

Throughthisway,wecanimproveresourceutilization,cpucoreallocation,affinity/anti-affinityofapplicationsandso

onBatch

placementBatch

placementBatchAllocationPlanCRDforcreatingbufferPodswithcandidate

nodesBatchAdoptionCRDforhelpingworkloadadoptbuffer

PodsApplication

upgradeAlldefaultworkloadsrecreatePodsduringrollingupdate.Strategiesthey

offered:maxSurge/maxUnavailable/partition/…What’sthe

problem?IfweusethedefaultupdatepolicyinK8s,thedeterminacywillbebrokenTopologychanges,imagere-warm,unexpectedoverhead,resourceallocationchurn

…Introduce:in-place

updateAdvanced

StatefulSetPod-1appsidecarPod-0appPod-1sidecar appsidecarNodeNodeKubeletUpdateappimageintemplatePartition=1Patchimage

ofapp

containerFindhash

changedRecreatecontainerwithnew

imageKubeletComparationRecreate

updateInPlaceupdateClusterdeterminacyEfficiencyofimage

downloadingRequirementof

resourceReschedulingandservice

registrationRecovery

automaticallySupportallfields

updaterolling

updateinplace

updateSidecar managementDefinedinapp

workloadsHardtomanagewhentherearelotsofapplicationsand

workloadsApplicationdevelopersdon’tknowwhichsidecartouseHowtoupdatesidecarintoomany

workloads……Injectedby

SidecarSetDefinesidecarsalone,partfromapplicationworkloadsApplicationdevelopershavenottocareaboutsidecarsUpdatesidecarcontainersin-place,noeffecton

applicationsSidecarSet

injectionWhenwehavethousandssidecars,we’ll

need:ASidecarSetCRD:Describeallsidecarsneedtobe

operatedA

SidecarOperator:Injectsidecarcontainerstoselected

PodsUpgradesidecarcontainersfollowingrolloutpolicywhenSidecarSetis

updatedDeletesidecarcontainerswhenSidecarSetisdeletedWhatcanSidecarSet

doSidecarSetcan

doIstiocan

doUpdate

strategyIn-placeupdateMountdata

volumesResourcelimitwithpodMore

configurationLabel

selectorInject

locationBasic

injectionapp

containersidecar

containerpod1.copy

data2.

trigger3.read

datasharedvolume0.updatesidecar

in-placeHotupgradeusing

sidecarOpenKruise/openkruise/kruiseSharewhathaveextremelyhelpedusmovetoCloud

NativeAdvancedStatefulSetAdvancedStatefulSetMaxUnavailableImprovespeedof

updating.(ForaStatefulSetwith500Podsandupdatein10batches,thismayaccelerate50times.)InPlaceUpdateUpdatePodswithoutrecreate.AvoidPodsrescheduleand

reshuffleReadinessGateensuresPodsstayNotReadyduring

updating.Morefeaturescoming

soon…SidecarSetand

BroadcastJobSidecarSetInjectsidecatcontainerintomatchedPods.Currently,itcanjustdoinjectionduringpodcreating,andwewillsupportsidecarupdatein-place

soonBroadcastJobAblendofDaemonSetand

JobRunpodsonallmachinesexactly

onceUsecase:softwareupgrade,nodevalidator,nodelabeler

etcScalabilityMattersin

AlibabaScalabilityboundaryofupstreamK8s

(v1.14)Nomorethan5k

nodesNomorethan150ktotal

podsNomorethan300ktotal

containersNomorethan100podsper

nodeScalabilitygoalinourweb-scale

clusterMorethan10k

nodesMorethan300k

podsNon-goal:Totalcontainers&podsper

nodeQuestion:Howtodiscoverscalabilityissuein10knodes

cluster?PerformanceBenchmark

ToolkitkubemarkwithHTTP

interfaceHollow-NodePodscmd/kubemark/hollow-node.goTaintanddrainnodesforperftest,andrun

itTypicaltestcasesin10knodes

cluster:Startuptimeduringscaling

podsTimeofcreatinganddeletingpodsPodlisting

RTFailure

countscurl-XPOST-H"Content-Type:application/json"

\"/api/kubemark/test"

\-d

'{"test_focus":"\\[Feature:Performance\\]","test_skip":"handle","node_count":10000,"pods_per_node":30}'Howto

run?DiscoverPerformance

BottlenecksFixingPerformanceBottleneckswith

Upstream#9296#9384#9511#9418bbolt#141#10283PodListIndexing:

~35x

(upstream

soon)Watch

Bookmark:#75474

(New!)etcd APIServer KubeletBug

fix:#72709LockAlgorithm

~24x*Cherrypick:Incrementalheartbeat

#14733PerfConfigureEtcd100clients,1millionrandomkeyvaluepairs,5000

QPSCompletiontime:

~200sLatency:99.9%in

97.6msK