無(wú)服務(wù)器架構(gòu)設(shè)計(jì)模式和最佳實(shí)踐

1、無(wú)服務(wù)器架構(gòu)設(shè)計(jì)模式和最佳實(shí)踐技術(shù)創(chuàng)新 變革未來(lái)AWS所提供的服務(wù)類(lèi)型AWSLambdaAmazonKinesisAmazon S3Amazon APIGatewayAmazon SQSAmazon DynamoDBAWS IoTAmazon EMRAmazon ElastiCacheAmazon RDSAmazon RedshiftAmazon Elasticsearch ServiceServerlessAmazon EC2Microsoft SQL ServerAmazon CognitoAmazonCloudWatch開(kāi)源服務(wù)自動(dòng)化、自有服務(wù)開(kāi)放化?，F(xiàn)在？無(wú)服務(wù)器化！“On EC2”M

2、anagedAWS Serverless 客戶S3 Bucket(s)Amazon AthenaKeyAWSManagement CloudTrail ServiceAmazon CognitoAWS IAMAmazon Kinesis StreamsAmazon Kinesis FirehoseAmazon QuickSightAWS GlueAmazon ESAmazonDynamoDBAmazon MacieAmazon API GatewayAWS IAMAmazon Redshift SpectrumAWS Direct Connect攝入Catalog & 搜索安全 & 審計(jì)API

3、/UI分析 & 處理AWS GlueAWSLambdaAWS Serverless Data LakeAWS Lambda客戶案例 - YublAWS Lambda 客戶案例 - YublA Lambda function that processes Kinesis events should also:retry failed events X times depending on processing timesend failed events to a DLQ after exhausting X retries/2017/04/aws-lambda-3-pro-tips-for-w

4、orking-with-kinesis-streams/設(shè)計(jì)模式: Lambda優(yōu)化Amazon API GatewayAmazon AlexaAWS IoTAmazon KinesisAmazon SNSAmazon SESAWS Step Functions213Invocations Functions InteractionsAmazon S3Amazon DynamoDBCustom endpointsAmazonAmazonElasticsearch CloudWatchEC2 instanceAWS Lambda 優(yōu)化Your functionLanguageruntimeExe

5、cution EnvironmentCompute substrateLambda函數(shù)無(wú)服務(wù)器的設(shè)計(jì)模式建立在Functions之上Functions是最小的部署和擴(kuò)展單元按照請(qǐng)求來(lái)自動(dòng)擴(kuò)展，用戶無(wú)需為這些關(guān)注這些擴(kuò)展的開(kāi)銷(xiāo)無(wú)需為idle的請(qǐng)求付費(fèi)跳過(guò)了最無(wú)聊的部分，也是越過(guò)了最難的部分AWS Lambda并發(fā)問(wèn)題 - 流量激增問(wèn)題增加您的并發(fā)執(zhí)行函數(shù)/zh_cn/lambda/latest/dg/concurrent-executions.htmlAWS Lambda 將根據(jù)增加的流量動(dòng)態(tài)擴(kuò)展容量，具體取決于您的AWS賬戶的并發(fā)執(zhí)行限制。為了處理突增流量，Lambda 將立即根據(jù)預(yù)定量增加您

6、的并發(fā)執(zhí)行函數(shù)。3000直到達(dá)到賬戶的安全限制， 或并發(fā)執(zhí)行的函數(shù)數(shù)量足 以成功處理增加的負(fù)載Lambda函數(shù)時(shí)間軸（Lambda不在VPC里面）Bootstrapthe runtimeStart yourcodeFull cold startPartial cold startWarm startDownloadyour codeStart newcontainerAWS optimizationYour optimizationAWS X-Ray跟蹤Lambda冷啟動(dòng)時(shí)間Before the function handler code executes, there is an also

7、additional 200ms of “Initialization”thats the time it takes the function to load its dependencies and run code outside of the handler function.AWS X-Ray跟蹤Lambda冷啟動(dòng)時(shí)間In the second “warm start” case, the function handler code starts almost immediately after the service receives the request to invoke t

8、he functionand no initialization occurs. Minimizing time spent in the Lambda service and avoiding function initialization on requests can have significant performance implications.Lambda函數(shù)代碼Avoid “fat”/monolithic functionsControl the dependencies in your functions deployment packageSeparate Lambda h

9、andler (entry point) from core logicOptimize for your languageNode Browserfy, MinifyLambda的注意事項(xiàng)和最佳實(shí)踐你的Lambda Function能在冰天 雪地中幸存下來(lái)么?(Code Start)要注意初始化AWS客戶端連接或 者數(shù)據(jù)庫(kù)連接的時(shí)候的變量scope， 盡可能的復(fù)用連接利用CloudWatch Events來(lái)做預(yù)熱ENIs for VPC將會(huì)在Code Start時(shí) 候被加載import sys import loggingimport rds_config import pymysqlrds

10、_host = rds-instance db_name = rds_config.db_name try:conn = pymysql.connect( except:logger.error(ERROR: def handler(event, context):with conn.cursor() as:cur僅在調(diào)用時(shí)被執(zhí) 行冷啟動(dòng)時(shí)執(zhí)行流失處理針對(duì)敏感的延時(shí)問(wèn)題 - Cold StartLambda是否使用VPC？Should my Lambda function be in a VPC?Does my function need to access any specific reso

11、urces in a VPC?Does it also need to access resources or services in the public internet?Dont put the function in a VPCPut the function in a private subnetPut the function in a subnet with a NATd route to the internetYesYesNoNoLambda配合VPC的時(shí)間軸（Lambda在VPC內(nèi)）Downloadyour codeStart newcontainerStart yourc

12、odeCreateAttachVPC ENI VPC ENIFull cold startWarm startBootstrapruntimePartial cold startAWS optimizationYour optimizationALWAYS configure a minimum of 2 Availability ZonesGive your Lambda functions their own subnetsGive your Lambda subnets a large IP rangeto handle potential scaleIf your functions

13、need to talk to a resource onthe internet, you need a NAT!Lambda的注意事項(xiàng)和最佳實(shí)踐 - Lambda in VPC了解重試機(jī)制同步模式不會(huì)重試異步模式重試兩次Stream-Base模式會(huì)一直重試使用Dead Letter Queues（私信隊(duì)列）使用SQS 或者 SNS 來(lái)重試記住: 重試次數(shù)也算一次調(diào)用Lambda“重試”機(jī)制Lambda的注意事項(xiàng)和最佳實(shí)踐文件系統(tǒng)如何考量?別忘記我們還有/tmp (512 MB 空間)exports.ffmpeg = function(event,context)new ffmpeg(./t

14、humb.MP4, function (err, video)if (!err) video.fnExtractFrameToJPG(/tmp) function (error, files) if (!error) console.log(files); context.done();.設(shè)計(jì)模式: 批處理批處理的特征大批的數(shù)據(jù)量定期的或者被計(jì)劃的任務(wù)Extract Transform Load (ETL)的工作模式大部分情況下不會(huì)長(zhǎng)期運(yùn)行大部分問(wèn)題都能用MapReduce的模式來(lái)解決無(wú)服務(wù)器架構(gòu) 批處理AWS Lambda: SplitterAmazon S3 ObjectAmazon S3

15、:Mapper ResultsAWS Lambda: Mappers.AWS Lambda: ReducerAmazon S3 Results無(wú)服務(wù)器架構(gòu) 批處理AWS Lambda: SplitterAmazon S3 ObjectAmazon DynamoDB:Mapper ResultsAWS Lambda: Mappers.AWS Lambda: ReducerAmazon S3 ResultsAWS Lambda處理map-reduce類(lèi)型工作Lambda Counting Function 0S3 BucketUsersLambda Counting Function 0Lambd

16、a Counting Function nSource DataSource DataLambda CountingFunction nSource DataSource DataLambda CascadeFunction 0Lambda CascadeFunction nNode.js Web AppRequest ResponseRequestResponseRequest ResponseRequest ResponseRequest ResponseRequest ResponseQuery responds with URL for the event stream. App in

17、vokes Lambda and stream resultServer-Sent Event streaming responseWeb UICascadeProcessQueryAWS中國(guó)（北京）區(qū)域由光環(huán)新網(wǎng)運(yùn)營(yíng)注意事項(xiàng)和最佳實(shí)踐串聯(lián)mapper functionsLambda批處理模式 vs. SQL (Hadoop / Spark生態(tài))處理數(shù)據(jù)和Lambda function的并發(fā)和限制需要取得平衡使用DynamoDB/ElastiCache/S3來(lái)管理mapper functions的狀態(tài)Lambda MapReduce Reference ArchitectureLambda已經(jīng)

18、集成了SQS事件觸發(fā)Lambda已經(jīng)集成了SQS事件觸發(fā)設(shè)計(jì)模式: 流式處理流式處理的注意事項(xiàng)高吞吐近乎實(shí)時(shí)的處理能力爆發(fā)性的間歇性流量消息持久化消息處理的保序無(wú)服務(wù)器架構(gòu)流式處理SensorsAmazon Kinesis:StreamLambda:Stream ProcessorS3:Final Aggregated OutputLambda: Periodic Dump to S3CloudWatch Events: Trigger every 5 minutesS3:Intermediate Aggregated DataLambda: Scheduled DispatcherKPL:P

19、roducerLambda流式處理并發(fā)問(wèn)題事件源Amazon Kinesis目的地 1Lambda目的地 2ShardsScale Amazon Kinesis by splitting or merging shardsPolls a batch Waits for responseLambda 自動(dòng)擴(kuò)展一個(gè) Kinesis Shard:If no record, wait some time (1s)Sub-batch in-memory and format records into Lambda payloadInvoke Lambda with synchronous invokeL

20、ambda blocks on ordered processing for each individual shardIncreasing # of shards with even distribution allows increased concurrencyBatch size may impact duration if the Lambda function takes longer to process more recordsSourceAmazon KinesisDestination 1ShardsScale Amazon Kinesis by splitting or

21、merging shardsDestination 2Lambda 自動(dòng)擴(kuò)展LambdaPolls a batWchaits for responseLambda流式處理并發(fā)問(wèn)題數(shù)據(jù)源ShardsTrim horizonCheckpointLatestCheckpointLambda流式處理并發(fā)問(wèn)題Event received by Lambda function is a collection of records from the stream: Records: kinesis: partitionKey: partitionKey-3,kinesisSchemaVersion: 1.0

22、,data: SGVsbG8sIHRoaXMgaXMgYSB0ZXN0IDEyMy4=,sequenceNumber: 49545115243490985018280067714973144582180062593244200961 ,eventSource: aws:kinesis,eventID: shardId-000000000000:49545115243490985018280067714973144582180062593244200961,invokeIdentityArn: arn:aws:iam:account-id:role/testLEBRole,eventVersio

23、n: 1.0, eventName: aws:kinesis:record,eventSourceARN: arn:aws:kinesis:us-west-2:35667example:stream/examplestream,awsRegion: us-west-2 Lambda流式處理并發(fā)問(wèn)題Lambda流式處理并發(fā)問(wèn)題Lambda流式處理Polls and blocks on synchronous invocation per shardIf put/ingestion rate is greater than the theoretical throughput, your proc

24、essing is at risk of falling behind Maximum theoretical throughput # shards * 2 MB / Lambda function duration (s)Effective theoretical throughput # shards * batch size (MB) / Lambda function duration (s)Retries 流失處理的重試機(jī)制Will retry on execution failures until the record is expiredThrottles and errors

25、 impact duration and directly impact throughputBest practiceRetry with exponential backoffUse case for service integrationRealtime clickstream logs to Kinesis streamReal-time Clickstream Anomaly Detection Reference:/cn/blogs/big-data/real-time-clickstream-anomaly-detection-with-amazon-kinesis-analyt

26、ics/扇出模式（fade-out）Kinesis shards的數(shù)量等同于Lambda的調(diào)用并發(fā)數(shù)量高吞吐和低延時(shí) vs 消息保序SensorsAmazon Kinesis:StreamKPL:ProducerLambda: ProcessorsIncrease throughput, reduce processing latency如果Kinesis在單個(gè)shard處理的時(shí)候超過(guò)了這個(gè)限制 級(jí)的處理能力，就需要考慮使用Fade out的模式了。分發(fā)者 Lambda: Dispatcher缺點(diǎn)？ 保序性！最佳實(shí)踐調(diào)教Kinesis每次調(diào)用 Lambda觸發(fā)器batch size的 值，默認(rèn)

27、是100，更大的值 能夠使得Lambda有更少的 調(diào)用次數(shù)調(diào)教Lambda的內(nèi)存 更快的處理速度盡量使用KPL的Batch Message功能來(lái)批量發(fā)送數(shù) 據(jù)，這樣可以更充分的利用Kinesis的高吞吐特性Lambda architectureData SourcesServing LayerSpeed LayerBatch Layer模式4: 自動(dòng)化處理自動(dòng)化處理所具備的特征處理報(bào)警和各類(lèi)事件周期性的計(jì)劃任務(wù)審計(jì)和通知對(duì)AWS服務(wù)功能性上的擴(kuò)展高可用 + 自動(dòng)擴(kuò)展自動(dòng)化處理: 動(dòng)態(tài)為EC2分配DNSAWS Lambda: Update Route53Amazon CloudWatch Eve

28、nts: Rule TriggeredAmazon EC2 Instance State ChangesAmazon DynamoDB:EC2 Instance PropertiesAmazon Route53: Private Hosted ZoneTag:CNAME = A 34/awslabs/route53-dynamic-dns-with-lambda自動(dòng)化處理: 從S3加載生成圖片縮略圖AWS Lambda:Resize ImagesUsers upload photosS3:Source BucketS3:Destination BucketTriggered on PUTs圖片

29、打水?。↙ambda + S3）/bjsdemo/Lambda-watermark-demo/Lambda_watermark_demo.zip圖片壓縮/裁剪（Lambda + S3）/bjsdemo/Lambda-CreateThumbnail-demo/CreateThumbnail.zip根據(jù)輸入?yún)?shù)線上裁剪圖片（API Gateway + Lambda + S3）/bjsdemo/Serverless-Image-Resizing-Demo/serverless-image- resizing-demo-Danrong.zip自動(dòng)化處理 - 配合Lambda還有很多常用場(chǎng)景EC2實(shí)例定時(shí)開(kāi)機(jī)、關(guān)機(jī)：/awslabs/ec2-scheduler/blob/master/code/ec2-scheduler.pyEBS定時(shí)打快照，并記錄打快照的是否成