阿里云-歸檔存儲(chǔ)-最佳實(shí)踐-D

1、歸檔存儲(chǔ)最佳實(shí)踐彈性伸縮/常見(jiàn)問(wèn)題彈性伸縮/常見(jiàn)問(wèn)題 PAGE 15 PAGE 15最佳實(shí)踐Python SDK 代碼示例Python SDK 簡(jiǎn)易使用示例本節(jié)以示例的方式展示如何使用SDK高級(jí)接口進(jìn)行開(kāi)發(fā)。用戶(hù)在閱讀本節(jié)后，可模仿示例，并參考高級(jí)接口一 節(jié)進(jìn)行開(kāi)發(fā)。其中，方括號(hào)內(nèi)的參數(shù)用戶(hù)應(yīng)根據(jù)實(shí)際需求進(jìn)行替換。- 創(chuàng)建 Vaultfrom oas.oas_api import OASAPI from oas.ease.vault import Vaultfrom oas.oas_api import OASAPI from oas.ease.vault import Vault# 創(chuàng)建 O

2、ASAPI 對(duì)象api = OASAPI(Server Host, Access Key ID, Access Key Secret)# 創(chuàng)建 Vaultvault = Vault.create_vault(api, Vault Name)- 查找Vault# 創(chuàng)建 OASAPI 對(duì)象# 創(chuàng)建 OASAPI 對(duì)象api = OASAPI(Server Host, Access Key ID, Access Key Secret)# 根據(jù)名稱(chēng)獲取 Vaultvault = Vault.get_vault_by_name(api, Vault Name)# 根據(jù) ID 獲取 Vaultvault

3、= Vault.get_vault_by_id(api, Vault ID)上傳文件archive_id = vault.upload_archive(File Path)archive_id = vault.upload_archive(File Path)Archivevault.delete_archive(Archive ID)vault.delete_archive(Archive ID)MultipartUploaduploader = vault.recover_uploader(Upload ID) uploader.resume(File Path)uploader = va

4、ult.recover_uploader(Upload ID) uploader.resume(File Path)Archivejob = vault.retrieve_inventory() job = vault.retrieve_inventory() job.download_to_file(File Path)Archivejob = vault.retrieve_archive(Archive ID) job.download_to_file(File Path)job = vault.retrieve_archive(Archive ID) job.download_to_fi

5、le(File Path)從OSS上轉(zhuǎn)儲(chǔ)Object到OASjob = vault.pull_from_oss(conf.osshost, conf.bucket, conf.object, test desc)job = vault.pull_from_oss(conf.osshost, conf.bucket, conf.object, test desc)從OAS上轉(zhuǎn)儲(chǔ)Archive到OSSjob = vault.push_to_oss(archive_id, conf.osshost, conf.bucket, archive_id, test desc)job = vault.pus

6、h_to_oss(archive_id, conf.osshost, conf.bucket, archive_id, test desc)Python SDK 完整使用演示代碼下面函數(shù)中test_single_archive_upload提供單一文件archive上傳； 函數(shù)test_multi_upload() 使用sdk低級(jí)接口實(shí)現(xiàn)分段上傳；函數(shù)test_uploader()使用sdk高級(jí)接口實(shí)數(shù)據(jù)上傳（當(dāng)數(shù)據(jù)大于64MB時(shí)，會(huì)自動(dòng)選擇分段上傳）； 函數(shù)test_vault_retrieve()實(shí)現(xiàn)vault信息查詢(xún)；函數(shù)test_download_archive(archive_id)

7、實(shí)現(xiàn)archive下載； 函數(shù)test_delete_archive(archive_id)實(shí)現(xiàn)archive刪除。import random函數(shù)test_pull_from_oss（） 實(shí)現(xiàn)從OSS直接轉(zhuǎn)儲(chǔ)到OAS 函數(shù)test_push_to_oss() 實(shí)現(xiàn)從OAS直接轉(zhuǎn)儲(chǔ)到OSSimport random容器服務(wù)/服務(wù)編排文檔容器服務(wù)/服務(wù)編排文檔import time import loggingimport logging.handlersfrom oas.oas_api import OASAPI from oas.ease.api import APIProxy from oa

8、s.ease.exceptions import * from oas.ease.response import * from oas.ease.utils import *from oas.ease.vault import * from oas.ease.uploader import * from oas.ease.job import *import os LOG_FILE=test.loghandler = logging.handlers.RotatingFileHandler(LOG_FILE, maxBytes = 1024*1024, backupCount = 5)fmt=

9、%(asctime)s-%(filename)s:%(lineno)s-%(name)s-%(message)s formatter =logging.Formatter(fmt)handler.setFormatter(formatter) log.addHandler(handler) log.setLevel(logging.DEBUG)class TestConf(object): def init (self):self.host = oas域名self.accesskey_id = 您 的 access key Id self.accesskey_secret = 您的access

10、 key secret self.vault_name = normal self.vault_name_test = testself.osshost = 您要轉(zhuǎn)儲(chǔ)的oss域名 self.bucket = 您要轉(zhuǎn)儲(chǔ)的bucket self.object = 您要轉(zhuǎn)儲(chǔ)的Objectconf = TestConf()class TestDemo():_MEGABYTE = 1024 * 1024def init (self):self.api=OASAPI(conf.host,conf.accesskey_id,conf.accesskey_secret) self.api_proxy =API

11、Proxy(self.api)self.vault_name = conf.vault_name self.vault_name_test = conf.vault_name_testself.file_big = random100M.bin self.file_big_size = 200*self._MEGABYTEwith open(self.file_big, wb+) as f: self.write_random_data(f, self.file_big_size)self.file_small = random30M.bin self.file_small_size = 30

12、*self._MEGABYTEwith open(self.file_small, wb+) as f: self.write_random_data(f, self.file_small_size)def write_random_data(self, file, size): remaining = sizewhile remaining 0:n = min(remaining, 1024) random_data = os.urandom(n) file.write(random_data) remaining = remaining - nn = min(remaining, 1024

13、 * random.randint(256, 1024) file.write(0 * n)remaining = remaining - ndef test_single_archive_upload(self): print #Using the low level api method to upload a small archive#res = self.api_proxy.create_vault(self.vault_name) vault_id = resx-oas-vault-idetag = compute_etag_from_file(self.file_small) t

14、ree_etag = compute_tree_etag_from_file(self.file_small) with open(self.file_small, rb) asf:content = f.read()res=self.api_proxy.post_archive(vault_id,content,etag,tree_etag) archive_id =resx-oas-archive-idprint archive_id,archive_id#test the multipart upload by the proxy api, and this is low level a

15、pi def test_multi_upload(self):print nnn#Using the low level api to invoke the multipart api and implement the archive uplod#res = self.api_proxy.create_vault(self.vault_name) vault_id = resx-oas-vault-idpart_size = 1024 * 1024 * 64etag_array= tree_etag_array= offset = 0cur_part_num=0 cur_start_pos

16、= 0cur_end_pos = 0print 1. comput the etag,tree-etagprint 1.1 comput the etag , tree_etag of part while True:tmpsize = part_sizeif(cur_start_pos + tmpsize self.file_big_size): tmpsize = self.file_big_size - cur_start_pos;cur_end_pos += tmpsize -1etag_array.append(compute_etag_from_file(self.file_big

17、, cur_start_pos, tmpsize) ) tree_etag_array.append(compute_tree_etag_from_file(self.file_big, cur_start_pos, tmpsize)cur_start_pos += tmpsize cur_part_num += 1if(cur_start_pos = self.file_big_size-1): break;print 1.2 comput the total tree_etag of the archivetree_etag_total = compute_combine_tree_eta

18、g_from_list( tree_etag_array) ;print 2.1 init the upload task, and get the uploadIdres=self.api_proxy.create_multipart_upload(vault_id,part_size) upload_id =resx-oas-multipart-upload-idprint upload_id,upload_idf = open(self.file_big, rb) cur_part_num = 0cur_start_pos = 0cur_end_pos = 0 while True:tm

19、psize = part_sizeif(cur_start_pos + tmpsize self.file_big_size): tmpsize = self.file_big_size - cur_start_pos;cur_end_pos += tmpsizeprint2.2uploadeveryparttooasserver,andtheetagofthepartwillbeused.currentpartis:, cur_part_num+1self.api_proxy.post_multipart_from_reader(vault_id, upload_id, f, tmpsize

20、,(%d-%d %(cur_start_pos, cur_end_pos-1), etag_arraycur_part_num,tree_etag_arraycur_part_num)cur_start_pos += tmpsize cur_part_num += 1if(cur_end_pos= self.file_big_size -1): break;print2.3completethemultiparttask,andthetotaletagwillbeused res =self.api_plete_multipart_upload(vault_id, upload_id, sel

21、f.file_big_size, tree_etag_total) print output the archive idarchive_id = resx-oas-archive-id print archive_id,archive_idreturn archive_id#test the uploader to upload big file, and this is high level api def test_uploader(self):print nnn#Using the High level api to invoke the multipart api#vault = V

22、ault.create_vault(self.api, self.vault_name)print initial a uploadIduploader = vault.initiate_uploader(self.file_big) uploader_id = uploader.idprint uploader_id,uploader_idprint start the multipart archive_id = uploader.start()print finish the upload, and output the archive_id print archive_id,archi

23、ve_idreturn archive_id#to inquire the archive list of vault def test_vault_retrieve(self):print nnn#Retrieve the vault info, and inquire the archive list of the vault#vault = Vault.create_vault(self.api, self.vault_name) job = vault.retrieve_inventory() job.download_to_file(job.id)#to test download

24、archivedef test_download_archive(self, archive_id): print nnn#Submit the archive job and download the job to local#vault = Vault.create_vault(self.api, self.vault_name) job = vault.retrieve_archive(archive_id) job.update_status()if not pleted:job.download_to_file(file_path = job.id, block = True)#te

25、st delete archivedef test_delete_archive(self, archive_id): print nnn#Delete the archive#vault = Vault.create_vault(self.api, self.vault_name) vault.delete_archive(archive_id)def test_pull_from_oss(self): print nnn#submit a pull-from-oss job and OAS will finish the pull of OSS object to OAS#vault =

26、Vault.create_vault(self.api, self.vault_name)# create vaultjob = vault.pull_from_oss(conf.osshost, conf.bucket, conf.object, test desc)job._check_status(block=True)job._check_status(block=True)# delete archiveprint bucket:%s object:%s finish pull from oss,n archiveId:%s % (conf.bucket, conf.object,

27、job.archive_id)def test_push_to_oss(self): print nnn#submit a push-to-oss job and OAS will finish the push of OAS archive to OSS#vault = Vault.create_vault(self.api, self.vault_name) archive_id = vault.upload_archive(self.file_big)job=vault.push_to_oss(archive_id,conf.osshost,conf.bucket,archive_id,

28、testdesc) job._check_status(block=True)print archive_id + finish push to ossif name = main : t =TestDemo()t.test_single_archive_upload() t.test_multi_upload();archive_id = t.test_uploader();t.test_vault_retrieve(); t.test_download_archive(archive_id) t.test_delete_archive(archive_id)t.test_pull_from

29、_oss() t.test_push_to_oss()JAVA示例程序AliyunOASClient和ArchiveManagerAliyunOASClient和ArchiveManager分別是低階接口、高級(jí)接口的入口對(duì)象，在每個(gè)功能中都會(huì)使用到它們，為避免實(shí)例程序的冗長(zhǎng)，將AliyunOASClient和ArchiveManager的創(chuàng)建在這里單獨(dú)介紹。注：在本文檔中出現(xiàn)的 aliyunOASClient 均是 AliyunOASClient 實(shí)例，archiveManager 均是ArchiveManager實(shí)例/ AccessId和AccessKey配置ServiceCredentia

30、ls credentials = new ServiceCredentials(AliyunOASClient實(shí)例化/ AccessId和AccessKey配置ServiceCredentials credentials = new ServiceCredentials(yourAccessId, yourAccessKey);yourAccessId, yourAccessKey);/ 配置客戶(hù)端設(shè)置ClientConfiguration clientConf = new ClientConfiguration();/ 指定要連接的服務(wù)地址ServiceHost serviceHost =

31、new HYPERLINK / ServiceHost();/ 用OAS工廠獲得AliyunOASClient對(duì)象AliyunOASClient aliyunOASClient = OASFactory.aliyunOASClientFactory( serviceHost, credentials, clientConf);/ 或者像下面這樣，通過(guò)ServiceCredentials和url直接創(chuàng)建AliyunOASClient對(duì)象AliyunOASClient aliyunOASClient = OASFactory.aliyunOASClientFactory(credentials,

32、HYPERLINK / );/ 使用默認(rèn)配置ServiceCredentials credentials = new ServiceCredentials(yourAccessKeyID, yourAccessKeySecret);ArchiveManager manager = OASFactory.archiveManagerFactory( credentials, HYPERLINK / / 使用默認(rèn)配置ServiceCredentials credentials = new ServiceCredentials(yourAccessKeyID, yourAccessKeySecret

33、);ArchiveManager manager = OASFactory.archiveManagerFactory( credentials, HYPERLINK / );/ 自定義配置ClientConfigurationServiceCredentials credentials = new ServiceCredentials(yourAccessKeyID, yourAccessKeySecret);/ 指定要連接的服務(wù)地址ServiceHost serviceHost = new HYPERLINK / ServiceHost();/ 配置客戶(hù)端設(shè)置ClientConfigura

34、tion clientConf = new ClientConfiguration();ArchiveManager archiveManager = new ArchiveManager(serviceHost, credentials, clientConf);/ 使用AliyunOASClient初始化AliyunOASClient aliyunOASClient = OASFactory.aliyunOASClientFactory( credentials, clientConf).withLogger();ArchiveManager archiveManager = OASFac

35、tory.archiveManagerFactory(aliyunOASClient);Vault操作創(chuàng)建Vault如果指定名稱(chēng)的Vault不存在，則創(chuàng)建Vault；如果指定名稱(chēng)的Vault已存在，不執(zhí)行創(chuàng)建動(dòng)作，返回Vault的信息。/ vaultName必須滿(mǎn)足已下2個(gè)條件：/ vaultName必須滿(mǎn)足已下2個(gè)條件：/ 1. 總長(zhǎng)度在363之間（包括3和63）；/ 2. 只允許包含以下字符：/ 2. 只允許包含以下字符：/ 0-9(數(shù)字),/ a-z(小寫(xiě)英文字母),/ -(短橫線(xiàn)),/ _(下劃線(xiàn))/ 其中 短橫線(xiàn) 和 下劃線(xiàn) 不能作為vaultName的開(kāi)頭和結(jié)尾； String v

36、aultName = oas_demo;/ 發(fā)送創(chuàng)建vault請(qǐng)求/ 創(chuàng)建Vault的名稱(chēng)，用CreateVaultRequest來(lái)指定CreateVaultRequest createRequest = new CreateVaultRequest().withVaultName(vaultName);/ 發(fā)起創(chuàng)建Vault的請(qǐng)求/ 如果有同名的vault存在，OAS會(huì)返回已有的vault的vaultId，而不會(huì)執(zhí)行創(chuàng)建動(dòng)作try CreateVaultResult result = aliyunOASClient.createVault(createRequest); (Vault cre

37、ated vaultId=, result.getVaultId(); (Vault created vaultLocation=, result.getLocation(); catch (OASClientException e) logger.error(OASClientException Occured:, e); catch (OASServerException e) logger.error(OASServerException Occured:, e);/ 刪除vault是高危動(dòng)作，因此只能通過(guò)vaultId來(lái)刪除DeleteVaultRequest deleteVaultR

38、equest = new DeleteVaultRequest().withVaultId(yourVaultId);try OASResult result = aliyunOASClient.deleteVault(deleteVaultRequest); (Delete/ 刪除vault是高危動(dòng)作，因此只能通過(guò)vaultId來(lái)刪除DeleteVaultRequest deleteVaultRequest = new DeleteVaultRequest().withVaultId(yourVaultId);try OASResult result = aliyunOASClient.de

39、leteVault(deleteVaultRequest); (DeleteSuccess!,result.getRequestId(),result.getDate(); catch (OASClientException e) logger.error(OASClientException Occured:, e); catch (OASServerException e) logger.error(OASServerException Occured:, e);上傳Archive快速上傳Multipart上傳，無(wú)須用戶(hù)選 擇/ java sdk 會(huì)幫助用戶(hù)完成 vaultName 到 v

40、aultId 的轉(zhuǎn)換/ 由此避免了用戶(hù)對(duì)一長(zhǎng)串vaultId的記憶負(fù)擔(dān)關(guān)于高級(jí)接口ArchiveManager的詳細(xì)文檔，參考 OAS Java SDK高級(jí)接口文檔 / java sdk 會(huì)幫助用戶(hù)完成 vaultName 到 vaultId 的轉(zhuǎn)換/ 由此避免了用戶(hù)對(duì)一長(zhǎng)串vaultId的記憶負(fù)擔(dān)String yourVaultName = oas_demo;String yourVaultName = oas_demo;File file = new File(oas_demo_data/random10M.bin); try UploadResultuploadResult=archiv

41、eManager.upload(yourVaultName,file); (Fileuploadedcomplete.ArchiveId=,md5=,treeEtag=,file.getAbsolutePath(), uploadResult.getArchiveId(), uploadResult.getHashValue(), uploadResult.getHashTreeValue(); catch (OASClientException e) logger.error(OASClientException Occured:, e); catch (OASServerException

42、 e) logger.error(OASServerException Occured:, e);File file = new File(oas_demo_data/random120M-2.bin);/獲得uploadId/文件大小必須大于100MB，否則會(huì)拋異常提示用普通上傳接口進(jìn)行上傳String uploadId = archiveManager.initiateMultipartUpload(vaultName, file, Hello OAS!);File file = new File(oas_demo_data/random120M-2.bin);/獲得uploadId/文件

43、大小必須大于100MB，否則會(huì)拋異常提示用普通上傳接口進(jìn)行上傳String uploadId = archiveManager.initiateMultipartUpload(vaultName, file, Hello OAS!);/如果是已有的uploadId，直接使用之前獲取過(guò)的uploadId/String uploadId = yourUploadId; System.out.println(Get uploadId= + uploadId);UploadResultuploadResult=archiveManager.uploadWithUploadId(vaultName,fi

44、le,uploadId); System.out.println(Archive ID= +uploadResult.getArchiveId();任務(wù)執(zhí)行下載任務(wù)是通過(guò)提交任務(wù)、下載任務(wù)輸出的異步過(guò)程執(zhí)行的，任務(wù)的詳細(xì)介紹，參考 OAS Java SDK高級(jí)接口文檔。String yourVaultName = oas_demo; String yourArchiveId =6086C62DBFFD4A68BAEE9D1B680EC77C8FE6AE617FF47BCD3DB3CB71AFAE29B33431AED14D06D9C4AD251F17C 173F3CD;JobMonitor j

45、obMonitor = archiveManager.downloadAsync(yourVaultName, yourArchiveId);JobMonitor jobMonitor = archiveManager.downloadInventoryAsync(oas_demo);String yourVaultName = oas_demo; String yourArchiveId =6086C62DBFFD4A68BAEE9D1B680EC77C8FE6AE617FF47BCD3DB3CB71AFAE29B33431AED14D06D9C4AD251F17C 173F3CD;JobM

46、onitor jobMonitor = archiveManager.downloadAsync(yourVaultName, yourArchiveId);JobMonitor jobMonitor = archiveManager.downloadInventoryAsync(oas_demo);執(zhí)行Inventory 下載Job輸出數(shù)據(jù)傳輸/常見(jiàn)問(wèn)題FAQ數(shù)據(jù)傳輸/常見(jiàn)問(wèn)題FAQarchiveManager.downloadJobOutput(yourVaultName, jobId, archiveManager.downloadJobOutput(yourVaultName, job

47、Id, new File(oas_demo_data/mySaveFile.bin);配置ProcessListener以上傳Archive為例：ServiceCredentials credentials = new ServiceCredentials( DemoConstants.ACCESS_ID, DemoConstants.ACCESS_KEY);/ 通過(guò)工廠類(lèi)獲得archiveManager接口ArchiveManager archiveManager = OASFactory.archiveManagerFactory(credentials); File file = new

48、 File(pathToYourFile);final BaseTransfer bt = archiveManager.uploadAsync( yourVaultName, file);/設(shè)置最大并發(fā)數(shù)，默認(rèn)為3，最大為10 bt.setNumConcurrence(5);bt.addProgressListener(new ProgressListener() Overridepublic void onStart(String id) / 任務(wù)開(kāi)始時(shí)調(diào)用，其中id為Multipart Upload任務(wù)ID，對(duì)于一般上傳任務(wù)id為空System.out.println(Start! Up

49、load ID: + id);Overridepublic boolean onError(Range range, Throwable t) / 出錯(cuò)時(shí)調(diào)用，range是出錯(cuò)的字節(jié)范圍，t是相應(yīng)的錯(cuò)誤/ 當(dāng)返回true時(shí)，BaseTransfer會(huì)進(jìn)行重試，false則放棄System.out.println(ERROR!);return false;Overridepublic void onCompleted() / 任務(wù)完成時(shí)調(diào)用System.out.println(Upload complete);Overridepublic void onProgressed(long curre

50、nt, long total) / 上傳進(jìn)度，total為文件字節(jié)大小，current為當(dāng)前已上傳字節(jié)數(shù)System.out.println(Progress: + current + / + total););bt.start();final Timer timer = new Timer(); timer.scheduleAtFixedRate(new TimerTask() OverrideOverridepublic void run() System.out.println(Running time: + bt.getRunningTime() + seconds); System.

51、out.println(Completed size: + bt.getSizeCompleted() + bytes); System.out.println(Total size: + bt.getSizeTotal() + bytes); System.out.println(Averagespeed:+bt.getAverageSpeedInBytesPerSecond()+ B/s);if (bt.isComplete() timer.cancel(); synchronized (bt) bt.notify();, 0, 1000);synchronized (bt) try bt

52、.wait(); catch (InterruptedException e) logger.error(, e);/ 任務(wù)結(jié)束System.out.println(=);System.out.println(Running time: + bt.getRunningTime() + seconds); System.out.println(Completed size: + bt.getSizeCompleted() + bytes); System.out.println(Total size: + bt.getSizeTotal() + bytes); System.out.printl

53、n(Averagespeed:+bt.getAverageSpeedInBytesPerSecond()+B/s); UploadResult uploadResult =bt.getResult();System.out.println(Archive ID: + uploadResult.getArchiveId(); System.out.println(Hash Value: + uploadResult.getHashValue();adup備份實(shí)踐簡(jiǎn)介adup是基于開(kāi)源備份工具duplicity( HYPERLINK /duplicity/ /duplicity/ )，使用pyth

54、on語(yǔ)言實(shí)現(xiàn)，支持文件全量增量備份，支持小文件打包合并，提供了阿里云歸檔存儲(chǔ)的備份工具。duplicity本身是一個(gè)多功能本地和 遠(yuǎn)程的備份軟件，支持多種后端存儲(chǔ)介質(zhì)如ftp、ssh和各類(lèi)云存儲(chǔ)。同時(shí)adup參照 HYPERLINK /thornomad/dt-s3-backup /thornomad/dt-s3-backup進(jìn)行優(yōu)化封裝，方便進(jìn)行定時(shí)任務(wù)的配置。工具下載adup_0.1.0.tar.gz 開(kāi)發(fā)包特性簡(jiǎn)單易用的命令行使用方式使用rsync對(duì)變化數(shù)據(jù)進(jìn)行增量備份，提高帶寬和存儲(chǔ)的高效利用使用標(biāo)準(zhǔn)文件打包壓縮格式GNU-tar，能夠?qū)υ隽繑?shù)據(jù)進(jìn)行追加多種遠(yuǎn)程存儲(chǔ)方式的支持，阿里云歸檔存儲(chǔ)，AWS等安裝sudo bash deploy_debian.shdebian發(fā)行版本sudo bash deploy_debian.shredhat/centos發(fā)行版本sudo bash deploy_rh.shsudo bash deploy_rh.sh安裝完成以后，備份腳本dt-oas-backup.sh 存放在/usr/local/bin/下。使用步驟配置# 在redhat/centos發(fā)行版本中需要設(shè)置，debian環(huán)境中可以將其注釋export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib#