模型INT8轉(zhuǎn)換整理文檔

1、模型INT8轉(zhuǎn)換整理文檔為評估在GPUT4模型推理性能和資源消耗，需要考慮將模型進(jìn)行量化后的處理。FP16模型的量化比較簡單，只需要改動API的一些參數(shù)即可；INT8的實(shí)現(xiàn)比較復(fù)雜，特作此整理，以方便以后重復(fù)此操作。2 .環(huán)境服務(wù)器：03CPU:48xE5-2650v42.20GHz內(nèi)存：62GGPUI:TeslaT4顯存：15GTensorRT：CUDA10.1GPU驅(qū)動版本：418.116.00CUDNN:7.5.0CUBLAS:10.1模型：googlenet3 .實(shí)現(xiàn)方法參考文檔TensorRT/samples/sampleINT8/R

2、EADME.md說明，卞據(jù)非Caffe模型轉(zhuǎn)換方法實(shí)現(xiàn)，原文部分如下：Generatingbatchfilesfornon-CaffeusersFordevelopersthatarenotusingCaffe,orcannoteasilyconverttoCaffe,thebatchfilescanbegeneratedviathefollowingsequenceofstepsontheinputtrainingdata.Subtractoutthenormalizedmeanfromthedataset.Cropalloftheinputdatatothesamedimensions.S

3、plitthedataintobatchfileswhereeachbatchfilehasNpreprocessedimagesandlabels.GeneratethebatchfilesbasedontheformatspecifiedinBatchfilesforcalibration.4 .操作步驟4.1. 準(zhǔn)備校正集文件CalibrationFile編譯命令：g+createCalibrationCache.cpp-lopencv_imgproc-lopencv_core-lopencv_imgcodecs-occc-g執(zhí)行命令：./ccccachelmgs.calibILSVRC

4、2012_val_00001234.JPEG注：1、這里ccc即測試程序，cacheImgs.calib是保存的校正集文件第三個參數(shù)是使用的圖片2、可以多次使用不同的圖片，增加校正集中圖片的數(shù)量，這里只為測試只采用了一張圖片4.2. 使用校正集進(jìn)行模型轉(zhuǎn)換并序列化CudaEngine保存這里使用了TensorRT提供的trtexec樣例程序，執(zhí)行命令：cd/home/reny/downloads/TensorRT-/samples/trtexec./trtexec-deploy=data/googlenet/totxt-output="prob&qu

5、ot;-model=data/googlenet/googlenet.caffemodel-int8-batch=4-saveEngine=googlenet_int8_bs4.engine-calib=CalibratorcreateCache/cacheImgs.calib4.3. 加載CudaEngine文件并反序列化這里采用了TensorRT提供的SampleGoogleNet樣例程序主要改動：增加一個加載CudaEngine文件的函數(shù)，該函數(shù)思路源于trtexec樣例代碼/addbyreny2019/12/27staticnvinfer1:ICudaEngine*LoadEngine

6、(conststring&sSavedEngineFile)ICudaEngine*engine;std:vector<char>trtModelStream;size_tsize0;std:ifstreamfile(sSavedEngineFile.data(),std:ios:binary);if(file.good()file.seekg(0,file.end);size=file.tellg();file.seekg(0,file.beg);trtModelStream.resize(size);file.read(trtModelStream.data(),siz

7、e);file.close();)IRuntime*infer=createInferRuntime(gLogger.getTRTLogger();/*if(.useDLACore>=0)infer->setDLACore(gParams.useDLACore);)*/engine=infer->deserializeCudaEngine(trtModelStream.data(),size,nullptr);gLogInfo<<sSavedEngineFile<<"hasbeensuccessfullyloaded."<&l

8、t;std:endl;infer->destroy();returnengine;)修改后測試程序路徑：/home/reny/downloads/TensorRT-/samples/sampleGoogleNet_ry4.4. 執(zhí)行正常步驟進(jìn)行推理cd/home/reny/downloads/TensorRT./bin/sample_googlenet-loadEngine/home/reny/nvidia/googlenet_int8_bs4.engine5 .實(shí)現(xiàn)截圖goqgiian0t_intB_b!t4.ongino_1/bin/s>jnpls_

9、g'=KigLand.-.lladdlEnginst/haiw/rony/mvidItkLUal唱jndwMgQ制InferenLi*也唱山也for5逅1電lietllgoliHfonMT1RTTensorPT叼、量*?審打鈍工呷1具51中事$butisHNk說乎tCuOLMi博I/hcinr-j'renjr/nwidijB/jooglenrtint8wbs4.engineha3beea5uuuit3>3ifiJ】lyloaded_interiigoingon町TftTJTw5<h*RF2號，顰加暈工14bytisJUftkediCiiBLAS工乳工用inpiitbuiferiiib：京嗎M41Kinfflp&fice53H5irfxtr2與18褪&tir*es汨+封已收一coststirw:868mgIRan./bin/sasplejgaoElenetwith；£lInput(a)：dataIJOuLputfG：prccb，原MiPAS5DT力sorffT.5nRMgJWNt*,-loadFnglftp|rgtkEs-art：ftrTans口rtiT.5pwd6.參考資料1代碼TensorRT/s