《自然語言處理技術》 高階案例 基于改進PANNs的環(huán)境音識別

《自然語言處理技術》代碼42所示。代碼STYLEREF1\s4SEQ代碼\*ARABIC\s12特征工程frompaddleaudio.featuresimportLogMelSpectrogramf_min=50.0f_max=14000.0#-sr:音頻文件的采樣率。#-n_fft:FFT樣本點個數(shù)。#-hop_length:音頻幀之間的間隔。#-win_length:窗函數(shù)的長度。#-window:窗函數(shù)種類。#-n_mels:梅爾刻度數(shù)量。feature_extractor=LogMelSpectrogram(sr=sr,n_fft=n_fft,hop_length=hop_length,win_length=win_length,window='hann',f_min=f_min,f_max=f_max,n_mels=64)x=paddle.to_tensor(data).unsqueeze(0)#[B,L]log_fbank=feature_extractor(x)#[B,D,T]log_fbank=log_fbank.squeeze(0)#[D,T]print('LogFBank.shape:{}'.format(log_fbank.shape))plt.figure()plt.title("梅爾頻率頻譜圖")plt.xlabel('時間',fontsize='14')plt.ylabel('頻率',fontsize='14')plt.imshow(log_fbank.numpy(),origin='lower')plt.show()運行REF_Ref97730183\h代碼42，得到LogFBank矩陣的形狀和矩陣如下，LogFBank頻譜圖如REF_Ref134114069\h圖41所示。LogFBank形狀為：[64,221]LogFBank矩陣為：Tensor(shape=[64,221],dtype=float32,place=Place(cpu),stop_gradient=True,[[-23.95069122,-22.08021164,-17.53287888,...,-42.92661285,-43.09761429,-38.32103348],[-20.52371788,-17.36217880,-14.43681240,...,-51.71458817,-48.15540314,-38.29315948],[-12.41798973,-12.07518482,-11.05051422,...,-46.21258545,-46.95867538,-40.08457947],...,[-59.85544586,-69.29409790,-76.78606415,...,-80.47877502,-80.00144958,-75.48116302],[-56.54590988,-66.36911011,-78.42842865,...,-79.01375580,-79.38146973,-75.62033844],[-53.41579437,-63.53484344,-81.39573669,...,-80.55462646,-81.66659546,-75.75375366]])圖STYLEREF1\s4SEQ圖\*ARABIC\s11LogFBank頻譜圖構建改進PANNs模型這個步驟，定義了一個名為SoundClassifier的音頻分類器類。該類繼承了PaddlePaddle框架的nn.Layer。其中backbone是預訓練的cnn14神經(jīng)網(wǎng)絡模型，用于提取音頻特征，使用了三層全連接層（self.fc_1、self.fc_2和self.fc_3）來代替原始的單層全連接層（self.fc）進行分類，如REF_Ref100677394\h代碼43所示。代碼STYLEREF1\s4SEQ代碼\*ARABIC\s13改進PANNs模型importpaddle.nnasnnfrompaddlespeech.cls.modelsimportcnn14backbone=cnn14(pretrained=True,extract_embedding=True)classSoundClassifier(nn.Layer):def__init__(self,backbone,num_class,dropout=0.1):super().__init__()self.backbone=backboneself.dropout=nn.Dropout(dropout)#self.fc=nn.Linear(self.backbone.emb_size,num_class)self.fc_1=nn.Linear(self.backbone.emb_size,128)self.fc_2=nn.Linear(128,64)self.fc_3=nn.Linear(64,num_class)defforward(self,x):x=x.unsqueeze(1)x=self.backbone(x)x=self.dropout(x)logits=self.fc_1(x)logits=self.fc_2(logits)logits=self.fc_3(logits)returnlogitsmodel=SoundClassifier(backbone,num_class=len(ESC50.label_list))print('模型結構：\n',model)運行REF_Ref100677394\h代碼43，得到改進PANNS的模型結構如下。[2023-05-0416:01:28,615][INFO]-PaddleAudio|unique_endpoints{''}[2023-05-0416:01:28,615][INFO]-PaddleAudio|unique_endpoints{''}[2023-05-0416:01:28,619][INFO]-PaddleAudio|Found/data0/lizhuoxuan/.paddlespeech/models/panns/panns_cnn14.pdparams[2023-05-0416:01:28,619][INFO]-PaddleAudio|Found/data0/lizhuoxuan/.paddlespeech/models/panns/panns_cnn14.pdparams模型結構：SoundClassifier((backbone):CNN14((bn0):BatchNorm2D(num_features=64,momentum=0.9,epsilon=1e-05)(conv_block1):ConvBlock((conv1):Conv2D(1,64,kernel_size=[3,3],padding=(1,1),data_format=NCHW)(conv2):Conv2D(64,64,kernel_size=[3,3],padding=(1,1),data_format=NCHW)(bn1):BatchNorm2D(num_features=64,momentum=0.9,epsilon=1e-05)(bn2):BatchNorm2D(num_features=64,momentum=0.9,epsilon=1e-05))(conv_block2):ConvBlock((conv1):Conv2D(64,128,kernel_size=[3,3],padding=(1,1),data_format=NCHW)(conv2):Conv2D(128,128,kernel_size=[3,3],padding=(1,1),data_format=NCHW)(bn1):BatchNorm2D(num_features=128,momentum=0.9,epsilon=1e-05)(bn2):BatchNorm2D(num_features=128,momentum=0.9,epsilon=1e-05))(conv_block3):ConvBlock((conv1):Conv2D(128,256,kernel_size=[3,3],padding=(1,1),data_format=NCHW)(conv2):Conv2D(256,256,kernel_size=[3,3],padding=(1,1),data_format=NCHW)(bn1):BatchNorm2D(num_features=256,momentum=0.9,epsilon=1e-05)(bn2):BatchNorm2D(num_features=256,momentum=0.9,epsilon=1e-05))(conv_block4):ConvBlock((conv1):Conv2D(256,512,kernel_size=[3,3],padding=(1,1),data_format=NCHW)(conv2):Conv2D(512,512,kernel_size=[3,3],padding=(1,1),data_format=NCHW)(bn1):BatchNorm2D(num_features=512,momentum=0.9,epsilon=1e-05)(bn2):BatchNorm2D(num_features=512,momentum=0.9,epsilon=1e-05))(conv_block5):ConvBlock((conv1):Conv2D(512,1024,kernel_size=[3,3],padding=(1,1),data_format=NCHW)(conv2):Conv2D(1024,1024,kernel_size=[3,3],padding=(1,1),data_format=NCHW)(bn1):BatchNorm2D(num_features=1024,momentum=0.9,epsilon=1e-05)(bn2):BatchNorm2D(num_features=1024,momentum=0.9,epsilon=1e-05))(conv_block6):ConvBlock((conv1):Conv2D(1024,2048,kernel_size=[3,3],padding=(1,1),data_format=NCHW)(conv2):Conv2D(2048,2048,kernel_size=[3,3],padding=(1,1),data_format=NCHW)(bn1):BatchNorm2D(num_features=2048,momentum=0.9,epsilon=1e-05)(bn2):BatchNorm2D(num_features=2048,momentum=0.9,epsilon=1e-05))(fc1):Linear(in_features=2048,out_features=2048,dtype=float32)(fc_audioset):Linear(in_features=2048,out_features=527,dtype=float32))(dropout):Dropout(p=0.1,axis=None,mode=upscale_in_train)(fc_1):Linear(in_features=2048,out_features=128,dtype=float32)(fc_2):Linear(in_features=128,out_features=64,dtype=float32)(fc_3):Linear(in_features=64,out_features=50,dtype=float32))模型訓練這個步驟，首先設置訓練輪數(shù)、每輪訓練步數(shù)、日志輸出頻率和評估頻率。然后開始訓練循環(huán)，如REF_Ref133953943\h代碼44所示。代碼STYLEREF1\s4SEQ代碼\*ARABIC\s14使用網(wǎng)格搜索進行模型調(diào)優(yōu)frompaddleaudio.utilsimportlogger#設置訓練輪數(shù)epochs=20#設置每輪訓練步數(shù)steps_per_epoch=len(train_loader)#設置日志輸出頻率log_freq=10#設置評估頻率eval_freq=10#開始訓練循環(huán)forepochinrange(1,epochs+1):model.train()#初始化損失和評估指標avg_loss=0num_corrects=0num_samples=0#遍歷訓練數(shù)據(jù)forbatch_idx,batchinenumerate(train_loader):waveforms,labels=batchfeats=feature_extractor(waveforms)feats=paddle.transpose(feats,[0,2,1])#[B,N,T]->[B,T,N]logits=model(feats)#計算損失loss=criterion(logits,labels)loss.backward()optimizer.step()ifisinstance(optimizer._learning_rate,paddle.optimizer.lr.LRScheduler):optimizer._learning_rate.step()optimizer.clear_grad()#更新?lián)p失avg_loss+=loss.numpy()[0]#計算指標preds=paddle.argmax(logits,axis=1)num_corrects+=(preds==labels).numpy().sum()num_samples+=feats.shape[0]#輸出日志if(batch_idx+1)%log_freq==0:lr=optimizer.get_lr()avg_loss/=log_freqavg_acc=num_corrects/num_samplesprint_msg='Epoch={}/{},Step={}/{}'.format(epoch,epochs,batch_idx+1,steps_per_epoch)print_msg+='loss={:.4f}'.format(avg_loss)print_msg+='acc={:.4f}'.format(avg_acc)print_msg+='lr={:.6f}'.format(lr)logger.train(print_msg)avg_loss=0num_corrects=0num_samples=0#每個評估周期進行一次評估ifepoch%eval_freq==0andbatch_idx+1==steps_per_epoch:model.eval()num_corrects=0num_samples=0#評估模型在驗證集上的性能withcessing('Evaluationonvalidationdataset'):forbatch_idx,batchinenumerate(dev_loader):waveforms,labels=batchfeats=feature_extractor(waveforms)feats=paddle.transpose(feats,[0,2,1])logits=model(feats)preds=paddle.argmax(logits,axis=1)num_corrects+=(preds==labels).numpy().sum()num_samples+=feats.shape[0]print_msg='[Evaluationresult]'print_msg+='dev_acc={:.4f}'.format(num_corrects/num_samples)logger.eval(print_msg)運行REF_Ref133953943\h代碼44，得到訓練數(shù)據(jù)如下。[2023-05-0416:07:14,880][TRAIN]-Epoch=1/20,Step=10/100loss=6.3594acc=0.0187lr=0.000100[2023-05-0416:07:14,880][TRAIN]-Epoch=1/20,Step=10/100loss=6.3594acc=0.0187lr=0.000100[2023-05-0416:12:57,082][TRAIN]-Epoch=1/20,Step=20/100loss=6.6290acc=0.0250lr=0.000100[2023-05-0416:12:57,082][TRAIN]-Epoch=1/20,Step=20/100loss=6.6290acc=0.0250lr=0.000100[2023-05-0416:18:27,851][TRAIN]-Epoch=1/20,Step=30/100loss=6.1544acc=0.0187lr=0.000100[2023-05-0416:18:27,851][TRAIN]-Epoch=1/20,Step=30/100loss=6.1544acc=0.0187lr=0.000100[2023-05-0416:23:53,470][TRAIN]-Epoch=1/20,Step=40/100loss=7.3926acc=0.0312lr=0.000100[2023-05-0416:23:53,470][TRAIN]-Epoch=1/20,Step=40/100loss=7.3926acc=0.0312lr=0.000100…模型測試這個步驟將加載一個訓練好的音頻分類模型（SoundClassifier）的權重，并使用該模型對給定音頻文件（dog.wav）進行分類。首先加載權重并設置模型狀態(tài)，然后讀取音頻文件并提取其特征。最后，將特征傳遞給模型以獲取預測結果，并打印出前top_k個預測類別及其概率，如REF_Ref133953958\h代碼45所示。代碼STYLEREF1\s4SEQ代碼\*ARABIC\s15模型評估checkpoint_path='10_model.pdparams'model_dict=paddle.load(checkpoint_path)model.set_state_dict(model_dict)top_k=10wav_file='../data/dog.wav'waveform,sr=load(wav_file,sr=sr)feature