09-3-文本分類實(shí)戰(zhàn)：中文垃圾郵件分類

機(jī)器學(xué)習(xí)與自然語言處理Python自然語言處理第十章課前回顧無監(jiān)督學(xué)習(xí)任務(wù)文本聚類K-means算法流程文本分類實(shí)戰(zhàn)：中文垃圾郵件分類03實(shí)現(xiàn)代碼評價(jià)指標(biāo)實(shí)現(xiàn)代碼

數(shù)據(jù)列表：數(shù)據(jù)名稱數(shù)據(jù)類型ham_data.txt正常郵件spam_data.txt垃圾郵件實(shí)現(xiàn)代碼

代碼組織：-data_loader.py：讀取郵件數(shù)據(jù)、停用詞列表-data_processing.py：數(shù)據(jù)清洗以及數(shù)據(jù)集轉(zhuǎn)化-classifier.py：模型訓(xùn)練、預(yù)測、評價(jià)實(shí)現(xiàn)代碼defget_data(path=DATA_DIR):withopen(path+"/ham_data.txt",encoding="utf8")asham_f,\open(path+"/spam_data.txt",encoding="utf8")asspam_f:ham_data=[x.strip()forxinham_f.readlines()ifx.strip()]spam_data=[x.strip()forxinspam_f.readlines()ifx.strip()]ham_label=np.ones(len(ham_data)).tolist()spam_label=np.zeros(len(spam_data)).tolist()corpus=ham_data+spam_datalabels=ham_label+spam_labelreturncorpus,labelsdefget_stopwords(path=DATA_DIR):withopen(path+"/stop_words.utf8",encoding="utf8")asf:stopword_list=set([x.strip()forxinf.readlines()]+list(r'!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'))returnstopword_list

data_loader.py實(shí)現(xiàn)代碼defnorm_corpus(corpus,stopword_list):result_corpus=[]##匹配連續(xù)2個(gè)以上的英文+空格符號，后面替換成一個(gè)空格

pattern=pile('[{}\\s]'.format(re.escape(r'!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'))+r"{2,}")fortextincorpus:#分詞，按停用詞表過濾

seg_text=''.join([x.strip()forxinjieba.cut(text)ifx.strip()notinstopword_list])result_corpus.append(pattern.sub("",seg_text))returnresult_corpusdefconvert_data(norm_train,norm_test,vectorizer):##fit把數(shù)據(jù)集中所有文字按規(guī)則（默認(rèn)空格）切分成詞元以后每個(gè)詞元記錄一個(gè)數(shù)字

##transform對切分文字匹配出數(shù)字id，作為向量維度下標(biāo)

##fit_transform：兩個(gè)功能合在一起

train_features=vectorizer.fit_transform(norm_train)test_feature=vectorizer.transform(norm_test)returntrain_features,test_feature,vectorizer

data_processing.py實(shí)現(xiàn)代碼DATA_DIR=data_loader.DATA_DIRcorpus,labels=data_loader.get_data(DATA_DIR)stopwords=data_loader.get_stopwords(DATA_DIR)#train_test_split將數(shù)據(jù)集按test_size劃分測試、訓(xùn)練集，random_state相同每次隨機(jī)結(jié)果都一樣corpus_train,corpus_test,y_train,y_test=train_test_split(corpus,labels,test_size=0.4,random_state=1)norm_train=data_processing.norm_corpus(corpus_train,stopwords)norm_test=data_processing.norm_corpus(corpus_test,stopwords)#文本數(shù)據(jù)轉(zhuǎn)成矩陣形式；CountVectorizer是轉(zhuǎn)換器，保留篇頻最小值為2x_train,x_test,vectorizer=data_processing.convert_data(norm_train,norm_test,CountVectorizer(min_df=2))

classifier.py加載、轉(zhuǎn)換數(shù)據(jù)實(shí)現(xiàn)代碼print("全部數(shù)據(jù)數(shù)量：",len(corpus_train)+len(corpus_test))print("訓(xùn)練數(shù)據(jù)數(shù)量：",len(corpus_train),"\n")print("分詞后的文本樣例：\n",norm_train[1])print("訓(xùn)練集特征詞數(shù)量：",len(vectorizer.get_feature_names()))

classifier.py打印數(shù)據(jù)信息實(shí)現(xiàn)代碼fromsklearn.naive_bayesimportBernoulliNBLABELS=["垃圾郵件","正常郵件"]defshow_prediction(idx_list):model=BernoulliNB()#選擇模型

model.fit(x_train,y_train)#訓(xùn)練模型

y_pred=model.predict(x_test)#模型預(yù)測，每條記錄返回0,1foridxinidx_list:print("原來的郵件類別：",LABELS[int(y_test[idx])])print("預(yù)測的郵件類別：",LABELS[int(y_pred[idx])])print("正文：\n",corpus_test[idx])print("=========================")show_prediction([0,1])

classifier.py模型訓(xùn)練與預(yù)測實(shí)現(xiàn)代碼

classifier.py模型訓(xùn)練與預(yù)測文本分類實(shí)戰(zhàn)：中文垃圾郵件分類03實(shí)現(xiàn)代碼評價(jià)指標(biāo)評價(jià)指標(biāo)分類結(jié)果的四種可能：-Truepositives(TP):實(shí)正分正（正確）-Falsepositives(FP):實(shí)負(fù)分正（錯(cuò)誤）-Falsenegatives(FN):實(shí)正分負(fù)（錯(cuò)誤）-Truenegatives(TN):實(shí)負(fù)分負(fù)（正確）評價(jià)指標(biāo)混淆矩陣（ConfusionMatrix）：評價(jià)指標(biāo)準(zhǔn)確率（Pression）、召回率（Recall）、精確率（Accuray）的公式：評價(jià)指標(biāo)準(zhǔn)確率和召回率函數(shù)圖：評價(jià)指標(biāo)F1-measure：評價(jià)指標(biāo)fromsklearn.linear_modelimportLogisticRegressionfromsklearn.metricsimportconfusion_matrix,classification_reportdefevaluate_models():models={"樸素貝葉斯":BernoulliNB(),"邏輯回歸":LogisticRegression()}formodel_name,modelinmodels.items():print("分類器：",model_name)model.fit(x_train,y_train)y_pred=model.predict(x_test)report=classification_report(y_test,y_pred,target_n