單元10：聊天機(jī)器人3

10.4Elasticsearch10.4.1Elasticsearch的安裝10.410.4.1

Elasticsearch安裝由于Elasticsearch是用Java編寫的，所以安裝之前確保JavaJDK已安裝以及配置了正確的環(huán)境變量。打開網(wǎng)址https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.5.4.zip，下載并解壓，運(yùn)行bin目錄下的elasticsearch.bat文件，即可啟動(dòng)Elasticsearch服務(wù)，如圖10-4所示。圖10-4es啟動(dòng)文件圖10.4.2

ik插件10.410.4.2

ik插件ik是ES的一個(gè)分詞插件，安裝ik分詞器，訪問/medcl/elasticsearch-analysis-ik找releases找到對(duì)應(yīng)的es版本，在es的安裝目錄的plugins文件夾下新建ik文件夾，將下載的壓縮包解壓到該文件夾下，如圖10-5所示。圖10-5ik插件安裝示意圖10.4.3Python操作Elasticsearch10.410.4.3Python操作ElasticsearchPython運(yùn)行ES的話，需要先安裝依賴包，通過pipinstallelasticsearch安裝即可。接下來學(xué)習(xí)ES的用法。整個(gè)流程如下：（1）連接ES；（2）創(chuàng)建索引；（3）插入數(shù)據(jù)。10.410.4.3Python操作Elasticsearch首先導(dǎo)入需要的庫：importtimeimportjsonfromelasticsearchimportElasticsearchfromelasticsearch.helpersimportbulkimportplatformimportos10.410.4.3Python操作Elasticsearch把整個(gè)流程封裝成了一個(gè)類ProcessIntoES。classProcessIntoES:def__init__(self):self._index="crime_data"self.es=Elasticsearch([{"host":"","port":9200}],max_retries=3,retry_on_timeout=True)self.doc_type="crime"if(platform.system()=="Linux"):cur='/'.join(os.path.abspath(__file__).split('/')[:-1])self.music_file=os.path.join(cur,'data/qa_corpus.json')10.410.4.3Python操作Elasticsearchelif(platform.system()=='Windows'):cur=os.getcwd()+"\\"self.music_file=os.path.join(cur,'data\\qa_corpus.json')else:raise"系統(tǒng)不是Windows也不是Linux"print("初始化ProcessIntoES類完成")10.410.4.3Python操作Elasticsearch'''創(chuàng)建ES索引，確定分詞類型'''defcreate_mapping(self):print("開始創(chuàng)建ES索引")node_mappings={"mappings":{self.doc_type:{#type"properties":{"question":{#field:問題"type":"text","analyzer":"ik_max_word","search_analyzer":"ik_smart",10.410.4.3Python操作Elasticsearch"index":"true"},"answers":{#field:答案"type":"text","analyzer":"ik_max_word","search_analyzer":"ik_smart","index":"true"},}}}}10.410.4.3Python操作Elasticsearchifnotself.es.indices.exists(index=self._index):self.es.indices.create(index=self._index,body=node_mappings)print("Create{}mappingsuccessfully.".format(self._index))else:print("index({})alreadyexists.".format(self._index))print("創(chuàng)建ES索引結(jié)束")'''批量插入數(shù)據(jù)'''definsert_data_bulk(self,action_list):print("開始插入數(shù)據(jù)")success,_=bulk(self.es,action_list,index=self._index,raise_on_error=True)print("Performed{0}actions._:{1}".format(success,_))10.410.4.3Python操作Elasticsearch__init__方法中連接上了ES，默認(rèn)端口號(hào)是9200，并指定了_index和doc_type，其作用相當(dāng)于在創(chuàng)建數(shù)據(jù)庫的時(shí)候指定數(shù)據(jù)庫的名字和表的名字。create_mapping方法用來創(chuàng)建索引。analyzer字段的作用：（1）插入文檔時(shí)，將text類型字段做分詞，然后插入倒排索引；（2）在查詢時(shí)，先對(duì)text類型輸入做分詞，再去倒排索引搜索。如果想要“索引”和“查詢”，使用不同的分詞器，那么只需要在字段上使用search_analyzer。這樣，索引只看analyzer，查詢就看search_analyzer。ik_max_word：會(huì)對(duì)文本做最細(xì)力度的拆分；ik_smart：會(huì)對(duì)文本做最粗粒度的拆分。insert_data_bulk方法用來批量插入數(shù)據(jù)。action_list就是每批次要插入的數(shù)據(jù)。10.410.4.3Python操作Elasticsearch每個(gè)action的格式如下：action={"_index":pie._index,"_type":pie.doc_type,"_source":{"question":item['question'],"answers":'\n'.join(item['answers']),}}10.410.4.3Python操作Elasticsearchinit_ES函數(shù)是程序的主函數(shù)，新建ProcessIntoES對(duì)象，創(chuàng)建索引，并插入數(shù)據(jù)。definit_ES():pie=ProcessIntoES()#創(chuàng)建ES的indexpie.create_mapping()start_time=time.time()index=0count=0action_list=[]BULK_COUNT=1000#每BULK_COUNT個(gè)句子一起插入到ES中10.410.4.3Python操作Elasticsearchforlineinopen(pie.music_file,encoding='utf-8'):ifnotline:continueitem=json.loads(line)print('\n'.join(item['answers']))index+=1

action={#'_op_type':'insert',"_index":pie._index,"_type":pie.doc_type,"_source":{"question":item['question'],"answers":'\n'.join(item['answers']),}}10.410.4.3Python操作Elasticsearchforlineinopen(pie.music_file,encoding='utf-8'):ifnotline:continueitem=json.loads(line)print('\n'.join(item['answers']))index+=1

action={#'_op_type':'insert',"_index":pie._index,"_type":pie.doc_type,"_source":{10.410.4.3Python操作Elasticsearch"question":item['question'],"answers":'\n'.join(item['answers']),}}action_list.append(action)ifindex>BULK_COUNT:pie.insert_data_bulk(action_list=action_list)inde