ElasticSearch與Solr搜索引擎特性對(duì)比-new

1、ElasticSearch與Solr搜索引擎特性對(duì)比1、關(guān)鍵特性對(duì)比對(duì)比項(xiàng)Solr 5.xElasticSearch 2.x開(kāi)發(fā)人員開(kāi)發(fā)人員少，官方提供功能比ElasticSearch多開(kāi)發(fā)人員多，官方功能少，但是第三方插件很豐富，擴(kuò)展能力更強(qiáng)建立索引和查詢效率建立索引的速度和ElasticSearch差不多，當(dāng)索引完全建立好了之后搜索會(huì)很快，但是一邊建立索引一邊搜索就會(huì)非常慢。經(jīng)過(guò)詳細(xì)對(duì)比測(cè)試后發(fā)現(xiàn)，Solr在建索引時(shí)的查詢性能非常之差，因?yàn)镾olr在建索引時(shí)會(huì)產(chǎn)生io的阻塞，造成搜索性能的下降。建立索引的速度和Solr差不多，第一次搜索速度比Solr慢一些，但之后會(huì)緩存結(jié)果，第二次搜索就會(huì)

2、很快。另外ElasticSearch邊建立索引邊搜索速度沒(méi)有太大變化，因?yàn)樗窍劝阉饕膬?nèi)容保存到內(nèi)存之中，當(dāng)內(nèi)存不夠時(shí)再把索引持久化到硬盤(pán)中，同時(shí)它還有一個(gè)隊(duì)列，是在系統(tǒng)空閑時(shí)自動(dòng)把索引寫(xiě)到硬盤(pán)中。Sharding沒(méi)有自動(dòng)shard rebalancing的功能shard 必須一次設(shè)置好，之后不能修改，如果要修改則需要重新建立索引Schema模式需要 Schema (但是4.3版本之后有schemaless模式)無(wú)需Schema，創(chuàng)建索引更加靈活高級(jí)查詢沒(méi)有Query DSL有Query DSL，能夠支持更加高級(jí)和復(fù)雜的查詢語(yǔ)法，而且還可以以此擴(kuò)展實(shí)現(xiàn)類(lèi)SQL語(yǔ)法的查詢實(shí)時(shí)性傳統(tǒng)搜索應(yīng)用實(shí)時(shí)

3、搜索應(yīng)用（這個(gè)是ElasticSearch作者說(shuō)的）可擴(kuò)展性不支持插件開(kāi)發(fā)模式支持插件開(kāi)發(fā)模式，提供豐富的開(kāi)源插件庫(kù)社區(qū)社區(qū)活躍社區(qū)活躍并有大數(shù)據(jù)公司支持Hadoop集成 支持支持，v2.3.0提供Graph圖結(jié)構(gòu)數(shù)據(jù)存儲(chǔ)存儲(chǔ)Solr暫不支持ElasticSearch支持多盤(pán)存儲(chǔ)2、基于ElasticSearch數(shù)據(jù)交叉模型功能測(cè)試2.1、環(huán)境情況1.測(cè)試環(huán)境（市局環(huán)境）IP43/slave77/slave35CPU16* Intel(R) Xeon(R) CPU E5-2650 v2 2.60GHzMEM64G(ElasticSearch獨(dú)享16G)OS

4、CentOS release 6.5 (Final)2.測(cè)試數(shù)據(jù)數(shù)據(jù)來(lái)源省廳數(shù)據(jù)：VW_SHARE_ST_JJ_KKXX（交警_卡口信息）時(shí)間范圍2015100120160331數(shù)據(jù)量kettle清洗，數(shù)據(jù)量為11.1億測(cè)試數(shù)據(jù)查詢地址43:8200/_plugin/head/ 3.測(cè)試地址url41:18099/swagger-ui.html2.2、測(cè)試結(jié)果場(chǎng)景一：一個(gè)車(chē)牌交叉滿足N個(gè)條件序號(hào)交叉條件個(gè)數(shù)測(cè)試用例預(yù)期結(jié)果實(shí)際結(jié)果3次測(cè)試耗時(shí)(s)備注13datestart: 2016-01-05T08:40:59, location

5、: 犍為縣, dateend: 2016-01-05T08:53:59, querystr: 犍為外灘, datestart: 2016-02-03 09:48:00, location: 樂(lè)山市, dateend: 2016-02-03 19:48:00, querystr: 月卡口, datestart: 2016-01-07 19:47:18, location: 南充市, dateend: 2016-01-07 19:57:18, querystr: 綠心路 川LFC550 3 川LFC550 3 10在之前的測(cè)試時(shí)，曾出現(xiàn)結(jié)果為川LFC550 2的情況，經(jīng)定位是對(duì)“犍為縣”的分詞不完

6、善，修改代碼后，結(jié)果正確2223datestart: 2016-01-15T07:08:00, location: 成華區(qū), dateend: 2016-01-15T07:18:22, querystr: 雙橋子, datestart: 2016-01-05T06:39:44, location: 錦江區(qū), dateend: 2016-01-05T06:41:44, querystr: 老成仁路口, datestart: 2016-01-25T20:40:46, location: 武侯區(qū), dateend: 2016-01-25T20:44:46, querystr: 科華北路 川AW7D1

7、6 3 川AW7D16 3 1在之前測(cè)試中，曾出現(xiàn)結(jié)果為川AW7D16 0 的情況，經(jīng)定位，是分詞不夠完善，修改代碼后，結(jié)果正確1133datestart: 2016-02-06T17:48:41, location: 四川省, dateend: 2016-02-06T17:49:41, querystr: 海棠路, datestart: 2016-02-04T10:36:43, location: 富順縣, dateend: 2016-02-04T10:36:44, querystr: 鐘秀街東段, datestart: 2016-02-04T13:10:50, location: 富順縣,

8、 dateend: 2016-02-04T13:10:53, querystr: 吉安莊路 貴JL7321 3 貴JL7321 3112144datestart: 2016-02-16T15:06:57, location: 瀘定縣, dateend: 2016-02-16T15:06:57, querystr: 石油路, datestart: 2015-12-09T16:10:58, location: 新龍縣, dateend: 2015-12-09T16:10:58, querystr: 卡口抓拍機(jī), datestart: 2016-01-07T10:46:17, location: 瀘

9、定縣, dateend: 2016-01-07T10:46:17, querystr: 船頭, datestart: 2016-02-16T18:10:20, location: 雅安市, dateend: 2016-02-16T18:10:20, querystr: 名山紅星 川AE15F8 4 川AE15F8 425352datestart: 2015-11-05T13:39:46, location: 內(nèi)江市, dateend: 2015-11-05T13:39:46, querystr: 蘭桂, datestart: 2015-11-21T11:24:24, location: 內(nèi)江市

10、, dateend: 2015-11-21T11:24:24, querystr: 牌樓路 川K0103學(xué) 2 川K0103學(xué) 2119165datestart: 2015-12-01T15:06:39, location: 廣安市華鎣市, dateend: 2015-12-01T15:06:39, querystr: 古橋職中, datestart: 2015-09-30T15:13:34, location: 廣安市, dateend: 2015-09-30T15:16:34, querystr: 會(huì)展街, datestart: 2016-02-12T12:15:10, location:

11、 廣安市, dateend: 2016-02-12T12:15:10, querystr: 高速, datestart: 2016-02-10T08:11:49, location: 廣安市, dateend: 2016-02-10T08:11:59, querystr: 前華段出城, datestart: 2016-02-05T12:36:06, location: 廣安市, dateend: 2016-02-05T12:39:06, querystr: 馬石梯 渝B18D02 5 渝B18D02 5 46376datestart: 2016-01-01T22:06:37, location

12、: 青羊區(qū), dateend: 2016-01-01T22:06:37, querystr: 蘇坡立交, datestart: 2016-01-22T21:17:18, location: 武侯區(qū), dateend: 2016-01-22T21:17:18, querystr: 人民南路三段, datestart: 2016-02-14T00:01:26, location: 金牛區(qū), dateend: 2016-02-14T00:01:26, querystr: 營(yíng)門(mén)口, datestart: 2016-02-08T11:47:08, location: 成華區(qū), dateend: 2016

13、-02-08T11:47:08, querystr: 桃溪路, datestart: 2015-11-02T15:00:22, location: 錦江區(qū), dateend: 2015-11-02T15:00:22, querystr: 琉璃, datestart: 2015-10-13T09:34:12, location: 武侯區(qū), dateend: 2015-10-13T09:34:12, querystr: 雙楠路內(nèi)側(cè) 川ATS958 6 川ATS958 6 206186datestart: 2016-01-01T22:06:37, location: 市青羊, dateend: 20

14、16-01-01T22:06:37, querystr: 蘇坡立交, datestart: 2016-01-22T21:17:18, location: 武侯區(qū), dateend: 2016-01-22T21:17:18, querystr: 人民南路三段, datestart: 2016-02-14T00:01:26, location: 金牛區(qū), dateend: 2016-02-14T00:01:26, querystr: 營(yíng)門(mén)口, datestart: 2016-02-08T11:47:08, location: 成華區(qū), dateend: 2016-02-08T11:47:08, q

15、uerystr: 桃溪路, datestart: 2015-11-02T15:00:22, location: 錦江區(qū), dateend: 2015-11-02T15:00:22, querystr: 琉璃, datestart: 2015-10-13T09:34:12, location: 武侯區(qū), dateend: 2015-10-13T09:34:12, querystr: 雙楠路內(nèi)側(cè) 川ATS958 5川ATS958 532291datestart: 2016-01-05T08:40:59, location: 犍為縣, dateend: 2016-01-05T08:53:59, qu

16、erystr: 犍為外灘 川LFC550 1 川LFC550 1 121場(chǎng)景二：關(guān)注時(shí)間和地點(diǎn)，多個(gè)車(chē)牌滿足其中1N個(gè)條件序號(hào)交叉條件個(gè)數(shù)測(cè)試用例預(yù)期結(jié)果實(shí)際結(jié)果3次測(cè)試耗時(shí)(s)備注13datestart: 2016-02-07T16:44:10, location: 廣安市, dateend: 2016-02-07T16:44:20, querystr: 人民南路, datestart: 2016-02-10T11:08:08, location: 廣安市, dateend: 2016-02-10T11:09:09, querystr: 民族小學(xué), datestart: 2016-01-1

17、4T19:53:10, location: 武勝縣, dateend: 2016-01-14T19:53:12, querystr: 58KM 粵YQJ221 2川XW6238 2 粵YQJ221 2川XW6238 1 84924datestart: 2016-01-01T00:00:00, location: 德陽(yáng)市, dateend: 2016-01-04T00:00:00, querystr: 西山, datestart: 2016-02-22T08:10:55, location: 德陽(yáng)市, dateend: 2016-02-22T08:10:55, querystr: 108國(guó)道,

18、datestart: 2015-11-16T06:17:58, location: 羅江縣, dateend: 2015-11-16T06:17:58, querystr: 學(xué)校, datestart: 2016-01-03T16:53:16, location: 綿陽(yáng)市, dateend: 2016-01-03T16:53:16, querystr: 東路 川FAH567 2川FL1930 2川A9EB55 2 川FAH567 2川FL1930 2川A9EB55 23163、ElasticSearch與Solr的壓力測(cè)試對(duì)比3.1、測(cè)試環(huán)境1.集群環(huán)境此次用于測(cè)試Solr的集群包括4個(gè)節(jié)點(diǎn)：

19、0（master），13（slave1slave3）。用于測(cè)試ElasticSearch的集群包括4個(gè)節(jié)點(diǎn)：0(datanode,不可選舉為masternode)，13（datanode、可選舉為masternode）。測(cè)試集群信息：IP03CPU8* Intel(R) Xeon(R) CPU E5-2609 0 2.40GHzMemSolr：4G， ElasticSearch：4G(sys32G)OSCentOS re

20、lease 6.5測(cè)試客戶機(jī)（臨時(shí)使用slave3）：IP3CPU8* Intel(R) Xeon(R) CPU E5-2609 0 2.40GHzMemSolr4G ElasticSearch4G(sys32G)OSCentOS release 6.52.數(shù)據(jù)環(huán)境Solr中單索引庫(kù)中索引條數(shù)為5000W，ElasticSearch中單索引庫(kù)中索引條數(shù)為1億。3.2、測(cè)試工具工具名稱(chēng)http_load輸入?yún)?shù)說(shuō)明http_load -p 5 -f 1000 train_hc.txt-parallel簡(jiǎn)寫(xiě)-p：并發(fā)的用戶進(jìn)程數(shù)-fetches簡(jiǎn)寫(xiě)-f：總計(jì)的訪問(wèn)次數(shù)-rat

21、e簡(jiǎn)寫(xiě)-r：每秒的訪問(wèn)頻率-seconds簡(jiǎn)寫(xiě)-s：總計(jì)的訪問(wèn)時(shí)間輸出參數(shù)說(shuō)明1000 fetches, 5 max parallel, 1.13104e+06 bytes, in 4.50961 seconds運(yùn)行的請(qǐng)求數(shù)、最大的并發(fā)進(jìn)程數(shù)、總計(jì)傳輸?shù)臄?shù)據(jù)、運(yùn)行的時(shí)間1131.04 mean bytes/connection平均每個(gè)請(qǐng)求連接傳輸?shù)臄?shù)據(jù)量221.748 fetches/sec, 250807 bytes/sec每秒響應(yīng)請(qǐng)求數(shù)、每秒傳輸數(shù)據(jù)量msecs/connect: 0.485064 mean, 1.337 max, 0.384 min連接平均響應(yīng)時(shí)間、最大響應(yīng)時(shí)間、最小響應(yīng)

22、時(shí)間msecs/first-response: 21.7613 mean, 436.541 max, 13.468 min連接平均返回時(shí)間321 bad byte countsHTTP response codes:請(qǐng)求響應(yīng)碼返回個(gè)數(shù)code 200 - 10003.3、測(cè)試結(jié)果注：每組測(cè)試均重復(fù)進(jìn)行三次。場(chǎng)景一：模糊數(shù)字查詢查詢條件：q:31997統(tǒng)計(jì)測(cè)試結(jié)果如下：并發(fā)數(shù)總訪問(wèn)次數(shù)搜索引擎平均每秒響應(yīng)請(qǐng)求數(shù)(fetches/s)總消耗時(shí)間(s)連接平均響應(yīng)時(shí)間(ms)連接平均返回時(shí)間(ms)平均成功次數(shù)平均失敗次數(shù)1010000Solr239.66441.7250.16340.4671000

23、00ElasticSearch552.47118.1010.13017.6411000002510000Solr530.81918.8390.77744.675100000ElasticSearch677.38514.7630.12836.4281000005010000Solr664.68015.0450.20573.096100000ElasticSearch686.34014.5700.13072.28710000010010000Solr718.75113.9130.243136.493100000ElasticSearch699.83814.2890.440141.773100000

24、20010000Solr631.86915.8260.289310.823100000ElasticSearch632.78915.8034.370310.03010000050010000Solr38.925256.9081.9711848.6906479353ElasticSearch680.43214.6974.941716.01010000080010000Solr166.50360.0592.433278.3741139887ElasticSearch521.34919.18139.6641425.5809782218100010000Solr166.12360.1967.80238

25、6.7981489852ElasticSearch692.36114.44381.5321276.0009848152120010000Solr41.609240.33192.813407.1501019899ElasticSearch164.70560.715187.910841.63715528448場(chǎng)景二：精確中文查詢查詢條件：q:人民南路統(tǒng)計(jì)測(cè)試結(jié)果如下：并發(fā)數(shù)總訪問(wèn)次數(shù)搜索引擎平均每秒響應(yīng)請(qǐng)求數(shù)(fetches/s)總消耗時(shí)間(s)連接平均響應(yīng)時(shí)間(ms)連接平均返回時(shí)間(ms)平均成功次數(shù)平均失敗次數(shù)1010000Solr274.66536.4080.15935.091100000

26、ElasticSearch606.13516.4980.13316.0401000002510000Solr496.52420.1400.17348.379100000ElasticSearch832.06612.0180.12229.5811000005010000Solr528.96718.9050.39291.614100000ElasticSearch822.15112.1630.13260.28210000010010000Solr499.61420.0160.313196.558100000ElasticSearch848.29911.7881.236115.98910000020

27、010000Solr463.91021.5560.358422.760100000ElasticSearch785.50912.7312.071249.89810000050010000Solr155.62664.25610.400183.3117989202ElasticSearch827.33112.0879.174576.56710000080010000Solr166.10660.2037.612343.5382209780ElasticSearch814.86012.27230.071912.044100000100010000Solr166.01860.2342.354451.48

28、92289772ElasticSearch867.41811.52946.7301026.2909859141120010000Solr165.70260.35079.148428.1622789722ElasticSearch165.51660.417185.596734.681159984013.4、總結(jié) 在本次測(cè)試中，每次測(cè)試總的訪問(wèn)次數(shù)為10000，從以上圖表中可以看出隨著并發(fā)數(shù)的增加，Solr和ElasticSearch的每秒響應(yīng)請(qǐng)求數(shù)都是先增加，然后穩(wěn)定在一定數(shù)值，在超過(guò)某個(gè)閾值之后極速下降； 從平均每秒響應(yīng)請(qǐng)求次數(shù)來(lái)看，ElasticSearch性能指標(biāo)優(yōu)于Solr；在不同的并發(fā)數(shù)下，ElasticSearch的每秒響應(yīng)請(qǐng)求數(shù)在500800之間，而Solr的每秒響應(yīng)請(qǐng)求數(shù)在300600之間，ElasticSearch采用默認(rèn)分詞進(jìn)行中文模糊搜索時(shí)速度非常慢，經(jīng)分析更換ik中文分詞后性能有明顯提高； 從連接平均返回時(shí)間來(lái)看，在低并發(fā)時(shí)，ElasticSearch和Solr相差不大，但是隨著并發(fā)數(shù)量的增加，ElasticSearch返回時(shí)間的增長(zhǎng)速度大于Solr； 從不同的并發(fā)數(shù)下的成功次數(shù)來(lái)看，ElasticSearch性能指標(biāo)優(yōu)于Solr，Solr在并發(fā)數(shù)達(dá)到500的時(shí)候出現(xiàn)大面積錯(cuò)誤；ElasticSear