ElasticSearch與Solr搜索引擎特性對(duì)比-new

ElasticSearch與Solr搜索引擎特性對(duì)比1、關(guān)鍵特性對(duì)比對(duì)比項(xiàng)Solr5.xElasticSearch2.x開(kāi)發(fā)人員開(kāi)發(fā)人員少，官方提供功能比ElasticSearch多開(kāi)發(fā)人員多，官方功能少，但是第三方插件很豐富，擴(kuò)展能力更強(qiáng)建立索引和查詢(xún)效率建立索引的速度和ElasticSearch差不多，當(dāng)索引完全建立好了之后搜索會(huì)很快，但是邊建立索引邊搜索就會(huì)非常慢。經(jīng)過(guò)詳細(xì)對(duì)比測(cè)試后發(fā)現(xiàn)，Solr在建索引時(shí)的查詢(xún)性能非常之差，因?yàn)镾olr在建索引時(shí)會(huì)產(chǎn)生io的阻塞，造成搜索性能的下降。建立索引的速度和Solr差不多，第一次搜索速度比Solr慢些，但之后會(huì)緩存結(jié)果，第一次搜索就會(huì)很快。另外ElasticSearch邊建立索引邊搜索速度沒(méi)有太大變化，因?yàn)樗窍劝阉饕膬?nèi)容保存到內(nèi)存之中，當(dāng)內(nèi)存不夠時(shí)再把索引持久化到硬盤(pán)中，冋時(shí)它還有一個(gè)隊(duì)列，是在系統(tǒng)空閑時(shí)自動(dòng)把索引寫(xiě)到硬盤(pán)中。Sharding沒(méi)有自動(dòng)shardrebalancing的功能shard必須一次設(shè)置好，之后不能修改，如果要修改則需要重新建立索引Schema模式需要Schema（但是4.3版本之后有schemaless模式）無(wú)需Schema，創(chuàng)建索引更加靈活高級(jí)查詢(xún)沒(méi)有QueryDSL有QueryDSL，能夠支持更加高級(jí)和復(fù)雜的查詢(xún)語(yǔ)法，而且還可以以此擴(kuò)展實(shí)現(xiàn)類(lèi)SQL語(yǔ)法的查詢(xún)實(shí)時(shí)性傳統(tǒng)搜索應(yīng)用實(shí)時(shí)搜索應(yīng)用（這個(gè)是ElasticSearch作者說(shuō)的）可擴(kuò)展性不支持插件開(kāi)發(fā)模式支持插件開(kāi)發(fā)模式，提供豐富的開(kāi)源插件庫(kù)社區(qū)社區(qū)活躍社區(qū)活躍并有大數(shù)據(jù)公司支持Hadoop集成支持支持，v2.3.0提供Graph圖結(jié)構(gòu)數(shù)據(jù)存儲(chǔ)存儲(chǔ)Solr暫不支持ElasticSearch支持多盤(pán)存儲(chǔ)2、基于ElasticSearch數(shù)據(jù)交叉模型功能測(cè)試2.1、環(huán)境情況1?測(cè)試環(huán)境（市局環(huán)境）IP43/slave1~77/slave35CPU16*Intel(R)Xeon(R)CPUE5-2650v2@2.60GHz

OSCentOSrelease6.5(Final)OS2.測(cè)試數(shù)據(jù)數(shù)據(jù)來(lái)源省廳數(shù)據(jù)：VWSHARESTJJKKXX(交警—卡口信息)時(shí)間范圍20151001~20160331數(shù)據(jù)量kettle清洗，數(shù)據(jù)量為11.1億測(cè)試數(shù)據(jù)查詢(xún)地址43:8200/plugin/head/3.測(cè)試地址url41:18099/swagger-ui.html2.2、測(cè)試結(jié)果一個(gè)車(chē)牌交叉滿(mǎn)足N個(gè)條件場(chǎng)景:序號(hào)交叉條件個(gè)數(shù)測(cè)試用例預(yù)期結(jié)果實(shí)際結(jié)果3次測(cè)試耗時(shí)(s)備注13[{"datestart":"2016-01-05T08:40:59","location":"犍為縣"，"dateend":〃2016-01-05T08:53:59〃，"querystr":"犍為外灘"},{"datestart":"2016-02-0309:48:00","location":"樂(lè)山市","dateend":"2016-02-0319:48:00","querystr":"月卡口"},{"datestart":"2016-01-0719:47:18","location":"南充市"，"dateend":"2016-01-0719:57:18","querystr":"綠心路"}]川LFC5503川LFC550310在之前的測(cè)試時(shí)，曾出現(xiàn)結(jié)果為川LFC5502的情況，經(jīng)定位是對(duì)“犍為縣”的分詞不完善，修改代碼后，結(jié)果正確2223[{"datestart":"2016-01-15T07:08:00","location":"成華區(qū)"，"dateend":"2016-01-15T07:18:22","querystr":"雙橋子"},{"datestart":"2016-01-05T06:39:44","location":"錦江區(qū)"，"dateend":"2016-01-05T06:41:44","querystr":"老成仁路口"},{"datestart":川AW7D163川AW7D1631在之前測(cè)試中，曾出現(xiàn)結(jié)果為川AW7D160的情況，經(jīng)定位，是分詞不夠完善，修改代碼后，結(jié)果正確1111〃2016-01-25T20:40:46〃，"location":"武侯區(qū)"，"dateend":〃2016-01-25T20:44:46〃，"querystr〃：〃科華北路〃}][{"datestart":"2016-02-06T17:48:41","location":"四川省"，"dateend":"2016-02-06T17:49:41","querystr":"海棠路"},{"datestart":"2016-02-04T10:36:43","location":"富順縣"，"dateend":"2016-02-04T10:36:44","querystr":"鐘秀街東段"},{"datestart":"2016-02-04T13:10:50","location":"富順縣"，"dateend":"2016-02-04T13:10:53","querystr":"吉安莊路"}][{"datestart":"2016-02-16T15:06:57","location":"瀘定縣"，"dateend":"2016-02-16T15:06:57","querystr":"石油路"},{"datestart":"2015-12-09T16:10:58","location":"新龍縣"，"dateend":"2015-12-09T16:10:58","querystr":"卡口抓拍機(jī)"},{"datestart":"2016-01-07T10:46:17","location":"瀘定縣"，"dateend":"2016-01-07T10:46:17","querystr":"船頭"},{"datestart":"2016-02-16T18:10:20","location":"雅安市"，"dateend":"2016-02-16T18:10:20","querystr":"名山紅星"}][{"datestart":"2015-ll-05T13:39:46","location":"內(nèi)江市"，"dateend":"2015-ll-05T13:39:46",〃」〃〃aa4-4-〃iquerystr:二桂},{"datestart":"2015-n-21Tll:24:24",貴JL73213貴JL73213川AE15F84川AE15F84川K0103學(xué)2川K0103學(xué)2111111""location"":""內(nèi)江市""，""dateend"":〃2O15T1-21T11:24:24〃，""querystr"":""牌樓路""}][{""datestart"":〃2015-12-01T15:06:39"",""location"":""廣安市華鎣市""，"dateend"":""2015-12-01T15:06:39"","querystr"":""古橋職中""},{""datestart"":""2015-09-30T15:13:34"",""location"":""廣安市""，""dateend"":""2015-09-30T15:16:34"","querystr"":""會(huì)展街""},{""datestart"":""2016-02-12T12:15:10"",渝B18D025渝B18D025""location"":""渝B18D025渝B18D025""2016-02-12T12:15:10"",""querystr"":""高速""},{""datestart"":""2016-02-10T08：n：49"",""location"":""廣安市""，""dateend"":""2016-02-10T08：n：59"","querystr"":""前華段出城""},{""datestart"":""2016-02-05T12:36:06"",""location"":""廣安市""，""dateend"":""2016-02-05T12:39:06"",""querystr"":""馬石梯""}][{""datestart"":""2016-01-01T22:06:37"",""location"":""青羊區(qū)""，""dateend"":""2016-01-01T22:06:37"",""querystr"":""蘇坡立交""},{""datestart"":""2016-01-22T21:17:18"",""location"":""武侯區(qū)""，""dateend"":""2016-01-22T21:17:18"","querystr"":""人民南路三段""},{""datestart"":""2016-02-14T00:01:26"",""location"":""金牛區(qū)""，""dateend"":""2016-02-14T00:01:26"",""querystr"":""營(yíng)門(mén)口""},{""datestart"":""2016-02-08Tn：47:08"",""location"":""成華區(qū)""，""dateend"":川ATS9586川ATS958620〃2016-02-08Tn：47:08〃，"querystr":"桃溪路"},{"datestart":〃2015-n-02T15:00:22〃，"location":"錦江區(qū)"，"dateend":"2015Tl-02T15:00:22","querystr":"琉璃"},{"datestart":"2015T0T3T09:34:12","location":"武侯區(qū)"，"dateend":"2015T0T3T09:34:12","querystr":"雙楠路內(nèi)側(cè)"}][{"datestart":"2016-01-01T22:06:37","location":"市青羊"，"dateend":"2016-01-01T22:06:37","querystr":"蘇坡立交"},{"datestart":"2016-01-22T21:17:18","location":"武侯區(qū)"，"dateend":"2016-01-22T21:17:18","querystr":"人民南路三段"},{"datestart":"2016-02-14T00:01:26","location":"金牛區(qū)"，"dateend":"2016-02-14T00:01:26","querystr":"營(yíng)門(mén)口"},{"datestart":"2016-02-08Tll:47:08","location":"成華區(qū)"，"dateend":"2016-02-08Tll:47:08","querystr":"桃溪路"},{"datestart":"2015Tl-02T15:00:22","location":"錦江區(qū)"，"dateend":"2015Tl-02T15:00:22","querystr":"琉璃"},{"datestart":"2015T0T3T09:34:12","location":"武侯區(qū)"，"dateend":"2015T0T3T09:34:12","querystr":"雙楠路內(nèi)側(cè)"}]川ATS9585川ATS9585[{"datestart":"2016-01-05T08:40:59","location":"犍為縣"，"dateend":"2016-01-05T08:53:59",川LFC5501川LFC5501—1""querystr":"犍為外灘""}]場(chǎng)景二：關(guān)注時(shí)間和地點(diǎn)，多個(gè)車(chē)牌滿(mǎn)足其中1~N個(gè)條件序號(hào)交叉條件個(gè)數(shù)測(cè)試用例預(yù)期結(jié)果實(shí)際結(jié)果3次測(cè)試耗時(shí)(s)備注13[{""datestart":〃2016-02-07T16:44:10〃，""location"":""廣安市""，""dateend"":〃2016-02-07T16:44:20〃，"querystr":""人民南路"},{"datestart"":〃2016-02-10Tll:08:08〃，""location"":""廣安市""，""dateend"":〃2016-02-10Tll:09:09〃，""querystr"":""民族小學(xué)""},{"datestart"":""2016-01T4T19:53:10"",""location"":""武勝縣""，""dateend"":""2016-01T4T19:53:12"",""querystr"":""58KM""}]粵YQJ2212川XW62382粵YQJ2212川XW6238184924[{"datestart"":""2016-01-01T00:00:00"",""location"":""德陽(yáng)市""，""dateend"":""2016-01-04T00:00:00"",""querystr"":""西山""},{"datestart"":""2016-02-22T08:10:55"",""location"":""德陽(yáng)市""，""dateend"":""2016-02-22T08:10:55"",""querystr"":""108國(guó)道""},{"datestart"":"2015-11-16T06:17:58",""location"":""羅江縣""，""dateend"":"2015-11-16T06:17:58",""querystr"":""學(xué)校""},{"datestart"":""2016-01-03T16:53:16"",""location"":""綿陽(yáng)市""，""dateend"":""2016-01-03T16:53:16"",""querystr"":""東路""}]川FAH5672川FL19302川A9EB552川FAH5672川FL19302川A9EB5523163、ElasticSearch與Solr的壓力測(cè)試對(duì)比3.1、測(cè)試環(huán)境1■集群環(huán)境此次用于測(cè)試Solr的集群包括4個(gè)節(jié)點(diǎn)：0(master),1~3(slave1~slave3)。用于測(cè)試ElasticSearch的集群包括4個(gè)節(jié)點(diǎn)：0(datanode,不可選舉為masternode),1~3(datanode、可選舉為masternode)。測(cè)試集群信息：IP0~3CPU8*Intel(R)Xeon(R)CPUE5-26090@2.40GHzMemSolr：4G,ElasticSearch：4G(sys32G)OSCentOSrelease6.5測(cè)試客戶(hù)機(jī)（臨時(shí)使用slave3）：IP3CPU8*Intel(R)Xeon(R)CPUE5-26090@2.40GHzMemSolr4GElasticSearch4G(sys32G)OSCentOSrelease6.52?數(shù)據(jù)環(huán)境Solr中單索引庫(kù)中索引條數(shù)為5000W,ElasticSearch中單索引庫(kù)中索引條數(shù)為1億。3.2、測(cè)試工具工具名稱(chēng)http_load輸入?yún)?shù)說(shuō)明http_load-p5-f1000train_hc.txt-parallel簡(jiǎn)寫(xiě)-p：并發(fā)的用戶(hù)進(jìn)程數(shù)-fetches簡(jiǎn)寫(xiě)-f：總計(jì)的訪問(wèn)次數(shù)-rate簡(jiǎn)寫(xiě)-r:每秒的訪問(wèn)頻率-seconds簡(jiǎn)寫(xiě)-s：總計(jì)的訪問(wèn)時(shí)間輸出參1000fetches,5maxparallel,1.13104e+06bytes,in4.50961seconds運(yùn)行的數(shù)說(shuō)明請(qǐng)求數(shù)、最大的并發(fā)進(jìn)程數(shù)、總計(jì)傳輸?shù)臄?shù)據(jù)、運(yùn)行的時(shí)間1131.04meanbytes/connection平均每個(gè)請(qǐng)求連接傳輸?shù)臄?shù)據(jù)量221.748fetches/sec,250807bytes/sec每秒響應(yīng)請(qǐng)求數(shù)、每秒傳輸數(shù)據(jù)量msecs/connect:0.485064mean,1.337max,0.384min連接平均響應(yīng)時(shí)間、最大響應(yīng)時(shí)間、最小響應(yīng)時(shí)間msecs/first-response:21.7613mean,436.541max,13.468min連接平均返回時(shí)間321badbytecountsHTTPresponsecodes:請(qǐng)求響應(yīng)碼返回個(gè)數(shù)code200—10003.3、測(cè)試結(jié)果注：每組測(cè)試均重復(fù)進(jìn)行三次。場(chǎng)景一：模糊數(shù)字査詢(xún)查詢(xún)條件："q":"31997"統(tǒng)計(jì)測(cè)試結(jié)果如下:并發(fā)數(shù)總訪問(wèn)次數(shù)搜索引擎平均每秒響應(yīng)請(qǐng)求數(shù)(fetches/s)總消耗時(shí)間(s)連接平均響應(yīng)時(shí)間(ms)連接平均返回時(shí)間(ms)平均成功次數(shù)平均失敗次數(shù)1010000Solr239.66441.7250.16340.467100000ElasticSearch552.47118.1010.13017.6411000002510000Solr530.81918.8390.77744.675100000ElasticSearch677.38514.7630.12836.4281000005010000Solr664.68015.0450.20573.096100000ElasticSearch686.34014.5700.13072.28710000010010000Solr718.75113.9130.243136.493100000ElasticSearch699.83814.2890.440141.77310000020010000Solr631.86915.8260.289310.823100000ElasticSearch632.78915.8034.370310.03010000050010000Solr38.925256.9081.9711848.6906479353

ElasticSearch680.43214.6974.941716.01010000080010000Solr166.50360.0592.433278.3741139887ElasticSearch521.34919.18139.6641425.5809782218100010000Solr166.12360.1967.802386.7981489852ElasticSearch692.36114.44381.5321276.0009848152120010000Solr41.609240.33192.813407.1501019899ElasticSearch164.70560.715187.910841.63715528448場(chǎng)景二：精確中文查詢(xún)查詢(xún)條件："q"："人民南路"統(tǒng)計(jì)測(cè)試結(jié)果如下：并發(fā)數(shù)總訪問(wèn)次數(shù)搜索引擎平均每秒響應(yīng)請(qǐng)求數(shù)(fetches/s)總消耗時(shí)間(s)連接平均響應(yīng)時(shí)間(ms)連接平均返回時(shí)間(ms)平均成功次數(shù)平均失敗次數(shù)1010000Solr274.66536.4080.15935.091100000ElasticSearch606.13516.4980.13316.0401000002510000Solr496.52420.1400.17348.379100000ElasticSearch832.06612.0180.12229.5811000005010000Solr528.96718.9050.39291.614100000ElasticSearch822.15112.1630.13260.28210000010010000Solr499.61420.0160.313196.558100000ElasticSearch848.29911.7881.236115.98910000020010000Solr463.91021.5560.358422.760100000ElasticSearch785.50912.7312.071249.89810000050010000Solr155.62664.25610.400183.3117989202ElasticSearch827.33112.0879.174576.56710000080010000Solr166.10660.2037.612343.5382209780ElasticSearch814.86012.27230.071912.044100000100010000Solr166.01860.2342.354451.4892289772ElasticSearch867.41811.52946.7301026.2909859141120010000Solr165.70260.35079.148428.1622789722ElasticSearch165.51660.417185.596734.681159984013.4、總結(jié)?在本次測(cè)試中，每次測(cè)試總的訪問(wèn)次數(shù)為10000,從以上圖表中可以看出隨著并發(fā)數(shù)的增加，Solr和ElasticSearch的每秒響應(yīng)請(qǐng)求數(shù)都是先增加，然后穩(wěn)定在一定數(shù)值，在超過(guò)某個(gè)閾值之后極速下降；從平均每秒響應(yīng)請(qǐng)求次數(shù)來(lái)看，ElasticSearch性能指標(biāo)優(yōu)于Solr；在不同的并發(fā)數(shù)下，ElasticSearch的每秒響應(yīng)請(qǐng)求數(shù)在500~800之間，而Solr的每秒響應(yīng)請(qǐng)求數(shù)在300~600之間，ElasticSearch采用默認(rèn)分詞進(jìn)行中文模糊搜索時(shí)速度非常慢，經(jīng)分析更換ik中文分詞后性能有明顯提高；?從連接平均返回時(shí)間來(lái)看，在低并發(fā)時(shí)，ElasticSearch和Solr相差不大，但是隨著并發(fā)數(shù)量的增加，ElasticSearch返回時(shí)間的增長(zhǎng)速度大于Solr；?從不同的并發(fā)數(shù)下的成功次數(shù)來(lái)看，ElasticSearch性能指標(biāo)優(yōu)于Solr，Solr在并發(fā)數(shù)達(dá)到500的時(shí)候出現(xiàn)大面積錯(cuò)誤；ElasticSearch直到并發(fā)數(shù)為1000的時(shí)候才出現(xiàn)了錯(cuò)誤；Solr集群不太穩(wěn)定，12月30日簡(jiǎn)單測(cè)試過(guò)一次，當(dāng)時(shí)并發(fā)達(dá)到1000時(shí)才開(kāi)始出現(xiàn)性能大幅下降，并且平均每秒響應(yīng)請(qǐng)求數(shù)在600左右，而31日則在并發(fā)500時(shí)表現(xiàn)有較大差距。4、ElasticS