Logstash+ElasticSearch+Kibana3架構(gòu)日志系統(tǒng)操作手冊

Logstash+ElasticSearch+Kibana3架構(gòu)日志系統(tǒng)一，分析系統(tǒng)簡介日志監(jiān)控和分析在保障業(yè)務穩(wěn)定運行時，起到了很重要的作用，不過一般情況下日志都分散在各個生產(chǎn)服務器，且開發(fā)人員無法登陸生產(chǎn)服務器，這時候就需要一個集中式的日志收集裝置，對日志中的關(guān)鍵字進行監(jiān)控，觸發(fā)異常時進行報警，并且開發(fā)人員能夠查看相關(guān)日志。logstash+elasticsearch+kibana3就是實現(xiàn)這樣功能的一套系統(tǒng)，并且功能更強大。logstash是一個管理日志和事件的工具，你可以收集它們，解析它們，并存儲它們以供以后使用(例如日志搜索)，logstash有一個內(nèi)置的web界面，用來搜索你的所有日志。logstash在部署時有兩種運行模式：standalone和centralized：*standalone：standalone的意思是所有的事情都在一臺服務器上運行，包括日志收集、日志索引、前端WEB界面都部署在一臺機器上。*centralized：就是多服務器模式，從很多服務器運輸(ship)日志到一臺總的日志(collector)服務器上用來索引和查找。需要注意的是logstash本身并沒有什么shipper和collector這種說法，因為不論是運輸日志的進程還是匯集總的日志的進程運行的都是同一個程序，只是使用的配置文件不同而已。elasticsearch:基于lucene的開源搜索引擎，是一個分布式的搜索分析系統(tǒng)，主要特點有：realtimedata、realtimeanalytics、distributed、highavailability、multi-tenancy、fulltextsearch、documentoriented、conflictmanagement、schemafree、restfulapi等等。kibana3:可視化日志和數(shù)據(jù)系統(tǒng)，作為WEB前端可以很容易的和elasticsearch系統(tǒng)結(jié)合。kibana有版本2和版本3的區(qū)分，版本2采用ruby編寫，部署起來很麻煩，需要安裝很多ruby依賴包(目前網(wǎng)上多是這個版本的部署)，版本3采用純html+css編寫，因此部署起來很方便，解壓即用。二，測試步驟測試架構(gòu)兩臺機器試驗，CentOS6.540：logstashagent，redis，jdk35：logstashindex，elasticsearch，kibana，jdk其中redis上安裝agent端，agent和index的logstash安裝是一樣的，唯一不同點是conf文件不同。2，redis和logstashagent安裝（40）1），安裝java環(huán)境/en/download/manual.jsprpm-ivhjdk-7u67-linux-x64.rpmPreparing...###########################################[100%]1:jdk###########################################[100%]UnpackingJARfiles... rt.jar... jsse.jar... charsets.jar... localedata.jar... jfxrt.jar...#java-versionjavaversion"1.7.0_67"Java(TM)SERuntimeEnvironment(build1.7.0_67-b01)JavaHotSpot(TM)64-BitServerVM(build24.65-b04,mixedmode)2），安裝rediswgethttp://download.redis.io/releases/redis-2.8.17.tar.gztar-zxvfredis-2.8.17.tar.gzcd/redis-2.8.17.tar.gz先安裝tcl,否則下面會報錯yuminstalltcl-ymakeMALLOC=libcmaketestmakeinstall#pwd/soft/redis-2.8.17/utils./install_server.shWelcometotheredisserviceinstallerThisscriptwillhelpyoueasilysetuparunningredisserverPleaseselecttheredisportforthisinstance:[6379]Selectingdefault:6379Pleaseselecttheredisconfigfilename[/etc/redis/6379.conf]Selecteddefault-/etc/redis/6379.confPleaseselecttheredislogfilename[/var/log/redis_6379.log]Selecteddefault-/var/log/redis_6379.logPleaseselectthedatadirectoryforthisinstance[/var/lib/redis/6379]Selecteddefault-/var/lib/redis/6379Pleaseselecttheredisexecutablepath[/usr/local/bin/redis-server]Selectedconfig:Port:6379Configfile:/etc/redis/6379.confLogfile:/var/log/redis_6379.logDatadir:/var/lib/redis/6379Executable:/usr/local/bin/redis-serverCliExecutable:/usr/local/bin/redis-cliIsthisok?ThenpressENTERtogoonorCtrl-Ctoabort.Copied/tmp/6379.conf=>/etc/init.d/redis_6379Installingservice...Successfullyaddedtochkconfig!Successfullyaddedtorunlevels345!StartingRedisserver...Installationsuccessful!#pwd/soft/redis-2.8.17/src[root@logserversrc]#./redis-cli-h-p6379:6379>pingPONG:6379>setnamefooOK:6379>getname"foo":6379>bye(error)ERRunknowncommand'bye':6379>quit3），安裝logstashagent#rpm-ivhlogstash-1.4.2-1_2c0f5a1.noarch.rpmPreparing...###########################################[100%]1:logstash###########################################[100%][root@logserversoft]#rpm-ivhlogstash-contrib-1.4.2-1_efd53ef.noarch.rpmPreparing...###########################################[100%]1:logstash-contrib###########################################[100%]修改配置文件cat/etc/logstash/conf.d/logstash.agent.confinput{file{path=>"/var/log/messages" type=>"syslog"}file{path=>"/var/log/httpd/access_log" type=>"apache"}}output{redis{host=>"40"data_type=>"list"key=>"logstash"}}啟動logstashagentservicelogstashstartroot142030.732.61397132119108pts/0SNl14:231:02/usr/java/jdk1.7.0_67/bin/java-Djava.io.tmpdir=/var/lib/logstash-Xmx500m-XX:+UseParNewGC-XX:+UseConcMarkSweepGC-Djava.awt.headless=true-XX:CMSInitiatingOccupancyFraction=75-XX:+UseCMSInitiatingOccupancyOnly-jar/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar-I/opt/logstash/lib/opt/logstash/lib/logstash/runner.rbagent-f/etc/logstash/conf.d-l/var/log/logstash/logstash.loglogstashindex安裝（35）1），安裝java環(huán)境/en/download/manual.jsprpm-ivhjdk-7u67-linux-x64.rpmPreparing...###########################################[100%]1:jdk###########################################[100%]UnpackingJARfiles... rt.jar... jsse.jar... charsets.jar... localedata.jar... jfxrt.jar...#java-versionjavaversion"1.7.0_67"Java(TM)SERuntimeEnvironment(build1.7.0_67-b01)JavaHotSpot(TM)64-BitServerVM(build24.65-b04,mixedmode)2），安裝logstash#rpm-ivhlogstash-1.4.2-1_2c0f5a1.noarch.rpmPreparing...###########################################[100%]1:logstash###########################################[100%][root@logserversoft]#rpm-ivhlogstash-contrib-1.4.2-1_efd53ef.noarch.rpmPreparing...###########################################[100%]1:logstash-contrib###########################################[100%]修改配置文件cat/etc/logstash/conf.d/logstash.index.confinput{redis{host=>"40"data_type=>"list"port=>"6379"key=>"logstash"type=>"redis-input"}}output{elasticsearch{host=>"35"port=>"9300"}}啟動logstashindexservicelogstashstartpsaux|greplogstashroot63651.138.51319172140912?SNl14:221:27/usr/bin/java-Djava.io.tmpdir=/var/lib/logstash-Xmx500m-XX:+UseParNewGC-XX:+UseConcMarkSweepGC-Djava.awt.headless=true-XX:CMSInitiatingOccupancyFraction=75-XX:+UseCMSInitiatingOccupancyOnly-jar/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar-I/opt/logstash/lib/opt/logstash/lib/logstash/runner.rbagent-f/etc/logstash/conf.d-l/var/log/logstash/logstash.log3)，安裝elasicsearchwget/elasticsearch/elasticsearch/elasticsearch-1.3.4.noarch.rpmrpm-ivhelasticsearch-1.3.4.noarch.rpmPreparing...###########################################[100%]1:elasticsearch#########################################