培訓(xùn)-ELK日志監(jiān)控報警實戰(zhàn)-張人杰-20181104

ELK日志監(jiān)控報警實戰(zhàn)磁云科技張人杰2018.10.16什么是ELKE:

ElasticSearchL:

LogstashK:

Kibana運行效果當(dāng)服務(wù)器宕機(jī)時,立即發(fā)送郵件通知環(huán)境搭建(一)

ElasticSearch安裝1、安裝elasticsearch的yum源的密鑰rpm--importhttps://artifacts.elastic.co/GPG-KEY-elasticsearch2、配置elasticsearch的yum源vim/etc/yum.repos.d/elasticsearch.repo[elasticsearch-6.x]name=Elasticsearchrepositoryfor6.xpackagesbaseurl=https://artifacts.elastic.co/packages/6.x/yumgpgcheck=1gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearchenabled=1autorefresh=1type=rpm-md3、安裝elasticsearchyuminstall-yelasticsearch

環(huán)境搭建(一)

ElasticSearch環(huán)境搭建1、需要安裝jdk1.8版本以上的java-version2、創(chuàng)建elasticsearchdata的存放目錄，并修改該目錄的屬主屬組mkdir-p/data/es-datachown-Relasticsearch:elasticsearch/data/es-data3、修改elasticsearch的日志屬主屬組chown-Relasticsearch:elasticsearch/var/log/elasticsearch/4、修改elasticsearch的配置文件vim/etc/elasticsearch/elasticsearch.yml

/etc/elasticsearch/elasticsearch.yml編輯找到配置文件中的，打開該配置并設(shè)置集群名稱:elk-tang找到配置文件中的，打開該配置并設(shè)置節(jié)點名稱:elk-tang-1修改data存放的路徑path.data:/data/es-data修改logs日志的路徑path.logs:/var/log/elasticsearch/注釋配置內(nèi)存使用用交換分區(qū)#bootstrap.memory_lock:true監(jiān)聽的網(wǎng)絡(luò)地址network.host:開啟監(jiān)聽的端口http.port:9200增加新的參數(shù)，這樣head插件可以訪問es(5.x版本，如果沒有可以自己手動加)http.cors.enabled:truehttp.cors.allow-origin:"*"啟動ElasticSearch/etc/init.d/elasticsearchstart

創(chuàng)建開機(jī)自啟動服務(wù)chkconfigelasticsearchon其他需要修改的參數(shù)vim/etc/security/limits.conf在末尾追加以下內(nèi)容（elk為啟動用戶，當(dāng)然也可以指定為*）elksoftnofile65536elkhardnofile65536elksoftnproc2048elkhardnproc2048elksoftmemlockunlimitedelkhardmemlockunlimitedvim/etc/security/limits.d/XXX-nproc.conf將里面的1024改為2048（ES最少要求為2048）*softnproc2048vim/etc/elasticsearch/elasticsearch.yml加入以下內(nèi)容bootstrap.system_call_filter:false再次啟動/etc/init.d/elasticsearchrestart環(huán)境搭建（二）

安裝elasticsearch-head插件安裝node.jssudocurl-sL-o/etc/yum.repos.d/khara-nodejs.repo/coprs/khara/nodejs/repo/epel-7/khara-nodejs-epel-7.reposudoyuminstall-ynodejsnodejs-npm安裝headgitclonegit:///mobz/elasticsearch-head.gitcdelasticsearch-headnpminstallnpmrunstart環(huán)境搭建（三）

安裝Logstash環(huán)境Logstash需要安裝到產(chǎn)生日志的服務(wù)器上rpm--importhttps://artifacts.elastic.co/GPG-KEY-elasticsearchyuminstall-ylogstashrpm-qllogstashln-s/usr/share/logstash/bin/logstash/bin/Logstash配置input{file{path=>["/var/log/nginx/access.log"]start_position=>"beginning"ignore_older=>0}}filter{grok{patterns_dir=>"/opt/logstash/patterns"match=>{"message"=>"%{NGINXACCESS}"}add_field=>[“resp_code”,“%{response}”]}geoip{source=>"http_x_forwarded_for"target=>"geoip"database=>"/etc/logstash/GeoLite2-City.mmdb"add_field=>["[geoip][coordinates]","%{[geoip][longitude]}"]add_field=>["[geoip][coordinates]","%{[geoip][latitude]}"]}mutate{convert=>["[geoip][coordinates]","float"]convert=>["response","integer"]convert=>["bytes","integer"]replace=>{"type"=>"nginx_access"}remove_field=>"message"}date{match=>["timestamp","dd/MMM/yyyy:HH:mm:ssZ"]}mutate{remove_field=>"timestamp"}}output{elasticsearch{hosts=>[":9200"]index=>"logstash-nginx-access-%{+YYYY.MM.dd}"}stdout{codec=>rubydebug}}建立grok使用的表達(dá)式mkdir-pv/opt/logstash/patternsvi/opt/logstash/patterns/nginxNGUSERNAME[a-zA-Z\.\@\-\+_%]+NGUSER%{NGUSERNAME}NGINXACCESS%{IPORHOST:clientip}-%{NOTSPACE:remote_user}\[%{HTTPDATE:timestamp}\]\"(?:%{WORD:verb}%{NOTSPACE:request}HTTP/%{NUMBER:httpversion}|%{DATA:rawrequest})\"%{NUMBER:response}(?:%{NUMBER:bytes}|-)%{QS:referrer}%{QS:agent}\"(?:%{IPV4:http_x_forwarded_for}|-)\"GeoIP的數(shù)據(jù)庫解析ipwget/download/geoip/database/GeoLite2-City.tar.gztar-xzvfGeoLite2-City.tar.gzmvGeoLite2-City_20181030/GeoLite2-City.mmdb/etc/logstash/.測試配置文件并啟動Logstash服務(wù)logstash-t-f./elk.confnohuplogstash–f./elk.conf2>&1>/dev/null&環(huán)境搭建（四）

Kibanawgethttps://artifacts.elastic.co/downloads/kibana/kibana-6.4.2-linux-x86_64.tar.gz#注意需要與ES對應(yīng)的版本tar-xzfkibana-6.4.2-linux-x86_64.tar.gzmvkibana-6.4.2-linux-x86_64/usr/localln-s/usr/local/kibana-6.4.2-linux-x86_64//usr/local/kibanavim/usr/local/kibana/config/kibana.yml/usr/local/kibana/config/kibana.yml編輯server.port:5601server.host:""elasticsearch.url:"http://localhost:9200"kibana.index:".kibana"安裝screen,以便于kibana在后臺運行yum-yinstallscreenscreen/usr/local/kibana/bin/kibanaKibana安裝完成打開瀏覽器并設(shè)置對應(yīng)的indexhttp://localhost:5601在Kibana上安裝sentinl插件用于發(fā)送郵件提醒1、到/sirensolutions/sentinl/releases上選擇合適的版本2、在服務(wù)器上運行：/usr/local/kibana/bin/kibana-plugininstall/sirensolutions/sentinl/releases/download/tag-6.4.2-0/sentinl-v6.4.2.zip3、重啟kibana4、在kibana界面上配置sentinl注意：需要定時清理日志文件定期清理nginx日志文件echo'::1--[03/Nov/2018:20:28:03+0800]"GET/HTTP/1.1"2000"-""Mozilla/5.0(X11;Li