ELK企业级日志分析平台（二）——Filebeat日志采集与Kibana数据可视化

文章目录filebeat日志采集filebeat 采集web 服务日志输出到ES输出到logstashkibana数据可视化部署定制数据可视化网站访问量访问量排行榜创建dashboard大屏展示filebeat日志采集官方文档https://www.elastic.co/guide/en/beats/filebeat/7.6/filebeat-overview.html​ Filebeat 是一个 Elastic Beat组件一款轻量级的日志数据转发器用于转发和集中日志数据。Filebeat 作为代理安装在指定的服务器上会监控指定的日志文件或位置收集日志事件并将其转发到Elasticsearch或 Logstash进行索引。​ Filebeat 的工作原理启动 Filebeat 时它会启动一个或多个输入用于查找指定的日志数据位置。对于 Filebeat 找到的每个日志Filebeat 都会启动一个 Harvester收割机。每个 Harvester收割机 会读取单个日志以查找新内容并将新的日志数据发送到 libbeatlibbeat 会聚合事件并将聚合数据发送到 Filebeat 配置的输出。filebeat 采集web 服务日志https://www.elastic.co/guide/en/beats/filebeat/7.6/filebeat-module-apache.html[rootserver4 ~]# yum install -y httpd[rootserver4 ~]# echo server4 /var/www/html/index.html[rootserver4 ~]# systemctl enable --now httpd.service[rootserver4 ~]# rpm -ivh filebeat-7.6.1-x86_64.rpm[rootserver4 ~]# cd /etc/filebeat/modules.d[rootserver4 modules.d]# filebeat modules enable apache #激活apache模块Enabled apache[rootserver4 modules.d]# vim apache.yml- module: apache# Access logsaccess: enabled:truevar.paths:[/var/log/httpd/access_log*]error: enabled:truevar.paths:[/var/log/httpd/error_log*]输出到EShttps://www.elastic.co/guide/en/beats/filebeat/7.6/elasticsearch-output.html[rootserver4 modules.d]# cd ..[rootserver4 filebeat]# vim /etc/filebeat/filebeat.ymloutput.elasticsearch:# Array of hosts to connect to.hosts:[192.168.117.131:9200]#检查配置文件语法[rootserver4 filebeat]# filebeat test config -c /etc/filebeat/filebeat.ymlConfig OK#测试与 ES 的连接[rootserver4 filebeat]# filebeat test output -c /etc/filebeat/filebeat.ymlelasticsearch: http://192.168.117.131:9200... parse url... OK connection... parse host... OK dns lookup... OK addresses:192.168.117.131 dial up... OK TLS... WARN secure connection disabled talk to server... OK version:7.6.1#启动 Filebeat[rootserver4 filebeat]# systemctl enable --now filebeat输出到logstash[rootserver5 conf.d]# vim beats.confinput{beats{port5044}}filter{grok{match{message%{HTTPD_COMBINEDLOG}}}}output{stdout{}elasticsearch{hosts192.168.117.131:9200indexapachelog-%{YYYY.MM.dd}}}#启动logstash[rootserver5 conf.d]# /usr/share/logstash/bin/logstash -rf /etc/logstash/conf.d/beats.conf看到Successfully后不要退出进程修改filebeat输出[rootserver4 filebeat]# vim /etc/filebeat/filebeat.ymloutput.logstash:# The Logstash hostshosts:[192.168.117.135:5044][rootserver4 filebeat]# systemctl restart filebeat.service测试在server3上curl server4拷贝软件包[rootserver4 ~]# scp filebeat-7.6.1-x86_64.rpm server3:server3上部署[rootserver3 ~]# yum install -y httpd[rootserver3 ~]# echo server3 /var/www/html/index.html[rootserver3 ~]# systemctl enable --now httpd.service[rootserver3 ~]# rpm -ivh filebeat-7.6.1-x86_64.rpm[rootserver3 ~]# cd /etc/filebeat/modules.d[rootserver3 modules.d]# filebeat modules enable apache[rootserver3 modules.d]# scp server4:/etc/filebeat/modules.d/apache.yml .[rootserver3 modules.d]# vim /etc/filebeat/filebeat.ymloutput.logstash:# The Logstash hostshosts:[192.168.117.135:5044][rootserver3 modules.d]# systemctl enable --now filebeat测试在server4上curl server3kibana数据可视化​ Kibana 是 Elastic Stack原 ELK Stack中的数据可视化仪表盘工具专门用于对 Elasticsearch 中存储的数据进行搜索、分析和可视化展示。​ 一句话概括Kibana Elasticsearch 的可视化仪表盘 数据分析平台。它让你通过 Web 界面而非命令行来查看、搜索、图表化 Elasticsearch 中的数据。部署server4上部署kibana[rootserver4 ~]# rpm -ivh kibana-7.6.1-x86_64.rpm[rootserver4 ~]# cd /etc/kibana/[rootserver4 kibana]# vim kibana.ymlserver.host:0.0.0.0elasticsearch.hosts:[http://192.168.117.132:9200]i18n.locale:zh-CN[rootserver4 kibana]# systemctl enable --now kibana[rootserver4 kibana]# netstat -antlp |grep :5601tcp000.0.0.0:56010.0.0.0:* LISTEN2805/node访问web页面: 192.168.117.134:5601定制数据可视化创建索引模式网站访问量将日期选择today保存视图访问量排行榜保存视图创建dashboard大屏展示添加创建的可视化保存仪表板