其他
ES 慢查询日志收集实战总结
点击上方“民工哥技术之路”选择“星标”
每天10点为你分享不一样的干货
前言
搜索慢日志
# vim /etc/elasticsearch/elasticsearch.yml
# 记录获取慢日志
index.search.slowlog.threshold.fetch.warn: 1s
index.search.slowlog.threshold.fetch.info: 200ms
index.search.slowlog.threshold.fetch.debug: 60ms
index.search.slowlog.threshold.fetch.trace: 50ms
# 记录查询慢日志
index.search.slowlog.threshold.query.warn: 1s
index.search.slowlog.threshold.query.debug: 500ms
索引慢日志
# vim /etc/elasticsearch/elasticsearch.yml
index.search.slowlog.threshold.index.warn: 10s
index.search.slowlog.threshold.index.info: 5s
index.search.slowlog.threshold.index.debug: 2s
index.search.slowlog.threshold.index.trace: 500ms
index.search.slowlog.level: info
index.search.slowlog.source: 1000
Logging配置
# vim /etc/elasticsearch/logging.yml
index.search.slowlog: TRACE, index_search_slow_log_file
index.indexing.slowlog: TRACE, index_indexing_slow_log_file
additivity:
index.search.slowlog: true
index.indexing.slowlog: true
deprecation: false
index_search_slow_log_file:
type: dailyRollingFile # 日志类型,每天一个文件
file: ${path.logs}/${cluster.name}_index_search_slowlog.log # 文件命名格式
datePattern: "'.'yyyy-MM-dd" # 每日备份的后缀
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n" #记录日志的开头格式
index_indexing_slow_log_file:
type: dailyRollingFile
file: ${path.logs}/${cluster.name}_index_indexing_slowlog.log
datePattern: "'.'yyyy-MM-dd"
layout:
type: pattern
conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"
通过API动态设置慢日志
这时一个索引级别的日志,也就是说可以独立应用给索引:
PUT /my_index/_settings
{
"index.search.slowlog.threshold.query.warn" : "10s", # 查询慢于10秒输出一个WARN日志
"index.search.slowlog.threshold.fetch.debug": "500ms", # 获取慢于500毫秒输出一个DEBUG日志
"index.indexing.slowlog.threshold.index.info": "5s" # 索引慢于5秒输出一个INFO日志
}
logstash收集ES慢日志到ES存储中
input{
file{
start_position => "beginning"
path=> ["填入你的ES慢日志路径"]
sincedb_path => "./slowlogdb"
}
}
filter {
ruby{
code => "temp=event['message'].split(', ');
t1= temp[0]
common_attr=t1.split(']')
event['time']=common_attr[0].split('[')[1]
event['loglevel']=common_attr[1].split('[')[1]
event['slowtype']=common_attr[2].split('[')[1]
event['indexname']=common_attr[3].split('[')[1]
t2= temp[1]
time_attr=t2.split('[')
event['took_millis']= time_attr[1].split(']')[0]
t3= temp[2]
t4= temp[3]
t5= temp[4]
t6= temp[5]
shards_attr=t6.split('[')
event['total_shards']= shards_attr[1].split(']')[0]
t7= temp[6]
t8= temp[7]
event['search_type']= t5
event['message']= t7
event['extra_source']= t8
"}
mutate{
convert => ["took_millis","integer"] #设置took_millis的类型为integer类型
}
mutate{
convert => ["total_shards","integer"] #设置total_shards的类型为integer类型
}
}
output{
elasticsearch{
index => "es-slowlog-%{+YYYY-MM}"
hosts=> [填入你的ES集群主机列表]
flush_size => 3000
}
}
在logstash调试模式输出ES慢日志各字段含义说明:
{
# 慢查询的语句
"message" => "source[{\"fields\":[\"_parent\",\"_source\"],\"query\":{\"bool\":{\"must\":[],\"must_not\":[],\"should\":[{\"match_all\":{}}]}},\"from\":0,\"size\":50,\"sort\":[],\"aggs\":{},\"version\":true}]",
"@version" => "1",
"@timestamp" => "2018-03-15T12:20:40.091Z",
# 慢查询日志路径
"path" => "/root/test.log",
# 慢查询主机名
"host" => "c7-node1.fblinux.com",
# 慢查询产生时间
"time" => "2018-03-15 11:26:30,318",
# 慢查询级别
"loglevel" => "INFO ",
# 慢查询类型
"slowtype" => "index.search.slowlog.query",
# 索引名称
"indexname" => "test-2018-03",
# 慢查询时间,单位毫秒
"took_millis" => 64,
# 总shards数量
"total_shards" => 1188,
"search_type" => "search_type[QUERY_THEN_FETCH]",
"extra_source" => "extra_source[],"
}
版权申明:作者:西门飞冰,一名90后it男,一直在北京工作,热爱运动,热爱冒险,热爱旅行。原文:http://www.fblinux.com/?p=1334,由作者原创投稿,版权归原创者所有。除非无法确认,我们都会标明作者及出处,如有侵权烦请告知,我们会立即删除并表示歉意,谢谢。
关注 民工哥技术之路 微信公众号对话框回复关键字:1024 可以获取一份最新整理的技术干货:包括系统运维、数据库、redis、MogoDB、电子书、Java基础课程、Java实战项目、架构师综合教程、架构师实战项目、大数据、Docker容器、ELK Stack、机器学习、BAT面试精讲视频等。
点击【阅读原文】发现更多精彩内容~~