172173.com

架构说明

  • rsyslog client上应用程序的原生日志不做处理，直接通过relp协议发送到rsyslog server
  • rsyslog server接收日志队列，通过配置tag+msg配合设施（管道）识别原生日志再写到自定义的文件中分类保存。

一、Rsyslog Client

CentOS 7.6 为例

1.安装relp协议模块

1
2
3
4
5
# CentOS
yum install rsyslog-relp

# Ubuntu
apt-get install rsyslog-relp

2.配置rsyslog.conf

加载模块

1
2
3
4
5
6
7
8
9
10
11
12
# 加载输出模块 omrelp
$ModLoad omrelp

# 加载输入模块 三选一，我这边用的是 imtcp
#$ModLoad imudp
#$UDPServerRun 514

$ModLoad imtcp
$InputTCPServerRun 514

#$ModLoad imrelp
#$InputRELPServerRun 514

参数优化

1
2
3
$SystemLogRateLimitInterval 0   # Interval 设置率计算的时间间隔，0 表示关闭
$SystemLogRateLimitBurst 0      # Burst 设置该间隔内允许的日志数，0 表示关闭
# $MaxMessageSize 16k           # 日志最大大小，太大的值需要考虑传输协议，如 UDP

配置输入设施

  • 这边要看应用程序生产日志配置的是什么设施，推荐使用自定义设施local0-local6
  • 但是java log4j2模块即便配置了local3，也无法识别，生产设置走的是默认的user.notice
  • 所以这边以user.notice为设施
    • 禁止user.*设施生产的日志写到本地
    • user.*设施生产的所有级别日志发送到远端rsyslog server
1
2
3
4
5
6
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none;user.none                /var/log/messages

# Save boot messages also to boot.log
user.*                                                :omrelp:10.10.0.19:20514

3.检查语法

1
2
3
rsyslogd -N1
或
rsyslogd -f /etc/rsyslog.conf -N1

4.重启rsyslogd

1
systemctl restart rsyslog.service

一、Rsyslog Server

Ubuntu 16.04

1.安装relp协议模块

1
2
3
4
5
# CentOS
yum install rsyslog-relp

# Ubuntu
apt-get install rsyslog-relp

2.配置rsyslog.conf

加载模块

加载输入模块，主要是在tcp20514端口接收日志

1
2
3
4
5
$ModLoad imrelp
$InputRELPServerRun 20514

$ModLoad imtcp
$InputTCPServerRun 514

参数优化

1
2
3
4
5
6

$SystemLogRateLimitInterval 0   # Interval 设置率计算的时间间隔，0 表示关闭
$SystemLogRateLimitBurst 0      # Burst 设置该间隔内允许的日志数，0 表示关闭
# $MaxMessageSize 16k           # 日志最大大小，太大的值需要考虑传输协议，如 UDP
# $InputTCPMaxSessions 1024
$EscapeControlCharactersOnReceive off

3.配置/etc/rsyslog.d/50-default.conf

禁止来自设施user.none的日志写到本地文件。

1
2
#*.*;auth,authpriv.none     -/var/log/syslog
*.*;auth,authpriv.none;local3.none;local4.none;user.none        -/var/log/syslog

4.过滤切割

应用程序原生日志

1
Aug 14 17:05:59 xxxx xxxx 2019-08-14 17:05:59,386 [LogDataUtil.java:44][INFO]:{"account_info":{"age":"","aid":"yyt_201908226050558812_518","area":"","gamekey":"notSdk","sex":"0","sid":0,"uid":55397},"create_time":1565773559380,"device_info":{"device_model":"","device_type":"","os_info":"","pt":0,"uuid":""},"ip":"220.249.166.153","log_id":"06b8090e-93ee-4478-b0d2-8dc8749e41e2","log_type":"register_log","role_info":{"job":"","nickname":"yyt_201908226050558812_518","role_id":55397,"sex":""}}

过滤切割/etc/rsyslog.d/gamelog-pxmbd.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# $template cocsFormat, "%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg%\n"
# $template DEBUG, "/data/rsyslog/%fromhost-ip%/DEBUG_%$year%%$month%%$day%.log"


$template xxxx_REGISTER_LOG, "/data/rsyslog/%fromhost-ip%/%$year%%$month%%$day%/xxxx_register_%$year%%$month%%$day%.log"
$template xxxx_LOGIN_LOG, "/data/rsyslog/%fromhost-ip%/%$year%%$month%%$day%/xxxx_login_%$year%%$month%%$day%.log"
$template xxxx_RESOURCE_LOG, "/data/rsyslog/%fromhost-ip%/%$year%%$month%%$day%/xxxx_resource_%$year%%$month%%$day%.log"
$template xxxx_PAYMENT_LOG, "/data/rsyslog/%fromhost-ip%/%$year%%$month%%$day%/xxxx_payment_%$year%%$month%%$day%.log"
$template xxxx_ACTION_LOG, "/data/rsyslog/%fromhost-ip%/%$year%%$month%%$day%/xxxx_action_%$year%%$month%%$day%.log"
$template xxxx_UNKNOWN_LOG, "/data/rsyslog/%fromhost-ip%/%$year%%$month%%$day%/xxxx_unknown_%$year%%$month%%$day%.log"
$template xxxx_BUGS_LOG, "/data/rsyslog/%fromhost-ip%/%$year%%$month%%$day%/xxxx_bugs_%$year%%$month%%$day%.log"


if $programname startswith 'xxxx' and $msg contains 'register_log' then ?xxxx_REGISTER_LOG
& stop
if $programname startswith 'xxxx' and $msg contains 'login_log' then ?xxxx_LOGIN_LOG
& stop
if $programname startswith 'xxxx' and $msg contains 'resource_log' then ?xxxx_RESOURCE_LOG
& stop
if $programname startswith 'xxxx' and $msg contains 'payment_log' then ?xxxx_PAYMENT_LOG
& stop
if $programname startswith 'xxxx' and $msg contains 'action_log' then ?xxxx_ACTION_LOG
& stop
if $programname startswith 'xxxx' then ?xxxx_BUGS_LOG
& stop
if $programname startswith 'xxxx' and $syslogfacility-text == 'user' and $syslogseverity <= '5' then ?xxxx_UNKNOWN_LOG
& stop

5.重启rsyslogd

1
2
3
4
5
6
7
8
9
10
11
# 检查语法
rsyslogd -N1    # 或 rsyslogd -f /etc/rsyslog.conf -N1

# 授权
chown -R syslog.syslog /data/rsyslog

# 重启rsyslogd
systemctl restart rsyslog.service 

# 开机自启动
echo '/etc/init.d/rsyslog start' >> /etc/rc.local

三、Client logger命令测试

client

1
2
3
4
5
logger -it xxxx -p user.warning 111111111register_log111111111
logger -it xxxx -p user.warning 111111111login_log111111111
logger -it xxxx -p user.warning 111111111resource_log111111111
logger -it xxxx -p user.warning 111111111payment_log111111111
logger -it xxxx -p user.warning 111111111action_log111111111

rsyslog server

1
2
3
4
5
6
ls
-rw-r----- 1 syslog adm      72 Aug 14 15:54 xxxx_user_4_xxxx_action_20190814.log
-rw-r----- 1 syslog adm     142 Aug 14 15:54 xxxx_user_4_xxxx_login_20190814.log
-rw-r----- 1 syslog adm      73 Aug 14 15:54 xxxx_user_4_xxxx_payment_20190814.log
-rw-r----- 1 syslog adm      74 Aug 14 15:54 xxxx_user_4_xxxx_register_20190814.log
-rw-r----- 1 syslog adm      74 Aug 14 15:54 xxxx_user_4_xxxx_resource_20190814.log
 上一页

Rsyslog

 评论