Install
> sudo apt-get install syslog-ng
|
Pretty easy so far…J
Looking at the /etc/syslog-ng/syslog-ng.conf
> sudo vi /etc/syslog-ng/syslog-ng.conf
|
I am going to go over this line
log {
source(s_src); filter(f_syslog3); destination(d_syslog); };
|
Let me take this line apart…
log { source(s_src); filter(f_syslog3); destination(d_syslog);
};
|
Source
log { source(s_src);
filter(f_syslog3); destination(d_syslog); };
|
First source, this is what I am going to log. Here you can see that source contains s_src. This is just a variable go look for a line
containing "source s_src {
You should find this.
source s_src
{
system();
internal();
};
|
This a source that syslog will look at for logs. See https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-source-system.html
[1]
system() will look at what it considers system
locations. If you look down a little
further in the page you will see this.
For a linux system this will look at /dev/log a socket.
The internal() is for all message generated internally by
syslog-ng see https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-sources-internal.html
[2]
filter
log {
source(s_src); filter(f_syslog3);
destination(d_syslog); };
|
Here you can see that
filter contains f_syslog3
Looking around you will
find a line with "filter f_sylog3" in it.
filter
f_syslog3 { not
facility(auth, authpriv, mail) and not filter(f_debug ); };
|
Docs for filter can be found here https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-filters.html
[3]
Message have to pass all filters to go on to the
destination.
In this specific example what does not facility and not
filter mean? Here is a list of filter functions
Facility
Specific documentation can be found at https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-filters.html
[5]
The short of it is that logs coming in are going to have
codes. Here is a list of them
Numerical Code
|
Facility name
|
Facility
|
0
|
kern
|
kernel
messages
|
1
|
user
|
user-level
messages
|
2
|
mail
|
mail
system
|
3
|
daemon
|
system
daemons
|
4
|
auth
|
security/authorization
messages
|
5
|
syslog
|
messages
generated internally by syslogd
|
6
|
lpr
|
line
printer subsystem
|
7
|
news
|
network
news subsystem
|
8
|
uucp
|
UUCP
subsystem
|
9
|
cron
|
clock
daemon
|
10
|
authpriv
|
security/authorization
messages
|
11
|
ftp
|
FTP
daemon
|
12
|
ntp
|
NTP
subsystem
|
13
|
security
|
log
audit
|
14
|
console
|
log
alert
|
15
|
solaris-cron
|
clock
daemon
|
16-23
|
local0..local7
|
locally
used facilities (local0-local7)
|
So for this part
{ not facility(auth, authpriv, mail)
|
So in this case the log cannot be a auth, authpriv, or mail
type. Whose numerical codes are 4, 10,
2.
So for this part
not filter(f_debug )
|
f_debug refers to another filter in the same file and here
it is.
filter
f_debug { level(debug)
and not facility(auth, authpriv, news, mail); };
|
Here we again see the not facility but this time it has
news. So I guess we really did not need
this part in the first filter as it was in the nested filter. … oh wait the first one does a
not on this so it's an inverse of this filter… duh!
Level
Now for level
Here is the documentation for it https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-filters.html#filter-priority
[6]
This is just the simple log levels the options are
Value
|
Severity
|
Keyword
|
0
|
Emergency
|
emerg
|
1
|
Alert
|
alert
|
2
|
Critical
|
crit
|
3
|
Error
|
err
|
4
|
Warning
|
warn
|
5
|
Notice
|
notice
|
6
|
Informational
|
Info
|
7
|
Debug
|
debug
|
So in this case
filter
f_debug { level(debug)
and not facility(auth, authpriv, news, mail); };
|
So in this case it must be a debug level in order to pass
this filter.
Destination
log {
source(s_src); filter(f_syslog3); destination(d_syslog); };
|
Here you can see the destination that refers to the variable
d_syslog
Looking around you should find this.
destination d_syslog
{ file("/var/log/syslog"); };
|
This is fairly simple basically that is the file to log to
/var/log/syslog
So reading it all together
log {
source(s_src); filter(f_syslog3); destination(d_syslog); };
|
Here are the other pieces it references.
source s_src
{
system();
internal();
};
filter
f_syslog3 { not
facility(auth, authpriv, mail) and not filter(f_debug ); };
filter
f_debug { level(debug) and not
facility(auth, authpriv, news, mail);
};
destination d_syslog
{ file("/var/log/syslog"); };
|
As I read this.
1.
For logs going to system ie /dev/logs
2.
That are filtered
a. All
levels except debug
b. Not
faciilty Auth, AuthPriv, Mail
3.
And put the results in /var/log/syslog
Testing it out
Is there a simple way to echo a log to /dev/log as a test?
I am going to just try a simple python programing using
their logging system.
> vi log_test.py
|
Here is my code
#!/usr/bin/python3
import sys
import syslog
import logging
import logging.handlers
#Set up Logger
logger = logging.getLogger("10x13")
logger.setLevel(logging.DEBUG)
handler = logging.handlers.SysLogHandler(address='/dev/log',facility=syslog.LOG_LOCAL0)
#handler = logging.StreamHandler(sys.stdout)
log_format = logging.Formatter('%(asctime)s, %(message)s')
handler.setFormatter(log_format)
logger.addHandler(handler)
logger.info("Info Log Message")
logger.debug("Debug Log Message")
logger.error("Error log Message")
|
Make it executable
> chmod u+x log_test.py
|
Open another terminal and tail /var/log/syslog
> tail -f
/var/log/syslog
|
Run it
> ./log_test.py
|
Here are my results after running it twice. You cans see that I got the INFO and the ERROR but the DEBUG was filtered
out .
More specific docs on Python logging can be found here https://docs.python.org/2.6/library/logging.html#syslog-handler
[9]
New log file
I am going to make a new setting to send this type of log
file to a new location.
Open up the config file
> sudo vi /etc/syslog-ng/syslog-ng.conf
|
First the log (added near the end of the file
log { source(s_src); filter(f_app); destination(d_app); };
|
I am using the same source as before s_src which will listen
to /dev/log in my case on linux.
Filter
Now to make my filter
filter f_app { facility(local0); }
|
This says filter only those that are sent to facility local0
which is what I am doing in the python code.
This part
handler
= logging.handlers.SysLogHandler(address='/dev/log',facility=syslog.LOG_LOCAL0)
#handler = logging.StreamHandler(sys.stdout)
|
Destination
Set the destination
destination d_app { file("/var/log/app.log"); }
;
|
That should be it!
Now you need to reload syslog-ng
> sudo systemctl reload syslog-ng
|
Tailing the /var/log/syslog I see an error!
Mar 23 13:23:04 syslog syslog-ng[2359]: Error parsing configuration;
filename='/etc/syslog-ng/syslog-ng.conf'
|
Ahh I see the issue I forgot a ; at the end of filter f_app
It should have been
filter f_app { facility(local0); };
|
OK reload again
> sudo systemctl reload syslog-ng
|
Looks like it took that time.
Open another terminal and tail /var/log/syslog
> tail -f
/var/log/syslog
|
Run it
> ./log_test.py
|
Logs are coming in…
But if I look for /var/log/app.log
> ls -alh /var/log/app.log
|
I see nothing! The
file is not auto created… What am I doing wrong?
As a quick test I changed my filter to a known filter
log { source(s_src); filter(f_syslog3); destination(d_app); };
|
Reloaded and ran my test and it worked! So what is wrong with my filter?
A few more test and I found out it's my code the filter is
fine as is.
syslog.LOG_LOCAL0 = 128
rather than the 16 needed for local0 in syslog
Running a quick test in my python code.
print(syslog.LOG_LOCAL0)
print(syslog.LOG_LOCAL1)
print(syslog.LOG_LOCAL2)
print(syslog.LOG_LOCAL3)
print(syslog.LOG_LOCAL4)
print(syslog.LOG_LOCAL5)
print(syslog.LOG_LOCAL6)
print(syslog.LOG_LOCAL7)
|
Values
Type
|
Python
3 val
|
Needed
value
|
LOCAL0
|
128
|
16
|
LOCAL1
|
136
|
17
|
LOCAL2
|
144
|
18
|
LOCAL3
|
152
|
19
|
LOCAL4
|
160
|
20
|
LOCAL5
|
168
|
21
|
LOCAL6
|
176
|
22
|
LOCAL7
|
184
|
23
|
The solution is to tweak the code to use the number
#!/usr/bin/python3
import sys
import syslog
import logging
import logging.handlers
#Set up Logger
logger = logging.getLogger("10x13")
logger.setLevel(logging.DEBUG)
handler = logging.handlers.SysLogHandler(address='/dev/log',facility=16)
#handler = logging.StreamHandler(sys.stdout)
log_format = logging.Formatter('%(asctime)s, %(message)s')
handler.setFormatter(log_format)
logger.addHandler(handler)
logger.info("Info Log Message")
logger.debug("Debug Log Message")
logger.error("Error log Message")
|
Now tail the new file and run the command.
> tail /var/log/app.log
|
Run it
> ./log_test.py
|
There we go we are getting logs now.
Logrotate
Syslog-ng does not do anything about log rotation. For that job I am going to set up logrotate.
Create a logrotate
file for this log
> sudo vi
/etc/logrotate.d/myapp
|
And place the following in it.
#Logrotate for app
/var/log/app.log { copytruncate dateext dateformat _%Y-%m-%d extension .log daily missingok rotate 15 } |
Now check to see if its valid
> sudo logrotate -d
/etc/logrotate.conf 2> >(grep
app.log -A 4)
|
If it's valid you should see details of your rotation.
Here you can see it rotates after 1 day and keeps 15
rotations.
There is no need to restart any service as during the next
run it will pick up the changes it finds.
You can force the run of it to confirm its output with this
command.
> sudo logrotate
--force /etc/logrotate.d/myapp
|
And doing a quick check
> ls -alh /var/log/app*
|
Removing syslog extra info
How do I remove the extra information, timestamp and syslog,
that syslog prepends the log messages?
Mar
23 17:38:04 syslog 2017-03-23 17:38:04,107, Error log Message
|
How do I remove that part?
In other words how do I send the record the raw log only?
The answer is templates!
Open up the config file
> sudo vi /etc/syslog-ng/syslog-ng.conf
|
And edit this line
destination
d_app { file("/var/log/app.log" template("${MSG}\n") ); } ;
|
Adding the inline template section
Now you need to reload syslog-ng
> sudo systemctl reload syslog-ng
|
Now test it out!
There we go now we are getting just the raw message.
For more details see https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-macros.html
[10]
One last tweak
I noticed I am getting debug info… which in this case I do
not want. I need one more tweak to my
file.
Open up the config file
> sudo vi /etc/syslog-ng/syslog-ng.conf
|
And edit this line
filter
f_app { facility(local0) and
not level(debug); };
|
Now one last try.
Now you need to reload syslog-ng
> sudo systemctl reload syslog-ng
|
There we are I filtered out the debug level. Wahoo!
References
[1] Syslog-ng docs (system)
https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-source-system.html
Accessed 3/2017
Accessed 3/2017
[2] Syslog-ng docs (internal)
https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-sources-internal.html
Accessed 3/2017
Accessed 3/2017
[3] Syslog-ng docs (filter)
https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-filters.html
Accessed 3/2017
Accessed 3/2017
[4] Syslog-ng docs (filter
functions)
https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-filters.html
Accessed 3/2017
Accessed 3/2017
[5] Syslog-ng docs (facility)
https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-filters.html
Accessed 3/2017
Accessed 3/2017
[6] Syslog-ng docs (level)
https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-filters.html#filter-priority
Accessed 3/2017
Accessed 3/2017
[7] Wikipedia syslog-ng (level)
[8] Syslog-ng docs (destination
file)
https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-destination-file.html
Accessed 3/2017
Accessed 3/2017
[9] Python logging docs
[10] Templates and Loggin
https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-macros.html
Accessed 3/2017
Accessed 3/2017
No comments:
Post a Comment