Syslog-ng listen on port

Posted on Saturday, March 25, 2017





I recently had to get up to speed on setting up and using syslog-ng. which I wrote up on how to install it and get it logging a simple python app.  See http://www.whiteboardcoder.com/2017/03/syslog-ng-install-and-configure.html [1]


In this post I am going to go over how to get syslog-ng to listen on a port for incoming logs.




Getting it to listen on a port




Edit syslog-ng.conf


  > sudo vi /etc/syslog-ng/syslog-ng.conf


And create the most simple network source just network()



source s_network {
    network( transport(tcp) port(601));
};


This source will listen on the default port 601 and on TCP. It will also listen from any IP address.


Now let me create the log section




log { source(s_network); filter(f_network); destination(d_networl); };


Next create a filter section


filter f_network { facility(local0) and not level(debug); };


Next create a destination section


destination d_network { file("/var/log/app_network.log" ); };



Now reload  the syslog-ng services


  > sudo systemctl reload syslog-ng




To double check that the conf file was correctly formatted.


  > tail -f /var/log/syslog


And you should see something like this.



On the top there is an error.


Now do a quick check and make sure syslog-ng is listening on that port.


  > sudo netstat -peant | grep syslog-ng





Yep all looks good.


Looking at the docs I see this note.

A TCP source listening for messages using the IETF-syslog message format. Note that for transferring IETF-syslog messages, generally you are recommended to use the syslog() driver on both the client and the server, as it uses both the IETF-syslog message format and the protocol. For details, see Section 6.11, Collecting messages using the IETF syslog protocol (syslog() driver).

So it is listening but expecting a specific protocol to come in.  What is the simplest way to test this?

A simple way to send a message is to use the logger command https://linux.die.net/man/1/logger [3]




Sending logs to the port


Tail the /var/log/syslog


  > tail -f /var/log/syslog



Now Send a log message to the port


  > logger -p local0.info --server 127.0.0.1 --tcp --port 601 "My Test Message"



You should see 3 messages in syslog






Mar 25 07:38:56 syslog syslog-ng[2359]: Syslog connection accepted; fd='13', client='AF_INET(127.0.0.1:39584)', local='AF_INET(0.0.0.0:601)'
Mar 25 07:38:56 127.0.0.1 1 2017-03-25T07:38:56.889550-06:00 syslog patman - - [timeQuality tzKnown="1" isSynced="1" syncAccuracy="313500"] My Test Message
Mar 25 07:38:56 syslog syslog-ng[2359]: Syslog connection closed; fd='13', client='AF_INET(127.0.0.1:39584)', local='AF_INET(0.0.0.0:601)'





The first messages just says that the connection was accepted.  The last that the connection is was closed.  The one in the middle is the actual message.


Mar 25 07:38:56 127.0.0.1 1 2017-03-25T07:38:56.889550-06:00 syslog patman - - [timeQuality tzKnown="1" isSynced="1" syncAccuracy="313500"] My Test Message


There is my message.

Now just tail the /var/log/app_network.app


  > tail -f /var/log/app_network.log




For me that log is a little too verbose.

That can be changed by a few tweaks in the sylog-ng.con file






Simplify the message



I am going to do a simple inline template

Here are a few variables

Variable
Definition
${DATE}
Date-related macros, FULLDATE, C_FULLDATE, R_FULLDATE, S_FULLDATE
${HOST}

${HOUR12}
AMPM
${HOUR}
Date-related macros
${ISODATE}

${LEVEL}
LEVEL_NUM, PRIORITY or LEVEL
${MSGHDR}
Templates and macros, MSG or MESSAGE
${MSGONLY}
MSG or MESSAGE
${MSG}
Message representation in syslog-ng OSE, MSG or MESSAGE, substr
${PID}
Comparing macro values in filters


Here are some variables ${ISODATE} ${HOST} ${MSGHDR}${MSG}\n. (The ${MSGHDR}${MSG}

Edit syslog-ng.conf


  > sudo vi /etc/syslog-ng/syslog-ng.conf


I am going to use MSG and LEVEL

Update this line


destination d_network {
   file("/var/log/app_network.log" template("${LEVEL}::${MSG}\n"));
};






Now reload  the syslog-ng services


  > sudo systemctl reload syslog-ng



Now Send a log message to the port


  > logger -p local0.info --server 127.0.0.1 --tcp --port 601 "My Test Message"




Not exactly what I expected I do see the info:: (level) there.   But the message is rather robust.  Is logger adding this additional information to the message?

Poking around I finally figured it out.   Here is how to get logger to not send that extra information.



  > logger -p local0.info --server 127.0.0.1 --tcp --port 601\
      --rfc3164 "My Test Message"







Wahoo that worked !

Now since this is listening on a port other servers can send messages to this port.

I can run this command from cygwin on another machine and it works (The logger is an older version and by default uses the rfc3164 standard)



  > logger -p local0.info --server 192.168.0.52 --tcp --port 601 \
     "From another server"




Works!



Python code sending to a port



Now that I got that working I wanted to create a simple python logger example to send messages to that port.

Looks like for python the only option to send logs on a port is via UDP, so I need to tweak the syslog-ng.conf file to also listen to udp



  > sudo vi /etc/syslog-ng/syslog-ng.conf


And create the most simple network source just network()


source s_network {
    network( transport(tcp) port(601));
    network( transport(udp) port(602));
};




Now do a quick check and make sure syslog-ng is listening on that port.


  > sudo netstat -peanut | grep syslog-ng




There we go


Now for the code


  > vi port_log_test.py


 Here is my code


#!/usr/bin/python3
import sys
import syslog
import logging
import logging.handlers

#Set up Logger
logger = logging.getLogger("10x13")
logger.setLevel(logging.DEBUG)
handler = logging.handlers.SysLogHandler(address=('localhost',602),facility=16)
log_format = logging.Formatter('%(asctime)s, %(message)s')
handler.setFormatter(log_format)
logger.addHandler(handler)

logger.info("Info Log Message")
logger.debug("Debug Log Message")
logger.error("Error log Message")




Make it executable and run it


  > chmod u+x port_log_test.py
  >./port_log_test.py


Now looking at the logs

 


Done deal!    The debug message was filtered out by the syslog f_network filter settings


filter f_network { facility(local0) and not level(debug); };



Wahoo it's all working !!



Oh and now since it is listening on UDP I can send a logger message over UDP


  > logger -p local0.info --server 127.0.0.1 --udp --port 602 \
      --rfc3164 "My UDP Message"





Nice!






References


[1]        syslog-ng install and configure.
[2]        network
[3]        Logger man page
[4]        Syslog-ng templates
[5]        Syslog-ng symbols





No comments:

Post a Comment