What Is Syslog and How Does It Work?

Syslog is a good thing. It’s a standard network-based logging protocol that works on an extremely wide variety of different types of devices and applications, allowing them to send free text-formatted log messages to a central server.

Essentially every device on your network—whether it’s a storage box or a server, a switch or a firewall—likely has a syslog agent you can use to send messages to a common central location.

When is syslog used?

People who are familiar with SNMP might be a little confused. Why do we need syslog if we already have SNMP traps? Surely they both provide similar functionality.

They’re both used to send alerts and messages to a central server without needing to be polled. As soon as an event occurs, the message can be sent without waiting for the server to poll for status.

The biggest difference is SNMP traps have a special predefined format that’s contained in a MIB file. If an interface on a switch goes down, the MIB file defines an “ifDown” trap message that includes useful information like the specific interface. That’s great if you know ahead of time exactly what information the message will contain. But sometimes you don’t.

There are two really important cases where it’s impractical to have a MIB file with predefined message types. The first is the one for which syslog was originally invented: application alerting.

Suppose you’ve written an in-house application that performs some business function. Such applications will generally have a huge range of log messages, and each new release of the software could have a bunch of new messages. It’s simply not practical to create a new MIB file for each of these messages and load it into your central logging server for every software release. The MIB file will always be a release or two behind, and the central server will be unable to parse any new message types.

The second important case is log messages for security devices like IDS (intrusion detection systems). These systems are constantly being updated with new signatures. There could be so much variation in the information detected by these signatures that it’s almost impossible to standardize the event messages. And many commercial IDS systems subscribe to services that download new signatures daily.

So SNMP traps are used for well-defined events like interface resets, particularly on network devices, while syslog events are best for events that are more general in nature and harder to predict.

The free-form nature of syslog messages is syslog’s greatest strength, but it’s also syslog’s greatest problem. It’s very difficult to parse through a log including events from dozens of different systems from different vendors and make sense of them simultaneously. Which messages represent what functions? And which ones represent critical events versus mere informational messages?

To deal with these questions, the syslog protocol (which is defined in RFC 5424) provides these free-form messages with special fields called “facility” and “severity.”

(Note that RFC 5424 is the standard for syslog, but not all syslog implementations are RFC 5424-compliant. The syslog protocol has been around for a very long time and there are some pre-standard implementations still in use.)

What are severity and facility codes?

The severity value is easily understood. It includes a single number between 0 and 7 that indicates how important the message is.

Numerical Code	Severity	Meaning
0	Emergency	System is unusable
1	Alert	Action must be taken immediately
2	Critical	Critical conditions
3	Error	Error conditions
4	Warning	Warning conditions
5	Notice	Normal but significant condition
6	Informational	Informational messages
7	Debug	Debug-level messages

In practice, you don’t normally see emergency level messages because if the system is that badly broken, it probably can’t send a message. And you probably don’t want to see debugging messages in your log because there will be too many of them, and they’re rarely important for operational purposes. So typical production systems will normally be set to a logging level of 5 or 6. The sending system might keep a local copy of the less severe messages, but it won’t send them to the central server.

The facility code requires a little more explanation. Early implementations of syslog server software generally just dumped the incoming messages into one or more log files. The server system used the facility code to sort related messages into the same file. Modern syslog implementations just dump all messages into a common database and use the facility code simply as one of many possible search keys.

Facility codes are also numerical values, which are listed in the following table. Notice many of them are highly specific to systems that are, in many cases, not commonly used anymore. These are the facility codes defined in RFC 5424, which are somewhat different from those used by BSD Unix. The differences are mostly an historical curiosity and aren’t actually all that important for most purposes.

Numerical Code	Facility Name	Usage
0	Kernel messages	Unix kernel
1	User-level messages	User application alerts
2	Mail system	Unix mail
3	System daemons	Unix system processes
4	Security/authorization messages	Unix authentication/authorization messages
5	Messages generated internally by syslogd	Syslog process itself
6	Line printer subsystem	Unix line printer
7	Network news subsystem	Unix “news” system
8	UUCP subsystem	Unix-to-Unix Copy Protocol
9	Clock daemon
10	Security/authorization messages
11	FTP daemon
12	NTP subsystem
13	Log audit
14	Log alert
15	Clock daemon
16 (local0)	Local use 0
17 (local1)	Local use 1
18 (local2)	Local use 2
19 (local3)	Local use 3
20 (local4)	Local use 4
21 (local5)	Local use 5
22 (local6)	Local use 6
23 (local7)	Local use 7

Notice a lot of these facility codes are specific to ancient legacy Unix systems. Nobody really uses UUCP anymore, for example. It was an asynchronous way of copying files around automatically between servers that could only exchange data by means of analog dial-up.

The facility and severity codes appear at the start of each syslog message, and are surrounded by angle brackets. For example, “<166>” would indicate a “local0” facility message of severity 6.

These days most network devices will use one of the “local” codes for their syslog messages. By default, Cisco ASA firewalls will use facility code 20 (local4), while most Cisco switches and routers will use code 23 (local7). These codes exist purely for the convenience of the syslog server to sort the incoming messages into useful categories. Most people won’t need to change them because, as noted earlier, in modern syslog implementations, the facility code is just one of many possible key values that are used to search the database of messages.

How is syslog transported?

There are two common ways to send syslog messages. Although RFC 5424 requires that all syslog implementations must support an encrypted TLS network transport over TCP, most syslog messages are still delivered using the older UDP method.

In the UDP version, messages are simply put into the data portion of a UDP packet and sent to the server over UDP port 514. Each message will generally fit into a single packet. UDP is stateless and sessionless, so there’s no acknowledgement. Packets are just sent into the network. This has the obvious problem that any kind of network problem could prevent delivery of the packet, in which case you might not know the network is down because it can’t tell you. It also means that sometimes important packets can be lost or damaged in transit.

The other important thing to remember about the UDP syslog transport is that it’s not encrypted. So the packets can be intercepted and read, and they can even be forged relatively easily. So it’s never recommended to send UDP syslog packets through the public internet unless they’re also tunneled in an encrypted VPN of some kind.

Fortunately, there’s also an encrypted session-based version of syslog transport that uses TLS for security. This uses TCP port 6514 and the same type of certificates for authentication as HTTPS.

Because TCP is session-based, the remote devices will open a TCP session with the server and will generally keep it active while delivering all queued messages. In the case of devices like firewalls, which send a huge number of syslog messages, this might mean a continuous connection for a long period of time.

This method has several key advantages. First, because the session is encrypted, it isn’t possible to read them in flight. Second, because every device has an unique certificate, it’s possible for the server to authenticate that the device really is who it says it is, which prevents forging of bogus messages. Third, because the session is TCP based, the delivery of each message is guaranteed, with none getting lost, and they can be retransmitted if they are lost or damaged in transit.

But it also has some disadvantages. If the central server is accepting messages from a large number of devices, it could suffer from resource problems due to the number of simultaneous TCP connections. It’s also more complicated to configure.

---

Now, we’ve covered the basics of what syslog is. In my next article, I’ll show you how to enable both UDP and TLS syslog versions on a Cisco device.