Articles setup, updated 2026-11-16

Syslog sender: collecting a site's syslog

The Oversight syslog sender is a small program that runs on one Linux machine at a site and collects the syslog from every device on its LAN: routers, switches, firewalls, UPS cards, other servers. It keeps everything it receives in a file per day on that machine, picks out the few lines that matter using rules you write, and sends just those to Oversight every five minutes. They appear on the Logs page, under the channel syslog.

That split is the point of it. A normal site sends hundreds of thousands of syslog lines a day, nearly all of it firewall logging, logins by monitoring systems and routine chatter. None of that crosses the internet. On a good day nothing at all is shipped; on a bad one, the loop warning, the switch that rebooted itself or the port flapping between 10M and 1G is in Oversight within five minutes, with the full day's record still on the site if you need to look further.

It needs no rsyslog, syslog-ng or logrotate: it has its own listener, writes and rotates its own files, and only ever connects out to Oversight, over HTTPS. For a host's own log files, such as fail2ban's or the login log, or its systemd journal (journald) without rsyslog, see Log sender; the rules are the same.

How it is installed

It goes on one Linux machine at the site that is always on and that the devices can reach: a server, a VM or a small box. It is installed with one command, as root, which checks the build against its published checksum and starts it as a service under an unprivileged user that may listen on the syslog port and do nothing else. It never needs root once installed, and the same command upgrades it later, leaving its settings alone.

The install prints a token, which GEN registers against your account. Until it is registered, Oversight refuses everything the sender uploads. A machine that already runs a log sender keeps the token it has.

Keep the machine's syslog port to the LAN. Syslog has no authentication, and the sender trusts the address a line came from.

What sends to it

Anything that can send syslog: MikroTik routers and switches, Linux servers through their own rsyslog, which passes on what journald holds as well, and the switches, firewalls, NAS units and UPS cards that have a syslog server setting somewhere in their web interface. There is no need to send a router's own firewall logging, which is the bulk of any router's syslog and is better left on the router.

Windows has no syslog of its own. The Oversight Windows agent sends the event log on, with each event's log, provider, ID and level as fields in the line, so a rule can pick out failed logons on a domain controller or services that crashed by name.

What it keeps on the site

Everything received goes into a file per day by UTC date, one line per message:

2026-10-14T23:56:52.914Z 172.16.128.254 udp interface,warning ether8 - CRS317 Switch excessive broadcasts/multicasts, probably a loop

That is when it arrived, which device sent it, how, and the message exactly as received. After midnight the day before is compressed, and by default files older than 90 days are deleted; some customers keep every file for ever and copy them off to their own storage. This is the place to go when something in Oversight needs its context.

The rules

The rules are the sender's filter: they decide which lines leave the site. Each line is tested against the exclude rules first, and dropped if any matches. Then the include rules, in order: the first that matches sends the line, and its name goes with it to Oversight. A line no include rule matches is not sent. A rule can name the devices it applies to, by address or range, the message's severity and facility, and words to look for, as a regular expression. Aim for a normal day to send nothing: the routine is better watched by sensors.

MikroTik devices send no severity at all, so a severity rule never matches them; their rules match the words at the front of the line instead, such as interface,warning.

At a real site, a MikroTik router and switches with a few Linux servers, checked against a full day of its traffic, seven include rules and three excludes send a loop warning, link changes and real VPN failures, and nothing else. The excludes take out what would otherwise match: the router's firewall logging, internet scanners knocking on its VPN port, and password guessing.

Rules can be tried against the live traffic before they are relied on. A dry run shows each line a rule would send the moment it matches, with a count for each rule, and sends nothing.

In Oversight

The Logs page shows each channel's lines, newest first, for today unless a longer range is chosen, filtered by device, rule, severity and words in the message. It shows at most the 2,000 newest that match; to see older ones, narrow the filter. Export CSV gives every matching line. Your users need the Logs permission, under Setup, Users.

A device that matches a device in your tree by address is shown by that device's name. Below the lines is every device the sender has heard from, with how many lines it sent today and how many of them were shipped, with the percentage. Some logs are worth sending whole, so this is a figure to check the rules against, not a warning.

The page also says when the rules need attention:

  • Lines dropped at the limit, in orange: more than 8,192 lines matched in five minutes, and the rest were not sent. A rule is far too broad.
  • A config fault or disk full, in red: the sender is sending nothing until it is put right.
  • Gaps, among the lines: a stretch when nothing was received, so a silence is never mistaken for a quiet spell.

GEN is told if a sender stops uploading for three intervals.

Installing it

Pick a Linux machine at the site that is always on and that the devices can reach: a server, a VM or a small box. As root:

The command or address for this step is for signed-in customers. Sign in to see it.

The script picks the build for the machine (x86_64, arm64 or 32-bit ARM), checks it against its published checksum, and installs /opt/oversight/syslog, an oversight system user, a starter config at /etc/oversight/syslog.json and the oversight-syslog service, which it starts. It ends by printing a token:

token: 9f2c6e0a4b7d1385c2e9f04a6b8d3e7f1a5c9b2d6e0f4a8c3b7d1e5f9a2c6b0d
register this under Probes > Senders before it can upload

Send that token to GEN. We register it against your account, and until we do Oversight refuses everything the sender uploads. It is kept in the config as well, so it can be read back off the machine later. A machine that already runs a log sender keeps the token it has.

The service runs as the oversight user and is allowed to listen on port 514, nothing else; it never needs root once installed. Running the same line again is how it is upgraded, and leaves the config alone.

The machine's firewall needs a rule for syslog from the LAN. Nothing in the install opens it. On AlmaLinux or Rocky, for a LAN of 172.16.128.0/24:

firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address=172.16.128.0/24 port port=514 protocol=udp accept'
firewall-cmd --reload

On Debian or Ubuntu with ufw: ufw allow from 172.16.128.0/24 to any port 514 proto udp. Keep it to the LAN: syslog has no authentication, and the sender trusts the address a line came from.

If something else on the machine already listens on 514, usually rsyslog with its UDP input turned on, the sender cannot start. ss -ulnp | grep ':514 ' shows what has it.

Pointing the devices at it

Every device sends its syslog to the machine's address on UDP 514. For a MikroTik, with the machine at 172.16.128.55:

/system logging action set remote remote=172.16.128.55 remote-port=514
/system logging add topics=critical action=remote
/system logging add topics=error action=remote
/system logging add topics=warning action=remote
/system logging add topics=interface action=remote
/system logging add topics=system action=remote

There is no need to send the firewall's own logging (firewall); it is the bulk of any router's syslog and is better left on the router. For a Linux server running rsyslog, one line in /etc/rsyslog.d/oversight.conf, then systemctl restart rsyslog:

*.* @172.16.128.55:514

Switches, firewalls, NAS units and UPS cards each have a syslog server setting somewhere in their web interface: give it the machine's address and port 514.

Windows servers

Windows has no syslog of its own. The Oversight Windows agent sends the event log on, from version 0.1.5. On each server, as Administrator:

"C:\Program Files\Oversight\agent.exe" --syslog 172.16.128.55:514

That sends the System and Application logs at warning and worse; which logs, and how much of each, is set on the server, under Forwarding the event log in that article. The event's channel, provider, ID and level travel as fields in the line, so a rule can name them exactly. Failed logons on a domain controller, with Security set to failure, and services that crashed:

{"name": "logon-failures", "match": "eventID=\"4625\""},
{"name": "service-crashed", "match": "channel=\"System\".*eventID=\"(7031|7034)\""}

What it keeps on the site: where

Everything received goes into /var/log/oversight-syslog/messages_YYYYMMDD.log, one file per day by UTC date, one line per message:

2026-10-14T23:56:52.914Z 172.16.128.254 udp interface,warning ether8 - CRS317 Switch excessive broadcasts/multicasts, probably a loop

That is when it arrived, which device sent it, how, and the message exactly as received. After midnight the day before is compressed, and files older than 90 days are deleted. Both are settings, below; some customers keep every file for ever and copy them off to their own storage. This is the place to go when something in Oversight needs its context: zgrep works on the compressed days.

The config

Everything the sender is told is in /etc/oversight/syslog.json. Only root and the oversight user can read it, because it holds the token:

{
  "token": "9f2c6e0a...",
  "channel": "syslog",
  "listen_udp": "0.0.0.0:514",
  "logdir": "/var/log/oversight-syslog",
  "keep_days": 90,
  "compress": true,
  "names": {
    "172.16.128.254": "router"
  },
  "exclude": [ ... ],
  "include": [ ... ]
}
KeyWhat it does
tokenThis machine's token, as the install printed it.
channelWhat the lines are called in Oversight. syslog unless you have reason to change it.
listen_udp, listen_tcpWhere it listens. UDP on 514 by default; TCP is off unless given an address, such as 0.0.0.0:514, for the rare device that only sends over TCP.
logdirWhere the day files go. If you change it, the oversight user must own the new directory: run the install line again and it is created for you.
keep_daysDay files older than this are deleted. 0 keeps them for ever.
compressCompress each day after midnight. On unless set to false.
intervalSeconds between uploads: 300 or more.
namesNames for devices that do not send a useful hostname, by address. Oversight also names a device itself where it has one in the tree with that address, which takes precedence.
exclude, includeThe rules: what is never sent, and what is. Below.

A misspelt key is refused rather than ignored. The service checks the file every five minutes and applies a change by itself; systemctl reload oversight-syslog applies it at once. A file with a mistake in it is not applied: the sender carries on with the last good one, and the Logs page shows the fault in red.

Writing the rules

A line is tested against the exclude rules first, and dropped if any matches. Then the include rules, in order: the first that matches sends the line, and its name goes with it to Oversight. A line no include rule matches is not sent. Aim for a normal day to send nothing: the routine is better watched by sensors.

FieldWhat it does
nameRequired, and different for every rule. Shown beside each line in Oversight.
sourceDevices it applies to: addresses, ranges such as 172.16.128.0/24, or the last part of an address as a span, 172.16.128.10-16.
severityThe message's own severity, this or worse: emerg, alert, crit, err, warning, notice, info, debug.
facilityA list, such as ["kern"], for messages from that part of a system.
matchText to look for, as a regular expression. (?i) at the front ignores case; \b marks the edge of a word, so that oom does not find bloom.

Every field a rule sets must match. Three things trip people up:

  • Backslashes are doubled, because the file is JSON: \blink down\b is written "\\blink down\\b".
  • No look-ahead or look-behind ((?!...), (?<=...)) and no back-references (\1). The sender refuses them and says which rule. Where you would reach for one, write an exclude rule instead.
  • MikroTik devices send no severity at all, so a severity rule never matches them. Match the words at the front of the line instead: interface,warning, system,error,critical.

This is the config at a real site, a MikroTik router and switches with a few Linux servers, checked against a full day of its traffic. It sends a loop warning, link changes and real VPN failures, and nothing else:

"exclude": [
  {"name": "firewall-logging", "match": "\\bfirewall,info\\b"},
  {"name": "ipsec-scanners", "match": "\\bipsec,error\\b.*(negotiation failed\\.$|no suitable proposal|valid proposal|pre-process)"},
  {"name": "login-failures", "match": "(?i)\\blogin failure for user\\b"}
],
"include": [
  {"name": "critical", "severity": "crit"},
  {"name": "mikrotik-errors", "match": "^([a-z0-9-]+,)*(critical|error)\\b"},
  {"name": "kernel-errors", "facility": ["kern"], "severity": "err"},
  {"name": "panic-reboot", "match": "(?i)(kernel panic|\\bpanic\\b|kernel failure|watchdog (reset|reboot|timeout|timer|expired)|rebooted without proper shutdown|\\bcold ?start\\b|system restarted)"},
  {"name": "link", "match": "(?i)(\\blink (up|down)\\b|changed state to (up|down)|\\bis now (on|off)-line\\b|duplex mismatch)"},
  {"name": "flood", "match": "(?i)(syn flood|flood(ing)? (attack|detected)|\\bdos attack\\b|table full|neighbou?r table overflow|broadcast storm|storm control|excessive (broadcasts|multicasts)|probably a loop|loop (detected|protect))"},
  {"name": "kernel-faults", "match": "(?i)(out of memory|oom-kill|\\bi/o error\\b|ext4-fs error|xfs .*(corruption|error)|hardware error|machine check|soft lockup|blocked for more than \\d+ seconds)"}
]

The excludes take out what would otherwise match: the router's firewall logging, internet scanners knocking on its VPN port, and password guessing. If you only want link changes from the core switches and not every desk port going to sleep at night, give the link rule a source.

Trying rules before relying on them

Check the file for mistakes:

/opt/oversight/syslog --validate

Then watch what the rules pick out of the real traffic, as it arrives. The service has to stop for this, since only one program can listen on 514; nothing is sent to Oversight and nothing is kept while it runs:

systemctl stop oversight-syslog
/opt/oversight/syslog --dry-run | jq -c 'select(.rule) | [.src, .rule, .msg]'
systemctl start oversight-syslog

Each line that would be sent appears the moment it matches. Edit the rules in another window and pkill -HUP -f 'syslog --dry-run' to try the change at once. A minute's count goes to the screen as well, so you can see how much each rule would send.

The day files are the other test bed: grep a day for what you expect a rule to catch, and look at what it would have sent.

If it does not work

The journal says what it is doing: journalctl -u oversight-syslog. One line every five minutes is the healthy state:

uploaded syslog, 09:15:00 to 09:20:00: 2 shipped of 1412 received
The journal saysWhat to do
Oversight does not know this tokenThe token is not registered yet, or has been disabled. Send it to GEN.
udp 0.0.0.0:514 is already in useSomething else has the port, usually rsyslog. Turn its UDP input off.
cannot create /path or day file: ... permission deniedThe logdir is somewhere the oversight user cannot write. Run the install line again, which creates and hands it over.
config: ...A mistake in the config, named; the last good one carries on meanwhile. --validate lists them all.
upload: ... with batches waitingOversight cannot be reached from here. Batches are kept for an hour and sent when it can; anything older is counted and shown as a gap.

Uploading, but a device's lines never appear: check the device is sending at all, from the machine: tcpdump -ni any udp port 514 and host 172.16.128.16. If they arrive but are not sent, the rules do not match them, and the day file shows exactly what the device sends.

Received but the device is missing from the Logs page's list: the firewall rule does not cover its address.