Lightmeter 1.5: Historical Insights and Logstash support

Sam Tuke Mar 30 2021 Share

New installations of Lightmeter now analyse historical logs and generate Insights, providing a digest of past problems. Monitoring remote mailservers with Lightmeter is also now supported: from Logstash via sockets, and from any source of logfiles via Rsync synchronisation.

23 Merge requests and 24 commits went into this release. Check the Roadmap to see what features are coming next.

Note that Lightmeter is still in early development and intended for small scale deployments. Known issues are documented in README.

Analyse past delivery using historical Insights

On first-run Lightmeter now analyses past logs and provides related Insights, displayed as a convenient summary. This reveals problems which may have gone undetected at the time they occurred, and helps identify for how long a recurrent problem has existed.

A new “Archived” category of insight now exists to separate these historical Insights from others to keep the contents of the Observatory (default Lightmeter view) always relevant. You can filter Insights by category or status, or go straight to view all historical Insights by following the link in the digest.

For larger log sets, generating historical Insights can take some time, so progress indicators have been added to the registration page and Observatory to keep you informed.

In future Lightmeter will be able to learn what configuration options to use based on this assessment of historical events, such as what is a “normal” period of mail inactivity for the server in question, and therefore after what time interval Lightmeter should warn you of an abnormally long period of inactivity because the server is likely unreachable and effectively down.

Get remote logs from Logstash using sockets

Lightmeter now supports reading log data via both TCP/IP and Unix sockets. This means that any application or service which can send mail logs over sockets can be a source for Lightmeter, either on the same server or remotely, including from Logstash (part of Elastic’s ELK stack).

If your mail logs are already part of a data processing pipeline then they can be forwarded to Lightmeter with little configuration.

Note that this feature is currently experimental. See the README for details. Thanks to Alexander Landmesser for requesting and testing this feature!

Read remote logs synchronised using Rsync

Easy monitoring of remote servers is now available — simply synchronise your mail log directory from mailserver to Lightmeter server via Rsync, and Lightmeter will do the rest.

Adding this features was surprisingly complex due to how Rsync writes files and Lightmeter detects changes. Fortunately using it is simple; check the README for details. Thanks to Michiel Dethmers for requesting and Dominik Holland for testing this feature!

What’s else is new?

1. Several improvements to log parsing including support for milter-reject and smtpd reject
2. Fix reporting of some successful deliveries being ignored
3. Remove Mailspike reputation list from RBL scanner — thanks to Luc Didry!
4. Do not trigger error during notification generation if settings are missing
5. Automatically run npm audit fix for fixing security issue within front-end dependencies

The full list of merge requests and commits is on GitLab.

Get involved

Join us in making email peaceful and transparent — request or add what you’re missing to make your own mail server steadfast and efficient:

Go to the GitLab repo, fork it, build it, and submit your first pull request (there are many open issues to choose from if you’d like inspiration, including some quick wins). And if you get stuck at any point, drop by the forum, the Fediverse, Telegram, Twitter, or email ?

Download

• Docker Hub: https://hub.docker.com/r/lightmeter/controlcenter/tags
• Binaries & source code: https://gitlab.com/lightmeter/controlcenter/-/releases

Missing your favourite packaging system? Request it in the comments.

Supported by

This project has received funding from the European Union’s Horizon 2020 research and innovation programme within the framework of the NGI-POINTER Project funded under grant agreement No 871528.