The power and necessity of log management solutions
(republished on linkedin)
“Logs” is a general term in software development which describes the recording of status updates or other events of an application, for example from a server or operating system. Logs contain information at a central location for developers and administrators to understand the software’s behavior. They are used to solve issues. A common way to handle logs is to attach entries to the end of a file so that it can be viewed via text editing tools.
About the necessity of log management in distributed environments
Application landscapes are getting more complex every day. This is mainly due to the increasing distribution of software. Just imagine a mobile app that requests data from a server; the simplest setup already results in two different log files. Therefore, a developer needs to check at least two different locations for maintenance. If you imagine IoT and Industry 4.0 scenarios where many different systems and parties are involved the usefulness of log management systems gets clear. It helps to centralize and relate log entries from different applications.
To simplify daily work the concept of log management was developed. It makes use of a central system where all recorded logs are stored. With the help of a query language, log transformation features, dashboards and alerts developers are able to handle logs more efficiently. For example: if you experience some strange network behavior at 08:43 you are able to define a search query that displays recorded logs between 08:40 and 08:50. Additionally, you can filter the logs so that only network issues are displayed.
With the right logging strategy applied to your applications operations can be simplified drastically. Relations among different applications can be made transparent, impacts of a fault on the application landscape can be made visible and processes or workflows are traceable.
The traceability is one of my favorites. If you have a specific identifier, for example an order number of your online shop, you are able to track and trace the order process. Just log the order number as part of each log entry. Changes made in the shopping system can be related to changes in the billing system. You just need to search for the order number and all log entries related to it are displayed.
Added value for non-technicians
Typically, logs do not only contain technical data. Developers also record status changes like a successful order. Therefore, log management and its additional features like dashboards might be interesting for departments, too. This means, that log management provides mechanisms to make business processes more transparent and to provide an initial solution for real-time dashboards. It might be a great tool to gather first insights and learnings. In IoT and Industry 4.0 scenarios you have many different sensors, actuators and systems which generate data. Log management software can be a great tool to gather these data and to provide a frontend for users to browse and analyze data easily before specific frontends are implemented. Therefore, log management helps to understand data and situations within your processes before your applications are changed or new software is developed.
Let us have a look at a simple demonstration we developed at Schacht One.
Example: our Graylog at Schacht One
We used a raspberry pi (a single board computer) and attached a temperature sensor to record temperature and humidity every 30s. Both values are logged and send to our industrial server which runs Graylog as log management software.
Graylog which is the leading log management software stores and prepares the data in a way that users can search for log entries of the last three months. Of course, this interval can be changed. In the image below a query was executed that searches for the term “temperature” in log messages which are not older than 5 minutes.
Based on the temperature and humidity data from the raspberry pi we configured two widgets for Graylog’s dashboard feature which visualizes data of the last 24 hours in two diagrams. The image shows the humidity and temperature values around the 26/06/2019 of June.
Alerts
As we have absolute values of the temperature and humidity, we are able to define simple alerts. We just defined a threshold for the temperature so that an alert is fired when the temperature is higher than 30°C.
A short summary
Besides the traditional use of logs to monitor applications’ states and behavior logs also provide business value. Log management can help to make it available to non-technicians. Moreover, with the help of dashboards and alerts data can be visualized and people can be made aware of interesting situations. Additionally, log management is a great tool to get a deeper insight of your systems and processes. It helps to understand your data before application development is started.
Appendix: A selection of log management software
There is a huge variety of log management software on the market. My favorite ones are:
Graylog The leading log management software on the market. It is easy to install and simple to use.
ELK ELK stands for Elastic search (basically a search engine), Kibana (dashboard technology) and Logstash (data processing pipeline for logs). Great tools which can be combined to a log management environment. Using this technology stack, you are able to adapt the software to your specific needs. However, compared to Graylog it is more difficult to install and maintain.
Splunk Splunk is a great tool and can be run in cloud and on premise. Besides classical log management further features for machine learning and predictive maintenance are provided. It is an extremely interesting piece of software for IoT scenarios. However, it is not for free, but they have a free version for testing. It is called Splunk Free.
SignalFx Last but not least, there is SignalFx which has great dashboards and is especially designed for integrating with the big cloud providers. They use stream analytics as core feature to create insights.