Monitoring systems are, without a doubt, a useful and critical tool. Everyone recognizes their necessity. However, for most organizations, implementing these systems is fraught with difficulties. It often happens that engineers spend most of their time maintaining the monitoring system itself rather than having it help them solve problems.
Let me share some personal experience that will likely help many people better understand the problem.
Monitoring "As Usual"
So, management concludes that a monitoring system is critically important and assigns the task of implementing it. An engineer, who is already busy with their core duties—like configuring networks—is told to "deploy a monitoring system." Often, no one knows how to do this because it’s their first experience in this area. Let's say the company has a hundred remote branches, and they need to monitor the availability of communication links with each of them. In this situation, they start looking for free solutions, studying available options, and picking one.
Usually, they choose the simplest solution based on checking endpoint availability via ping. After spending significant time installing and configuring it, they expect immediate results. However, the system starts throwing false positives. For example, a remote branch has a primary and a backup link, and the system monitors both. The engineer gets an alert about link issues, checks the availability, and finds that everything is working fine. The monitoring system cried wolf. Despite cases where there are actual problems, most alerts turn out to be false. As a result, instead of helping, the monitoring system simply scatters your focus, leading people to eventually neglect it.