Newsletter:1/Hacking Event Notifications
Hacking Event Notifications
Meet Foo. Foo is a rickety old CentOS 5 box running an Apache httpd server. (Ed: what about nginx?) The Apache httpd server tends to crash, a lot, but given that Foo isn’t mission critical, Foo isn’t going to get time dedicated to it for several weeks to determine why Apache httpd keeps crashing.
Now meet Dick, the on-call engineer who is starting to look like a zombie because of sleep deprivation. That can’t be good. Let’s try to get Dick some sleep, and prevent a zombie uprising. Maybe Zenoss can help!
Zenoss 4.x’s Event trigger/notification system supports command notifications, so with five minutes of effort, we can get Dick more sleep! Let's see how.
First, set up a trigger that looks like this for foo, with a Count of less than 5:
Next, make sure you exclude httpd on Foo from your main critical notification. Since Zenoss will execute every matching trigger, we don't want Foo's httpd crashing to hit the catch-all, because Dick will lose sleep!
Add another trigger that looks like this:
This trigger will fire if, after attempting to restart Apache four times, Apache still won't restart. At this point we'll have to to notify poor Dick that it's still a problem.
Next copy the Zenoss ssh key to root@Foo, and set up a remote restart of Apache with the following command notification:
To wrap this up, if Zenoss generates five events that the Web server is down, we tell someone by adding the second trigger we created to our main e-mail notification:
Zenoss will now attempt to restart the web server 4 times before notifying someone of the outage. The event will still show up in the event console so this problem can be addressed normally during working hours, but it should save Dick some sleep. Sleep Dick, sleep.