Notify Me of Important Events
A key aspect of Zenoss is to be notified when events important to you take place, so you do not need to watch the monitoring screen all day. In Zenoss, this is done using a system of triggers and notifications. Below, we will explain how this works and how to configure your own triggers and notifications.
An important concept in Zenoss is that triggers apply to events, and notifications define actions that are taken when trigger rules are satisfied.
To navigate to the Triggers definition page, go to the Events tab, and select Triggers from the sub-menu.
Triggers define criteria (called 'rules') that are compared to events as they are received by Zenoss, and will apply to all events that match the rules we define in the trigger. These rules cover nearly every aspect of monitoring and are extremely flexible, and are available via drop-down boxes. But a trigger by itself does not cause any action to take place. Actions are defined using Notifications, which we'll cover in the second half of this document. But first, let's learn about triggers.
To create a new trigger, navigate to the Triggers definition page and click the + icon to add a new trigger. You will be asked to provide a ID/name for the trigger. Once created, your new trigger will be displayed in the drop-down Triggers definition list. Double-click on the new trigger to begin defining rules for your trigger (or, alternatively, select the trigger and click the gear icon.)
You will notice that there is a drop-down menu that allows you to select either any' or all to apply to the rules that follow below. If all is selected, then all rules that you have defined below must apply to the event for your trigger to activate for the event. If any is selected, then one or more rule must apply to the event for the trigger to activate for the event.
To add additional rules, press the plus icon, or the branching icon at the right of each line. The plus icon will simply add another rule to your list, which will be combined with any existing rules in either an any (logical "or") or all (logical "and") fashion, depending on what you selected.
The branching icon () allows you to further qualify any existing rule. You can add additional conditions to your rule, and you can also select whether any or all of these sub-conditions must exist for this rule to be considered satisfied.
The most basic rule that a beginning Zenoss administrator will probably look at are Production State, and Severity. A good first basic trigger is to activate on any event that has a Critical Severity on a Production device. To set this rule, perform the following steps in the Edit Trigger dialog:
- Create a new trigger using the plus icon on the triggers definition page.
- Enter in an id/name for your trigger.
- Edit the new trigger by double-clicking on it or selecting it and clicking on the gear icon.
- Set the trigger to fire when all of the following rules are satisfied.
- Create this rule: Device Production State / equals / Production
- Click the + icon to add another rule.
- Create this rule: Severity / equals / Critical
- Click the Submit button.
Once your trigger is defined, there is still an additional step -- to define a notification that will occur when your trigger rules are satisfied.
Notifications are actions taken when the criteria for a Trigger is met, and that trigger is activated. To associate a Notification with a Trigger, after creating the Notification, open it, and select the Triggers from the drop-down box in the "Notification" tab.
Zenoss supports 4 types of notifications:
In Prepare Remote Device, we discussed SSH keys, and with a command plugin we can run remote scripts or attempt to restart services via ssh. This is also where the "count" criteria in Triggers comes in handy. If an attempt to restart a service fails several times, we can then escalate by adding a "count" criteria to a second notification and let someone know that the service is down and won't restart.
Commands are specified in the "Content" tab of the Notification.
The E-mail tab allows a formatted e-mail with event information to be sent. The body of the message can be modified in the "Content" tab, and the recipients specified in the "Subscribers" tab. A test event e-mail is reproduced below, the last four lines are links to your Zenoss instance.
Device: localhost Component: /Status/Ping Severity: 5 Time: 2013/03/24 17:51:58.000 Message: Purple! Event Detail Acknowledge Close Device Events
This isn't used much anymore, but for those of you who still own pagers, this plugin is available to interface with your pager.
SNMP Traps Notifications can be sent to an existing legacy NMS for additional handling, routing, and escalation.