OP5 Monitor ["OP5 Monitor"]
["Technical Reference"]

# Manage notifications

## Overview

The procedures below describe how to perform some advanced notification configuration tasks. Only advanced users with a good knowledge of how OP5 Monitor notifications work should make the updates described.

### Background information

This page describes advanced configuration options of notifications. For background information on notifications, see the following:

- For a general description of how notifications work in OP5 Monitor, see Checks, alerts, and notifications in Key concepts of OP5 Monitor.
- For descriptions of the notification options you can set on hosts and services, see Host options in Manage templates and time periods and Service options in Manage templates and time periods.

### Notification process

OP5 Monitor determines whether it needs to send out notifications through the service check and host check logic. It sends out host and service notifications in the following instances:

- When a hard state change occurs.
- When a host or service remains in a hard non-OK state and the time specified by the configuration setting `notification_interval` in the host or service configuration has passed since the last notification was sent out for that specific host or service.
- If the notification filters permit it. For more information, see Notification filters.

Note: If a host is in a soft or hard `DOWN` or `UNREACHABLE` state, the service notifications for services belonging to that host will be blocked.

Each host and service definition has a `contact` and `contact_groups` option that specifies which contacts receive notifications for that particular host or service.

When OP5 Monitor sends out a host or service notification, it notifies each individual contact or member of any contact groups specified in the `contact_groups` option of the service definition. If a contact is a member of more than one contact group, OP5 Monitor removes duplicate contact notifications before sending notifications out.

The default behaviour is that if a host has contacts configured in the host configuration, the specified contacts receive notifications for the host and the services on the host. There are exceptions to this default behaviour:

- If a service on a host has different contacts set to the host, the contacts on the host receive all the notifications, except for the service that has contacts defined.
- If a service in a host group has contacts set, only those specific contacts receive the notification.

The following notification options inherit their values from the host they are associated with, if you do not specify a value on the service:

- `contacts` and `contact_groups`
- `check_period`
- `notification_interval`
- `notification_period`

### Notification filters

OP5 Monitor uses a number of filters to determine whether to send a notification or not. You can find more information about time periods and service, host, and contact settings in Manage templates and time periods.

Filter | Description
--- | ---
Program-wide | Whether notifications are enabled on a program-wide basis. Program-wide notification settings are managed in the Operations menu of Manage > Process information. For more details, see Program-wide management.
Service and host filters | Is the host or service in scheduled downtime? Is the host or service in a flapping state? Are the host or service notification options enabled for this type of notification? Are we in the right time period for notifications at the moment? Have we already sent a notification about this alert? Has the host or service remained in the same non-OK state as it was when the last notification went out?
Contact filters | Do the contact notifications options says that this type of notification is supposed to be sent? Are the contact notification options enabled for this type of notification?

### Naemon notification documentation

The notifications in OP5 Monitor follow an extensive rule set that is inherited from the core Naemon daemon. You can find more information about Naemon notifications in the Naemon notifications documentation.

## Notification scripts

Notification commands use scripts in the same way as normal check commands. The information below describes the content of the scripts, which you can use for guidance when updating the scripts.

All default notification scripts delivered with OP5 Monitor are located in `/opt/monitor/op5/notify`.

### Notification macros

Many of the arguments sent to the notification commands are macros. Macros are a type of variable containing, in most cases, a program-wide value. You can find more information about macros in the Naemon macro documentation.

One of the most important macros used with notifications is `$NOTIFICATIONTYPE$`. This macro tells you what type of notification is to be sent.

The `$NOTIFICATIONTYPE$` macro can have one of the following values:

Notification Type | Description
--- | ---
PROBLEM | The service or host has just entered (or is still in) a problem state.
RECOVERY | The service or host has recovered from a problem state.
ACKNOWLEDGEMENT | A service or host in a problem state has been acknowledged by a user.
FLAPPINGSTART | The host or service has entered a flapping state.
FLAPPINGSTOP | The host or service has left a flapping state.
FLAPPINGDISABLED | Host or service flapping detection has stopped and it has therefore left a flapping state.
DOWNTIMESTART | The host or service has entered scheduled downtime.
DOWNTIMESTOP | The host or service has left scheduled downtime.
DOWNTIMECANCELLED | The scheduled downtime for a host or service has been cancelled.

## Change notification email settings

### Update the host name link in the notification email

The notification email includes a link back to the OP5 Monitor server that sent the notification. You can configure the email settings to link to a different host.

This is useful if OP5 Monitor is configured for load-balanced monitoring, where notifications can be sent out from different sources, but one of the peers is the preferred system for configuration and viewing data. This is the recommended way to configure load-balanced environments.

You can update the URL linking back to OP5 Monitor by updating file `/etc/op5/notify.yml` with a different `hostname` value. For example:

    ...
    # The notification email includes a link back to the OP5 Monitor server that
    # sent the notification. You can configure the email settings to link to a
    # different host.
    # If not specified, the system hostname is used.
    hostname: master1.op5.com
    ...

In the above example, a notification sent from `master2.op5.com` in a load-balanced configuration appears as follows:

    op5 Monitor

    Service CUSTOM detected  2016-07-03  16:36:11.
    'Certificate Expiration Check'  on host  'master01'  has passed the CRITICAL threshold.
    https://master01.op5.com/monitor/index.php/status/service/master01

    Additional info;
    CRITICAL - File /opt/plugins/custom/certificate-expire is  21770079  seconds old

    Host:    master01
    Address:  172.27.0.12
    Alias:   OP5 Monitor Server
    Status:  UP
    Comment: /etc/op5/notify.yml configured on master2

The link in the notification email takes you to `master1` to view details of the problem.

For more information on `/etc/op5/notify.yml`, see Default configuration file.

### Update the sender email address in the notification email

By default, OP5 Monitor sends notifications from email address `op5monitor` without any domain. The mail transfer agent (MTA) adds the local domain name, which by default is `@localhost.localdomain`.

To change the email address that OP5 Monitor sends notifications from, you can use the `--from` argument in the notification command, or reconfigure your MTA and host name in OP5 Monitor to send the message from the correct domain.

To change the sender email address in the notification command from `op5monitor@localhost.localdomain` to `op5notification@mycompany.com`:

1. In the OP5 Monitor user interface, click Manage > Configure.
2. In the Checks and Notifications section, click Commands.
3. In the Command to edit field, search for `host-notify` or select the command from the drop-down list, then click Go.
4. In the command_line field, add `--from op5notification@mycompany.com`, as in the following example:

    $USER3$/notify/notify --from op5notification@mycompany.com -c "$CONTACTNAME$" -h "$HOSTNAME$" -f "$NOTIFICATIONTYPE$" -m "$CONTACTEMAIL$" -p "$CONTACTPAGER$" "HOSTALIAS=$HOSTALIAS$" HOSTADDRESS=$HOSTADDRESS$ " " HOSTSTATE=$HOSTSTATE$ " " HOSTSTATEID=$HOSTSTATEID$ " " HOSTSTATETYPE=$HOSTSTATETYPE$ " " HOSTATTEMPT=$HOSTATTEMPT$ " " HOSTLATENCY=$HOSTLATENCY$ " " HOSTEXECUTIONTIME=$HOSTEXECUTIONTIME$ " " HOSTDURATION=$HOSTDURATION$ " " HOSTDURATIONSEC=$HOSTDURATIONSEC$ " " HOSTDOWNTIME=$HOSTDOWNTIME$ " " HOSTPERCENTCHANGE=$HOSTPERCENTCHANGE$ " " HOSTGROUPNAME=$HOSTGROUPNAME$ " " HOSTGROUPALIAS=$HOSTGROUPALIAS$ " " LASTHOSTCHECK=$LASTHOSTCHECK$ " " LASTHOSTSTATECHANGE=$LASTHOSTSTATECHANGE$ " " LASTHOSTUP=$LASTHOSTUP$ " " LASTHOSTDOWN=$LASTHOSTDOWN$ " " LASTHOSTUNREACHABLE=$LASTHOSTUNREACHABLE$ " " HOSTOUTPUT=$HOSTOUTPUT$ " " HOSTPERFDATA=$HOSTPERFDATA$ " " HOSTACKAUTHOR=$HOSTACKAUTHOR$ " " HOSTACKCOMMENT=$HOSTACKCOMMENT$ " " NOTIFICATIONNUMBER=$NOTIFICATIONNUMBER$ " " CONTACTALIAS=$CONTACTALIAS$ " " DATETIME=$DATETIME$ " " SHORTDATETIME=$SHORTDATETIME$ "DATE=$DATE$" "TIME=$TIME$" "TIMET=$TIMET$" "HOSTACTIONURL=$HOSTACTIONURL$" "HOSTNOTESURL=$HOSTNOTESURL$" "ADMINPAGER=$ADMINPAGER$" "ADMINEMAIL=$ADMINEMAIL$" "NOTIFICATIONCOMMENT=$NOTIFICATIONCOMMENT$"

5. Click Submit.
6. Click Save at the top of the configuration page.

Note: To change this for service notifications, repeat the steps above for the command `service-notify`.

## Change notification date/time format

Beginning OP5 Monitor 8.3.2, you can change the date/time format for notifications.

By default, notification time stamps follow the RFC 3339 format. You can change the date/time settings for notifications in `/etc/op5/notify.yml`. For more information, see Default configuration file.

Where no date/time format is defined, notifications will assume the `YYYY-MM-DD HH:MM:SS` format.

### Options for date/time format

The following configuration settings are available for `/etc/op5/notify.yml`:

Option | Description
--- | ---
`timestamp_format` | Sets the time format following strftime(3). For more information, see strftime(3) — Linux manual page. The following presets are provided in the `/etc/op5/notify.yml`: RFC 3339 `%Y-%m-%d %H:%M:%S%z` Example output: `2021-05-25 14:16:26+0100` Default: Enabled System locale preferred format `%c` Default: Disabled US-style time stamp with time zone `%d/%m/%Y %I:%M:%S %p %Z` Example output: `25/05/2021 03:16:26 PM CET` Default: Disabled
`utc_timestamp` | If set to yes, then notifications will use the UTC time stamps instead of the local system time. Default value: `no`

### Naemon macros that use Unix time

There are a few Naemon macros that get reported in Unix time (`time_t`) by default. These are converted to your configured date/time format:

- LASTHOSTCHECK

- LASTHOSTSTATECHANGE

- LASTHOSTUP

- LASTHOSTDOWN

- LASTHOSTUNREACHABLE

- LASTSERVICECHECK

- LASTSERVICESTATECHANGE

- LASTSERVICEOK

- LASTSERVICEWARNING

- LASTSERVICEUNKNOWN

- LASTSERVICECRITICAL

- TIMET

- PROCESSSTARTTIME

- EVENTSTARTTIME

Caution: Macros not included in this list do not necessarily reflect Unix time. Before you convert a reported time stamp, verify the date/time format used for the macro.

## Default configuration file

Beginning OP5 Monitor 8.3.2, the `/etc/op5/notify.yml` configuration file is shipped with OP5 Monitor by default:

    ---
    # Configuration for OP5 Monitor notifications.
    # Location: /etc/op5/notify.yml
    # Lines starting with # are ignored.

    # The notification email includes a link back to the OP5 Monitor server that
    # sent the notification. You can configure the email settings to link to a
    # different host.
    # If not specified, the system hostname is used.
    #hostname: "monitor-server-hostname.example.com"

    # Format in notifications for timestamp macros like LASTHOSTCHECK, LASTSERVICESTATECHANGE etc.
    # See strftime(3) for details of formatting (https://man7.org/linux/man-pages/man3/strftime.3.html).

    # Internet Date/Time Format (rfc3339)
    timestamp_format: "%Y-%m-%d %H:%M:%S%z"

    # Use system locales preferred format:
    #timestamp_format: "%c"

    # US style timestamp, with timezone:
    #timestamp_format: "%d/%m/%Y %I:%M:%S %p %Z"

    # Set to yes to use UTC (Universal Coordinated Time/GMT) timestamps in
    # notifications instead of systems local time.
    utc_timestamps: no

If you have already defined `/etc/opt/notify.yml` in the past, then this default file does not affect your existing implementation. Instead, the package manager appends `.rpmnew` to the default `/etc/op5/notify.yml` being shipped with the updated OP5 Monitor package. In this case, you must manually merge your existing `/etc/op5/notify.yml` file with the contents of the new `/etc/op5/notify.yml.rpmnew` file to use the new date/time formats available to notifications.

## Notification skins

The three basic notification types (email, SMS and HTML POST notifications) all use notification skins. The notification skins are templates describing how the notification is supposed to look.

The `/notify` directory includes the following skins directories:

- `skins.htmlpost/`
- `skins.mail/`
- `skins.sms/`

Each directory contains a number of notification skins divided into host and service notification filters:

- `host.ACKNOWLEDGEMENT`
- `host.FLAPPINGSTART`
- `host.FLAPPINGSTOP`
- `host.PROBLEM`
- `host.RECOVERY`
- `service.ACKNOWLEDGEMENT`
- `service.FLAPPINGSTART`
- `service.FLAPPINGSTOP`
- `service.PROBLEM`
- `service.RECOVERY`

### The SMS service.PROBLEM skin

The SMS `service.PROBLEM` skin is very simple, since there is a limit to how much data you can send by SMS:

    #SERVICEDESC# on #HOSTNAME# is #SERVICESTATE#. #SERVICEOUTPUT#

### The mail service.PROBLEM skin

Mail notifications can contain a lot more data, so the mail skin is more complex:

    From: #FROMEMAIL#
    To: #CONTACTEMAIL#
    Subject: [op5] #NOTIFICATIONTYPE#: '#SERVICEDESC#' on '#HOSTNAME#' is #SERVICESTATE#
    Content-Type: text/plain; charset=utf-8

    #extra_host_vars#

    ITRS OP5 Monitor

    Service #NOTIFICATIONTYPE# detected #LASTSERVICESTATECHANGE#.
    '#SERVICEDESC#' on host '#HOSTNAME#' has passed the #SERVICESTATE# threshold.
    #STATUS_URL#

    Additional info;
    #SERVICEOUTPUT#

    Host:    #HOSTNAME#
    Address: #HOSTADDRESS#
    Alias:   #HOSTALIAS#
    Status:  #HOSTSTATE#
    Comment: #NOTIFICATIONCOMMENT#

    Service: #SERVICEDESC#
    Status : #SERVICESTATE#
    Latency: Check was #SERVICELATENCY# seconds behind schedule
    Misc   : Check took #SERVICEEXECUTIONTIME# seconds to complete

    Additional links (requires configuration);
    Host actions: #HOSTACTIONURL#
    Host notes:   #HOSTNOTESURL#
    Service actions: #SERVICEACTIONURL#
    Service notes:   #SERVICENOTESURL#

### Keywords

Skins contain keywords in the following format:

- `#SERVICEDESC#`
- `#HOSTNAME#`

Keywords are replaced with the value of a command line argument in the format `X=Y`. So a command line such as this generates a keyword with the name `X` having the value `Y`.

If a notification macro, or other object supplying a keyword value, is missing from the notification command, it does not stop the notification from being sent; only the replacement value will be missing.

### Create a custom notification skin

If you need to change the default notification skins, you need to create them in a different directory.

To create custom notification skins:

1. From the OP5 Monitor command line, change to the `/notify` directory:

       cd /opt/monitor/op5/notify

2. Create a directory called `custom-skins`:

       mkdir custom-skins

3. Copy the `skins.*` directories to the `custom-skins` directory:

       cp -R skins.* custom-skins/

4. Make your changes to the skins. The new skins are used directly after you save the changes.

Note: If you have a peered configuration, you need to make these changes on all peers.

## Configure SNMP trap notifications

You can send OP5 Monitor notifications as SNMP traps. To configure SNMP notifications, you need to perform the following:

- Add SNMP notification commands.
- Configure contacts.

### Add SNMP notification commands

You need to add one command for host notifications and one for service notifications.

1. In the OP5 Monitor user interface, click Manage > Configure.
2. In the Checks and Notifications section, click Commands.
3. Add two new commands with the following arguments, changing `snmp.trap.host` and `SNMPCOMMUNITY` to their correct values in both commands.

`host_notify_by_snmp`:

    $USER3$/notify/notify_by_snmp.pl -h snmp.trap.host -C SNMPCOMMUNITY -t nHostNotify "NOTIFICATIONTYPE=$NOTIFICATIONTYPE$" "NOTIFICATIONNUMBER=$NOTIFICATIONNUMBER$" "HOSTACKAUTHOR=$HOSTACKAUTHOR$" "HOSTACKCOMMENT=$HOSTACKCOMMENT$" "HOSTNAME=$HOSTNAME$" "HOSTSTATEID=$HOSTSTATEID$" "HOSTSTATETYPE=$HOSTSTATETYPE$" "HOSTATTEMPT=$HOSTATTEMPT$" "HOSTDURATIONSEC=$HOSTDURATIONSEC$" "HOSTGROUPNAME=$HOSTGROUPNAME$" "LASTHOSTCHECK=$LASTHOSTCHECK$" "LASTHOSTSTATECHANGE=$LASTHOSTSTATECHANGE$" "HOSTOUTPUT=$HOSTOUTPUT$"

`service_notify_by_snmp`:

    $USER3$/notify/notify_by_snmp.pl -h snmp.trap.host -C SNMPCOMMUNITY -t nSvcNotify "NOTIFICATIONTYPE=$NOTIFICATIONTYPE$" "NOTIFICATIONNUMBER=$NOTIFICATIONNUMBER$" "SERVICEACKAUTHOR=$SERVICEACKAUTHOR$" "SERVICEACKCOMMENT=$SERVICEACKCOMMENT$" "HOSTNAME=$HOSTNAME$" "HOSTSTATEID=$HOSTSTATEID$" "SERVICEDESCRIPTION=$SERVICEDESCRIPTION$" "SERVICESTATEID=$SERVICESTATEID$" "SERVICEATTEMPT=$SERVICEATTEMPT$" "SERVICEDURATIONSEC=$SERVICEDURATIONSEC$" "SERVICEGROUPNAME=$SERVICEGROUPNAME$" "LASTSERVICECHECK=$LASTSERVICECHECK$" "LASTSERVICESTATECHANGE=$LASTSERVICESTATECHANGE$" "SERVICEOUTPUT=$SERVICEOUTPUT$"

4. Click Submit.
5. Click Save at the top of the configuration page.

### Configure contacts

To configure the notification contacts:

1. In the OP5 Monitor user interface, click Manage > Configure.
2. In the Checks and Notifications section, click Contacts.
3. Either open an existing contact or create a new one.
4. From the host_notification_commands drop-down list, select host_notify_by_snmp.
5. From the service_notification_commands drop-down list, select service_notify_by_snmp.
6. Set contactgroups to a contact group associated with the required monitoring objects.
7. Click Submit.
8. Click Save at the top of the configuration page.

Was this topic helpful?
Your thoughts...
How can we improve this topic?
It did not answer my question(s)
Confusing
Missing screenshot or graphics
Missing technical details
Information is out of date
Your thoughts...
Thank you for your feedback!