How to use Alerta to monitor Nagios alerts on CentOS 7

Introduction

Alerta is a Web application that integrates and regenerates alerts from multiple monitoring systems and visualizes on a single screen. Alerta can be integrated with a number of well-known monitoring tools such as Nagios, Zabbix, Sensu, InfluxData Kapacitor.

In this tutorial, you will set up Alerta and configure it to display notifications from Nagios(Universal Open Source Monitoring System).

prerequisites

To follow this tutorial, you will need:

• Set up two CentOS 7 servers by following the CentOS 7 Initial Server Setup Guide , including a sudo non-root user and a firewall.
• On the first CentOS server, you will run Nagios and install the following components:
  • Apache, MySQL and PHP, follow this tutorial on how to install Linux, Apache, MySQL, PHP (LAMP) on CentOS 7 .
  • Nagios 4, through the following tutorials to install how to install Nagios 4 on CentOS 7 and monitor your server and its configuration prerequisites.
• On the second CentOS server, we will install Alerta in this tutorial to install the following components:
  • Nginx, install the following tutorial on how to install Nginx on CentOS 7 .
  • MongoDB, install the following tutorial on how to install MongoDB on CentOS 7 .
  • Alerta, through the following steps 1 to 6, how to use the Alerta on CentOS 7 to monitor Zabbix alerts .

Step 1 - Install the Nagios-to-Alerta event agent module

You can use the Nagios Event Broker (NEB) module to extend Nagios functionality. NEB is Nagios event integration mechanism, NEB module is the other services can be integrated with Nagios shared library. In this step, we installed Nagios to Alerta Gateway , the NEB module that will send notification to Alerta.

Log on to the Nagios server as a non-root user:

ssh mrdragonx@your_nagios_server_ip

Nagios to Alerta Gateway does not have a preconfigured system package, so you must build it from the source code. To do this, you need to install some development tools and files. You also need to install Git so that you can get the source code from GitHub.

yum install -y git curl gcc make libcurl-devel

With the prerequisites installed, use Git to clone the source code from the project's GitHub repository:

git clone https://github.com/alerta/nagios-alerta.git

Then change to the new directory:nagios-alerts

cd nagios-alerta

Then use the makecompilation module:nagios-alerta

make nagios4

You will see the following output:

Outputcd ./src && make nagios4
make[1]: Entering directory `/root/nagios-alerta/src'
gcc -fPIC -g -O2 -DHAVE_CONFIG_H -I../include -I../include/nagios4 -lcurl -o alerta-neb.o alerta-neb.c -shared  -lcurl
make[1]: Leaving directory `/root/nagios-alerta/src'

If you see different content, make sure you have installed all the prerequisites.

Now run the installation task:

sudo make install

You will see this output, indicating that the module is installed in :/usr/lib/nagios

Outputcd ./src && make install
make[1]: Entering directory `/root/nagios-alerta/src'
[ -d /usr/lib/nagios ] || mkdir /usr/lib/nagios
install -m 0644 alerta-neb.o /usr/lib/nagios
make[1]: Leaving directory `/root/nagios-alerta/src'

After installing the module, we can configure Nagios to use this new module.

Step 2 - Configure the Nagios-to-Alerta module

Let's configure Nagios to send a notification message to Alerta.

First, enable the newly installed Alerta proxy module in the Nagios main configuration file. Open the Nagios profile in the editor:

sudo vi /usr/local/nagios/etc/nagios.cfg

Find broker_modulethe part that contains the instruction:

/usr/local/nagios/etc/nagios.cfg

...
# EVENT BROKER MODULE(S)
# This directive is used to specify an event broker module that should
# by loaded by Nagios at startup.  Use multiple directives if you want
# to load more than one module.  Arguments that should be passed to
# the module at startup are separated from the module path by a space.
#
[...]
#broker_module=/somewhere/module1.o
#broker_module=/somewhere/module2.o arg1 arg2=3 debug=0
...

To configure the Alerta module, you need to provide two mandatory parameters:

• URL : The address used to communicate with the Alerta API. You have configured how to use Alerta to monitor Zabbix alerts on CentOS 7 in step 3 of this tutorial .
• key : How does the API key you created in step 4 of this tutorial use Alerta on CentOS 7 to monitor the Zabbix alert . You need this feature to be able to authenticate with Alerta and publish events.

Add this line to a file to configure Alerta integration:

/usr/local/nagios/etc/nagios.cfg

...
broker_module=/usr/lib/nagios/alerta-neb.o http://your_alerta_server_ip/api key=ALERTA_API_KEY
...

There are some additional optional parameters that can also be specified:

• env : Specifies the environment name. The default environment name is " Production.
• hard_only : forward only in hard state. You can find more information about the Nagios status type in the Nagios documentation . Set it 1to enable this mode.
• Debug : - Enables the debug mode of the module. Set it 1to enable this mode.

To specify all of these options, use this line instead:

/usr/local/nagios/etc/nagios.cfg

...
broker_module=/usr/lib/nagios/alerta-neb.o http://your_alerta_server_ip/api key=ALERTA_API_KEY env=Production hard_only=1 debug=1
...

Save the file and exit the editor.

In order to identify alerts based on the environment and service name, you need to use the Nagios custom object variable to set the environment and service name. To do this, use the configuration _Environmentand _Servicevariables. Now let's configure them.

Open the default Nagios host object configuration file, which you can find in the directory:/usr/local/nagios/etc/objects/

sudo vi /usr/local/nagios/etc/objects/localhost.cfg

We will mark all alarms for this host as production alerts, and we will call the default service Nagios . Find the following host definitions:

/usr/local/nagios/etc/objects/localhost.cfg

...
define host{
        use                     linux-server            ; Name of host template to use
                                                        ; This host definition will inherit all variables that are defined
                                                        ; in (or inherited by) the linux-server host template definition.
        host_name               localhost
        alias                   localhost
        address                 127.0.0.1
        }

...

The _Environmentand _Servicevalues added to the configuration:

/usr/local/nagios/etc/objects/localhost.cfg

...
        host_name               localhost
        alias                   localhost
        address                 127.0.0.1
        _Environment            Production
        _Service                Nagios
        }
...

All events associated with the system partitio space are now marked as a system alert. Find this part of the file that defines how to check the available space:

/usr/local/nagios/etc/objects/localhost.cfg

...
define service{
        use                             local-service         ; Name of service template to use
        host_name                       localhost
        service_description             Root Partition
        check_command                   check_local_disk!20%!10%!/
        }
...

Modify it to Systemassociate it with the service:

/usr/local/nagios/etc/objects/localhost.cfg

...
define service{
        use                             local-service         ; Name of service template to use
        host_name                       localhost
        service_description             Root Partition
        check_command                   check_local_disk!20%!10%!/
        _Service                        System
        }
...

Save the file and exit the editor. Reboot Nagios to apply these new settings:

sudo systemctl restart nagios.service

By checking its status to ensure that the service is running:

systemctl status nagios.service

You will see the following output:

Output...
Jul 01 08:44:31 nagios nagios[8914]: [alerta] Initialising Nagios-Alerta Gateway module, v3.4.1
Jul 01 08:44:31 nagios nagios[8914]: [alerta] debug is off
Jul 01 08:44:31 nagios nagios[8914]: [alerta] states=Hard/Soft
Jul 01 08:44:31 nagios nagios[8914]: [alerta] Forward service checks, host checks and downtime to http://your_alerta_server_ip/api
Jul 01 08:44:31 nagios nagios[8914]: Event broker module '/usr/lib/nagios/alerta-neb.o' initialized successfully.
Jul 01 08:44:31 nagios nagios[8914]: Successfully launched command file worker with pid 8920

Nagios will now send a notification as soon as any system or service is closed. Let's generate a test event.

Step 3 - Generate a test alert to verify Nagios-Alerta integration

Let's generate a test alert to ensure all connections. By default, Nagios tracks the size of disk space available on the server. We will create a large enough temporary file to trigger the Nagios file system to use the alert.

First, determine how much free space is on the Nagios server. You can use the dfcommand to find out:

df -h

You will see the following output:

Output    Filesystem      Size  Used Avail Use% Mounted on
    /dev/vda1        20G  3.1G   16G   17% /

Look at the size of the available space. In this case, the available space is 16GB. Your available space may be different.

Use the fallocatecommand to create a file that occupies more than 80% of the available disk space, which should be sufficient to trigger an alert:

fallocate -l 14G /tmp/temp.img

Within a few minutes, Nagios triggers an alert about the amount of free disk space and sends a notification message to Alerta. You will see this new notification in the Alerta dashboard:

Now that you know that the alert is running, delete the temporary file you created so that you can reclaim disk space:

rm -f /tmp/temp.img

After a while, Nagios will send a recovery message. Then the alarm will disappear from the main Alerta dashboard, but you can see all closed events by selecting Off .

You can click the event line to see more details.

in conclusion

In this tutorial, you have configured Nagios to send notifications to another server running Alerta.

Alerta offers you a convenient place to track many system alerts. For example, if you use Nagios in some parts of your infrastructure and other parts use Zabbix, you can combine the notifications for both systems into one panel.
Source howtoforge