Icinga Notifications in Telegram

There are multiple solutions for mobile notifications from Icinga2, including native apps or via E-Mail. However I found that problems on my server have actually become rare and I don’t check my private mails as often as I used to. So I thought it would be great to just get my alerts via a communication channel I use all the time anyway and which supports push notifications: The messenger Telegram.

Telegram has a feature called bots which is more or less a fancy interface to send chat messages to clients. You create a bot from your Telegram-client and receive an authentication token which you can then use to run a handler on your server. After this, whenever the bot (which is running on the Telegram servers) receives a message, it gets dispatched (by the telegram servers) to the handler on your server and you can process it there.

In theory the bot-handler doesn’t necessarily need to run all the time, only for subscribe or unsubscribing in our case, so in theory we could fire up a temporary handler, subscribe to the bot in our Telegram client and then only ever run a notification script from Icinga that directly talks to the Telegram bot API.

However I decided for reasons of simplicity and potential extensibility to create a simple HTTP Gateway that will forward an incoming request via HTTP (which we can much more easily create in an Icinga notification script) and translate this to the API. That has the added benefit of letting the gateway handle new subscribe requests at any time.

The official Telegram Documentation describes the setup of a new bot in detail, the short form is: Add a contact called @BotFather, press ‘Start’ and follow the instructions. You will be given an authentication token which we will use later.

You can find my Telegram Gateway on GitHub. If you have a look at the file telegram-interface.py, you can see it functions very simple:

• start a Telegram bot (using the python-telegram-bot library)
• add a handler that saves the chat-id of anybody writing /start to the bot
• add a handler that allows people to unsubscribe via /unsubscribe (removing the respective chat-id)
• start a flask server listening locally for POST-Requests with a special JSON-payload
• if a request arrives, format a message as Markdown and write it to all chat-id’s (respectively all subscribed clients) in the save file

To start the gateway:

git clone https://github.com/FAUSheppy/telegram-http-gateway
cd telegram-http-gateway echo $AUTH_TOKEN > bot.token
python3 -m pip install -r req.txt 
./telegram-interface.py --port 6000

With $AUTH_TOKEN being the token you received when creating the bot. You can of course use any port, 6000 is port use in the later examples.

The following part assumes you have a working Icinga2-instance. Icingaweb is not needed. On most systems just installing Icinga2 via the package-manager will create a working instance ( apt install icinga2 on Debian). If you don't care about the explanation you can just clone the atlantishq monitoring-tools into /etc/icinga2/monitoring-tools/ and add an

include /etc/icinga2/monitoring-tools/commands.d/telegram-notify.conf;

at the end of icinga.conf. Make sure the directory and it’s contents can be read by Icinga.

Notification Command Object

Then we need the script which Icinga actually executes to create a request to the gateway. In theory this could just be inline-curl in a NotificationCommand-object:

object NotificationCommand "telegram-service-notification" {
    command = [ "/usr/bin/curl", "-X", "POST", "-H", "Content-Type:
              application/json", "--data", "
                 { 'service_name' : 'hello world', 
                    service_type : '$notification_type$' }",
               "localhost:6000/send-all-icinga"] 
}

But that gets very messy, very quickly so I created a small python-script that emulates the arguments of the mail notification script already in included in Icinga, so you can just copy and paste the NotificationCommand for mail notifications (usually in notifications.conf in /etc/icinga2) and replace the path/name. It should look like this afterwards:

object NotificationCommand "telegram-service-notification" {
  command = [ "/etc/icinga2/monitoring-tools/telegram-notify.py" ]

  arguments += {
    "-4" = "$notification_address$"
    "-6" = "$notification_address6$"
    "-b" = "$notification_author$"
    "-c" = "$notification_comment$"
    "-d" = {
      required = true
      value = "$notification_date$"
    }
    "-e" = {
      required = true
      value = "$notification_servicename$"
    }
    "-f" = {
      value = "$notification_from$"
      description = "Set from address."
    }
    "-i" = "$notification_icingaweb2url$"
    "-l" = {
      required = true
      value = "$notification_hostname$"
    }
    "-n" = {
      required = true
      value = "$notification_hostdisplayname$"
    }
    "-o" = {
      required = true
      value = "$notification_serviceoutput$"
    }
    "-r" = {
      required = true
      value = "$notification_useremail$"
    }
    "-s" = {
      required = true
      value = "$notification_servicestate$"
    }
    "-t" = {
      required = true
      value = "$notification_type$"
    }
    "-u" = {
      required = true
      value = "$notification_servicedisplayname$"
    }
    "-v" = "$notification_logtosyslog$"
  }

  vars += {
    notification_address = "$address$"
    notification_address6 = "$address6$"
    notification_author = "$notification.author$"
    notification_comment = "$notification.comment$"
    notification_type = "$notification.type$"
    notification_date = "$icinga.long_date_time$"
    notification_hostname = "$host.name$"
    notification_hostdisplayname = "$host.display_name$"
    notification_servicename = "$service.name$"
    notification_serviceoutput = "$service.output$"
    notification_servicestate = "$service.state$"
    notification_useremail = "$user.email$"
    notification_servicedisplayname = "$service.display_name$"
  }
}

Notification Object

We then need to apply this notification to something:

apply Notification "telegram-generic" to Service {
    import "mail-service-notification"
    user_groups = ["icingaadmins"]
    command = "telegram-service-notification"
    assign where host.address 
}

By convention we import the mail-service-notification object as a basis, then we define user_groups for which we want send notification. This (in our case) is not about whom to send notifications, but for which services we send notifications. Usually you just want the icingadmins group here because this group already exists by default and will get notifications for anything (unless you already changed this behavior). If you have a special notification group with additional filters, rules, etc, you can use this group instead (or use a single user with the “user = … “-directive).

Finally “ assign where host.address” is basically an “ assign everywhere” since every service likely has an address. Obviously you could use something more specific here, like assign where host.vars.telegram_notifications. Please refer to the Icinga2 Documentation for information about the exact workings of the assign-directive.

Now you can add your Bot as a contact in your telegram client, send a /start-message and you will now receive notification from your Icinga in Telegram.

Originally published at https://blog.atlantishq.de.