zabbix Archives

Integrating Zabbix w/Slack for Channel Notifications

January 16, 2019

I’ve been using Slack and Zabbix now for a few years, and figured since there’s not too many guides out there on how to integrate the two easily. I know there are a few examples so far, but this is what’s worked best for me.

First I use this script, and put it in your alertscripts directory, ex. /usr/lib/zabbix/alertscripts .

#!/usr/bin/python

import sys

import yaml

from slackclient import SlackClient

slack_config_file = '/etc/zabbix/slack.yml'

with open(slack_config_file, 'r') as ymlfile:

slack_config = yaml.load(ymlfile)

slack = SlackClient(slack_config['slack_api_token'])

channel = sys.argv[1]

message = "```Status: %s\n%s```" %(sys.argv[2], sys.argv[3])

response = send_slack_message(channel, message)

if response["ok"]:

print("Message posted successfully: " + response["message"]["ts"])

# If the message failed, check for rate limit headers in the response

elif response["ok"] is False and response["headers"]["Retry-After"]:

# The `Retry-After` header will tell you how long to wait before retrying

delay = int(response["headers"]["Retry-After"])

print("Rate limited. Retrying in " + str(delay) + " seconds")

time.sleep(delay)

slack.api_call("chat.postMessage", channel=channel, text=message)

Notice there’s no field for credentials. That’s because over time I found it easier to deploy with a configuration file that was in yaml format.

Make sure you create an API token for slack by creating a Slack App, then using it’s Verification Token. Hopefully in the future I revisit this and make use of the Client ID, Client Secret, Signing Secret configuration.

So to provide that file we need to create the following in your zabbix configuration directory ex. /ext/zabbix/slack.yml .

1 2	# Zabbix - Slack Configuration slack_api_token: {{ zabbix_slack_api_token }}

This makes it easier at least for me to deploy and separate configuration from the script itself.

Next steps include creating the Slack Media Type.

Navigate to Administration > Media types

Select “Create media type”, and fill out the following:

Name: Slack

Type: Script

Script name: zabbix_slack.py

Script parameters:

{ALERT.SENDTO}

{ALERT.SUBJECT}

{ALERT.MESSAGE}

Enabled: Checked

Don’t forget to save!

Your first value is going to be “Channel” the rest fill in the message details.

devops, notifications, operations, ops, slack, zabbix

Zabbix

Alertscript for Zabbix to Slack

February 23, 2016

Eric Anderson

Recently Slack has gained a lot of usage, requiring Operations teams to try and integrate Zabbix Alerting with their Slack Installation, here’s how to do so.

You will need to get your Slack Post URL, fill that value in the following script.

#!/usr/bin/python

import requests

import sys

requests.packages.urllib3.disable_warnings()

slack_post_url = 'https://slack_post_url'

def sendAlert(webhook_url, alert_msg):

payload = alert_msg

response = requests.post(webhook_url, json=payload)

print payload

sendAlert(

slack_post_url,

{

"channel": "%s" % sys.argv[1],

"username": "zabbix",

"text": "```Status: %s\n%s```" %(sys.argv[2], sys.argv[3]),

}

)

actions, alerts, notification, python, slack, zabbix

Zabbix

Configuring High Availability (HA) Zabbix Server on CentOS 7

February 18, 2016

Eric Anderson

Step 1: Configure Name Resolution

We need to configure our host to know it’s counterpart quickly without DNS (we don’t want false failures).

Open the following two entries to your /etc/hosts file.

1 2	primary_zabbixserver_ip zabbixserver1 zabbixserver1.example.com secondary_zabbixserver_ip zabbixserver2 zabbixserver2.example.com

Step 2: Install Zabbix Server

Now we need to install the Zabbix Server, however there are so many ways to deploy this I prefer Puppet (mainly because I contribute a lot to https://github.com/dj-wasabi/puppet-zabbix) We are going to do this the standard way.

On both Zabbix Servers we will need to install Zabbix, I like MySQL, but that’s just because I’ve been using it a long time.

At the time this is written current version is Zabbix 3.0

#install the zabbix repository

sudo rpm -ivh http://repo.zabbix.com/zabbix/3.0/rhel/7/x86_64/zabbix-release-3.0-1.el7.noarch.rpm

#install zabbix-server-mysql from the zabbix repository

sudo yum install zabbix-server-mysql

I don’t really want to go far in depth to showing the entire Zabbix Server configuration, if you still need help setting the rest up please visit:
https://www.zabbix.com/documentation/3.0/manual/installation/install_from_packages#red_hat_enterprise_linuxcentos

However there is one change that is very important. You will need to configure in your /etc/zabbix/zabbix_server.conf

1 2	SourceIP=<cluster_ip> ListenIP=<cluster_ip>

Step 3: Configure Pacemaker

#zabbixserver1

sudo yum install pacemaker pcs

sudo passwd hacluster

sudo systemctl start pcsd.service

sudo systemctl enable pcsd.service

#zabbixserver2

sudo yum install pacemaker pcs

sudo passwd hacluster

sudo systemctl start pcsd.service

sudo systemctl enable pcsd.service

Lets go back to zabbixserver1 and get these two authenticating to each other.

1 2	sudo pcs cluster auth zabbixserver1 zabbixserver2 Username: hacluster

Now we need to create the cluster and add the zabbix servers

1	sudo pcs cluster setup --name zabbixserver zabbixserver1 zabbixserver2

To start the cluster

1	sudo pcs cluster start --all

Make sure that Pacemaker and Corosync are started at boot on both hosts.

#zabbixserver1

sudo systemctl enable corosync.service

sudo systemctl enable pacemaker.service

#zabbixserver2

sudo systemctl enable corosync.service

sudo systemctl enable pacemaker.service

Check the status of your cluster after it’s been started by using

1	sudo pcs status cluster

Check the status of your nodes in the cluster by using

1	sudo pcs status nodes

For Zabbix we don’t need or want stonith especially since we just have 2 servers. Lets run:

1	sudo pcs property set stonith-enabled=false

Because we have 2 nodes our number of nodes is too low to have a quorum. To have a quorum you must have at least 2 nodes, and that would defeat the purpose of this. We will ignore the low quorum with the following.

1	pcs property set no-quorum-policy=ignore

Now we are into the fun part! Lets create the VIP which would be used between both of these Zabbix Servers.

1	pcs resource create cluster_vip ocf:heartbeat:IPaddr2 ip=<cluster_ip> cidr_netmask=<cluster_netmask> op monitor interval=20s

To test this you can try to ping the <cluster_ip> now.

1	ping -c1 <cluster_ip>

You should also see a resource now appear when you test the command

1	sudo pcs status

Lets disable systemd from controlling zabbix-server. Once the next step is completed, Pacemaker will actually control the process.

1 2	#zabbixserver1 sudo systemctl disable zabbix-server

1 2	#zabbixserver2 sudo systemctl disable zabbix-server

Now we need to setup the Zabbix Server resource which will make sure that the systemd (CentOS 7) has zabbix-server always running in this cluster.

1	sudo pcs resource create zabbix_server systemd:zabbix-server op monitor interval=10s

You should also see that resource now appear when you test the command

1	sudo pcs status

Now we may have a problem, in its current state the VIP and the zabbix_server can be running on different nodes. Definitely not what we want happening. To resolve this we need to add some constraints that tell pacemaker that they need to be on the same host, and that the cluster_ip needs to be completed before the zabbix_server