ericsysmin's DevOps Blog

Integrating Zabbix w/Slack for Channel Notifications

I’ve been using Slack and Zabbix now for a few years, and figured since there’s not too many guides out there on how to integrate the two easily. I know there are a few examples so far, but this is what’s worked best for me.

First I use this script, and put it in your alertscripts directory, ex.   /usr/lib/zabbix/alertscripts .

Notice there’s no field for credentials. That’s because over time I found it easier to deploy with a configuration file that was in yaml format.

Make sure you create an API token for slack by creating a Slack App, then using it’s Verification Token. Hopefully in the future I revisit this and make use of the Client ID, Client Secret, Signing Secret configuration.

So to provide that file we need to create the following in your zabbix configuration directory ex.  /ext/zabbix/slack.yml .

This makes it easier at least for me to deploy and separate configuration from the script itself.

Next steps include creating the Slack Media Type.

Navigate to Administration > Media types

Select “Create media type”, and fill out the following:

 

Don’t forget to save!

Your first value is going to be “Channel” the rest fill in the message details.

Continue reading...

Alertscript for Zabbix to Slack

Recently Slack has gained a lot of usage, requiring Operations teams to try and integrate Zabbix Alerting with their Slack Installation, here’s how to do so.

You will need to get your Slack Post URL, fill that value in the following script.

 

Continue reading...

Configuring High Availability (HA) Zabbix Server on CentOS 7

Step 1: Configure Name Resolution

We need to configure our host to know it’s counterpart quickly without DNS (we don’t want false failures).

Open the following two entries to your /etc/hosts file.

Step 2: Install Zabbix Server

Now we need to install the Zabbix Server, however there are so many ways to deploy this I prefer Puppet (mainly because I contribute a lot to https://github.com/dj-wasabi/puppet-zabbix) We are going to do this the standard way.

On both Zabbix Servers we will need to install Zabbix, I like MySQL, but that’s just because I’ve been using it a long time.

At the time this is written current version is Zabbix 3.0

I don’t really want to go far in depth to showing the entire Zabbix Server configuration, if you still need help setting the rest up please visit:
https://www.zabbix.com/documentation/3.0/manual/installation/install_from_packages#red_hat_enterprise_linuxcentos

However there is one change that is very important. You will need to configure in your  /etc/zabbix/zabbix_server.conf

Step 3: Configure Pacemaker

Lets go back to zabbixserver1 and get these two authenticating to each other.

Now we need to create the cluster and add the zabbix servers

To start the cluster

Make sure that Pacemaker and Corosync are started at boot on both hosts.

Check the status of your cluster after it’s been started by using

Check the status of your nodes in the cluster by using

For Zabbix we don’t need or want stonith especially since we just have 2 servers. Lets run:

Because we have 2 nodes our number of nodes is too low to have a quorum. To have a quorum you must have at least 2 nodes, and that would defeat the purpose of this. We will ignore the low quorum with the following.

Now we are into the fun part! Lets create the VIP which would be used between both of these Zabbix Servers.

To test this you can try to ping the <cluster_ip> now.

You should also see a resource now appear when you test the command

Lets disable systemd from controlling zabbix-server. Once the next step is completed, Pacemaker will actually control the process.

Now we need to setup the Zabbix Server resource which will make sure that the systemd (CentOS 7) has zabbix-server always running in this cluster.

You should also see that resource now appear when you test the command

Now we may have a problem, in its current state the VIP and the zabbix_server can be running on different nodes. Definitely not what we want happening. To resolve this we need to add some constraints that tell pacemaker that they need to be on the same host, and that the cluster_ip needs to be completed before the zabbix_server

Testing the cluster can be done by the following

Bonus Item

If you would like Zabbix Server to prefer a specific server you can use the following command to do so.

Used Resources:

https://www.digitalocean.com/community/tutorials/how-to-set-up-an-apache-active-passive-cluster-using-pacemaker-on-centos-7

Continue reading...