ericsysmin's DevOps Blog -

Install Postman on Ubuntu from Binary

June 10, 2020

This one is pretty straight-forward. Simply just download and untar and then create a desktop link, and /usr/bin link.

wget https://dl.pstmn.io/download/latest/linux64 -O postman.tar.gz

sudo tar -xzf postman.tar.gz -C /opt

rm postman.tar.gz

sudo ln -s /opt/Postman/Postman /usr/bin/postman

cat > ~/.local/share/applications/postman.desktop <<EOL

[Desktop Entry]

Encoding=UTF-8

Name=Postman

Exec=postman

Icon=/opt/Postman/app/resources/app/assets/icon.png

Terminal=false

Type=Application

Categories=Development;

EOL

Import Self-Signed Cert for Chrome/Edge/Chromium on Mac

Eric Anderson

May 16, 2020

With the latest versions of Chrome, Edge, or Chromium, it’s no longer possible to load pages with self-signed certs and they provide a NET::ERR_CERT_INVALID error.

ERR_CERT_INVALID

When you go to the “Advanced” button it does not allow you to ignore and proceed.

To work around this you can download the certificate. Then import it into the trusted certificate store.

I used OpenSSL to download the certificate, but there are other options. Here are the commands I used.

server=myserver.com

openssl s_client -connect ${server}:443 \

-showcerts </dev/null 2>/dev/null|\

openssl x509 -outform PEM >${server}.pem

sudo security add-trusted-cert -d -r trustAsRoot \

-p ssl -k /Library/Keychains/System.keychain ${server}.pem

This will add your certificate to the System Keychain and trust it as an SSL certificate.

If you get an the error:

1	SecTrustSettingsSetTrustSettings: One or more parameters passed to a function were not valid.

try replacing

1	-r trustAsRoot

1	-r trustRoot

Ansible Collections: Testing only what’s changed

Eric Anderson

May 11, 2020

Previously

When testing roles before GitHub Actions, it was assumed that you’d only have one repository for each role. But with the addition of collections, that is no longer the case. Your collection can now have multiple roles, modules, and often you do not need to test everything when a role or a set of modules has changed.

Using GitHub Actions, there’s a way to do this.

Now with GitHub Actions

Using GitHub Actions and workflow, we can configure what actions will trigger a test (workflow) run. In my example, which I do use on all of my collections, I set only on Pull Request and Push will the tests be triggered.

on:

push:

paths:

- 'roles/zabbix_agent/**'

- 'molecule/zabbix_agent/**'

- '.github/workflows/zabbix_agent.yml'

pull_request:

paths:

- 'roles/zabbix_agent/**'

- 'molecule/zabbix_agent/**'

- '.github/workflows/zabbix_agent.yml'

jobs:

zabbix_agent:

runs-on: ubuntu-18.04

env:

PY_COLORS: 1

ANSIBLE_FORCE_COLOR: 1

strategy:

fail-fast: false

...

So if you notice in the example, configured my test to run on both push and pull_request. Unfortunately, GitHub Actions doesn’t support anchors yet so I couldn’t use them.

Why did I choose those paths?

'roles/zabbix_agent/**' – sets GitHub actions to watch all the files underneath the role zabbix_agent

'molecule/zabbix_agent/**' – watches all the files part of the molecule testing for zabbix_agent

'.github/workflows/zabbix_agent.yml' – the file that runs the GitHub Action workflow itself

The code here helps ensure that only when a file used for testing or executing this role is modified will it run and ensures that you don’t waste a lot of testing time on GitHub Actions so other tests can run on other repositories. You can find more options here https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#on

Using a Dockerfile Repo for Molecule Dockerfiles

Eric Anderson

May 10, 2020

I’d like to share with you another design in testing your Ansible collections, modules, playbooks, and roles. Molecule used to include a file name Dockerfile.j2. This template, in the past, created your docker container on execution. It’s since moved away from that and now only uses the base image you provide it via molecule.yml. In some cases, you need more than what the base image offers, and you may not want to create docker images and upload them to Docker Hub, or Quay.io. I wanted a solution and test that didn’t require people to download my docker images from Docker Hub.

Dockerfile.j2 with lots of Jinja

I prefer building my images using Dockerfile each time I test. It’s relatively quick and ensures that my host is testing against the latest packages that are installed by the Dockerfile.

However, I have lots of roles, and this means each role had at least one Dockerfile, and the Dockerfiles were precisely the same. A simple change to one Dockerfile usually said I needed to update all of the others. What if I need systemd installed? SystemD is different on many operating systems, different files needed, as well as various install commands. Well, I initially started building a more complicated Dockerfile.j2,which used the platform values from Molecule. But then after adding CentOS, Debian, Ubuntu, Fedora, and many of their different versions, it got complicated.

100

101

102

103

104

105

106

107

108

109

110

111

# Molecule managed

{% if item.registry is defined %}

FROM {{ item.registry.url }}/{{ item.image }}

{% else %}

FROM {{ item.image }}

{% endif %}

ENV container=docker

{# Initial Package Installs and Container Prep #}

{% if item.image.split(':', 1)[0] in ["ubuntu"] %}

RUN apt-get update \

&& apt-get install -y --no-install-recommends \

locales software-properties-common rsyslog systemd systemd-cron sudo \

iproute2

RUN sed -i 's/^$$ModLoad imklog$/#\1/' /etc/rsyslog.conf

{% elif item.image.split(':', 1)[0] in ["debian"] %}

RUN apt-get update \

&& apt-get install -y --no-install-recommends \

sudo systemd systemd-sysv \

build-essential wget

{% elif item.image.split(':', 1)[0] in ["centos"] %}

{% if item.image in ["centos:7"] %}

RUN yum makecache fast && yum -y install deltarpm \

{% elif item.image in ["centos:8"] %}

RUN yum makecache --timer \

{% endif %}

&& yum -y install epel-release \

&& yum -y update \

&& yum -y install sudo which

{% endif %}

{# Install of Python2 #}

{% if item.image in ["ubuntu:16.04"] %}

RUN apt-get update \

&& apt-get install -y --no-install-recommends python-setuptools wget \

&& wget https://bootstrap.pypa.io/get-pip.py \

&& python get-pip.py

{% elif item.image in ["debian:9"] %}

RUN apt-get update \

&& apt-get install -y --no-install-recommends libffi-dev libssl-dev \

python-pip python-dev python-setuptools python-wheel

{% elif item.image in ["centos:7"] %}

RUN yum -y install python-pip

{% endif %}

{# Install of Python3 #}

{% if item.image in ["ubuntu:18.04", "ubuntu:20.04", "debian:10"] %}

RUN apt-get update \

&& apt-get install -y --no-install-recommends \

apt-utils python3-setuptools python3-pip

{% endif %}

{% if item.image in ["centos:8"] %}

RUN yum -y install hostname python3 python3-pip

{% endif %}

{# Steps for cleanup #}

{% if item.image.split(':', 1)[0] in ["ubuntu", "debian"] %}

RUN rm -Rf /var/lib/apt/lists/* \

&& rm -Rf /usr/share/doc && rm -Rf /usr/share/man \

&& apt-get clean

{% elif item.image.split(':', 1)[0] in ["centos"] %}

RUN yum clean all

{% endif %}

{# Steps for clenaup of systemd #}

{% if item.image in ["centos:7", "centos:8", "debian:9"] %}

RUN (cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == \

systemd-tmpfiles-setup.service ] || rm -f $i; done); \

rm -f /lib/systemd/system/multi-user.target.wants/*;\

rm -f /etc/systemd/system/*.wants/*;\

rm -f /lib/systemd/system/local-fs.target.wants/*; \

rm -f /lib/systemd/system/sockets.target.wants/*udev*; \

rm -f /lib/systemd/system/sockets.target.wants/*initctl*; \

rm -f /lib/systemd/system/basic.target.wants/*;\

rm -f /lib/systemd/system/anaconda.target.wants/*; \

mkdir -p /run/systemd/system

{% endif %}

{% if item.image in ["ubuntu:18.04", "ubuntu:20.04"] %}

# Remove unnecessary getty and udev targets that result in high CPU usage when using

# multiple containers with Molecule (https://github.com/ansible/molecule/issues/1104)

RUN rm -f /lib/systemd/system/systemd*udev* \

&& rm -f /lib/systemd/system/getty.target

{% endif %}

{% if item.image in ["centos:7", "centos:8"] %}

# Disable requiretty.

RUN sed -i -e 's/^$Defaults\s*requiretty$/#--- \1/' /etc/sudoers

{% endif %}

{% if item.image.split(':', 1)[0] not in ["centos", "debian"] %}

# Fix potential UTF-8 errors with ansible-test.

RUN locale-gen en_US.UTF-8

{% endif %}

# Install Ansible inventory file.

RUN mkdir -p /etc/ansible

RUN echo "[local]\nlocalhost ansible_connection=local" > /etc/ansible/hosts

{% if item.image in ["centos:7", "centos:8", "debian:9", "debian:10"] %}

VOLUME ["/sys/fs/cgroup"]

{% elif item.image in ["ubuntu:16.04", "ubuntu:18.04", "ubuntu:20.04"] %}

VOLUME ["/sys/fs/cgroup", "/tmp", "/run"]

{% endif %}

{% if item.image in ["centos:7", "centos:8"] %}

CMD ["/usr/sbin/init"]

{% elif item.image in ["ubuntu:16.04", "ubuntu:18.04", "ubuntu:20.04", "debian:9", "debian:10"] %}

CMD ["/lib/systemd/systemd"]

{% endif %}

It was overly complicated, and I was losing track of the if/then statements, “Which OS should run which commands?” and many other questions. I gave up. It’s not maintainable. Especially when there have been PR’s adding support for SUSE, and ArchLinux, so now I need to add those to my tests. Three words. OUT OF HAND. So I had to change how I tested. I’m not going to duplicate a Dockerfile that’s this complicated, 10+ times per collection. Maybe I can do file links? That worked, but then I had to manage the same files in each of my Roles/Collections. Again, not scalable. I wanted something easy to do and easy to maintain and add new OS support when needed. Then a couple of things hit me.

Molecule Uses Ansible (obviously)
Ansible has Lookup Filters

URL Lookup for Dockerfile.j2

What if I could do a URL lookup against a GitHub repository that allows me to manage the same Dockerfiles for SystemD and Ansible dependencies on all of my roles. So, I deleted all the contents of Dockerfile.j2 and replaced it with this:

{{ lookup('url', 'https://raw.githubusercontent.com/ericsysmin/docker-ansible-images/master/' ~ item.image ~ '/Dockerfile', split_lines=False) }}

So each time Molecule runs, it connects to this file, grabs the Dockerfile, and then uses it to build each docker container used by Molecule. Now I can centrally manage all of my Dockerfile files, and simplify my Dockerfiles by removing all of the if/then statements, and other logic. This does require that your system running Molecule requires internet access to the file location, if it fails, the Molecule execution will also fail.

Now in each of my roles, throughout my collections and standalone, I can modify by Dockerfiles and manage them from one location just as if I decided to produce Docker images from these Dockerfiles and then share them on Docker Hub or Quay.io.

Ansible Collections: Automating the Release Process to Galaxy

Eric Anderson

May 2, 2020

Since we are moving to Ansible Collections, some things are changing. Now when you create your collection and update your collection Ansible Galaxy will no longer automatically discover your collection via a Webhook. Now for Galaxy to know about your collection you have to upload a tar.gz file that containers the result of the ansible-galaxy collection build command.

However, many of us may still want to automate that process, and with @geerlingguy‘s help I was able to fully automate the release process, not from just tagging a release, but creating a release as we would before. So how does this work?

Creating the Build Directory

First, we need to create the build/ directory and include a couple of files.

build

├── galaxy_deploy.yml

└── templates

└── galaxy.yml.j2

Instead of having a galaxy.yml file in our root, we will need to generate the file when we execute the playbook.

This is the galaxy_deploy.yml playbook.

---

- hosts: localhost

connection: local

gather_facts: false

vars:

# This should be set via the command line at runtime.

tag: "{{ github_tag.split('/')[-1] }}"

pre_tasks:

- name: Ensure the ANSIBLE_GALAXY_TOKEN environment variable is set.

fail:

msg: ANSIBLE_GALAXY_TOKEN is not set.

when: "lookup('env','ANSIBLE_GALAXY_TOKEN') | length == 0"

- name: Ensure the ~/.ansible directory exists.

file:

path: ~/.ansible

state: directory

- name: Write the Galaxy token to ~/.ansible/galaxy_token

copy:

content: |

token: {{ lookup('env','ANSIBLE_GALAXY_TOKEN') }}

dest: ~/.ansible/galaxy_token

tasks:

- name: Template out the galaxy.yml file.

template:

src: templates/galaxy.yml.j2

dest: ../galaxy.yml

- name: Build the collection. # noqa 503

command: >

ansible-galaxy collection build

chdir=../

when: galaxy_yml.changed

- name: Publish the collection. # noqa 503

command: >

ansible-galaxy collection publish ./ericsysmin-system-{{ tag }}.tar.gz

chdir=../

when: galaxy_yml.changed

You’ll then need to create a build/templates folder, and create the galaxy.yml.j2 file within the templates folder.

Edit the values to fit your Ansible Collection, the only var I use is {{ tag }} which will be used later on.

### REQUIRED

# The namespace of the collection. This can be a company/brand/organization or product namespace under which all

# content lives. May only contain alphanumeric lowercase characters and underscores. Namespaces cannot start with

# underscores or numbers and cannot contain consecutive underscores

namespace: ericsysmin

# The name of the collection. Has the same character restrictions as 'namespace'

# The version of the collection. Must be compatible with semantic versioning

version: "{{ tag }}"

# The path to the Markdown (.md) readme file. This path is relative to the root of the collection

readme: README.md

# A list of the collection's content authors. Can be just the name or in the format 'Full Name <email> (url)

# @nicks:irc/im.site#channel'

authors:

- Eric Anderson (https://ericsysmin.com)

### OPTIONAL but strongly recommended

# A short summary description of the collection

description: Collection of System Administration Roles

# Either a single license or a list of licenses for content inside of a collection. Ansible Galaxy currently only

# accepts L(SPDX,https://spdx.org/licenses/) licenses. This key is mutually exclusive with 'license_file'

# license:

# - GPL-2.0-or-later

# The path to the license file for the collection. This path is relative to the root of the collection. This key is

# mutually exclusive with 'license'

license_file: 'LICENSE'

# A list of tags you want to associate with the collection for indexing/searching. A tag name has the same character

# requirements as 'namespace' and 'name'

tags:

- system

- chrony

- epel

- logrotate

- ntp

- remi_repo

- selinux

- rhel

- ubuntu

# Collections that this collection requires to be installed for it to be usable. The key of the dict is the

# collection label 'namespace.name'. The value is a version range

# L(specifiers,https://python-semanticversion.readthedocs.io/en/latest/#requirement-specification). Multiple version

# range specifiers can be set and are separated by ','

dependencies: {}

# The URL of the originating SCM repository

repository: https://github.com/ericsysmin/ansible-collection-system

# The URL to any online docs

documentation: https://github.com/ericsysmin/ansible-collection-system

# The URL to the homepage of the collection/project

homepage: https://github.com/ericsysmin/ansible-collection-system

# The URL to the collection issue tracker

issues: https://github.com/ericsysmin/ansible-collection-system/issues

Ok, so now that we’ve created the build components now we need to do the automation part of this. I chose to use GitHub Actions again, as they are the recommended path for the repositories sitting at https://github.com/ansible-collections.

Configuring the GitHub Action Workflow

In your .github/workflows/ folder you’ll need to create a release workflow. To do this I used the following GitHub Actions Workflow YAML. I called it release.yml , and it sits at .github/workflows/release.yml this is an example of what you can use.

on:

release:

types:

- created

jobs:

release:

runs-on: ubuntu-18.04

env:

ANSIBLE_GALAXY_TOKEN: ${{ secrets.ANSIBLE_GALAXY_TOKEN }}

ANSIBLE_FORCE_COLOR: 1

steps:

- name: Check out code

uses: actions/checkout@v1

- name: Set up Python 3.8

uses: actions/setup-python@v1

with:

python-version: 3.8

- name: Install dependencies

run: |

python -m pip install --upgrade pip

pip install ansible

- name: Run role test

run: >-

ansible-playbook -i 'localhost,' build/galaxy_deploy.yml -e "github_tag=${{ github.ref }}"

Using the on value we are able to set the workflow to only execute when a release is created in GitHub. This will ensure we have a GitHub ref to be used against the playbook. It will also sync your Ansible Galaxy release with GitHub release actions.

on:

release:

types:

- created

If you noticed we have a key here that provides our Ansible Galaxy token ${{ secrets.ANSIBLE_GALAXY_TOKEN }} for us to use this token we need to get it from Ansible Galaxy, and add it to our repository secrets. You can find your key here https://galaxy.ansible.com/me/preferences under API Key.

Ansible Galaxy Preferences Page

Within the GitHub repo go to the Settings -> Secrets.
GitHub Settings Page

Then when on that page we will add a new secret and name it ANSIBLE_GALAXY_TOKEN

GitHub Secrets Page

Now when the Workflow runs it will grab this secret from your GitHub and be able to authenticate to Ansible Galaxy.

This section tells GitHub Actions to only run this workflow when a release is created. That is done in the GitHub UI, just like you did in the past to release a new version of a role.

steps:

- name: Check out code

uses: actions/checkout@v1

- name: Set up Python 3.8

uses: actions/setup-python@v1

with:

python-version: 3.8

- name: Install dependencies

run: |

python -m pip install --upgrade pip

pip install ansible

- name: Run role test

run: >-

ansible-playbook -i 'localhost,' build/galaxy_deploy.yml -e "github_tag=${{ github.ref }}"

This section:

checks out the code
configures python 3.8 on the host
installs the latest version of python pip
installs ansible
then runs the playbook with the github.ref value from the GitHub Release action

Once this is done you will have the release version uploaded automatically to your Ansible Galaxy account.