dockerfile Archives

Recently, I started working on more repositories on GitLab. One of the common items in my Ansible testing is the use of URL lookups in the templating of my Dockerfiles in Molecule. There’s a completely different method which requires the use of the GitLab API endpoints that require different formatting and token auth. The details for this can be found here: https://docs.gitlab.com/ee/api/repository_files.html#get-raw-file-from-repository

Searching around I did find that you can pass the token via the private_token parameter to the url.

Because you need to include the folder directory as an encoded value, I had to do lots of trial and error to figure out how to do complicated strings.

Formats like this, DO NOT WORK:

1	{{ lookup('url', 'https://gitlab.com/api/v4/projects/xxxxxx/repository/files/' ~ 'dockerfiles/{}/Dockerfile'.format(item.image)\|urlencode\|regex_replace('/','%2F') ~ '/raw?ref=main&private_token=' ~ lookup('ansible.builtin.env', 'GITLAB_TOKEN'), split_lines=False) }}

But after a series of attempts, THIS WORKS:

{% set file_path = 'dockerfiles/{}/Dockerfile'.format(item.image) %}

{% set encoded_path = file_path|urlencode|regex_replace('/','%2F') %}

{{ lookup('url', 'https://gitlab.com/api/v4/projects/xxxxxx/repository/files/' ~ encoded_path ~ '/raw?ref=main&private_token=' ~ lookup('ansible.builtin.env', 'GITLAB_TOKEN'), split_lines=False) }}

Some explanations of my findings urlencode filter did not work when used inline in the lookup, it made no changes to the file path. To separate, I had to split it out into a jinja set to set the var to a string that included the value using format() jinja filter, then take the result and create an encoded path to meet the encoded requirements of GitLab’s API.

I’d like to share with you another design in testing your Ansible collections, modules, playbooks, and roles. Molecule used to include a file name Dockerfile.j2. This template, in the past, created your docker container on execution. It’s since moved away from that and now only uses the base image you provide it via molecule.yml. In some cases, you need more than what the base image offers, and you may not want to create docker images and upload them to Docker Hub, or Quay.io. I wanted a solution and test that didn’t require people to download my docker images from Docker Hub.

Dockerfile.j2 with lots of Jinja

I prefer building my images using Dockerfile each time I test. It’s relatively quick and ensures that my host is testing against the latest packages that are installed by the Dockerfile.

However, I have lots of roles, and this means each role had at least one Dockerfile, and the Dockerfiles were precisely the same. A simple change to one Dockerfile usually said I needed to update all of the others. What if I need systemd installed? SystemD is different on many operating systems, different files needed, as well as various install commands. Well, I initially started building a more complicated Dockerfile.j2,which used the platform values from Molecule. But then after adding CentOS, Debian, Ubuntu, Fedora, and many of their different versions, it got complicated.

100

101

102

103

104

105

106

107

108

109

110

111

# Molecule managed

{% if item.registry is defined %}

FROM {{ item.registry.url }}/{{ item.image }}

{% else %}

FROM {{ item.image }}

{% endif %}

ENV container=docker

{# Initial Package Installs and Container Prep #}

{% if item.image.split(':', 1)[0] in ["ubuntu"] %}

RUN apt-get update \

&& apt-get install -y --no-install-recommends \

locales software-properties-common rsyslog systemd systemd-cron sudo \

iproute2

RUN sed -i 's/^$$ModLoad imklog$/#\1/' /etc/rsyslog.conf

{% elif item.image.split(':', 1)[0] in ["debian"] %}

RUN apt-get update \

&& apt-get install -y --no-install-recommends \

sudo systemd systemd-sysv \

build-essential wget

{% elif item.image.split(':', 1)[0] in ["centos"] %}

{% if item.image in ["centos:7"] %}

RUN yum makecache fast && yum -y install deltarpm \

{% elif item.image in ["centos:8"] %}

RUN yum makecache --timer \

{% endif %}

&& yum -y install epel-release \

&& yum -y update \

&& yum -y install sudo which

{% endif %}

{# Install of Python2 #}

{% if item.image in ["ubuntu:16.04"] %}

RUN apt-get update \

&& apt-get install -y --no-install-recommends python-setuptools wget \

&& wget https://bootstrap.pypa.io/get-pip.py \

&& python get-pip.py

{% elif item.image in ["debian:9"] %}

RUN apt-get update \

&& apt-get install -y --no-install-recommends libffi-dev libssl-dev \

python-pip python-dev python-setuptools python-wheel

{% elif item.image in ["centos:7"] %}

RUN yum -y install python-pip

{% endif %}

{# Install of Python3 #}

{% if item.image in ["ubuntu:18.04", "ubuntu:20.04", "debian:10"] %}

RUN apt-get update \

&& apt-get install -y --no-install-recommends \

apt-utils python3-setuptools python3-pip

{% endif %}

{% if item.image in ["centos:8"] %}

RUN yum -y install hostname python3 python3-pip

{% endif %}

{# Steps for cleanup #}

{% if item.image.split(':', 1)[0] in ["ubuntu", "debian"] %}

RUN rm -Rf /var/lib/apt/lists/* \

&& rm -Rf /usr/share/doc && rm -Rf /usr/share/man \

&& apt-get clean

{% elif item.image.split(':', 1)[0] in ["centos"] %}

RUN yum clean all

{% endif %}

{# Steps for clenaup of systemd #}

{% if item.image in ["centos:7", "centos:8", "debian:9"] %}

RUN (cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == \

systemd-tmpfiles-setup.service ] || rm -f $i; done); \

rm -f /lib/systemd/system/multi-user.target.wants/*;\

rm -f /etc/systemd/system/*.wants/*;\

rm -f /lib/systemd/system/local-fs.target.wants/*; \

rm -f /lib/systemd/system/sockets.target.wants/*udev*; \

rm -f /lib/systemd/system/sockets.target.wants/*initctl*; \

rm -f /lib/systemd/system/basic.target.wants/*;\

rm -f /lib/systemd/system/anaconda.target.wants/*; \

mkdir -p /run/systemd/system

{% endif %}

{% if item.image in ["ubuntu:18.04", "ubuntu:20.04"] %}

# Remove unnecessary getty and udev targets that result in high CPU usage when using

# multiple containers with Molecule (https://github.com/ansible/molecule/issues/1104)

RUN rm -f /lib/systemd/system/systemd*udev* \

&& rm -f /lib/systemd/system/getty.target

{% endif %}

{% if item.image in ["centos:7", "centos:8"] %}

# Disable requiretty.

RUN sed -i -e 's/^$Defaults\s*requiretty$/#--- \1/' /etc/sudoers

{% endif %}

{% if item.image.split(':', 1)[0] not in ["centos", "debian"] %}

# Fix potential UTF-8 errors with ansible-test.

RUN locale-gen en_US.UTF-8

{% endif %}

# Install Ansible inventory file.

RUN mkdir -p /etc/ansible

RUN echo "[local]\nlocalhost ansible_connection=local" > /etc/ansible/hosts

{% if item.image in ["centos:7", "centos:8", "debian:9", "debian:10"] %}

VOLUME ["/sys/fs/cgroup"]

{% elif item.image in ["ubuntu:16.04", "ubuntu:18.04", "ubuntu:20.04"] %}

VOLUME ["/sys/fs/cgroup", "/tmp", "/run"]

{% endif %}

{% if item.image in ["centos:7", "centos:8"] %}

CMD ["/usr/sbin/init"]

{% elif item.image in ["ubuntu:16.04", "ubuntu:18.04", "ubuntu:20.04", "debian:9", "debian:10"] %}

CMD ["/lib/systemd/systemd"]

{% endif %}

It was overly complicated, and I was losing track of the if/then statements, “Which OS should run which commands?” and many other questions. I gave up. It’s not maintainable. Especially when there have been PR’s adding support for SUSE, and ArchLinux, so now I need to add those to my tests. Three words. OUT OF HAND. So I had to change how I tested. I’m not going to duplicate a Dockerfile that’s this complicated, 10+ times per collection. Maybe I can do file links? That worked, but then I had to manage the same files in each of my Roles/Collections. Again, not scalable. I wanted something easy to do and easy to maintain and add new OS support when needed. Then a couple of things hit me.

Molecule Uses Ansible (obviously)
Ansible has Lookup Filters

URL Lookup for Dockerfile.j2

What if I could do a URL lookup against a GitHub repository that allows me to manage the same Dockerfiles for SystemD and Ansible dependencies on all of my roles. So, I deleted all the contents of Dockerfile.j2 and replaced it with this:

{{ lookup('url', 'https://raw.githubusercontent.com/ericsysmin/docker-ansible-images/master/' ~ item.image ~ '/Dockerfile', split_lines=False) }}

So each time Molecule runs, it connects to this file, grabs the Dockerfile, and then uses it to build each docker container used by Molecule. Now I can centrally manage all of my Dockerfile files, and simplify my Dockerfiles by removing all of the if/then statements, and other logic. This does require that your system running Molecule requires internet access to the file location, if it fails, the Molecule execution will also fail.

Now in each of my roles, throughout my collections and standalone, I can modify by Dockerfiles and manage them from one location just as if I decided to produce Docker images from these Dockerfiles and then share them on Docker Hub or Quay.io.

ericsysmin's DevOps Blog

Accessing Raw Files on Authenticated GitLab

Using a Dockerfile Repo for Molecule Dockerfiles

Dockerfile.j2 with lots of Jinja

URL Lookup for Dockerfile.j2

Eric Anderson

Accessing Raw Files on Authenticated GitLab

Share:

Using a Dockerfile Repo for Molecule Dockerfiles

Dockerfile.j2 with lots of Jinja

URL Lookup for Dockerfile.j2

Share: