Category: linux

  • Automatically update your Docker base images with watchtower

    I’m an avid user of Docker containers, using base images pulled from the public registry DockedHub. As you may know, Docker containers are based on Docked base images, e.g. I run postgres containers that are based on Postgres base image.

    It occurs that base images could get updated by their respective author (in our case Postgres team) and pushed to DockerHub. But your container does not benefit from this update unless:

    • you pull the new image
    • stop and delete your container
    • spawn another container using the new base image (of course I’m considering a very simple setup without clusters and Kubernetes).

    What if I tell you that there is a way to automate the process for you?

    Enter watchtower: a Docker container (inception!) to automatically restart your Docker container to use the most recent published base image; of course, watchtower checks regularly for any updates of the base image and pulls the new version if necessary.

    Configuration is none existent as you just have to follow watchtower’s instructions and launch the container: after that, you are all set!

    Anybody said drawbacks? Yes, there might be drawbacks. What if your container is restarted during a transaction? What if the new base image is unstable?

    These are all factors that you should take into account if you want watchtower to update your containers or not. In my case, for some applications that I run in containers, I value the comfort of having watchtower handle the updates enormously compared to the problems it may generate (so far: none).

  • Docker and containerd on openSUSE: reaching the limit for cgroup (and how to overcome it!)

    I recently encountered a limitation during an experiment I was conducting; after some trial and error, I recognized that the limitation was due to cgroups.

    But let’s start from the beginning. I open sourced docker-salt, a small pet project I had in mind in order to have a full blown setup for SaltStack: a master with an army of minions. Now for the fun part: what if I really start a hundred of minions on a server that has 16GB of RAM ready to be stressed with SaltStack?

    yankee:~ # docker run -d --hostname saltmaster --name saltmaster -v `pwd`/srv/salt:/srv/salt -p 8000:8000 -ti mbologna/saltstack-master
    yankee:~ # for i in {1..100}; do docker run -d --hostname saltminion$i --name saltminion$i --link saltmaster:salt mbologna/saltstack-minion ; done                                                                                        
    

    When reaching around the ~50th container created, Docker cannot start containers anymore:

    [...]
    a9e72a3b9452d1ff23628ab431e1b3127a0cbf218bfa179d602230f676e3740
    docker: Error response from daemon: containerd: container not started.
    a827de31439a2937ceebd8769e742038c395c9543e548071f36058789b9b144c
    docker: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:237: starting init process command caused \\\"fork/exec /proc/self/exe: resource temporarily unavailable\\\"\"\n".
    [...]
    

    By looking at the logs, we can see a more verbose message:

    yankee containerd[2072]: time="2017-04-20T22:59:10.608383236+02:00" level=error msg="containerd: start container" error="oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:243: running exec setns process for init caused \\\"exit status 6\\\"\"\n" id=aa642284b64dc97a519f6d33004d4a1468c13b9ef52bb05338fc09396631567f
    

    The problem here is that we reached the limit of the cgroup imposed for containerd, so we cannot fork any new process to spawn a new container.

    The solution is pretty easy: open /usr/lib/systemd/system/containerd.service and add the directive TasksMax=infinity to overcome the problem:

    [Service]
    [...]
    TasksMax=infinity
    [...]
    

    Issue a systemctl daemon-reload followed by systemctl restart containerd and you are good to go. Now the army of 100 minions can be started (sky is the limit!)

  • Secure your SSH server against brute-force attacks with Fail2ban

    The problem: SSH can be brute-forced

    I usually leave an SSH server on a dedicated port on every server I administer and, as you may recall, I even linked two well-written guides to properly configure and harden SSH services.

    Now, Internet is a notoriously bad place: scanners and exploiters have always been there, but brute-forcers are on the rise and SSH is one of the services that is heavily targeted by them. Let’s gather some data:

    quebec:/var/log # ag "Invalid user" auth.log.2 auth.log.3 | wc -l
    4560
    quebec:/var/log # head -n 1 auth.log.3 | cut -d " " -f 1-3
    May  8 07:39:01
    quebec:/var/log # tail -n 1 auth.log.2 | cut -d " " -f 1-3
    May 21 07:39:01
    

    So, even if my SSH is allowing only PubkeyAuthentication, in the timespan of two weeks there has been 4560 brute-force attemps (~325 attempts per day).

    This is annoying and potentially insecure, depending on your configuration. What can we do?

    The solution: using Fail2ban

    I have recently read some posts about this problem, and luckily (for us) there are multiple solutions to this problem: the most popular one is Fail2ban. To cut a long story short, the idea behind Fail2ban is to monitor log files of the monitored services and keep track of which IP addresses are trying to use a brute-force attack to use the service. If the same IP address causes a number of bad events in the specified time frame, Fail2ban bans that IP (using netfilter/iptables) for a configure d time amount.

    So I just need to install Fail2ban and I am ready to go:

    # zypper in fail2ban
    

    NO. NO. NO. You will not be protected against brute-force attacks if you just install it without configuring it.

    You MUST configure it!

    Let’s take a step back. The core of Fail2ban is the configuration:

    # "bantime" is the number of seconds that a host is banned.
    bantime  = 600
    
    # A host is banned if it has generated "maxretry" during the last "findtime"
    # seconds.
    findtime = 600
    maxretry = 5
    

    These are the default values. And brute-forcers know them, so they can time accordingly their attempts not to break these limits (or to begin again their attempts after the bantime). Again, let’s gather some data. I noticed a frequent brute-forcer IP and follow his data:

    quebec:/var/log # ag "Invalid user" auth.log.2 auth.log.3 | ag <IP> | cut -d " " -f 1-3
    
    auth.log.2:8106:May 21 03:34:22
    auth.log.2:8112:May 21 03:43:41
    auth.log.2:8116:May 21 03:53:00
    auth.log.2:8120:May 21 04:02:18
    auth.log.2:8126:May 21 04:11:47
    auth.log.2:8132:May 21 04:21:08
    

    Can you believe it? It was just on the edge of the 600 seconds between every attempt!

    Key concept: outsmarting brute-forcers

    The key concept here is to provision a personalized version of your specific choosing of these values, in order to outsmart the brute-forcers (which is easily done). In order to do so, do not modify Fail2ban’s default config file (/etc/fail2ban/fail2ban.conf) but rather just override the defaults in another config file that you can create with:

    quebec:/etc/jail2ban # awk '{ printf "# "; print; }' /etc/fail2ban/jail.conf | sudo tee /etc/fail2ban/jail.local
    

    Your customizations have to be defined in /etc/fail2ban/jail.local, so your selected bantime, findtime and maxretry must go there.

    Further customization

    Some useful findings:

    • you can selectively whitelist a group of IPs, hosts, subnets, etc. in order to not being banned when accessing services from whitelisted IPs (e.g. ignoreip = <VPN subnet>/8 <another VPN subnet>/16)
    • you can receive an email everytime someone is banned with their whois (with good relief from the system administrator)
    • if you are using WordPress, you can extend Fail2ban to monitor failed WordPress login attempts with a WordPress plugin (did I mention that Fail2ban not only monitors sshd? It also monitors nginx, apache, vsftpd and other services)
    • you can have Fail2ban automatically sends the banned IPs to a shared blacklist (I have never used this)

    Disadvantages

    Everything seems perfect now, but what are the disadvantages of it? Given that Fail2ban reasons in terms of banning single IPs, it cannot protect you against distributed brute-force attacks. In this case Fail2ban is pretty useless and other solutions should be implemented that depends from case to case.

  • OpenSUSE Leap 42.2: this is how I work (my setup)

    Motivation

    I switched my distribution of choice to OpenSUSE. There are a lot of motivations behind this choice:

    1. I wanted an enterprise-grade quality of software in terms of stability, package choice, and supportability
    2. Growing interest in software non-distribution specific and/or customized, e.g. Gnome
    3. Dogfooding

    After nearly one year of usage, I can say that I am mostly satisfied with the setup I built.

    In this post I will cover a step-by-step advanced installation of OpenSUSE: we are going to mimic the exact setup I have on my machine. I want to share the setup first of all for myself, keeping track on why I did some decisions back then, and secondly for you, fellow readers: you can follow my example and setup a successful OpenSUSE box.

    Leap or Tumbleweed?

    OpenSUSE comes in two variants:

    • Leap: represent the stable branch. It’s a fixed schedule release, which means that a release comes out from time to time. You pick a release, and install it: every update is based on the release version.

    • Tumbleweed: represent the bleeding-edge branch, and it’s a rolling release (which means that you install a snapshot of that branch and apply updates from there on).

    On my machine, I always want stability over the bleeding-edge version of the latest package, so I choose Leap. Leap has version number 42: as the time of writing, two releases have been made available to the public:

    • 42.1 (released 2015-11-04)
    • 42.2 (released 2016-11-16)

    Let’s download 42.2 (the most recent one), burn it to a USB key (or a DVD, if your computer still has it) and follow the instructions.

    This post will not cover every choice. I will just point out what I changed from the default. If nothing is mentioned here, it means I followed the default.

    Installation choices

    Offline install

    Install the distribution in offline mode: if you use your laptop in clamshell mode, disconnect everything (even network): I want to install the distribution as it has been released (potential updates will be applied after the installation).

    Network Settings

    I enforce a static hostname and domain name. Feel free to name your machine and domain name, and check “Assign Hostname to Loopback IP”.

    hostname selection

    Partition layout

    Base partitioning: /boot and LVM

    For maximum compatibility, I want my hard drive partitioned with:

    • a primary partition that will contain /boot
    • an extended partition with 0x8E (Linux LVM) with a system Volume Group (VG) and at least two Logical Volumes (LV)

    partitioning hard drive

    /boot

    /boot should be an ext2 partition type and separated from the LVM partition (in case I need to take out my hard-drive and insert in another computer, this ensures compatibility with legacy BIOSes and older computers). The partition should be sized (roughly) at ~500 MB – I choose 487 MiB (1 Megabyte = 0.953674 Mebibyte).

    Linux LVM

    I use LVM everywhere for easiness of partition shrinking, growing, moving, etc. There is no motivation for not using it. If your BIOS support hard drive encryption, enable it there. If not, use encrypted LVM.

    LVM should have a VG named system that must have two LVs:

    • root that will contain all your data (I normally do not need a separated /home partition)
    • swap that will act as swap (this will be useful when you use suspend).

    For a system with more than one hard drive, I also create another VG (e.g. storage), or add them to system. Unless you use XFS, there is no need to do a final decision here (more on this in the following paragraph).

    root LV file-system

    I am a great fan of XFS. Over the many advantages of it, there is one major disadvantage: an XFS partition cannot be shrunk.

    So, think carefully here: if you think you are going to shrink your partition in the future for every reason, I would advise against XFS. Otherwise, go for XFS.

    In my experience, the aforementioned is non-existent for servers, although it can happen for desktop and laptop machines. For my main system I will not choose XFS, thus I will go with ext4.

    swap LV size

    A long time ago we reserved twice the size of RAM to the swap partition. Nowadays most computers have >= 8 GB of RAM, so I will just choose the same amount of RAM size for my swap partition.

    Clock and Time Zone

    I synchronize my system with an NTP server, and I chose to run NTP as a daemon for my system, saving the configuration.

    clock and timezone

    Desktop selection

    I usually go with Gnome or XFCE (it is a personal preference here so feel free to choose another one). During our customization (in the next post) we are going to also install i3, another great window manager that I like a lot.

    Local User

    My local user should be distinguished from the system administrator (root) account, so I deselected “Use this password for system administrator”. Of course, this will mean that root account will have another (different!) password.

    And I also do not want automatic login.

    local user creation

    Boot Loader Settings

    GRUB2 should be installed into Master Boot Record (MBR) and not into /boot. If you are installing via USB key, make sure to remove it from “Boot order”. Optional: set timeout to 0 in “Bootloader Options” (so you do not have to wait for booting).

    grub configuration disk order

    grub configuration timeout

    Software

    Just go with the default selection, or, if you cannot wait, go ahead and select packages. There is no rush, though: we will install packages that I need in the next post, during the customization phase.

    Firewall and SSH

    I suggest to unblock SSH port and enable SSH service. WARNING: make sure to config your ssh daemon properly in order to allow only key-based logins.

    summary of installation 1

    summary of installation 2

    Conclusion

    After the installation, we have a plain Gnome environment ready to rock. In the following post, we are going to customize every bit of it, installing all the packages that I think fundamental. Stay tuned!

    opensuse 42.2 desktop

  • OpenVPN with multiple configurations (TCP/UDP) on the same host (with systemd)

    OpenVPN with multiple configurations (TCP/UDP) on the same host (with systemd)

    As much more people are getting worried about their online privacy (including me), I started to use a server as a VPN termination (with OpenVPN) when I need to access the Internet via non-secure wired or wireless networks (e.g., hotel wireless network, airport Wi-Fi, etc.).

    Some overzealous network admins, though, try to lock down the network usage to users, for understandable reasons: fair usage, fear of abuse, and so on. To name some of such limitations:

    • non-encrypted traffic sniffing (who trusts HTTP nowadays for sensitive data? Surprisingly, there is still someone who deploys HTTP for that!);
    • traffic shaping (especially downstream);
    • destination ports limited to 80/tcp and 443/tcp;
    • dns locking and consequently leaking (yes, I’m paranoid).

    To overcome this limitations, I decided to use multiple configurations for OpenVPN, I wanted some flexibility on my side, offering multiple configurations of a VPN termination: one for TCP and one for UDP. I want to share some implementation notes that might save some time for whoever wants the same setup:

    • TCP subnets must be separated from UDP subnets (I use a /24 for each one; take a look at IANA Reserved addresses and do your math);
    • You can use the same tun adapter for both servers at the same time.

    Now for the tricky part:

    • Most OpenVPN implementations (depends on your distro) require that you supply a configuration file. In our case, we prepare two config files (one for TCP and one for UDP) under /etc/openvpn
    /etc/openvpn # ls *.conf
    tcp-server.conf  udp-server.conf
    • systemd must be informed on which configuration it must start whenever openvpn is launched via its service unit. To accomplish that, open /etc/default/openvpn and specify the VPN configurations that must be started:
    # Start only these VPNs automatically via init script.
    # Allowed values are "all", "none" or space separated list of
    # names of the VPNs. If empty, "all" is assumed.
    # The VPN name refers to the VPN configutation file name.
    # i.e. "home" would be /etc/openvpn/home.conf
    #
    # If you're running systemd, changing this variable will
    # require running "systemctl daemon-reload" followed by
    # a restart of the openvpn service (if you removed entries
    # you may have to stop those manually)
    #
    AUTOSTART="tcp-server udp-server"
    • Finally, we need to reload systemd as instructed above:
    # systemctl daemon-reload
    • Now, if you restart OpenVPN with systemctl restart openvpn and you check your logs, you should see that both your VPN are started:
      11:38:33 vpn02.lin.michelebologna.net systemd[1]: Starting OpenVPN connection to tcp-server...
      11:38:33 vpn02.lin.michelebologna.net systemd[1]: Starting OpenVPN connection to udp-server...
      11:38:33 vpn02.lin.michelebologna.net systemd[1]: Started OpenVPN connection to tcp-server.
      11:38:33 vpn02.lin.michelebologna.net systemd[1]: Started OpenVPN connection to udp-server.

      and you can also check that OpenVPN is listening with netstat:

      # netstat -plunt | grep -i openvpn
      tcp 0 0 0.0.0.0:1194 0.0.0.0:* LISTEN 1635/openvpn
      udp 0 0 0.0.0.0:1194 0.0.0.0:* 1644/openvpn

  • PSA: this website now is TLS-enabled

    After some thinking, I decided to switch my current domain registrar and hoster: in fact, I stayed for 5 years with Netsons.org for domain registration and hosting. I had a very pleasant experience with them, I will recommend their hosting to everyone (it’s very cheap in the plethora of Italian super-expensive hosters).

    Since I recently got a with at DigitalOcean, I thought: “hey, maybe I can host myself!”, and that’s what I have done: my new registrar will be namecheap, and my new hoster will be… me!

    Switched registrar, transferred the domain changed DNS to point to DigitalOcean, and that’s where the fun begins: configuring a webserver to serve my website.

    Why is that fun? Because I get the chance to decide whichever technologies of the L(A|E)MP stack can I use. In fact:

    • After years of Apache, I switched to nginx (with TLS and spdy HTTP2 enabled)
    • With some reckless considerations beforehand, I decided to switch to PHP7 (instead of 5) with php7.0-fpm
    • MySQL is the database of choice (no changes here)
    • TLS support: encryption everywhere, yay!
    • TLS certificate: obviously, I’d try (Let’s Encrypt)[https://letsencrypt.org/]

    After some headaches (and a little downtime), I finally managed to self-host myself: if you are seeing this, it means that you are automatically redirected to the encrypted version of this site (courtesy of nginx), and that the certificate is trusted (by the chain of Let’s Encrypt).

    Enjoy!

  • Packaging software for Debian/Ubuntu: eclipse

    Eclipse is my (Java, Python, Ruby, XML, <insert any other text format here) editor of choice, and it has been for many years. One thing that bothers me is that Eclipse package is outdated in Ubuntu: so, instead of using apt, I should resort to download/unpack/copy/create links to install it. These days are finished, though.

    In fact, I have been introduced to Debian packaging and I contributed to the Debian package of the latest version of the Eclipse IDE (4.5.1). EDIT: Repository has been removed as obsolete.

    This package is really simple (and in fact I used it to learn the packaging process for Debian/Ubuntu). How did I learn it? Recommended reading: How to package for Debian.

    In the following days I will try to publish a PPA with the built package. In the meanwhile, if you want to try to build the package on your own, just: 1. git clone -b eclipse_4.5.1
    2. cd eclipse-ide-java
    3. cd eclipse-ide-java_4.5.1
    4. debuild -i -us -uc -b
    5. cd ..

    Now you have a *.deb package waiting for you to be installed (via dpkg -i): upon installing it will fetch (via wget) the latest version of Eclipse, unpack, copy and create links.

  • Workaround for OpenVPN PAM authentication broken on Ubuntu 15.10

    After updating to Ubuntu 15.10 a box with an OpenVPN termination I am using to browse when I travel and use insecure networks, my VPN tunnel stops working. I am using, in this particular box, an OpenVPN server that relies on PAM plugin for authentication (and 2-step verification).

    Given the fact that I keep all my configuration files under etckeeper, the problem determination began with some git log under my /etc directory, both on server and client. Obviously, no configuration has changed during the upgrade.

    The problem has to be somewhere. I had a look at the logs:

    12:47:46 ovpn-3-rtr.bgo ovpn-server[982]: x.x.8.234:64484 TLS: Initial packet from [AF_INET]x.x.8.234:64484
    12:47:48 ovpn-3-rtr.bgo ovpn-server[982]: x.x.8.234:64484 PLUGIN_CALL: POST /usr/lib/openvpn/openvpn-plugin-auth-pam.so/PLUGIN_AUTH_USER_PASS_VERIFY status=1 
    00:47:48 ovpn-3-rtr.bgo ovpn-server[982]: x.x.8.234:64484 PLUGIN_CALL: plugin function PLUGIN_AUTH_USER_PASS_VERIFY failed with status 1: /usr/lib/openvpn/openvpn-plugin-auth-pam.so
    12:47:48 ovpn-3-rtr.bgo ovpn-server[982]: x.x.8.234:64484 TLS Auth Error: Auth Username/Password verification failed for peer
    12:47:50 ovpn-3-rtr.bgo ovpn-server[982]: x.x.8.234:64484 SENT CONTROL [UNDEF]: 'AUTH_FAILED' (status=1)
    12:47:50 ovpn-3-rtr.bgo ovpn-server[982]: x.x.8.234:64484 Connection reset, restarting [0]
    12:47:50 ovpn-3-rtr.bgo ovpn-server[982]: x.x.8.234:64484 SIGUSR1[soft,connection-reset] received, client-instance restarting
    

    (obviously I was providing the correct username and password).

    Ok, the problem was occurring with PAM plugin. After some research and trial, I came across Bug #1511524 “OpenVPN PAM authentication broken on 15.10 Server” : Bugs : openvpn package : Ubuntu: that is caused by a bug in Ubuntu package of OpenVPN (and specifically in OpenVPN systemd unit file).

    As described in the bug, you have three ways to restore a normal situation. Either:

    • stop the daemon and launch OpenVPN daemon
    • modify /lib/systemd/system/openvpn@.service and add CAP_AUDIT_WRITE to CapabilityBoundingSet property
    • or you can just wait while they ship a package with a correct systemd unit file.

    Don’t forget to systemctl restart openvpn to apply changes and use your VPN:

    13:03:49 ovpn-3-rtr.bgo ovpn-server[5186]: x.x.10.176:61423 PLUGIN_CALL: POST /usr/lib/openvpn/openvpn-plugin-auth-pam.so/PLUGIN_AUTH_USER_PASS_VERIFY status=0 
    13:03:49 ovpn-3-rtr.bgo ovpn-server[5186]: x.x.10.176:61423 TLS: Username/Password authentication succeeded for username 'x'
    
  • Getting started with cloud-init and CoreOS

    Lately I’ve been experimenting with CoreOS, a Linux distribution that enforces containerization (I made some experiments with Docker and I’d say that this area is fun!). CoreOS layer of containerization was based Docker, but now that they moved to Rocket. Not only Rocket, though: CoreOS brings some curious innovations to lightweight Linux distributions like clusterization with fleet and service discovery with etcd.

    Since I had to do some tweaks to run CoreOS in a virtual machine (VMware Fusion on OSX is the Type 2 hypervisor I used), I decided to write this post to better illustrate how should you customize CoreOS for the first run.

    First of all, download CoreOS! After importing it in VMware, the tricky part comes in. CoreOS is heavily focused on automation, so you have either two choices to login into your newly created CoreOS machine:

    • boot CoreOS kernel with coreos.autologin parameter active (debug)
    • prepare a cloud-init package that contains your customization. In this case, your ssh pubkey fingerprint. (preferred)

    Let’s see both ways.

    Boot CoreOS with coreos.autologin

    When you see the GRUB menu, edit the current entry and add coreos.autologin

    coreos.autologin

    and you are good to go.

    Note: that with this method only interactive local logins are allowed.

    Prepare a cloud-init package

    This one seems difficult, but it’s not! First of all: cloud-init is a set of scripts that customize a Linux distribution during boot: you can add users, groups, packages and execute commands before the Linux box comes up. CoreOS ships with cloud-init by default, so we only have to:

    1. Write a simple config file (cloud-config)
    2. Package the config into a config-drive (basically, an .iso file)
    3. Mount that iso as a drive for our virtual machine and reboot CoreOS to make your customizations effective

    Write a simple config file

    In our config we will:

    1. Add our ssh pubkey (be sure to have one, or generate it right now)
    2. Set the hostname (who wants a generic hostname anyway?)

    The cloud-config is straightforward: a YAML file where only the first line is equal for everyone:

    # cloud-config 
    ssh_authorized_keys: 
        - "ssh-rsa ... michele@fortknox" 
    hostname: "coreos-test"
    

    Remember to customize it with your pubkey fingerprint and the hostname you want and save it as user_data.

    Package cloud-config into a config-drive

    Now that we have a cloud-config file, we have to package it as a config-drive and make it available as a drive for CoreOS. Since I needed to repeat this process a couple of times, I decided to automate it and I wrote a simple script: configdrive_creator. Be sure to read the instruction: you prepare the config, put it in the same directory of the script, launch the script and the iso is created.

    Mount the iso file as a drive for CoreOS

    I am sure you are aware to do it on your own! After rebooting your CoreOS VM, you can finally ssh into it:

    coreos_ssh

    Happy hacking!

  • Playing with Docker: tips and tricks to write effective Dockerfiles

    Recently I have been playing with Docker containers, and I am sure you already know what Docker is. In this post I will describe what I have learnt while using Docker containers and preparing Dockerfiles.

    What is Docker?

    In a few words: Docker is a software to manage and run Linux containers in which you can deploy an application using Dockerfiles. The main concept here is divide-et-impera concept: a Docker container is just like a virtual machine, except that is very lightweight (it is only a container, so it is not virtualizing an entire machine).

    How do I run a Docker container?

    1. Install Docker
    2. docker run container command (e.g. docker run -ti fedora bash)

    Docker will fetch the container (from DockerHub, a central repo for Docker containers) and run the specified command.

    What is a Dockerfile?

    A Dockerfile is a set of instructions to prepare a Docker container on your own. It basically declare a base image (like a Linux distribution: for example Fedora, Ubuntu, etc.) and apply modifications on that container (fetch the software you want to run on container, compile it, run, etc). There is a basic set of instruction for Dockerfiles. For all Vagrant users: that’s right, a Dockerfile is just like a Vagrantfile: it states the steps to prepare a machine (except that in this case we are preparing a container).

    How do I use a Dockerfile?

    1. Install Docker
    2. Download a Dockerfile
    3. (if applicable): edit any files that will be copied from host to container. This is useful for configuration files if you want to customize your Dockerized application
    4. docker build -t containername .

    Docker will reproduce the steps to create the container, using the instructions found on the Dockerfile. After it has finished, you can run the Docker container as specified above.

    Docker: from theory to practice

    Ok, theory aside. I decided to create a Dockerfile for two applications because:

    • the application was not available from the official repos (e.g. alice)
    • the version in the official repos is outdated (e.g. bitlbee)

    Basically, we will declare two Docker containers in which we fetch our software, customize to our needs and run it inside the container. Both of them will declare a service, and the container will serve as a server for the application (alice/http and bitlbee/irc).

    bitlbee Dockerfile

    In this case we are using my preferred base image which is Fedora, we customize it to be able to fetch and compile the source code of bitlbee and then proceed to compile it. In this Dockerfile we also ADD two configuration files from the host to the Dockerfile. Again, we launch the service as daemon user and expose the 6667/tcp port. The final size of the Docker container image is 359MB.

    To use it, connect your IRC client to localhost:6667 (remember to map the correct port, see below).

    bitlbee Dockerfile on GitHub.

    Tips and caveats

    Docker

    First of all, some tips I learnt:

    • When running a container, it is always best to launch it with a name (it easier to reference the container afterwards): docker run --name bitlbee_container mbologna/docker/bitlbee
    • If you want to detach a container, supply -d option when running
    • You can inspect a running container by attaching to it: docker exec -ti bitlbee bash
    • Remember to clean up docker images and docker containers: show them with docker images and docker ps -a. Remove them with docker rmi and docker rm
    • If you are running Docker containers as a service (like in this example), you should remember to set the option --restart=always to make sure that your Docker container is started at boot and whenever it exits abnormally

    Everything on the docker container makes it apart from the host machine under all points of view (network, fs, etc.). Thus:

    • When using Docker containers (in particular you are running a service inside a Docker container), you can access your container ports by mapping the ports on the containers to ports on your host using the -p option: docker run -p 16667:6667 mbologna/docker-bitlbee (container maps 16667 port on the host machine to 6667 port on the container, so it can be accessed at 16667/tcp on the host machine)
    • When a container is restarted, everything on the container is reset (speaking of file-system too). In order to write non-volatile files, you should supply -v option that declares a volume; as with ports we have seen above, you specify first the directory on host and then the corresponding directory on the container. This is useful for config files (you want to keep them, right?): docker run -v /home/mbologna/docker-bitlee/var/lib/bitlbee:/var/lib/bitlbee mbologna/docker-bitlbee

    Dockerfiles

    • If you define a VOLUME in the Dockerfile:
      • if user is launching Docker container without specifying a volume, VOLUME directory will typically resides under /var/lib/docker/volumes (you can discover it using docker inspect <container>)
      • otherwise, VOLUME directory will resides on the specified directory using -v option.

      This exposes an issue of permissions on the VOLUME directory. I basically solved it by chowning twice the volume directory, otherwise either one of the two cases described above wouldn’t have the correct permissions: chown -R daemon:daemon /var/lib/bitlbee* # dup: otherwise it won't be chown'ed when using volumes
      VOLUME ["/var/lib/bitlbee"]
      chown -R daemon:daemon /var/lib/bitlbee* # dup: otherwise it won't be chown'ed when using volumes

    • When a final user pulls a container, it basically downloads your container from DockerHub. That’s why we want to minimize Docker container size. How can we do that when preparing a Dockerfile?
      • Every command you launch on a Dockerfile creates a new (intermediate) Docker container (the final result will be the application of every instruction on top of the instruction above it!) => minimize steps and group commands under RUN commands using &&. E.g.:

        RUN touch /var/run/bitlbee.pid && \
        chown daemon:daemon /var/run/bitlbee.pid && \
        chown -R daemon:daemon /usr/local/etc/* && \
        chown -R daemon:daemon /var/lib/bitlbee*

      • After you compiled/installed your software, be sure to remove it if unnecessary to clean up space: apt-get clean && \
        apt-get autoremove -y --purge make \
        rm -fr /var/lib/apt/lists/*

    • Every command you launch on the Docker container is run as root: be sure, before launching your software, to launch it with as minimal privileges as possible Principle of least privilege. For example, I launch alice and bitlbee daemons with the daemon user: USER daemon
      EXPOSE 8080
      CMD ["/home/alice/alice/bin/alice", "-a", "0.0.0.0"]

    Contributing

    You can pull my Docker containers on DockerHub:

    You can browse my Dockerfiles on GitHub:

    Future work

    Two interesting concepts I came across during my research and I will investigate in the future:

    • CoreOS, a Linux distribution in which every application is launched on a separate Docker container
    • Kubernetes, an orchestration layer for Docker containers