Thanks to letsencrypt it got way too easy to switch to a secure website, I ran out of excuses. So now, in all it's glory https://herbert.poul.at with a green lock, isn't that amazing? ;-)



It also only involved some small docker, DNS and certbot magic to get it all going. The hardest part was understanding that to use bind9 you would have to use the rfc2136 plugin. everything else is really well documented. Check out their documentation it's really well documented but it boils down to:

1. Configure a key to allow TXT dns changes

  update-policy {
    grant letsencrypt.key name _acme-challenge.poul.at. txt;
  };

(and make sure that your slave DNS are notified of changes)

2. Getting the certificate

Since I hate bloating my servers with software, I didn't like the thought of installing additional tools for certbot. But luckily, they have nice docker builds, so nothing is required, except docker. Which, luckily, I am already using exstensively. So I just created a little script:

docker run -it --rm \
    --name certbot \
    -v "`pwd`/etc/letsencrypt:/etc/letsencrypt" \
    -v "`pwd`/var/lib/letsencrypt:/var/lib/letsencrypt" \
    -v "`pwd`/var/log:/var/log" \
    -v "`pwd`/secret:/secret" \
    certbot/dns-rfc2136 certonly \
    --dns-rfc2136 \
    --dns-rfc2136-credentials /secret/rfc2136.ini \
    -d '*.poul.at'

3. Getting certificates to nginx

Now that the certificates are in etc/letsencrypt i just use rsync to copy them to the web server, and i'm ready to go.

The only thing to keep in mind are file permissions. certbot will create them only readable by root. But I don't really like logging into other systems as root, so permissions are not correctly synced by the default rsync setup, so i'm using --chmod option.

rsync -e "$SSH" -a --chmod=go-rwx etc/ $USER@$PROXY:letsencrypt-etc/

$SSH $USER@$PROXY sudo /etc/init.d/nginx reload

4. nginx

That's again very well documented and easy to do. (no surprises there)

server {
  server_name herbert.poul.at;
  return 301 https://herbert.poul.at$request_uri;
}

server {
  listen 192.168.9.5:443 spdy;
  server_name herbert.poul.at;

  ssl on;
  ssl_certificate XXX/letsencrypt-etc/letsencrypt/live/poul.at/fullchain.pem;
  ssl_certificate_key XXX/letsencrypt-etc/letsencrypt/live/poul.at/privkey.pem;

Just so I can remember a few commands.
If you ever happen to stumble upon kernel panics from a KVM guest after upgrading.. maybe a few commands which might be userful. (I have upgraded from debian 7 wheezy to debian 9 stretch) and encountered nice kernel panics.

Forcefully shutting down KVM

Even though it sounds scary. The correct way to just cut the power to your KVM guest is to use `destroy`.

root@host3 / # virsh start services

Mount file system from guest

I am using a LVM volume which itself has a partition table written from the guest system. So I can't mount it directly. Luckily with a bit of googling I found a helpful website: https://backdrift.org/mounting-a-file-system-on-a-partition-inside-of-an-lvm-volume

root@host3 / # kpartx -a /dev/vg0/services
root@host3 / # file -s /dev/mapper/vg0-services1 
/dev/mapper/vg0-services1: symbolic link to ../dm-13
root@host3 / # mount /dev/mapper/vg0-services1 /mnt

Updating grub config

root@host3 / # cd /mnt
root@host3 /mnt # mount -t proc proc proc/
root@host3 /mnt # mount -t sysfs sys sys/
root@host3 /mnt # mount -o bind /dev dev
root@host3 /mnt # chroot .

# Now change /etc/grub.conf to boot old kernel (or just delay automatic selection)

root@host3:/# update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.9.0-7-amd64
Found initrd image: /boot/initrd.img-4.9.0-7-amd64
Found linux image: /boot/vmlinuz-3.16.0-6-amd64
Found initrd image: /boot/initrd.img-3.16.0-6-amd64
Found linux image: /boot/vmlinuz-3.2.0-4-amd64
Found initrd image: /boot/initrd.img-3.2.0-4-amd64
done
root@host3:/#

🎉️🎉️🎉️🎉️🎉️

root@host3:/# exit
root@host3 /mnt # umount proc
root@host3 /mnt # umount sys
root@host3 /mnt # umount dev
root@host3 /mnt # cd /
root@host3 / # umount /mnt
root@host3 / # kpartx -d /dev/vg0/services
root@host3 / # virsh start services