Introduction Link to heading

For a few years I’ve been installing and configuring services at a company I contract for, exposing them internally with self-signed certificates most of the time due to the lack of an internal CA or mean to get custom certificates. As one could imagine, managing certificate validity (revocations as well as expiration), certificate bundles, trust stores, etc. was a real nightmare and lots of unsecured shortcuts were used to ease the burden on the few operators maintaining said services.

I’ve been meaning to change that for a long time, since it rapidly became unsustainable for me and for a few others inside this organization and the complaint of the few finally gained enough attention that I was able to start working on this issue.

The subject being vast: from the CA generation to its hosting, I decided to start a series of posts about it, as much as to document my process than to help others that could stumble on the same kind of issues I had: Self hosted PKI.

Following post on this series will cover

  • Root and intermediate CA initialisation and configuration with CFSSL;
  • CA, CRL and ocsp refreshing and hosting on kubernetes;
  • CA cert deployment on linux with ansible;
  • Certificate signing API hosted on kubernetes;
  • Configuration of cert-manager, trust-manager and use with ingresses.

Where will it run Link to heading

While I did implement a PKI at work, in this series I’ll be implementing a new one running in a temporary environment in my home-lab to prevent critical information from leaking. This home-lab consists of a bunch of hosts running on archlinux/archlinux-arm: network file system shares are hosted on two armv7l hosts, my router/dns/dhcp server architecture is amd64 and my kubernetes cluster is composed of five arm64 hosts.

The PKI will be hosted on this kubernetes cluster where I manage ingresses with traefik. Other hosts on my network will have to trust this self-hosted/self-signed CA.

Toolkit Link to heading

I looked at a few options to run the PKI. If I remember correctly I did read documentation and tried to fiddle with:

  • OpenSSL

    I discarded the idea of only using openssl since I needed to serve an API for signing and didn’t want to develop a custom server due to all the security/best practices implications and the added complexity.

  • Let’s Encrypt Boulder: https://github.com/letsencrypt/boulder

    Let’s Encrypt recommends other organization use another tool since this one is tailored to their requirements. Also, the company I contract for is not in the business of web hosting nor is a CA provider by trade. Using Boulder would probably be too complicated and a bit overkill.

  • HashiCorp Vault: https://www.hashicorp.com/products/vault/pki-with-vault

    I spent quite some time on this one since I already uses Vault with other secret engines but while it is a great product I rely on heavily for other tasks, after much consideration I decided to work with something else my coworkers would be more comfortable with.

  • Cloudflare cfssl: https://github.com/cloudflare/cfssl

    I finally settled on cfssl after I saw how easy it was to start and initialize a CA, both locally and using a remote cfssl instance. Due to this project’s lack of documentation my feelings quickly evolved into a mixture of love and hate. Fortunately, cfssl’s source code is quite easy to read and was a great help in understanding configuration options, arguments and parameters.

Conclusion Link to heading

When I started working on this project, I had but the most basic knowledge on how a CA worked, and I’ve learnt a lot through reviewing code, reading documentation and from lots of illuminating talks with some great peers. Which is to say: if you think I’ve made mistakes, I urge you, do not hesitate to contact me, so I can learn and fix them.