As I mentioned in my last post, I’ve been working through a book on Flask: Flask Web Development, by Miguel Grinberg. I’ve made it through Chapter 14, which was all about RESTful APIs. Those APIs were authenticated by passing the userid and password in the headers to an initial request. That is, of course, a security problem if the headers are sent in the clear, via HTTP. So that got me thinking about HTTPS and how to support it from my development web server.

I decided to take a break from the book and look into using HTTPS on my interal servers and services. And that generated a set of TILs:

  1. Create a CA cert on local machine
  2. Create server certificate (using local CA)
  3. Add a CA certificate to the OS X keychain
  4. Add CA certificate to Firefox

I’ll reference those TILs in this post.

The need for a CA certificate

X509 Certificates are required by the TLS handshaking to establish an HTTPS connection. A certificate authenticates an endpoint and provides a public key that is to be used in the TLS handshaking. It identifies the domain or domains to which the certificate applies, and identifies how the public key can be used.

A certificate will be cryptographically signed and the client must be able to verify that signature in order to trust the certificate. This means that the client must have a set of keys that it trusts, keys that are used to verify the cryptographic signatures of the server certificates it encounters.

The trivial case is that the client has a copy of valid certificates for every service that it communicates with. This is what is required for self-signed certificates. But this quickly breaks down because there may be an unbounded set of services a client is interested, making the management of each client’s trusted certificate store very onerous.

The solution to that problem is for each client to trust a small number of certificates, and have the various services publish certificates signed by one or more of those trusted certificates. Clients would verify the validity of server certificates by checking the cryptographic signatures using the public keys of well-known trusted certificates. The number of server certificates can remain unbounded, while the store of trusted certificates remains small, changes infrequently and is easily managed. This is the idea behind Certificate Authorities and CA certificates.

The need for a private CA certificate

A variety of commercial (DigiCert) and non-profit (Let’s Encrypt) organizations are Certificate Authorities and will generate and sign server certificates that you can use. Using a public CA can get expensive, however, and generating a new server certificate is tediously manual in most cases. For services that are not public, that only exist on private networks or perhaps within a single machine (e.g. a developer’s workstation), using a public CA may be impossible and is usually impractical.

It may be better to use a private Certificate Authority, operating within the private network, and having all the clients on the private network trust the CA certificates of that private CA. Doing this bypasses the cost issues of commercial CAs, and bypasses requirements of publicly accessible resources (needed by many CAs to validate that the certificate requester has ownership of the server to be protected by the requested certificate). It also allows the private network administrators to implement bespoke certificate management policies and automate the management and distribution of server certificates.

Establishing the CA and generating the CA certificate

It turns out that establishing and using a private CA is fairly easy to do, at least in a trivial way, if you have a Unix or Linux system.1 OpenSSL gets installed with the basics for running a private CA.

Note: What I’m describing here is a manual process, aided by pretty good scripting, but still a manual process. Automating certificate generation and renewal is a topic of great interest to me, but a topic for a later time.

I wrote the TIL “Create a CA cert on local machine” that describes how to setup a CA and generate a CA certificate.2 Included in the TIL is a script that drives the generation of the CA certificate. OpenSSL comes with it’s own “CA” script, but I wanted more control over the process and find my approach and my script to be better. Eventually I want to automate much of the internal CA’s operation, so building my own script is a great way to get familiar with the technical nitty gritty of certificate generation and CA maintenance.

Rather than my reiterating all the detail here, just read the TIL for the “how to”.

CA certificate validity period

I want to call out my philosophy for managing CA certificates in a private network environment. This is discussed in the TIL, but worth repeating here.

Certificates expire. They have a “valid after” timestamp and “valid before” timestamp. When working with public CAs, I find that the CA certificates are usually valid for a 10 year period, sometimes much longer, and server certificates are valid for a 1 year period (or several years for “extended validation” certs); Let’s Encrypt generates certs that expire after 90 days.

That means you need a way to issue and install new certificates as the existing ones expire. This is true for both server certs and CA certs. For a private network, I think it makes sense to use relatively short validity periods for certs. Why?

  • You probably aren’t going to maintain and publish a CRL on an internal network. Therefore certificate expiration is the only way to “get rid of” certificates that are no longer used, are mis-configured, or for any other reason should no longer be used.
  • Long lived certificates allow deprecated cryptographic hashes and protocols to be used longer than desired.
  • Long lived certificate can allow process rot to develop, in which people forget how to create and install certificates, forget where and how certificates are used; using short duration certs keep the certificate generation, renewal, and rotation processes in practice and evergreen.

So it seems sensible to me to generate server certificates that last only 90 days, with longer periods (up to 13 months) for special situations. CA certs should last three years, but should only be used to generate new certificates for just one year.

The rationale for the CA cert expiration and usage policy depends on these characteristics:

  • It may be onerous to update the trusted certificate store for all client systems (particularly for browsers and apps that don’t use the OS trusted store). Because this process might not be fully automated, it would be better to do it just once a year.3
  • A CA cert must remain valid through the union of expiration periods of all the server certificates it has signed. If you consider allowing up to 13 month validity periods for CA certs, and you use a given CA cert to sign certificates over a one year period, then the CA cert must have an validity period of at least 25 months.

My recommendation is to maintain a set of three active CA certificates: one used to sign certificates created in the current year, one used to sign certificates in the prior year, and one that was used to sign certificates in the year before that. Each new year you create a new CA certificate to be used in that new year, and the client trusted certificate stores get updated to remove the oldest CA cert and add the new CA cert.

If you restrict server certificates to validity periods no longer than 13 months then that means CA certs only need to be valid for a 25 month period (the 12 months you sign certs with them plus 13 months for the validity period of the last cert you sign with that CA cert). If you update client trusted stores once a year, then you have a 10 month period in which to create the CA certificate for the next year and to update all the clients with the new and elided CA certs.

Generating a CA-signed server certificate

Generating a standard domain validated server certificate, signed by a private CA certificate, is pretty straightforward. It’s very similar to creating the CA cert, except that the private key for the server certificate won’t have a passphrase and the cert will be signed by the CA certificate’s private key.

Look at the script in the TIL “Create server certificate (using local CA)”. The -nodes argument on the openssl req command means the private key won’t require a challenge passphrase. If the private key did have a passphrase then every time the webserver (that is using the server cert and private key) was restarted it would prompt for the passphrase; not good for automated operation of the server or system.

Generating the cert is pretty straightforward. The “how to” details can be found in the TIL, along with some guidelines about how the CA gets updated each time a server cert is generated.

In most cases when you supply the cert to a server or service then you’ll need to supply both the CA cert and the server cert, plus the server cert’s private key. You don’t need to, and should not, supply the CA cert private key. Most servers are setup to accept two files: a certificate file and a private key file.

So how do you supply two certificates (the CA cert and the server cert) in one file? The way to provide both the CA cert and signed server cert is to “chain” them. You do that by concatenating the certs into a single file. Order is important; first in the file should be the server cert and the CA cert that signed it should be next.4 This is also discussed in more detail in the TIL.

Updating client trusted stores with private CA certs

The remaining TILs cover updating the Mac OS X (or MacOS) trusted store with the private root CA certificate, and updating the Firefox browser with the private root CA certificate.

Updating the OS X trusted store (which is the OS X System keychain) will allow Safari and Chrome to validate the private server certificates signed with the private CA. But Firefox doesn’t use the OS X keychain; it has it’s own trusted root CA certificate store, so it needs to be updated independently.

See the TIL “Add a CA certificate to the OS X keychain” to add the CA cert to the OS X keychain. See the TIL “Add CA certificate to Firefox” to add the CA cert to the Firefox trusted certificate store.

  1. Seems harder to do on Windows. I’ve looked into it in years past and have not found a nice solution to running a CA on Windows. I have not looked recently, so something straightforward may be available now. ↩

  2. I got a big head start on using OpenSSL to generate CA certs from this Slashdot answer. I had to tweak some of what’s there, but the answer is largely correct and is quite useful. ↩

  3. Once all client’s trusted stores can be updated automatically, then the CA cert validity period could be reduced. ↩

  4. If you use intermediate CA certificates, then you order things from server cert to intermediate CA cert to root CA cert. ↩