Wildcard Certificate with letsencrypt

I have my own DNS, so I need to set it up myself to get letsencrypt to work as expected and generate a wildcard certificate for my websites.

They decided to test the DNS because that way they know you are in control of the domain and its sub-domains (only the owner  of a domain name would be able to allow such a test to work.) When creating a certificate for just one website they can ask you to place a file there, which is very easy, but for an entire domain, that wouldn't be quite enough, especially since some business endeavors actually use such a technique and either offer for free or sell sub-domains of a highly priced domain name.

Install certbot & Modules

If you don't have certbot installed yet, you can do the following to do so:

sudo apt-get install certbot
sudo apt-get install python3-certbot-apache
sudo apt-get install python3-certbot-dns-rfc2136-doc

About the Nameserver (BIND9)

So, I had to setup my DNS to let letsencrypt tweak a parameter to test that I do indeed own that domain name. I use BIND9 as the name server. I looked around and found a very good post about how to do it the hard way... or maybe it was how to do it when the letsencrypt wildcard feature came out. The fact is that it's actually very simple if you know what is required.

With BIND9 you define two types of files:

1) Configuration files, at least one of which has a list of zones

2) Zone files, one zone file per domain is required for this one

The BIND9 zone file format follows semantics that are very close to the technical aspect of the nameserver protocol.

The configuration files are easier. Somewhat based on a C-like structure definition.

What needs to be updated

In order to get a letsencrypt certificate, there are only two things you need to update in your configuration files. First the zone definition is given and update-policy entry as follow:

zone "restarchitect.com" IN {
  type "master";
  file "/var/lib/bind/restarchitect.com.zone";
  allow-transfer { trusted-servers; };
  check-names warn;
  update-policy {
    grant letsencrypt_wildcard. name _acme-challenge.restarchitect.com. txt;
  };
};

In that definition, I also added the check-names option because letsencrypt decided to use a name with an underscore (see below for more info.)

The rest is pretty much standard, but here I will define each of the instructions I use in my example here:

• type "master";

The type defines whether this domain name is being served by the master DNS or a slave. In my example here, this zone is from the master DNS.

• file "/var/lib/bind/restarchitect.com.zone";

The path and filename to the actual zone file. I have a sample below where you can see how it needs to be setup. Not much going on though. However, the important part is that this file can't reside under /etc.

IMPORTANT INFORMATION: The named service checks where the zone file is found. If it is defined under /etc/... then it does not get updated and that means updates to a zone are accepted but lost right away (I'm not too sure why it is accepted if it can't be saved, though). As mention in a comment by Adrian McElligott, this has to do with AppArmor. The profiles file includes named:

$ sudo less /sys/kernel/security/apparmor/profiles
...
/usr/sbin/named (enforce)
...

$ less /etc/apparmor.d/usr.sbin.named
...
/etc/bind/** r,
/var/lib/bind/** rw,
/var/lib/bind/ rw,
...

Although you can change the AppArmor settings from "r" to "rw", I recommend you keep those as they are and use the /var/lib/... folder instead. Files are expected to be writable in that other folder by default. Under Ubuntu, you can set the ownership to bind like so:

chown -R bind:bind /var/lib/bind

Although from my own experience, it works with files owned by root. But if you have a problem, your setup may be such that the named service may be running in a jail and not have write access to those files. You may also have issues with the writing of files when SE Linux is running.

You may have copies of your zones under /etc/bind/... if you only backup /etc and not /var/lib/.... That way you'll have the original available.

• allow-transfer { trusted-servers; };

I use this line to authorize full access to my secondary DNS systems. Especially, that gives my other DNS to retrieve the complete list of known sub-domain names for each domain I have.

This works because I have static IP addresses and the addresses of my other computers are specified in a named.conf option file:

options {
  ...
  allow-transfer { 10.0.0.1; 10.0.0.2; ... };
  ...
};

This example says that computers with IP address 10.0.0.1 and 10.0.0.2 gave permissions to do fast transfers from the DNS. (see about the axfr option below)

• check-names warn;

The check-names option is required in case the name letsencrypt adds _acme-challenge to your list of known sub-domains. The underscore character is not liked by BIND9. This is because it is not part of the domain name specification. It is not allowed at all. By default BIND will generate an error and log it and skip over that entry entirely (i.e. it will not serve that zone at all, albeit all the other zones will work just fine.)

You can also set this parameter to ignore. In that case, no warning is emitted in your logs.

Here is the error you get ("bad owner name") when a name uses characters that are not supposed to be used in a domain name:

09-Feb-2019 03:02:31.988 general: error:
           /var/lib/bind/restarchitect.com.zone:31:
          _acme-challenge.restarchitect.com:
          bad owner name (check-names)

The check-names option is currently the only way to fix this problem (i.e. you can't use an escape for that one specific letter.)

• update-policy { ... }

This line grants one specific permission to anyone with the HMAC secret key.

grant letsencrypt_wildcard.
      name _acme-challenge.restarchitect.com.
      txt;

Here is the line dissected:

1. The policy is a "grant"

This means we are going to allow a certain command to modify our DNS. The granularity is small enough that we can offer such with enough assurance that it won't allow hackers to completely destroy (take over) your DNS.

2. The name of the key: "letsencrypt_wildcard."

This is a reference to your encryption key. It is used to communicate between the client and server in such a way that it proves that the client knows us (the client has to have a copy of the key to be able to communicate with us.)

3. The name "_acme-challenge.restarchitect.com."

The sub-domain name which the client can temper with. This means only that one name is going to be updated by clients. All the other names can't be changed at all.

4. The type "txt"

Further, we specify the exact type of information can be modified. By using the "txt" type, letsencrypt limits permissions to the name which does not allow them to transformed that name with a valid IP address.

In other words, there is pretty much no hijacking possible with such a grant.

Create and Setup an HMAC Key

Now you want to define the HMAC key to be used with the bind setup.

letsencrypt expects such a key when you run their certbot command.

To generate the key, run the following commands:

$ sudo su -
(enter password when prompted)
# cd /etc/bind
# mkdir letsencrypt_keys
# chmod 700 letsencrypt_keys
# cd letsencrypt_keys
# dnssec-keygen -a HMAC-SHA512 -b 512 -n HOST letsencrypt_wildcard.
# sudo chmod 600 *

This is a relatively secure way of creating the key.

The output of the dnssec-keygen command are two files:

Kddns_update.+165+35602.key
Kddns_update.+165+35602.private

We are interested by the .private file. That file includes the key encoded using base64.

Private-key-format: v1.3
Algorithm: 165 (HMAC_SHA512)
Key: <here is a bunch of letters/digits/etc.>
Bits: AAA=
Created: 20190209013350
Publish: 20190209013350
Activate: 20190209013350

I show in blue what we are interested by. Copy that value from this file and paste it in a .conf file in BIND that you will name something like letsencrypt_wildcard_key.conf.

key "letsencrypt_wildcard" {
    algorithm hmac-sha512;
    secret "<here is a bunch of letters/digits/etc.>";
};

Now you need to include this key in your named.conf so it can be accessed by the zone declaration we mentioned earlier (and it has to appear before that zone declaration.) You can also copy/paste the whole key in your options, however, the key file should be in a safe folder as shown above (i.e. owned by root with read/write permissions only given to root. If you have SE Linux, you may instead need to use bind:bind for the ownership.)

IMPORTANT NOTE:

Here the key name is just letters and an underscore. No period (.) at the end of the name. Notice that in the grant definition above we have a period at the end.

To include the file, edit one of your configuration files and add an include command:

include "/etc/bind/letsencrypt_keys/letsencrypt_wildcard_key.conf";

RNDC Key Conflict

When I run the certbot command (see below), it gets stuck sending the DNS requests to the name server if the RNDC key is defined.

By default, you often have the rndc.key file included like so:

include "/etc/bind/rndc.key";

What I do is make sure that this include doesn't happen by commenting the line:

#include "/etc/bind/rndc.key";

Then try again.

Restart BIND9

To make sure that your settings are still valid, run a check:

named-checkconf /etc/named.conf

When no errors are discovered, the command returns as is (no output.) When errors are displayed, you may want to also look at the logs for additional hints about the potential problems.

Now is time for you to restart your BIND service. This is done with:

systemctl restart bind9

If you just added a new domain name, you can test the validity with dig:

dig @ns1.m2osw.com restarchitect.com

With the @... I specify the domain name server (DNS) to query. This way I directly query the source. The second parameter to dig is the name of the domain I'm testing.

Testing The Grant

To make sure that the grant we just defined in BIND9 works, we can use the nsupdate tool. This tool gives you a way to send commands to your DNS.

WARNING: to test properly, you want to test from a computer which is not the one with the DNS nor a secondary server (i.e. do not use a computer which IP address appears in the list of allow-transfer { ... } computers. Testing with one of these computers is likely to not check the grant itself.

The following are the instructions I suggest you use to test the grant:

$ nsupdate -k /path/to/letsencrypt_wildcard_key.conf -v
> server restarchitect.com
> debug yes
> zone restarchitect.com
> update add _acme-challenge.restarchitect.com. 60 TXT "test"
> show
Outgoing update query:
;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id:      0
;; flags:; ZONE: 0, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0
;; ZONE SECTION:
;restarchitect.com.        IN    SOA

;; UPDATE SECTION:
_acme-challenge.restarchitect.com. 86400 IN TXT    "test"

> send
Sending update to 10.0.0.1#53
Outgoing update query:
;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id:  60546
;; flags:; ZONE: 1, PREREQ: 0, UPDATE: 1, ADDITIONAL: 1
;; ZONE SECTION:
;restarchitect.com.        IN    SOA

;; UPDATE SECTION:
_acme-challenge.restarchitect.com. 86400 IN TXT    "test"

;; TSIG PSEUDOSECTION:
letsencrypt_wildcard.    0    ANY    TSIG    hmac-sha512.
                   1549778797 300 64 <key> 60546 NOERROR 0


Reply from update query:
;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id:  60546
;; flags: qr; ZONE: 1, PREREQ: 0, UPDATE: 0, ADDITIONAL: 1
;; ZONE SECTION:
;restarchitect.com.        IN    SOA

;; TSIG PSEUDOSECTION:
letsencrypt_wildcard.    0    ANY    TSIG    hmac-sha512.
                    1549778797 300 64 <key> 60546 NOERROR 0

> quit

Note: the 60 before the TXT is the number of seconds that info will stay in your DNS. You'll have to wait 1 min. before running letsencrypt again to make sure that it had time to clean up your test.

First, notice that I use the key defined in the configuration file I created above for BIND9. I changed the path because I'm on a different computer which is not a nameserver (otherwise the test may succeed even though the grant would not allow the user to otherwise change the zone settings.)

The update add ... command is the one defining what will be updated on the zone (which was specified on the line before.) Here we set a TXT field to the value "test".

The show command is not necessary. It gives you a way to see whether you type things right (if you can understand the output...)

The send command is what sends the UDP request to your server. The result shows "NOERROR" when it worked. It will say "FAILED" otherwise. For example, if you use the wrong key, it will fail. But assuming that you get everything right on your client computer, the command will fail only if the DNS does not authorize you to make the update. [NOTE: I broke up a couple of lines in two to make it easier to read on the website.]

Once the nsupdate says it worked (i.e. it returned a NOERROR message,) you can test with the dig command to see that it indeed was regitered:

$ dig @ns1.m2osw.com restarchitect.com axfr

; <<>> DiG 9.10.3-P4-Ubuntu <<>> @ns1.m2osw.com restarchitect.com axfr
; (1 server found)
;; global options: +cmd
restarchitect.com.    86400    IN    SOA    ns1.m2osw.com. hostmaster.m2osw.com. 1309082308 10800 180 1209600 300
restarchitect.com.    86400    IN    NS    ns1.m2osw.com.
restarchitect.com.    86400    IN    NS    ns2.m2osw.com.
restarchitect.com.    86400    IN    A    10.0.0.1
_acme-challenge.restarchitect.com. 86400 IN TXT    "test"
w.restarchitect.com.    86400    IN    A    10.0.0.1
ww.restarchitect.com.    86400    IN    A    10.0.0.1
www.restarchitect.com.    86400    IN    A    10.0.0.1
wwww.restarchitect.com.    86400    IN    A    10.0.0.1
restarchitect.com.    86400    IN    SOA    ns1.m2osw.com. hostmaster.m2osw.com. 1309082308 10800 180 1209600 300
;; Query time: 20 msec
;; SERVER: 10.0.0.1#53(10.0.0.1)
;; WHEN: Sat Feb 09 22:11:27 PST 2019
;; XFR size: 11 records (messages 1, bytes 313)

IMPORTANT: In 18.04, somehow, I can't see the _acme-challenge sub-domain, yet the certbot command works as expected. I'm not totally sure what happens. The nsupdate command returns NOERROR, so it would look like it worked as expected.

Note that the axfr requests a full transfer of the zone. For unknown clients, this feature is turned off by default. For this command to work you actually want to run it on one of your DNS servers (primary or secondary.)

As we can see in the output (highlighted in blue) the TXT field was added as expected. If not added, you're on your own buddy! Maybe re-read my document here and see whether something could have gone wrong. The logs often tell you why BIND9 refused the update transaction.

Once this works, the certbot will also be able to make such changes and that's sufficient for your system to receive a wildcard type of certificate.

Passing the Key to certbot

In order for certbot to access your DNS, it needs to have access to your key. The key will be transmitted to the letsencrypt servers which in turn verify that you own your domain before issuing the certificate.

Create a file, for example, /etc/bind/letsencrypt_keys/certbot.ini and copy the following, fixing a few parameters as required by your system:

# Target DNS server
dns_rfc2136_server = 10.0.0.1
# Target DNS port
dns_rfc2136_port = 53
# TSIG key name
dns_rfc2136_name = letsencrypt_wildcard.
# TSIG key secret
dns_rfc2136_secret = <here is a bunch of letters/digits/etc.>
# TSIG key algorithm
dns_rfc2136_algorithm = HMAC-SHA512

The dns_rfc2136_server parameter is the public IP address of your DNS server. (The example shows a private IP.)

The dns_rfc2136_name parameter defines the name of the key. I used "letsencrypt_wildcard" in my prior examples, this is that name.

The dns_rfc1236_secret parameter is the private key. The same we put in the letsencrypt_wildcard_key.conf file.

Generating the Wildcard Certificate

Now we are ready to generate a wildcard certificate with certbot:

sudo certbot certonly \
  --dns-rfc2136 \
  --dns-rfc2136-credentials /etc/bind/letsencrypt_keys/certbot.ini \
  -d '*.restarchitect.com'
  -d restarchitect.com

As we can see, the command references the certbot.ini file we just created.

The --dns-rfc2136 command line option tells certbot how to handle the domain name verification: directly with your DNS information as defined in the certbot.ini file.

The -d option specifies the name of the domain for which you want a certificate. Notice that to get a wildcard certificate, you want to use an asterisk. That way all the sub-domains created for restarchitect.com (in my example) will all benefit from the same certificate. However, you also want the name by itself because the wildcard only doesn't match the name without a subdomain.

If you want to protect multiple domain names with the same certificate, you can do so using additional -d command line options. For example, in my case I had the .com and .org so I could write:

... -d '*.restarchitect.com' -d restarchitect.com
    -d '*.restarchitect.org' -d restarchitect.org

There is a limit to the number of -d options you can use. Last time I checked it was 100, which I think is plenty, especially if you use a wildcard certificate.

Note: If you forget to include a certain domain, you can later add it using the --expand option. The one strange thing about the --expand is that you must keep a complete list of all the -d options that you first included and then add more of them. So for example, if you first had the .com and later you want to add the .org, you would still include the .com domains in the list of -d options.

Zone File

Here is a simple example of a minimal zone file.

$ORIGIN .
restarchitect.com       IN SOA  ns1.m2osw.com. hostmaster.m2osw.com. (
                                1309082307 ; serial
                                10800      ; refresh (3 hours)
                                180        ; retry (3 minutes)
                                1209600    ; expire (2 weeks)
                                300        ; minimum (5 minutes)
                                )
                        NS      ns1.m2osw.com.
                        NS      ns2.m2osw.com.
                        A       138.197.205.139
$ORIGIN restarchitect.com.
$TTL 86400      ; 1 day
w                       A       138.197.205.139
ww                      A       138.197.205.139
www                     A       138.197.205.139
wwww                    A       138.197.205.139

The file defines the 5 sub-domains I allow for pretty much all of my websites:

1. (nothing) — the A entry without a name

2. w, ww, wwww — for people who can't type

3. www — for the usual World Wide Web sub-domain (although many of my websites use case (1) and the "www" redirects there.)

You could add more entries such as a mail address or specialized sub-domains such as api for a REST access point.

This is all you need to have in your zone to make the letsencrypt wildcard system work as expected.

If the server decides it can't write to the file, you get (on one line, broken up to fit on this website):

09-Feb-2019 02:05:47.766 general: error:
     /etc/bind/zones/restarchitect.com.zone.jnl:
     create: permission denied

As we can see, we get a "create: permission denied", and that when the file is owned by BIND and rw in modification set.

Potential Errors

When the DNS challenge fails, you may get the following error:

Unable to determine base domain for _acme-challenge.ordermade.com
   using names: ['_acme-challenge.ordermade.com',
   'ordermade.com', 'com'].

More or less, this means that letsencrypt was not able to guarantee that you were the owner of the domain name (ordermade.com in my example.) It means your bind9 setup is not correct and it did not allow for adding a TXT field to test that you could allow letsencrypt to verify that you own that domain name.

Verify an SSL Certificate on a specific computer

If you have many computers that answer to the same domain name but different IP addresses (i.e. you have a round robbinds on a set of IP addresses), then you will want to test each computer and for that you can't just hope to hit them all with a regular access on your browser or a simple curl command.

However, it's actually possible to force resolve the IP address in a curl command like so:

curl -v --resolve example.com:443:1.2.3.4 https://example.com

The -v command line option is going to give you a lot of information (outside of the page content itself) and that includes the certificate information with the expiration date. However, that's a ton of data that fills your screen and as I just mentioned it's also going to download your page.

Another way is to use the openssl command. Here again, there is a limit, by default openssl shows the certificate owner and the encoded certificate but no dates. So the command needs to be a little longer like so:

echo \
    | openssl s_client -servername example.com -connect 1.2.3.4:443 2>/dev/null \
    | openssl x509 -noout -dates

Just like the curl command, we can resolve the server name using the -connect command line option. Then we want to decode the certificate and finally we just want the dates, so we use the -dates command line option.

That will tell you exactly when the certificate is valid (start and end dates).

Additional Information

Here are the pages I used to setup BIND9 and get my wildcard certificates:

certbot RFC 2136 documentation

Let's Encrypt Wildcard Certificate (with the acme.sh script)

Check BIND Server Configuration

List all DNS records

---