Today I had a problem with letsencrypt. I did not want to give me the certificate as it could not verify the domain name I was trying to get a certificate for.

Looking into why I would get the error:

The client lacks sufficient authorization

I only found references to mainly stupid answers. Especially, answers that would tell you to create a directory under the .well-known folder as in:

http://example.org/.well-known/acme-challenge/test

Then make sure you could access "test" by going to that URL.

Sure enough that worked just fine. But it has nothing to do with letsencrypt problems because the way they allow that folder to work is not like that!

What letsencrypt does is add two includes in your corresponding virtual host definition:

<VirtualHost *:80>
    Include /etc/apache2/le_http_01_challenge_pre.conf
    ...your code...
    Include /etc/apache2/le_http_01_challenge_post.conf
</VirtualHost>

In order to see the challenge file and the changes to your virtual host, add the "--debug-challenges" command line option. At some point the certbot script stops and asks you to hit enter. Once you do hit the enter key, it will finish up the verification process. But while stuck that way in the script, your settings are modified by letsencrypt and you can look at them (you shouldn't edit anything at that point, though.)

As we can see in the "pre" script:

    RewriteEngine on
    RewriteRule ^/\.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/letsencrypt/http_challenges/$1 [END]

The .well-known/acme-challenge/... file is handled by redirecting accesses to /var/lib/letsencrypt/http_challenges/... and NOT your installation folder. In other words, doing the test as shown above is totally useless.

For documentation, the "post" script looks like this:

    <Directory /var/lib/letsencrypt/http_challenges>
        Require all granted
    </Directory>
    <Location /.well-known/acme-challenge>
        Require all granted
    </Location>

As we can see, these two instruct the system to grant all privileges on the acme and challenge directories. So again, the check we are asked to perform above has nothing to do with anything (well, unless you have some form of proxy-ing, something that will prevent the letsencrypt setup from taking hold, etc.)

In my case, the script would go on and say that it would make a backup of many configuration files but not the one with the very domain I had problem with.

Creating backup of /etc/apache2/sites-enabled/000-m2osw.com.conf

In other words, it would copy like 7 configuration files where it would add the Include instructions but the very one where it was required did not get modified at all. So the Include were missing from the file where it is required.

My old installation was the culprit. The fact is that I had many old configuration files which I had named something like:

linux.m2osw.com

When the newer Apache2 convention (at least under Ubuntu) is to have:

linux.m2osw.com.conf

Without the .conf, somehow, letsencrypt ignores my file.

So... I went ahead and renamed all my files to include the .conf and updated the IncludeOptional from apache2.conf as follow:

IncludeOptional sites-enabled/*.conf

The old include looked like an ugly pattern:

IncludeOptional sites-enabled/[^#!.]*

and it worked in the old days, but now that was being ignored by letsencrypt.

It's not a bad thing to have renamed all my configuration files anyway. I think it's much better to have them with .conf than my old convension.

If you still have problems, make sure to check your Apache2 logs and the letsencrypt logs. That last one is found here on Ubuntu:

sudo less /var/log/letsencrypt/letsencrypt.log

The letsencrypt logs are protected so you will need sudo to look at them. You may have additional files (.1, .2, etc.)

And for more information, use the "-v" (verbose) flag of certbot as in:

certbot -v ...

Expect a lot of output when you use the -v option.

To install certbot on your Ubuntu computer do:

sudo apt-get install certbot
sudo apt-get install python3-certbot-apache
sudo apt-get install python3-certbot-nginx

If you have just apache or nginx you can install just one of them, you don't need both.

Here is the full message that I'd get when an error would occur:

$ sudo certbot --apache certonly -d example.org
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for example.org
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. example.org
 (http-01): urn:ietf:params:acme:error:unauthorized :: The
 client lacks sufficient authorization :: Invalid response
 from http://example.org/.well-known/acme-
 challenge/SemSXp5MwbH1gq0FJ9TW88X0bhsYWC9suRkz_vsFxOs:
 "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n
 <html><head>\n<title>403 Forbidden</title>\n</head><body>
 \n<h1>Forbidden</h1>\n<p"

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: apartments.snapwebsites.org
   Type:   unauthorized
   Detail: Invalid response from
   http://example.org/.well-known
   /acme-challenge/SemSXp5MwbH1gq0FJ9TW88X0bhsYWC9suRkz_vsFxOs:
   "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>403
   Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

Obviously, my domain name was not example.org. I use that here because the exact name is not relevant here.