Creating an SSL certificate using Let's Encrypt and setting up Nginx on Ubuntu 20

Introduction

I created an SSL certificate with Let's Encrypt and set up SSL for Nginx running on Ubuntu 20.

The aim is to encrypt the data that interacts with the Web API deployed in Azure VM + Nginx. Reference: Creation of Web API using Python + Flask + MongoDB and placement on Azure VM + Nginx (collecting hololive video distribution schedule 3)

environment

Ubuntu 20 LTS (Azure VM)
Nginx 1.18.0
certbot 1.10.1

Premise

Nginx is installed and can be accessed on port 80 (name resolution by DNS). To be on the safe side, make sure you can access the Web API over HTTP (port 80).

Preparation

Since SSL is configured to access on port 443, allow reception on port 443 as well. This time, since we used Azure as the cloud environment, we added the reception permission to port 443 on NSG.

SSL settings for Nginx using Certbot

I used Certbot to create an SSL certificate using Let's Encrypt.

For the combination of Ubuntu 20.04 and Nginx, follow the steps below. Official Procedures for Ubuntu 20.04 and Nginx

What is Certbot

A free open source software tool for enabling HTTPS with your Let's Encrypt certificate for your website.

What is Let's Encrypt?

A certificate authority operated by the Internet Security Research Group (ISRG), a non-profit organization, that issues TLS X.509 certificates free of charge.

Creating an SSL certificate

Make an SSH connection to the target server as a user with sudo privileges.
Install snapd to install the snap package.
```
$ sudo apt install snapd
```

Make sure you have the latest version of snapd.

$ sudo snap install core; sudo snap refresh core

Remove all existing OS packages for Certbot.
```
$ sudo apt remove certbot
```
Use snapd to install Certbot.
```
$ sudo snap install --classic certbot
```

Create a symbolic link to/snap/bin/certbot.

$ sudo ln -s /snap/bin/certbot /usr/bin/certbot

Run Certbot to generate the certificate. (Nginx automatic configuration change is not specified)

$ cd /etc/letsencrypt/
$ sudo certbot certonly --webroot -w <Public path> -d <domain> --renew-by-default --email <mail address>

When I followed the steps, there are no files in .well-known/acme-challenge. Since the following error occurred, the public path is explicitly specified at the same time as creating the certificate as described above. As a precaution, I also created a .well-known/acme-challenge directory directly under the public path.

```text
Challenge failed for domain hogehoge.cloudapp.azure.com
http-01 challenge for hogehoge.cloudapp.azure.com
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: hogehoge.cloudapp.azure.com
   Type:   unauthorized
   Detail: Invalid response from
   http://hogehoge.cloudapp.azure.com/.well-known/acme-challenge/...
   [xxx.xxx.xxx.xxx]: "<html>\r\n<head><title>404 Not
   Found</title></head>\r\n<body>\r\n<center><h1>404 Not
   Found</h1></center>\r\n<hr><center>nginx/1.18.0 (Ub"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.
```

Also, after trying several times, the following error has come to occur.

```bash
Another instance of Certbot is already running.
```

Look for the process and force it to stop

```bash
#Find the process and force it to stop
$ ps waux | grep cert
root      461580  0.0  0.1  11276  4984 pts/0    T    15:38   0:00 sudo certbot --nginx
root      461581  0.0  1.1  68672 46380 pts/0    T    15:38   0:00 /snap/certbot/793/bin/python3 /snap/certbot/793/bin/certbot --nginx
mcuser    462206  0.0  0.0   8160   672 pts/0    S+   15:52   0:00 grep --color=auto cert

$ sudo kill -9 461580
[1]+  Killed                  sudo certbot --nginx  (wd: ~)
(wd now: /etc/letsencrypt)

$ ps waux | grep cert
mcuser    462221  0.0  0.0   8160   740 pts/0    S+   15:53   0:00 grep --color=auto cert
```

The solution was to find and delete the .certbot.lock file.

```bash
# .certbot.Find and delete the lock file
$ sudo find / -type f -name ".certbot.lock"
/var/lib/letsencrypt/.certbot.lock
/var/log/letsencrypt/.certbot.lock
/etc/nginx/.certbot.lock
/etc/letsencrypt/.certbot.lock

$ sudo rm /var/lib/letsencrypt/.certbot.lock
$ sudo rm /var/log/letsencrypt/.certbot.lock
$ sudo rm /etc/nginx/.certbot.lock
$ sudo rm /etc/letsencrypt/.certbot.lock
```

When the certificate generation is complete, the log location and certificate and key information will be displayed.

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Requesting a certificate for hogehoge.cloudapp.azure.com

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/hogehoge.cloudapp.azure.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/hogehoge.cloudapp.azure.com/privkey.pem
   Your cert will expire on 2021-03-21. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Confirm that the certificate has been generated.

$ sudo -s
# ll /etc/letsencrypt/live/hogehoge.cloudapp.azure.com/

README
cert.pem -> ../../archive/hogehoge.cloudapp.azure.com/cert1.pem
chain.pem -> ../../archive/hogehoge.cloudapp.azure.com/chain1.pem
fullchain.pem -> ../../archive/hogehoge.cloudapp.azure.com/fullchain1.pem
privkey.pem -> ../../archive/hogehoge.cloudapp.azure.com/privkey1.pem

Nginx SSL settings

Edit the Nginx configuration file.

#Edit the Nginx configuration file created for the Web API.
$ vi ~/holoapi/config/nginx.conf

#Nginx's default config file looks like this
$ sudo vi /etc/nginx/sites-available/default

Enable SSL and specify fullchain.pem (certificate) for ssl_certificate and privkey.pem (key) for ssl_certificate_key.

...
server {
    listen  80;

    #SSL on port 443
    listen 443 ssl;
    #SSL certificate
    ssl_certificate /etc/letsencrypt/live/hogehoge.cloudapp.azure.com/fullchain.pem;
    #SSL key
    ssl_certificate_key /etc/letsencrypt/live/hogehoge.cloudapp.azure.com/privkey.pem;
    #Server name (domain)
    server_name hogehoge.cloudapp.azure.com;
...

Validate the Nginx configuration file.

$ sudo nginx -t -c ~/holoapi/config/nginx.conf
nginx: the configuration file /home/mcuser/holoapi/config/nginx.conf syntax is ok
nginx: configuration file /home/mcuser/holoapi/config/nginx.conf test is successful

I stopped and restarted the Nginx process.

#Process stop
$ ps ax | grep nginx
 468229 ?        Ss     0:00 nginx: master process nginx -c /home/mcuser/holoapi/config/nginx.conf
 468230 ?        S      0:00 nginx: worker process
$ cat /var/run/nginx.pid
468229
$ sudo kill -QUIT $( cat /var/run/nginx.pid )
#Start by specifying the configuration file
$ sudo nginx -c ~/holoapi/config/nginx.conf
#Reference: Reloading the configuration file
$ sudo nginx -s reload

#If you restart the daemon process, it looks like this
$ sudo systemctl restart nginx
#Check the status of the daemon process
$ sudo systemctl status nginx
#Reference: Reloading the configuration file
$ sudo systemctl reload nginx

When I check the port in use, port 443 is LISTEN.

$ ss -antu
Netid     State          Recv-Q     Send-Q          Local Address:Port               Peer Address:Port      Process
tcp       LISTEN         0          511                   0.0.0.0:443                     0.0.0.0:*

SSL connection confirmation

I was able to access it over HTTPS (port 443). The connection with this site is safe.

Deleting SSL certificate using Certbot

For reference, to delete the created SSL certificate, execute Certbot with delete specified as shown below.

$ sudo certbot delete

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Which certificate(s) would you like to delete?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: hogehoge.cloudapp.azure.com
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 1

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
The following certificate(s) are selected for deletion:

  * hogehoge.cloudapp.azure.com

Are you sure you want to delete the above certificate(s)?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y
Deleted all files relating to certificate hogehoge.cloudapp.azure.com.

With this alone, the Nginx configuration file etc. will remain as it is, so you need to manually modify the configuration file and restart (or reload).

Automatic renewal confirmation of SSL certificate using Certbot

The Certbot introduced in the previous steps will automatically renew the certificate expiration date on a regular basis, but you can test the renewal by running Certbot with renew --dry-run.

However, this time too, this caused an error, so

$ sudo certbot renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/hogehoge.cloudapp.azure.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
Simulating renewal of an existing certificate for hogehoge.cloudapp.azure.com
Performing the following challenges:
http-01 challenge for hogehoge.cloudapp.azure.com
Using default addresses 80 and [::]:80 ipv6only=on for authentication.
Waiting for verification...
Challenge failed for domain hogehoge.cloudapp.azure.com
http-01 challenge for hogehoge.cloudapp.azure.com
Cleaning up challenges
Attempting to renew cert (hogehoge.cloudapp.azure.com) from /etc/letsencrypt/renewal/hogehoge.cloudapp.azure.com.conf produced an unexpected error: Some challenges have failed.. Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/hogehoge.cloudapp.azure.com/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/hogehoge.cloudapp.azure.com/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: hogehoge.cloudapp.azure.com
   Type:   unauthorized
   Detail: Invalid response from
   http://hogehoge.cloudapp.azure.com/.well-known/acme-challenge/6uWFQz67cHvdXqevhQhIayObcJAgp-4uS3nQmwktOf8
   [20.46.183.35]: "<html>\r\n<head><title>404 Not
   Found</title></head>\r\n<body>\r\n<center><h1>404 Not
   Found</h1></center>\r\n<hr><center>nginx/1.18.0 (Ub"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

In this way, I explicitly specified the public path and executed it.

$ sudo certbot renew --dry-run --webroot -w <Public path>
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/hogehoge.cloudapp.azure.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator webroot, Installer None
Simulating renewal of an existing certificate for hogehoge.cloudapp.azure.com
Performing the following challenges:
http-01 challenge for hogehoge.cloudapp.azure.com
Using the webroot path <Public path> for all unmatched domains.
Waiting for verification...
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/hogehoge.cloudapp.azure.com/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/hogehoge.cloudapp.azure.com/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Renewal of SSL certificate using Certbot

To actually renew the SSL certificate, execute the following command.

$ sudo certbot renew

If it does not need to be updated, it will be skipped.

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/hogehoge.cloudapp.azure.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not yet due for renewal

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

The following certs are not due for renewal yet:
  /etc/letsencrypt/live/hogehoge.cloudapp.azure.com/fullchain.pem expires on 2021-03-22 (skipped)
No renewals were attempted.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Since the update is performed according to the setting of /etc/letsencrypt/renewal/hogehoge.cloudapp.azure.com.conf, change the setting if you want to intentionally specify the public path to access on port 80.

# renew_before_expiry = 30 days
version = 1.10.1
archive_dir = /etc/letsencrypt/archive/hogehoge.cloudapp.azure.com
cert = /etc/letsencrypt/live/hogehoge.cloudapp.azure.com/cert.pem
privkey = /etc/letsencrypt/live/hogehoge.cloudapp.azure.com/privkey.pem
chain = /etc/letsencrypt/live/hogehoge.cloudapp.azure.com/chain.pem
fullchain = /etc/letsencrypt/live/hogehoge.cloudapp.azure.com/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
authenticator = webroot
manual_public_ip_logging_ok = None
webroot_path = <Public path>,
server = https://...
[[webroot_map]]

This time, the public path was specified at the time of certificate generation and automatic renewal confirmation, so it was already set.

in conclusion

I was able to encrypt the data that interacts with the Web API deployed in Azure VM + Nginx. I've rarely used Nginx, but the settings to achieve what I want to do are simple and easy to understand.

Let's Encrypt was helpful because the SSL certificate is available for free. It seems that there are various advantages and disadvantages, but it is surprising that the encryption strength is almost the same as the SSL certificate that is generally sold.

The validity period of the SSL certificate is short because it is 90 days, but there seems to be no problem in normal use due to the automatic renewal mechanism. I will use it as it is and check for updates.