Opened 3 years ago

Last modified 2 years ago

#64 new task

Let's Encrypt HTTPS Certs

Reported by: chris Owned by: chris
Priority: major Milestone: Maintenance
Component: crin1 Version:
Keywords: Cc: matthew
Estimated Number of Hours: 0 Add Hours to Ticket: 0
Billable?: yes Total Hours: 4.05

Description (last modified by chris)

These certs, on Crin1, are due to expire soon:

The one on Crin2 has a while to run:

To save money I suggest that the certs on Crin1 are replaced with Let's Encrypt ones, for the above domains and in addition these sites which use CAcert:

I suggest we still use CACert for the MySQL certs as these are valid for two years and Let's Encrypt ones are only valid for 3 months and it would be complicated to automate the renewal and deployment for these certs.

Change History (8)

comment:1 Changed 3 years ago by chris

  • Add Hours to Ticket changed from 0 to 0.25
  • Total Hours set to 0.25

comment:2 Changed 3 years ago by chris

  • Description modified (diff)

comment:3 Changed 3 years ago by chris

  • Add Hours to Ticket changed from 0 to 1.5
  • Total Hours changed from 0.25 to 1.75

We might as well get one cert for all the domains on Crin1 and we can use the webroot method, which simply requires the Let's Encrypt client to have write access to /.well-known/acme-challenge/ to write files that are checked via HTTP.

So setting this up on Crin1:

cd /var/www
mkdir letsencrypt/.well-known/acme-challenge/ -p

Create /etc/apache2/conf-available/letsencrypt.conf containing:

<Directory "/var/www/letsencrypt">
        Require all granted
</Directory>

And add the following to all VirtualHosts:

Alias "/.well-known/acme-challenge" "/var/www/letsencrypt/.well-known/acme-challenge"

Some things were tided in the Apache config and also the DNS, these old entries were deleted:

web1		46.149.19.215
db1		46.149.17.37
ftp		213.138.117.132
enyaforum	205.153.117.25
enya		205.153.117.25
mirror		205.153.117.25
filemanager	205.153.118.149
web2		46.149.26.18

And this is the final zone file:

@ 86400 IN SOA ns0.1984.is. hostmaster.1984.is. 2015022684 86400 28800 604800 86400
@ 900 IN A 93.95.228.180
@ 86400 IN NS ns0.1984.is.
@ 86400 IN NS ns1.1984hosting.com.
@ 86400 IN NS ns1.1984.is.
@ 86400 IN NS ns2.1984hosting.com.
@ 86400 IN NS ns2.1984.is.
@ 900 IN MX 10 ASPMX.L.GOOGLE.COM.
@ 900 IN MX 20 ALT1.ASPMX.L.GOOGLE.COM.
@ 900 IN MX 20 ALT2.ASPMX.L.GOOGLE.COM.
@ 900 IN MX 30 ASPMX2.GOOGLEMAIL.COM.
@ 900 IN MX 30 ASPMX3.GOOGLEMAIL.COM.
@ 900 IN MX 30 ASPMX4.GOOGLEMAIL.COM.
@ 900 IN MX 30 ASPMX5.GOOGLEMAIL.COM.
@ 900 IN TXT "v=spf1 include:aspmx.googlemail.com ~all"
backup 900 IN A 93.95.228.198
cloud 900 IN A 93.95.228.179
www.cloud 900 IN A 93.95.228.179
crin1 900 IN A 93.95.228.179
*.crin1 900 IN A 93.95.228.179
crin2 900 IN A 93.95.228.180
*.crin2 900 IN A 93.95.228.180
crin3 900 IN A 93.95.228.198
crin4 900 IN A 93.95.228.222
*.crin4 900 IN A 93.95.228.222
dev 900 IN A 93.95.228.222
enoc 900 IN A 93.95.228.180
google73ec47ca095c0b26 900 IN CNAME google.com.
mail 900 IN CNAME ghs.google.com.
munin 900 IN A 93.95.228.179
www.munin 900 IN A 93.95.228.179
newprod 900 IN A 93.95.228.180
phpmyadmin 900 IN A 93.95.228.179
www.phpmyadmin 900 IN A 93.95.228.179
solr 900 IN A 93.95.228.180
stage 900 IN A 93.95.228.222
stats 900 IN A 93.95.228.179
www.stats 900 IN A 93.95.228.179
test 900 IN A 205.153.117.25
trac 900 IN A 93.95.228.179
trac 900 IN MX 10 mx.webarch.net.
www.trac 900 IN A 93.95.228.179
wiki 900 IN A 93.95.228.179
www.wiki 900 IN A 93.95.228.179
www 900 IN A 93.95.228.180

Test:

lynx -dump http://trac.crin.org/.well-known/acme-challenge/foo.txt
lynx -dump http://stats.crin.org/.well-known/acme-challenge/foo.txt
lynx -dump http://cloud.crin.org/.well-known/acme-challenge/foo.txt
lynx -dump http://trac.crin.org/.well-known/acme-challenge/foo.txt
lynx -dump http://wiki.crin.org/.well-known/acme-challenge/foo.txt
lynx -dump http://phpmyadmin.crin.org/.well-known/acme-challenge/foo.txt
lynx -dump http://munin.crin.org/.well-known/acme-challenge/foo.txt

The clients is in backports:

So added the following to /etc/apt/sources.list:

# backports
deb http://speglar.simnet.is/debian/ jessie-backports main 

And install:

aptitude install letsencrypt
The following NEW packages will be installed:
  dialog{a} letsencrypt python-acme{ab} python-cffi{a} python-configargparse{a} python-configobj{a} python-cryptography{a} 
  python-dialog{a} python-letsencrypt{ab} python-mock{a} python-ndg-httpsclient{a} python-openssl{a} python-parsedatetime{a} 
  python-ply{a} python-psutil{a} python-pyasn1{a} python-pycparser{a} python-pyicu{a} python-requests{a} python-rfc3339{a} 
  python-urllib3{a} python-werkzeug{a} python-zope.component{a} python-zope.event{a} python-zope.interface{a} 
0 packages upgraded, 25 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,103 kB of archives. After unpacking 10.9 MB will be used.
The following packages have unmet dependencies:
 python-letsencrypt : Depends: python-cryptography (>= 0.7) but 0.6.1-1 is to be installed.
 python-acme : Depends: python-openssl (>= 0.15) but 0.14-1 is to be installed.
               Depends: python-cryptography (>= 0.8) but 0.6.1-1 is to be installed.
The following actions will resolve these dependencies:

     Keep the following packages at their current version:
1)     letsencrypt [Not Installed]                        
2)     python-acme [Not Installed]                        
3)     python-letsencrypt [Not Installed]                 



Accept this solution? [Y/n/q/?] n
The following actions will resolve these dependencies:

     Install the following packages:                          
1)     python-cffi-backend [1.4.2-2~bpo8+1 (jessie-backports)]
2)     python-cryptography [1.1.1-1~bpo8+1 (jessie-backports)]
3)     python-enum34 [1.0.3-1 (stable)]                       
4)     python-idna [2.0-3~bpo8+1 (jessie-backports)]          
5)     python-ipaddress [1.0.16-1~bpo8+1 (jessie-backports)]  
6)     python-openssl [0.15.1-2~bpo8+1 (jessie-backports)]    
7)     python-pyasn1 [0.1.9-1~bpo8+1 (jessie-backports)]      

     Keep the following packages at their current version:    
8)     python-cffi [Not Installed]                            



Accept this solution? [Y/n/q/?] Y

The following NEW packages will be installed:
  dialog{a} letsencrypt python-acme{a} python-cffi-backend{a} python-configargparse{a} python-configobj{a} 
  python-cryptography{a} python-dialog{a} python-enum34{a} python-idna{a} python-ipaddress{a} python-letsencrypt{a} 
  python-mock{a} python-ndg-httpsclient{a} python-openssl{a} python-parsedatetime{a} python-psutil{a} python-pyasn1{a} 
  python-pyicu{a} python-requests{a} python-rfc3339{a} python-urllib3{a} python-werkzeug{a} python-zope.component{a} 
  python-zope.event{a} python-zope.interface{a} 
0 packages upgraded, 26 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,109 kB of archives. After unpacking 11.0 MB will be used.

And running the command to generate a key and get the cert:

letsencrypt certonly --webroot --webroot-path /var/www/letsencrypt --renew-by-default --email chris@webarchitects.co.uk --text --agree-tos -d stats.crin.org -d www.stats.crin.org -d trac.crin.org -d www.trac.crin.org -d cloud.crin.org -d www.cloud.crin.org -d wiki.crin.org -d www.wiki.crin.org -d phpmyadmin.crin.org -d www.phpmyadmin.crin.org -d munin.crin.org -d www.munin.crin.org


Failed authorization procedure. www.munin.crin.org (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: DNS problem: NXDOMAIN looking up A for www.munin.crin.org, www.phpmyadmin.crin.org (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: DNS problem: NXDOMAIN looking up A for www.phpmyadmin.crin.org

IMPORTANT NOTES:
 - If you lose your account credentials, you can recover through
   e-mails sent to chris@webarchitects.co.uk.
 - The following errors were reported by the server:

   Domain: www.munin.crin.org
   Type:   connection
   Detail: DNS problem: NXDOMAIN looking up A for www.munin.crin.org

   Domain: www.phpmyadmin.crin.org
   Type:   connection
   Detail: DNS problem: NXDOMAIN looking up A for
   www.phpmyadmin.crin.org

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.
 - Your account credentials have been saved in your Let's Encrypt
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Let's
   Encrypt so making regular backups of this folder is ideal.

So the DNS updates will take some time, trying without the extra domain names:

letsencrypt certonly --webroot --webroot-path /var/www/letsencrypt --renew-by-default --email chris@webarchitects.co.uk --text --agree-tos -d stats.crin.org -d www.stats.crin.org -d trac.crin.org -d cloud.crin.org -d www.cloud.crin.org -d wiki.crin.org -d phpmyadmin.crin.org -d munin.crin.org


IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/stats.crin.org/fullchain.pem. Your cert will
   expire on 2016-05-28. To obtain a new version of the certificate in
   the future, simply run Let's Encrypt again.
 - If you like Let's Encrypt, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

So that worked, the key, chain and certs:

 ls /etc/letsencrypt/live/stats.crin.org/ -lah
total 8.0K
drwxr-xr-x 2 root root 4.0K Feb 28 13:28 .
drwx------ 3 root root 4.0K Feb 28 13:28 ..
lrwxrwxrwx 1 root root   38 Feb 28 13:28 cert.pem -> ../../archive/stats.crin.org/cert1.pem
lrwxrwxrwx 1 root root   39 Feb 28 13:28 chain.pem -> ../../archive/stats.crin.org/chain1.pem
lrwxrwxrwx 1 root root   43 Feb 28 13:28 fullchain.pem -> ../../archive/stats.crin.org/fullchain1.pem
lrwxrwxrwx 1 root root   41 Feb 28 13:28 privkey.pem -> ../../archive/stats.crin.org/privkey1.pem

And also updating the cipher suite based on the Mozilla wiki:

In /etc/apache2/mods-enabled/ssl.conf we have:

        SSLCipherSuite ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS

        SSLCertificateFile      /etc/letsencrypt/live/stats.crin.org/cert.pem
        SSLCertificateKeyFile   /etc/letsencrypt/live/stats.crin.org/privkey.pem
        SSLCACertificateFile    /etc/letsencrypt/live/stats.crin.org/chain.pem

And the cert, key and root config can be commented out of the other config files.

Restart apache and it works:

Test:

Enable an alert for when the cert needs renewing, by adding this crontab:

30 09 * * * ssl-cert-check -qac /etc/letsencrypt/live/stats.crin.org/cert.pem -e "chris@webarchitects.co.uk" 

So I think that is sorted for Crin1, next Let's Encrypt could be set up on Crin4.

comment:4 Changed 3 years ago by chris

  • Add Hours to Ticket changed from 0 to 1.25
  • Total Hours changed from 1.75 to 3.0

Installing Let's Encrypt on Crin4:

echo "deb http://speglar.simnet.is/debian/ jessie-backports main " >> /etc/apt/sources.list
aptitude update ; aptitude install letsencrypt
mkdir /var/www/letsencrypt/.well-known/acme-challenge/ -p

Add the following the the Nginx config:

        # Let's Encrypt https://trac.crin.org.archived.website/trac/ticket/64
        location /.well-known/acme-challenge/ {
                alias /var/www/letsencrypt/.well-known/acme-challenge/;
                default_type  "text/plain";
                try_files $uri =404;
        }

Test:

https://solr.crin4.crin.org/.well-known/acme-challenge/foo
https://dev.crin.org/.well-known/acme-challenge/foo
https://www.dev.crin.org/.well-known/acme-challenge/foo
https://stage.crin.org/.well-known/acme-challenge/foo
https://www.stage.crin.org/.well-known/acme-challenge/foo

Dot files were blocked so change this:

        #location ~ (^|/)\. {
        #       return 403;
        #}
        location ~ \.(git|svn|htaccess|htpasswd) {
               return 403;
        }

Generate a key and get a get a cert:

letsencrypt certonly --webroot --webroot-path /var/www/letsencrypt --renew-by-default --email chris@webarchitects.co.uk --text --agree-tos -d dev.crin.org -d www.dev.crin.org -d stage.crin.org -d www.stage.crin.org -d solr.crin4.crin.org

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/dev.crin.org/fullchain.pem. Your cert will
   expire on 2016-05-29. To obtain a new version of the certificate in
   the future, simply run Let's Encrypt again.
 - If you like Let's Encrypt, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Update the cipher list from:

Install ssl-cert-check and add a check for the expiry date in root's crontab:

30 09 * * * ssl-cert-check -qac /etc/letsencrypt/live/dev.crin.org/cert.pem -e "chris@webarchitects.co.uk" 

Check the setup:

comment:5 follow-up: Changed 2 years ago by chris

  • Add Hours to Ticket changed from 0 to 0.45
  • Total Hours changed from 3.0 to 3.45

The Let's Encrypt certs new renewing, I'm getting this alert every day:

Date: Wed, 04 May 2016 09:30:01 +0000
From: root <root@crin1.crin.org>
Subject: Certificate for FILE "(CN: stats.crin.org)" will expire in 30-days or less

The SSL certificate for FILE "(CN: stats.crin.org)" will expire on May 28 12:29:00 2016 GMT

So looking at the documentation we need to run:

letsencrypt renew

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/stats.crin.org.conf
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/stats.crin.org/fullchain.pem
-------------------------------------------------------------------------------

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/stats.crin.org/fullchain.pem (success)

That good and it can be added to the root crontab, however we also need an Apache restart for the new cert to be used, if the script is run again:

letsencrypt renew

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/stats.crin.org.conf
-------------------------------------------------------------------------------

The following certs are not due for renewal yet:
  /etc/letsencrypt/live/stats.crin.org/fullchain.pem (skipped)
No renewals were attempted.

So I think an Apache restart could be scripted, but for now I think manually running this command ever two month and manually doing a restart would take less time -- scripting it would take a few hours to sort out., so doing it manually:

apache2ctl restart

And we now have a new cert:

comment:6 Changed 2 years ago by chris

  • Add Hours to Ticket changed from 0 to 0.15
  • Total Hours changed from 3.45 to 3.6

Email regarding Crin4:

Date: Thu, 19 May 2016 08:23:13 +0000
From: expiry@letsencrypt.org
To: chris@webarchitects.co.uk
Subject: Let's Encrypt certificate expiration notice

Hello,

Your certificate (or certificates) for the names listed below will expire in 10
days (on 29 May 16 09:58 +0000). Please make sure to renew your certificate
before then, or visitors to your website will encounter errors.

dev.crin.org
solr.crin4.crin.org
stage.crin.org
www.dev.crin.org
www.stage.crin.org

For any questions or support, please visit https://community.letsencrypt.org/.
Unfortunately, we can't provide support by email.

So:

letsencrypt renew


-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/dev.crin.org.conf
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/dev.crin.org/fullchain.pem
-------------------------------------------------------------------------------

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/dev.crin.org/fullchain.pem (success)

/etc/init.d/nginx restart
[ ok ] Restarting nginx (via systemctl): nginx.service.

Again this could do with being added as a crontab in the future.

comment:7 Changed 2 years ago by chris

  • Add Hours to Ticket changed from 0 to 0.1
  • Total Hours changed from 3.6 to 3.7

The replacement of letsencrypt with certbot, ticket:17#comment:95, has also come with a working cron job to renew the certs, as far as I'm aware, so all that might be needed on each server is a restart of apache2 / nginx every now and then...

comment:8 in reply to: ↑ 5 Changed 2 years ago by chris

  • Add Hours to Ticket changed from 0 to 0.35
  • Cc matthew added
  • Total Hours changed from 3.7 to 4.05

This morning Apache was reporting an expired cert for this site so I checked that the crontab was in place to send alerts when the cert is close to expiring, it was:

30 09 * * * ssl-cert-check -qac /etc/letsencrypt/live/stats.crin.org/cert.pem -e "chris@webarchitects.co.uk" 

Then I tried to renew the cert:

letsencrypt renew

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/stats.crin.org.conf
-------------------------------------------------------------------------------

The following certs are not due for renewal yet:
  /etc/letsencrypt/live/stats.crin.org/fullchain.pem (skipped)
No renewals were attempted.

And then I remembered that an Apache restart was needed to pick up the new cert... as documented above:

Replying to chris:

So I think an apache restart could be scripted, but for now I think manually running this command ever two month and manually doing a restart would take less time -- scripting it would take a few hours to sort out.

So that was done:

/etc/init.d/apache2 restart
[ ok ] Restarting apache2 (via systemctl): apache2.service.

That solved the issue, the current cert is valid from 1st September so a new one should be in place by 1st November, (they are valid 3 months and renewed after two months), so we could have a crontab like this to run every two months, but that would end up running on the wrong month:

# m h  dom mon dow   command
0 01 01 */2 * service apache2 restart

So a script could be written for this or we could simply restart apache once a month for now... that is what I have put in place:

# apache restart for Let's Encrypt
0 01 01 * * service apache2 restart

I'll leave this ticket open so we can revisit this if needs be.

Note: See TracTickets for help on using tickets.