Puppet certificate expired on granet

Since late yesterday afternoon the puppet certificate for the granet server had expired.
This morning all icinga alerts on this server were in “unkown” status.

First I tried the 'classic' renewal but puppet agent failed:

root@granet:~# puppet agent --test
Error: Could not download CA certificate: Bad Request
Error: Could not download CA certificate: Bad Request

Puppet version between pergamon and granet mismatch:

root@pergamon:~# awk '$1=="ii"' <(dpkg -l puppet-master)
ii  puppet-master  5.5.22-2     all          configuration management system, master service

root@granet:~# awk '$1=="ii"' <(dpkg -l puppet)
ii  puppet         7.23.0-1     all          transitional dummy package

added activity::MRO label

assigned to @guillaume

I tried to copy ca.pem and crl.pem on the new directory /var/lib/puppet/ssl on granet but puppet agent still failed:

root@granet:~# puppet agent -t
Info: Creating a new RSA SSL key for granet.internal.softwareheritage.org
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for granet.internal.softwareheritage.org
Info: Certificate Request fingerprint (SHA256): D2:86:3F:52:C7:2A:C6:D3:93:1A:61:81:20:61:CB:D0:28:0B:54:60:BC:82:20:6D:09:11:71:63:54:7B:85:27
Error: undefined method `message' for #<Puppet::HTTP::ResponseNetHTTP:0x00007f04d2374410 @url=#<URI::HTTPS https://pergamon.internal.softwareheritage.org:8140/puppet-ca/v1/certificate/granet.internal.softwareheritage.org>, @code=400, @reason="Bad Request", @nethttp=#<Net::HTTPBadRequest 400 Bad Request readbody=true>>

                 {certname: Puppet[:certname], message: e.response.message}, e)
                                                                  ^^^^^^^^
Error: undefined method `message' for #<Puppet::HTTP::ResponseNetHTTP:0x00007f04d2374410 @url=#<URI::HTTPS https://pergamon.internal.softwareheritage.org:8140/puppet-ca/v1/certificate/granet.internal.softwareheritage.org>, @code=400, @reason="Bad Request", @nethttp=#<Net::HTTPBadRequest 400 Bad Request readbody=true>>

                 {certname: Puppet[:certname], message: e.response.message}, e)

Then:

• generated a certificate on pergamon:

root@pergamon:/var/lib/puppet/ssl# puppet certificate generate granet.internal.softwareheritage.org --ca-location remote
Warning: 'puppet certificate' is deprecated and will be removed in a future release
   (location: /usr/lib/ruby/vendor_ruby/puppet/application/face_base.rb:244:in `main')
true

• sign the new certificate:

root@pergamon:/var/lib/puppet/ssl# puppet certificate sign granet.internal.softwareheritage.org --ca-location remote
Warning: 'puppet certificate' is deprecated and will be removed in a future release
   (location: /usr/lib/ruby/vendor_ruby/puppet/application/face_base.rb:244:in `main')
Error: Error 403 on SERVER: Not Authorized: Forbidden request: /puppet-ca/v1/certificate_status/granet.internal.softwareheritage.org [save]
Error: Try 'puppet help certificate sign' for usage

Although the command ends with an error, the files are created.

root@pergamon:~# find /var/lib/puppet/ssl -name "*granet.internal.softwareheritage.org*"
/var/lib/puppet/ssl/public_keys/granet.internal.softwareheritage.org.pem
/var/lib/puppet/ssl/ca/signed/granet.internal.softwareheritage.org.pem
/var/lib/puppet/ssl/certs/granet.internal.softwareheritage.org.pem
/var/lib/puppet/ssl/certificate_requests/granet.internal.softwareheritage.org.pem
/var/lib/puppet/ssl/private_keys/granet.internal.softwareheritage.org.pem

Finally just copy the public key, the private key and the certificate in granet with the right permissions:

root@granet:~# tree -paug /var/lib/puppet/ssl/
[drwxrwx--x puppet   puppet  ]  /var/lib/puppet/ssl/
|-- [drwxr-xr-x puppet   puppet  ]  certificate_requests
|-- [drwxr-xr-x puppet   puppet  ]  certs
|   |-- [-rw-r--r-- puppet   puppet  ]  ca.pem
|   `-- [-rw-r--r-- puppet   puppet  ]  granet.internal.softwareheritage.org.pem
|-- [-rw-r--r-- puppet   puppet  ]  crl.pem
|-- [drwxr-x--- puppet   puppet  ]  private
|-- [drwxr-x--- puppet   puppet  ]  private_keys
|   `-- [-rw-r----- puppet   puppet  ]  granet.internal.softwareheritage.org.pem
`-- [drwxr-xr-x puppet   puppet  ]  public_keys
    `-- [-rw-r--r-- puppet   puppet  ]  granet.internal.softwareheritage.org.pem

6 directories, 5 files

Puppet agent now runs successfully (the icinga certificate was renewed during the first execution):

root@granet:~# puppet agent -t
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Warning: The current total number of facts: 2401 exceeds the number of facts limit: 2048
Info: Caching catalog for granet.internal.softwareheritage.org
Info: Applying configuration version '1728986089'
Notice: Applied catalog in 7.43 seconds

I don't know if the CSR can be deleted on pergamon.

added 3h of time spent

closed

mentioned in issue #5415 (closed)