Updating custom certificates on Red Hat Satellite 6

Updating custom certificates on Red Hat Satellite 6

Overview

Satellite 6 hides the certificates used as part of installation away somewhere special and does not like them to be updated. This is a huge pain when certificates expire, the process to update current certificates is to run a command to re-install certificates.

Finding this information online took way longer than it should have and it made me mad, I wanted to add this post to help the next person struggling with an upgrade due to expired custom certificates with Red Hat Satellite 6.

Error message on upgrade of Satellite 6

Attempting a simple upgrade of Red Hat Satellite 6, things were going well, the upgrade check worked well and I started the actual upgrade. Everything was working as expected, the database was upgraded and on to the upgrade of the Satellite web components, this is where things got a little weird.

The below error was thrown signalling an error that was not recoverable by the upgrade process. The error failed the upgrade and I was left wondering where to go from here.

 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[satellite.company.org]: Failed to call refresh: Exception SSL_connect returned=1 errno=0 state=unknown state: excessive message size in get request to: https://satellite.company.org/api/v2/smart_proxies?search=name=%22satellite.company.org%22
 
/Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[satellite.company.org]: Exception SSL_connect returned=1 errno=0 state=unknown state: excessive message size in get request to: https://satellite.company.org/api/v2/smart_proxies?search=name=%22satellite.company.org%22

Invalid certificates

I checked the certificates provided by the Satellite GUI before starting the process and everything was valid. I tried to check the certificates for the GUI again from the browser, and the certificate was now showing as expired months ago.

This is where the wheels fell off for me. I could see the certificate pointed to in Apache was different to what it was before the upgrade, so I added back the valid certificate, restarted Apache, tested using openssl s_client -connect to ensure the certificate was valid and re-ran the upgrade. The upgrade failed with the same error and when I checked the certificate, it had reverted back to expired certificate again.

I was not able to find clear details on the error or where custom certificates where stored for Satellite 6, so I did what any person with a problem that needed fixing would do, I ran an md5sum on all files with crt or pem on the whole operating system looking for matches of the wrong certificate. I had a hunch that Satellite was obsessed with a certificate file somewhere on the OS and was replacing files as part of the upgrade. After replacing all offending crt and pem files with matching md5sum in turn, I re-ran the upgrade after each change, each time the same error was displayed and the upgrade failed.

Finding the real fix

Widening my online search terms, I found a very helpful blog post by Luc de Louw (https://blog.delouw.ch/2019/06/11/renew-letsencrypt-certificates-for-red-hat-satellite-6-and-capsule/) explaining how to rotate Satellite certificates when using LetsEncrypt as a CA. The blog post clearly described how updating certificates required running the satellite-installer command with certificate arguments.

After recovering from what appeared to be a very strange way to handle certificates, I crafted up a command to install the new certificates, a few attempts at getting the chain certificate sorted out, and I was able to update the custom certificates and completed the upgrade.

#> satellite-installer --scenario satellite --certs-server-cert "/etc/pki/tls/satellite/satellite.crt" --certs-server-key "/etc/pki/tls/satellite/satellite.key" --certs-server-ca-cert "/etc/pki/tls/satellite/satellite-server-ca.crt" --certs-update-server --certs-update-server-ca

Conclusion

So much documentation and many videos exist on the Red Hat website, but distilling it down to a simple question of updating custom certificates for Satellite 6 was very hard for me. I am sure your mileage may vary on finding pertinent information on this topic, but I had a difficult time getting an answer to what I felt was a simple question about the failed upgrade.

The important moment for me was finding the satellite-installer command was used to update certificates. I had the simple answer to my problem and found a way to update the certificates required.

The process was a pain for a few reasons, but the main issue I had with the process is how Satellite obfuscated certificates behind its installer. I life cycled (incorrectly it appears) the certificates when they expired by updating Apache, which worked with no issues for many months. It was only on upgrade the shifting of certificates from the base installation over took everything and I had no idea where files were being copied from.

I later found a way to see which files were used as part of the installation by looking into the /etc/foreman-installer/scenarios.d/satellite-answers.yaml file. I had looked in this file before and updated each cert during my mad md5sum period, but changing the certificates to valid ones did not seem to allow the correct certificates to be used. This could just be my mistake as my brain was slowly melting while working through this upgrade.

#> cat /etc/foreman-installer/scenarios.d/satellite-answers.yaml |grep -iE 'server_key|server_cert_req|server_ca_cert|server_cert'
server_cert: /etc/pki/tls/satellite/satellite.crt
server_key: /etc/pki/tls/satellite/satellite.key
server_cert_req: /root/ssl/cert/Satellite6_csr.csr
server_ca_cert: /etc/pki/tls/satellite/satellite-server-ca.crt
server_certname: satellite.company.org

Red Hat Satellite has been a pain for me for a long time, it does a great job at managing a fleet of RHEL servers and proxying RHN, but managing the tool itself has always felt harder than it should be. This is another example how something that should be simple, turned out to be far harder than it should be.

Useful resources