Gateways using MQTT / SSL with CA server signed cert from Lets Encrypt is broken

As planned, the Let’s Encrypt DST Root CA X3 cross-sign expired 30th September 2021

Video on the topic - DST Root CAX3 Expiration Sept 2021

They’ve made some changes to keep supporting older android clients, but this makes it incompatible with clients on older versions of openSSL, such as my Rak 7258 on openSSL 1.0.2j

I’ve manually edited the .pem file (on my server) to remove the offending certificate and now everything seems to be working.

I don’t pretend to fully understand this and hope it remains secure! Has anyone else run into this problem? How did you solve it?

I guess the proper fix is upgrading to openSSL 1.1.0+, but I believe that would require new firmware for the gateway

2 Likes

What file do you edit? I´m facing issues installing helium gateway-rs service into my RAK7258 gateway and trying to add my gateway to the blockchain. Everything point to the expired certificate will cause a lot of troubles when we need connect the gateways to new and present systems.

Just Like reference Improve curl requests errors handling · Issue #120 · helium/gateway-rs · GitHub

We have the same issue with the RAK7258, the latest firmware is still on OpenSSL 1.0.2j requiring a manual workaround every time on the certificate.

Request to RAK: Can you release a new firmware with OpenSSL 1.1.0 or newer?

Does anyone know if there has been any movement with this issue?

Just checked the new firmware: WisGateOS 1.3.1_RAK 20220119 b71
The OpenSSL is still: OpenSSL 1.0.2j 26 Sep 2016

Just checked the new firmware: WisGateOS v1.3.2 (1.3.2_RAK b75)
Still OpenSSL 1.0.2j 26 Sep 2016

Rak staff, is there any plan to update OpenSSL, please?

Just found out (after another major outage of gateways) that manually adjusting the Let’s Encrypt certificate after it was updated might not always be enough.
It seems that when the certificate was used before modification (which happens when you adjust it after the automatic update), the gateways not only fail in the period until the new certificate is adjusted, they keep failing on the correctly adjusted certificate. Not even a power cycle of the gateway is enough to resolve this, only by applying the LoRa Network Settings again or a firmware update the gateway starts to work with the adjusted certificate. Unfortunatly this requires us to physically visit each gateway location.

I pushed this issue via our sales contacts to RAK support. They raised it with their development team and I understand the plan is to update the OpenSSL to version 1.1.1. Lets see what is in the next release.

1 Like

Thanks @nkoot; that sounds very promising!

I’m not sure if this is the same problem or not, but sometimes when a gateway is power cycled, it fails to reconnect and the log is filled with this.

daemon.err mqttEv[1949]: mqttEvRdReady: mosquitto [[gwBridgeS Mqtt Client]] loop misc ERR: 4

It repeats so fast that the load averages are high and the tmpfs starts to fill up.

No additional power cycles will resolve it. Applying the same LoRa Network settings again (as mentioned by @nkoot) does solve it

We got a new Let’s Encrypt certificate for the MQTT broker today, we have now setup the Short/Alternate Chain (Long (default) and Short (alternate) Certificate Chains Explained - Issuance Policy - Let's Encrypt Community Support) so it is immediatelly the right one. That worked fine, the MQTT reconnected and kept on working.

However after a power cycle 1 hour later it would not reconnnect and flood the log with:
daemon.err mqttEv[1949]: mqttEvRdReady: mosquitto [[gwBridgeS Mqtt Client]] loop misc ERR: 4
I applied the LoRa network settings again and finally managed to resolve that as before (as also mentioned by @mtbspace). As that problem appears to be independent from the long certificate chain with OpenSSL issue I created a new topic on it: Daemon.err mqttEv[2031]: mqttEvRdReady: mosquitto [[gwBridgeS Mqtt Client]] loop misc ERR: 4

There is a new firmware 1.3.3 out, although there is no mention of OpenSSL nor certificates in the release notes I will give it a try shortly and check its OpenSSL version.

Unfortunatly firmware 1.3.3 still has the old OpenSSL 1.0.2j 26 Sep 2016

Just installed firmware 1.3.4, still OpenSSL 1.0.2j 26 Sep 2016.

I asked RAK again for a timeline on this.

They now promote WisGate OS 2 (WisGateOS 2 - Gateways Operating System - Built-in LoRaWAN Network Server - RAKwireless - IoT Made Easy) which has OpenSSL 1.1 as a feature. Unfortunatly it does not seem to support our RAK7258.

Quick response from RAK with hopefull news: OpenSSL will be updated in the next firmware update of the WisGateOS1.x in August (worst in early September)

1 Like

WisGateOS 1.3.5 is available and it has OpenSSL 1.1.1q 5 Jul 2022

This should work with Lets Encrypt again. I was not able to verify that yet as we would need all our gateways the be updated first.

The WisGateOS 1.3.5 upgrade looks fine so far on my RAK7258.

The WisGateOS 1.3.5 firmware isn’t available under the RAK7258 downloads yet, but you can get it directly here: https://downloads.rakwireless.com/LoRa/WisGateOS/WisGateOS_V1.3.5.zip

1 Like