Issue: A power cycle after a new SSL certificate on the MQTT broker does no longer connect MQTT for ChirpStack 3.x (JSON) and fills the log with hundrets of entries per second containing: daemon.err mqttEv[2031]: mqttEvRdReady: mosquitto [[gwBridgeS Mqtt Client]] loop misc ERR: 4
Setup:
RAK 7258
Firmware Version: 1.3.2_RAK b75
Region: EU863-870
Protocol: MQTT for ChirpStack 3.x (JSON)
SSL/TLS Mode: CA signed server certificate
TLS Version: TLSv1.2
LoRa® Server: ChirpStack EU868 @ 3.13.0
Details:
This only occurs after the MQTT broker certificate is updated, and not immediatelly because after reconnecting to the MQTT broker with the new certificate it keeps running fine. Only after a power cycle of the RAK7258 does it no longer connect and is the log filled up with:
daemon.err mqttEv[2031]: mqttEvRdReady: mosquitto [[gwBridgeS Mqtt Client]] loop misc ERR: 4
The above occurs hundrets of times per second, every now and then with this variant:
daemon.err mqttEv[2031]: mqttLoopMiscTimerHandler: mosquitto [[gwBridgeS Mqtt Client]] loop misc ERR: 4
As we only get a new certificate every 3 months (Let’s Encrypt) it is difficult to troubleshoot. I first noticed it 3 months ago and then tought it was related to: Gateways using MQTT / SSL with CA server signed cert from Lets Encrypt is broken
and the fact that we shortly had a certificate that doesn’t work with the old OpenSSL used by the RAK7258.
However we now have changed the Let’s Encrypt certificate generation so it is immediatelly correct, still the problem occured after we got a new certificate today.
The problem can be resolved by going to Lora Network > Network Settings and then press Save & Apply. Last time this resolved it immediately. Today it initially resolved it, but after a power cycle it occured again. Again pressed Save & Apply, got all kind of other errors about failing to start the gateway (forgot to capture those), but after another power cycle it was working again. It keeps on working now after several additional power cycles.
I have a RAK7258 I use for testing that is power cycled every day, it worked without issue for 3 months until the certificate was updated.
We have several RAK7258 in the field, they are still running ok, last time they disconnected only after they got power cycled days/weeks after the certificate update and we had to rush over to resolve it. This time we will visit them preemptive after the new certificate, but that is not a workable situation.