RAK3172 does not respond to AT+ commands after 48 days

beegee · April 12, 2023, 11:17am

R43 is required. Remove only R44 please.

guiff · April 12, 2023, 2:47pm

I removed R44 and do a reset, not working. I held reset for 1 second and not working too
I sent “AT+BOOTSTATUS”, but not receive any response.
It’s only responding when remove battery and put again.

Our R&D think it has something to do with the RAM, so when you remove the battery, it loses the RAM data and goes back to normal, make sense to you?

We have a product with 39 days of operation, we believe it will stop on the 48th day, can we have a chance to check something in its memory, any counters?

danak6jq · April 12, 2023, 7:07pm

I’d bet it’s a 32-bit value of milliseconds overflowing as I mention above. Way too close to be a coincidence; this works out to 2^32 / 1000mS / 3600 seconds / 24 hours → 49 days, 17h, 2m, 47 seconds

Slightly more than 48 days exactly but way too close to ignore.

guiff · April 12, 2023, 11:13pm

@danak6jq It’s possible.

Additional information, this issue not occurs in 1.0.4, we have products running for 223 days.

guiff · April 13, 2023, 1:58pm

I did another test, I took two products, one that the RAK3172 is not responding, and another that it is responding, the same FW in both. I measured the consumption without active low power mode, and performed a physical reset on both modules, both have a very similar consumption variation during the reset, the consumption in operation is the same, but one responds normally to AT commands, and the other does not respond. So I believe that the reset worked normally, but in the module that stops responding, he is not able to make it go back to normal operation.

Egodomenico · April 13, 2023, 5:51pm

I thought the same thing, in fact I manage timers in ms with uint_64

beegee · April 14, 2023, 2:34am

@guiff
Thank you for that test, I think that is an important information.
To bad you are controlling the JOIN manually from your host MCU. It would be interesting to know if only the UART is malfunctioning and the device would auto-join the network.

For the other reporters of this issue. Does anyone of you have auto-join enabled and do you see the hanging device joining after a hard reset, but just not responding on the UART?

Matejisko · April 14, 2023, 8:23am

Yesterday, i discovered that another one of my devices running fw v3.5.3 stopped working after 49 days and around 15-18 hours. Taking into account the oscillator stability, everything around 49 days could have the same cause.

guiff · April 14, 2023, 1:39pm

We still have a module that will probably fail in a few days. Is there any pin that we could see if the module is still running by its state?

@Matejisko, very interesting, thank you for sharing.

beegee · April 15, 2023, 9:11am

@guiff

There is no pin that would tell you the status. The only thing that could show if it re-joins after a reset would be if join would be set to automatic and you would see the join request on the gateway or LNS.
But I understood from your commands, that you initiate the join process manually from your host MCU. If the UART does not work, it will never try to join.

Matejisko · April 20, 2023, 11:36am

Hi,

I have some news,

During the last two days, 8 of my devices stopped working after 26-27 days of continuous operation. All of these devices were flashed with rui 4.0.0 + custom firmware, which only forces the baud rate to 9600. These devices were powered up within a few hours.

custom fw:

void setup(void)
{
   // Force Serial to 9600 Baud
   Serial.begin(9600);
}
void loop(void)
{
   // No need for the loop, kill it
   api.system.scheduler.task.destroy();
}

Another 3 devices with rui 3.5.3 (no custom fw) stopped working after 34 days of operation.

It is interesting that another 5 devices with RUI 4.0.0 + custom firmware are still alive, but they transmit every hour or so with low TX power and SF=7, compared to the previously mentioned devices with Txp=16 dBm, SF=12, and a transmit period of approximately 10 minutes.

Regarding this topic, any update on this?

Matejisko · April 21, 2023, 8:36am

Hi again,

An important update…

Today, i checked my devices and all of them seems to operate normally. I use the Helium network and past days there was something like Network migration to Solana. Although I read somewhere that it should not affect the actual data transfer, most likely there was a system outage so i could not see the data transfers.

In reference to the 5 devices that I previously mentioned as working normally (with different settings…), I’d like to note that they are connected to my “data only” gateway, which was likely not affected by this migration.

I apologize for the rushed conclusion.

Have a nice day

guiff · April 21, 2023, 11:28am

Hi,

We have not performed further tests, we are awaiting a response from R&D.

Regards,

beegee · April 28, 2023, 7:23am

For everyone who is experiencing this problem, I want to apologize for the inconvenience that this bug is causing all of you.

We found the root cause for the RAK3172 hanging after ~48 days.

We have tested the bug fix and will release patches for all RUI3 versions that have this bug.

The bug appears in the following RUI3 versions:
RUI3 V3.5.3
RUI3 V3.5.4
RUI3 V4.0.0
RUI3 V4.0.1

For users of the standard AT command firmware that have validated their product with one of these RUI3 versions, we will provide patches that will only fix this specific bug and do not change anything else.

For users of the RUI3 BSP, we will release new BSP versions for all affected versions.

Here is the list of the affected RUI3 versions and the new version numbers of the patched firmware and BSP:

Version	Comment
RUI_3.5.5	Patch for RUI_3.5.3
RUI_3.5.6	Patch for RUI_3.5.4
RUI_4.0.2	Patch for RUI_4.0.0
RUI_4.0.3	Patch for RUI_4.0.1

beegee · April 28, 2023, 11:34am

Updated AT command firmware are available for download:

Latest RUI3 BSP can be installed with ArduinoIDE board manager.
Latest version with the bug fix is 4.0.3.

[RUI_4.0.3] RAKwireless-Arduino-BSP-Index/RUI_4.0.3_233 at staging · RAKWireless/RAKwireless-Arduino-BSP-Index · GitHub

guiff · April 28, 2023, 12:27pm

That’s great news, we’ll test it out as soon as the AT+ version comes out, today we’ve chosen to use version 4.0.1, with a RAK3172 reset logic every 30 days.

beegee · April 28, 2023, 12:53pm

Patch for RUI3 V4.0.1 is already available, it is V4.0.3:

[RUI_4.0.3] RAKwireless-Arduino-BSP-Index/RUI_4.0.3_233 at staging · RAKWireless/RAKwireless-Arduino-BSP-Index · GitHub

pmjackson · April 28, 2023, 1:12pm

Can you tell us what the nature of the “bug” is?
I mean… what happens to hang the device?

beegee · April 29, 2023, 10:57am

It is a timer overflow (as Dana suggested), but the main problem is that the handler that should capture the overflow exception was not implemented correct. This led to a infinite loop that even disabled the hardware reset.

pmjackson · May 1, 2023, 4:09pm

@beegee . Question?
To go from RUI3 3.5.3 to RUI3 4.0.3_233 BSD, What would be the steps for this?
i.e. Do we need to use STM32CubeProgrammer and complete erase and then flash with RAK3172-E_latest_final.hex?
Side question… does using a BSD compiled file completely update the RAK3172 to that RUI?