[Rak2245] Packet forwarder stop working after 1 day with 2 nodes

Dear all,
I am running packet forwarder with 1 node in 1 week is ok. But it seem stop working after 1 day when running with 2 nodes. It only working again when i stop and start it. This is log show when it stop. Please give me recommence

journalctl -u packetfw -eroot@kumino:~# systemctl status packetfw
[[0;1;32m?[[0m packetfw.service - The Packet Forwarder
Loaded: loaded (/lib/systemd/system/packetfw.service; enabled; vendor preset:
enabled)
Active: [[0;1;32mactive (running)[[0m since Sat 2021-03-27 21:59:49 +07; 10h
ago
Main PID: 5574 (lora_pkt_fwd)
Tasks: 4 (limit: 1630)
Memory: 452.0K
CGroup: /system.slice/packetfw.service
mq5574 /opt/lora-packet-forwarder/lora_pkt_fwd

Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 61 m
s
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 159
ms
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 42 m
s
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 159
ms
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 56 m
s
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 150
ms
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 159
ms
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 41 m
s
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 159
ms
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 189
ms

root@kumino:~# systemctl status packetfwroot@kumino:~# journalctl -u packetfw -e
(END)Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 47 m
s
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 169
ms
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 46 m
s
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 159
ms
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 48 m
s
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 55 m
s
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 179
ms
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 46 m
s
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 160
ms
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 50 m
s
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 56 m
s
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 159
ms
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 60 m
s
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 61 m
s
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 159
ms
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 42 m
s
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 159
ms
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 56 m
s
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 150
ms
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 159
ms
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 41 m
s
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 159
ms
Mar 28 06:12:12 kumino semtech=pwd[5574]: INFO: [down] PULL_ACK received in 189
ms

~
(END)

Your log doesn’t show anything meaningful, it covers only a tiny slice of time during which nothing happened radio wise.

You need to examine logs of a meaningful period of time during which there was some radio traffic - which will typically be several minutes in between per node.

Hei @tuyenpham ,
@cstratton is right, however it might also help to give a little bit of context.
For example is this the rak firmware, is it raspberry pi os with the semtech packet forwarder, perhaps you have made modifications, etc.
How about the nodes, some details about how they are configured, activation mode, frequency band, ADR, Data rate, etc.

Regards
Vladislav

Because had a limited number of character. I attached link for full log at the gg drive link:
https://drive.google.com/file/d/1DyaJFw3m1pfbnCIaMslIIlsZWpPhDcek/view?usp=sharing
More information about node:

  • Freq: EU433
  • ADR: true
  • Datarate: SF12
  • Use chipstack build on server
  • CPU gateway use imx6 with package forwarder

thanks and regards,
Tuyen

Your log shows things working, and then there stop being any packets, but there’s no way to know from that alone if the issue is that none were received, or that none were transmitted. Do you have some independent means to verify that the node(s) are still transmitting?

If this is a custom system, one possibility could be that power problems have caused the radio subsystem to get into a bad state. Resetting the concentratrator and restarting the packet forwarder might bring it back. But really you want to make sure that there’s a clean and ample power supply - often people will use an insufficient power supply only able to handle the average current and not the peak, assume it’s okay because it works “most of the time” but have unexplained issues. You probably want a 4 or 5 amp supply.

Another issue could be SPI glitching causing similar problems. With the RAK2245 you probably want to use a 1 MHz SPI clock - some may say 2 MHz but 1 MHz is safer. Also make sure that your SPI traces or wires are very short.

Thanks cstratton,

  1. I can make sure that my node still working because i only restart packet forwarder then system can working again
  2. I am using adapter 10 amp for gateway
  3. Sorry, my bad. I am using rak2247 with USB interface. So we can’t any issue with SPI right?

regards,

That’s good, but also consider the power routing to the card and how it may respond to abrupt pulse loads

That means there’s an FTDI USB-SPI chip, but there’s still SPI on the far side of that. I’m not sure if the slow level shifter is still involved in that case. You could try configuring a slower SPI clock when talking through the FTDI, too.

hi cstratton,
How we can configurate spi speed via USB interface? any command from linux host?

thanks

It would be somewhere the in the lora hal or else packet forwarder source code