Demo software suddenly stopped working with TTN

Hi, I have a Pi with the RAK 2243 hat and a Sparkfun Explorable Thing Plus. I followed Sparkfun’s setup guides and had the Arduino sending data through the RAK gateway to TTN without much trouble at all. In the TTN console I can see the gateway status being updated and the live data coming from the Arduino.
This all worked fine for about a day. Setting it up again today and I can see that the gateway is sending updates to TTN, but their console says that my device has not sent any data. I don’t think this is correct, however, because the gateway view shows messages coming from my device id.

But then I look at the device activity and it says there hasn’t been activity since yesterday.
[removed my screenshot of that because I’m told I can only insert one media item]

I haven’t made any configuration changes or done anything that I think would have broken things. Indeed it looks to me like the Arduino is still sending messages over LoRa, through the RAK gateway, and on to TTN.

I also went through my previous day’s syslog files and I don’t see anything that looks like it logged my successfully sent messages. Right now I’m seeing this in syslog.

Any advice would be greatly appreciated and please let me know if there is any other info I can provide.
Cheers,
-Matt

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # PULL_DATA sent: 6 (100.00% acknowledged)

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # PULL_RESP(onse) datagrams received: 0 (0 bytes)

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # RF packets sent to concentrator: 0 (0 bytes)

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # TX errors: 0

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # BEACON queued: 0

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # BEACON sent so far: 0

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # BEACON rejected: 0

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: ### [JIT] ###

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # SX1301 time (PPS): 2822591235

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: src/jitqueue.c:448:jit_print_queue(): INFO: [jit] queue is empty

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: ### [GPS] ###

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # GPS sync is disabled

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: ##### END #####

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: JSON up: {“stat”:{“time”:“2023-06-16 20:07:14 GMT”,“rxnb”:0,“rxok”:0,“rxfw”:0,“ackr”:100.0,“dwnb”:0,“txnb”:0}}

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: [up] PUSH_ACK received in 33 ms

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: [down] PULL_ACK received in 33 ms

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: [down] PULL_ACK received in 33 ms

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: [down] PULL_ACK received in 41 ms

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: [down] PULL_ACK received in 35 ms

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: [down] PULL_ACK received in 35 ms

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: [down] PULL_ACK received in 32 ms

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: Disabling GPS mode for concentrator’s counter…

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: host/sx1301 time offset=(1686943181s:670467µs) - drift=-47µs

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: Enabling GPS mode for concentrator’s counter.

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: ##### 2023-06-16 20:07:44 GMT #####

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: ### [UPSTREAM] ###

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # RF packets received by concentrator: 0

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # CRC_OK: 0.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # RF packets forwarded: 0 (0 bytes)

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # PUSH_DATA datagrams sent: 1 (113 bytes)

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # PUSH_DATA acknowledged: 100.00%

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: ### [DOWNSTREAM] ###

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # PULL_DATA sent: 6 (100.00% acknowledged)

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # PULL_RESP(onse) datagrams received: 0 (0 bytes)

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # RF packets sent to concentrator: 0 (0 bytes)

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # TX errors: 0

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # BEACON queued: 0

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # BEACON sent so far: 0

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # BEACON rejected: 0

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: ### [JIT] ###

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # SX1301 time (PPS): 2882591654

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: src/jitqueue.c:448:jit_print_queue(): INFO: [jit] queue is empty

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: ### [GPS] ###

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # GPS sync is disabled

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: ##### END #####

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: JSON up: {“stat”:{“time”:“2023-06-16 20:07:44 GMT”,“rxnb”:0,“rxok”:0,“rxfw”:0,“ackr”:100.0,“dwnb”:0,“txnb”:0}}

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: [up] PUSH_ACK received in 33 ms

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: [down] PULL_ACK received in 33 ms

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: [down] PULL_ACK received in 35 ms

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: [down] PULL_ACK received in 34 ms

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: [down] PULL_ACK received in 34 ms

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: [down] PULL_ACK received in 32 ms

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: [down] PULL_ACK received in 32 ms

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: ##### 2023-06-16 20:08:14 GMT #####

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: ### [UPSTREAM] ###

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # RF packets received by concentrator: 0

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # CRC_OK: 0.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # RF packets forwarded: 0 (0 bytes)

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # PUSH_DATA datagrams sent: 1 (113 bytes)

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # PUSH_DATA acknowledged: 100.00%

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: ### [DOWNSTREAM] ###

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # PULL_DATA sent: 6 (100.00% acknowledged)

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # PULL_RESP(onse) datagrams received: 0 (0 bytes)

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # RF packets sent to concentrator: 0 (0 bytes)

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # TX errors: 0

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # BEACON queued: 0

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # BEACON sent so far: 0

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # BEACON rejected: 0

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: ### [JIT] ###

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # SX1301 time (PPS): 2882591654

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: src/jitqueue.c:448:jit_print_queue(): INFO: [jit] queue is empty

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: ### [GPS] ###

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # GPS sync is disabled

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: ##### END #####

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: JSON up: {“stat”:{“time”:“2023-06-16 20:08:14 GMT”,“rxnb”:0,“rxok”:0,“rxfw”:0,“ackr”:100.0,“dwnb”:0,“txnb”:0}}

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: [up] PUSH_ACK received in 36 ms

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: [down] PULL_ACK received in 34 ms

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: [down] PULL_ACK received in 33 ms

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: [down] PULL_ACK received in 34 ms

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: [down] PULL_ACK received in 34 ms

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: [down] PULL_ACK received in 33 ms

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: Received pkt from mote: 260CA5AA (fcnt=129)

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: JSON up: {“rxpk”:[{“tmst”:2939529491,“chan”:4,“rfch”:1,“freq”:904.700000,“stat”:1,“modu”:“LORA”,“datr”:“SF7BW125”,“codr”:“4/5”,“lsnr”:9.2,“rssi”:-79,“size”:20,“data”:“QKqlDCaAgQAB1nGqK2swulgDWQ0=”}]}

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: [up] PUSH_ACK received in 32 ms

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: [down] PULL_ACK received in 34 ms

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: Disabling GPS mode for concentrator’s counter…

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: host/sx1301 time offset=(1686943181s:670422µs) - drift=-45µs

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: INFO: Enabling GPS mode for concentrator’s counter.

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: ##### 2023-06-16 20:08:44 GMT #####

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: ### [UPSTREAM] ###

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # RF packets received by concentrator: 1

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # CRC_OK: 100.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # RF packets forwarded: 1 (20 bytes)

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # PUSH_DATA datagrams sent: 2 (315 bytes)

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # PUSH_DATA acknowledged: 100.00%

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: ### [DOWNSTREAM] ###

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # PULL_DATA sent: 6 (100.00% acknowledged)

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # PULL_RESP(onse) datagrams received: 0 (0 bytes)

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # RF packets sent to concentrator: 0 (0 bytes)

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # TX errors: 0

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # BEACON queued: 0

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # BEACON sent so far: 0

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # BEACON rejected: 0

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: ### [JIT] ###

Jun 16 13:08:44 rak-gateway ttn-gateway[392]: # SX1301 time (PPS): 2942592047

As a followup, I discovered a fix for this. In the TTN console, I went to my device’s “General settings” tab and expanded the “Network layer” section. Just to see what would happen I clicked the “Reset session and MAC state” button.
Almost immediately the console showed data flowing in from my device again. It has since been running correctly.
So my question now is: What happened that made it so I needed to reset the session? And how can I prevent it from happening in the future? This is just a demo application, but when I scale this application up to tens or hundreds of devices I don’t want to have to keep pressing that button to keep them working.
Thanks for any help.
-Matt