When "Automatic data recovery" on packet not forwarded to TTN

This was solved in V1.1.0063_Release_r205

I trying to write logic on my application side to handle cases like these. For RAK gateways I trust the gateway’s timestamp, as the packets can be buffered and sent to the network at quite some time later. Therefore the network time is incorrect.

However, the gateway metadata does not contain a time:

{
   "app_id":"myapp",
   "dev_id":"mydevice",
   "hardware_serial":"1122334455667788",
   "port":130,
   "counter":2388,
   "payload_raw":"JNAITQhmCIsI",
   "metadata":{
      "time":"2021-04-21T13:55:19.427237979Z",
      "frequency":868.1,
      "modulation":"LORA",
      "data_rate":"SF11BW125",
      "airtime":741376000,
      "coding_rate":"4/5",
      "gateways":[
         {
            "gtw_id":"eui-ac1f09fffe013d9b",
            "timestamp":1442497396,
            "time":"", // <== THIS IS EMPTY!!
            "channel":0,
            "rssi":-124,
            "snr":-12.8,
            "rf_chain":0
         }
      ]
   }
}

Because the time field for the gateway is empty, I do not know when the packets were received, and therefore I can not use the automatic data recovery feature to back fill missed data.

Is there a way we can have the time field populated so that the data recovery feature can be used?

You got this message from TTN, right ? Why not use timestamp as the time ? I guess the time field was for the gateway who has GPS inside.

Please see packet_forwarder/PROTOCOL.TXT at master · Lora-net/packet_forwarder · GitHub part 4.

  • time is the wall time as an iso string.
  • tmms is the GPS timestamp
  • tmst is an internal counter keeping track of when packets are received and when downlinks should be scheduled. All relative time, and has no connection to wall time. You can see this when trying to interpret 1442497396 above as a unix timestamp.

Two more examples. You will see how the timestamp value increased a lot more than what a unix timestamp would have.

{"app_id":"myapp","dev_id":"mydevice","hardware_serial":"redacted","port":130,"counter":2627,"payload_raw":"JDoHAghyCJEI","metadata":{"time":"2021-04-22T06:56:55.901018097Z","frequency":867.7,"modulation":"LORA","data_rate":"SF11BW125","airtime":741376000,"coding_rate":"4/5","gateways":[{"gtw_id":"eui-ac1f09fffe013d9b","timestamp":2609434412,"time":"","channel":6,"rssi":-116,"snr":-7,"rf_chain":0}]}}

{"app_id":"myapp","dev_id":"mydevice","hardware_serial":"redacted","port":31,"counter":2628,"payload_raw":"AAA=","metadata":{"time":"2021-04-22T06:58:01.280731299Z","frequency":868.3,"modulation":"LORA","data_rate":"SF11BW125","airtime":659456000,"coding_rate":"4/5","gateways":[{"gtw_id":"eui-ac1f09fffe013d9b","timestamp":2674798644,"time":"","channel":1,"rssi":-120,"snr":-9.5,"rf_chain":0}]}}

Timestamp is microseconds, gateway unique, rolls over about hourly and resets any time the concentrator chip does, so you can’t really use that.

Realistically packet archiving is incompatible with the LoRaWan spec prohibition of out of sequence frame counts, especially in a public network with gateways run by others. If you want to do that you probably need custom software to send the archived packets somewhere other than TTN for independent validation and decryption.

Or you can guess at archived packet times based on frame count and assuming a somewhat consistent interval between packet attempts.

Indeed, but in some situations there is no other way to prevent data loss during power outages.

A couple of gateway manufacturers implemented this feature (Kerlink).

Using Lacuna’s satellite system, which stores and forwards data to a LoRaWAN network, the same issue will happen. Therefore The Things Stack V3 will be adding a special channel via which buffered packets can be uploaded, without affecting the mac state.
See Support downstream resending traffic out-of-order · Issue #2708 · TheThingsNetwork/lorawan-stack · GitHub

Regarding the RAK gateway, it’s a workaround that is needed until TTS implements the new api.

@yutao can we get the gateway’s time added into the time field? If the gateway is NTP synced, this time is accurate enough.

Hi @jpmeijers . The time have already exist in the GWMP protocol. But it only be reported when GPS was fixed.
It means that the Indoor gateway with out GPS will not report time.

/* Packet RX time (GPS based), 37 useful chars */
            if (ref_ok == true) {
                /* convert packet timestamp to UTC absolute time */
                j = lgw_cnt2utc(local_ref, p->count_us, &pkt_utc_time);
                if (j == LGW_GPS_SUCCESS) {
                    /* split the UNIX timestamp to its calendar components */
                    x = gmtime(&(pkt_utc_time.tv_sec));
                    j = snprintf((char *)(buff_up + buff_index), TX_BUFF_SIZE-buff_index, ",\"time\":\"%04i-%02i-%02iT%02i:%02i:%02i.%06liZ\"", (x->tm_year)+1900, (x->tm_mon)+1, x->tm_mday, x->tm_hour, x->tm_min, x->tm_sec, (pkt_utc_time.tv_nsec)/1000); /* ISO 8601 format */
                    if (j > 0) {
                        buff_index += j;
                    } else {
                        MSG("ERROR: [up] snprintf failed line %u\n", (__LINE__ - 4));
                        exit(EXIT_FAILURE);
                    }
                }
                /* convert packet timestamp to GPS absolute time */
                j = lgw_cnt2gps(local_ref, p->count_us, &pkt_gps_time);
                if (j == LGW_GPS_SUCCESS) {
                    pkt_gps_time_ms = pkt_gps_time.tv_sec * 1E3 + pkt_gps_time.tv_nsec / 1E6;
                    j = snprintf((char *)(buff_up + buff_index), TX_BUFF_SIZE-buff_index, ",\"tmms\":%" PRIu64 "", pkt_gps_time_ms); /* GPS time in milliseconds since 06.Jan.1980 */
                    if (j > 0) {
                        buff_index += j;
                    } else {
                        MSG("ERROR: [up] snprintf failed line %u\n", (__LINE__ - 4));
                        exit(EXIT_FAILURE);
                    }
                }
            }

This is the code of packet forwarder

Indeed, what the Semtech code makes clearer than the description is that in the Semtech protocol that’s a GPS referenced time so shouldn’t be used for an approximate when feeding a public server which expects GPS accuracy or nothing.

Doing this usefully would need a new scheme, with new software on both ends - which if the infrastructure end is TTN cannot happen until they have a spec to target.

What could work right now is for jpmeijers to point the packet forwarder at their own local archiving code running on or next to the gateway that would also pass it to TTN when connected, add approximate timestamps there, and then when possible ship archived packets to their own decoding logic server outside of TTN.

Or to work out approximate times of the archived packets based on the increment of their frame count relative to packets timely processed through TTN.

Of course there are workarounds, but none solves the problem elegantly.

Currently the data recovery feature of the RAK7258 does not work. I bought these gateways for this feature. If RAK’s gateways’ data recovery feature doesn’t work, there is no reason for me to buy their gateways, and I’d be better off with a cheaper brand like Mikrotik.

I tried enabling fake gps in the hopes that the packet forwarder will then add the time field. Unfortunately that does not seem to work.

I filed an issue on the Semtech repo: time field empty when no GPS present · Issue #141 · Lora-net/packet_forwarder · GitHub

Fake GPS off:

{
   "gtw_id":"eui-60c5a8fffe761551",
   "timestamp":11257844,
   "time":"",
   "channel":5,
   "rssi":-38,
   "snr":9.5,
   "rf_chain":0
}

Fake GPS on:

{
   "gtw_id":"eui-60c5a8fffe761551",
   "timestamp":600364156,
   "time":"",
   "channel":5,
   "rssi":-38,
   "snr":9,
   "rf_chain":0,
   "latitude":"<REDACTED>",
   "longitude":"<REDACTED>",
   "altitude":190
}

Emm. Fake GPS will not work. Because the packet forwarder need to sync the time with GPS, not only the location.
Anyway. I will provide you a firmware version which include time field without GPS.
May I have your E-mail ? You can send it to [email protected].
Thanks you.

1 Like

The code does what it is supposed to, so that won’t go far with Semtech.

The only mistake is in your expectation.

To repeat, packet archiving is fundamentally incompatible with the design of LoRaWan.

It’s only really workable in the case of a custom network that starts from LoRaWan, but then diverges in allowing this.