Maybe we should be safe and only send a single uplink message per UDP packet? And not put more than one inside a UDP packet. We have no idea what the MTUs are of all the hops of the network. It could be very small if some VPNs are used inside eachother. Best is to remain on the safe side and stay far below the standard 1500byte MTU.
Emm. Two uplink message per UDP packet is also safe. The max sizeof per uplink message is 540. But this will cause a decrease in efficiency. Not every message has 540 bytes. Most messages are only a few dozen bytes.
I will try to dynamically adjust the number of messages in each UDP packet to ensure that the total length is less than the MTU.
1 Like
Perfect!
Path MTU discovery in practice says the following:
The minimal required MTU for all IPv6 hosts is 1,280, which is fair. Unfortunately for IPv4 the value is 576 bytes. On the other hand RFC4821 suggests that it’s “probably safe enough” to assume minimal MTU of 1,024.
So maybe use 1024?
Important thing to keep in mind is that the MTU you see on the gateway is not the minimum MTU of all the links on the path taken by the IP packets. It’s only the MTU for the first hop from the gateway to the first router.
Another problem is that the packet forwarder only send the next UDP packet after receive the ACK of the last one. If loragateway receives multiple long messages at once, it must be sent multiple times. If the network delay is large, the packets sent later may miss Rx1Window.
Good one! Definitely one of the issues with using the UDP packet forwarder. Basic Station should solve this.
But at this point Data Recovery is anyway very old data that missed the RX windows long time ago. So it’s fine to go on as we are.
I will try to send all udp pkt before collect the ACK.
1 Like
Hi @yutao. Are you succeeding to make the changes to the packet forwarder? Let me know if you have anything I can test.
This was solved in V1.1.0063_Release_r205
I trying to write logic on my application side to handle cases like these. For RAK gateways I trust the gateway’s timestamp, as the packets can be buffered and sent to the network at quite some time later. Therefore the network time is incorrect.
However, the gateway metadata does not contain a time:
{
"app_id":"myapp",
"dev_id":"mydevice",
"hardware_serial":"1122334455667788",
"port":130,
"counter":2388,
"payload_raw":"JNAITQhmCIsI",
"metadata":{
"time":"2021-04-21T13:55:19.427237979Z",
"frequency":868.1,
"modulation":"LORA",
"data_rate":"SF11BW125",
"airtime":741376000,
"coding_rate":"4/5",
"gateways":[
{
"gtw_id":"eui-ac1f09fffe013d9b",
"timestamp":1442497396,
"time":"", // <== THIS IS EMPTY!!
"channel":0,
"rssi":-124,
"snr":-12.8,
"rf_chain":0
}
]
}
}
Because the time field for the gateway is empty, I do not know when the packets were received, and therefore I can not use the automatic data recovery feature to back fill missed data.
Is there a way we can have the time field populated so that the data recovery feature can be used?
You got this message from TTN, right ? Why not use timestamp as the time ? I guess the time field was for the gateway who has GPS inside.
Please see packet_forwarder/PROTOCOL.TXT at master · Lora-net/packet_forwarder · GitHub part 4.
-
timeis the wall time as an iso string. -
tmmsis the GPS timestamp -
tmstis an internal counter keeping track of when packets are received and when downlinks should be scheduled. All relative time, and has no connection to wall time. You can see this when trying to interpret1442497396above as a unix timestamp.
Two more examples. You will see how the timestamp value increased a lot more than what a unix timestamp would have.
{"app_id":"myapp","dev_id":"mydevice","hardware_serial":"redacted","port":130,"counter":2627,"payload_raw":"JDoHAghyCJEI","metadata":{"time":"2021-04-22T06:56:55.901018097Z","frequency":867.7,"modulation":"LORA","data_rate":"SF11BW125","airtime":741376000,"coding_rate":"4/5","gateways":[{"gtw_id":"eui-ac1f09fffe013d9b","timestamp":2609434412,"time":"","channel":6,"rssi":-116,"snr":-7,"rf_chain":0}]}}
{"app_id":"myapp","dev_id":"mydevice","hardware_serial":"redacted","port":31,"counter":2628,"payload_raw":"AAA=","metadata":{"time":"2021-04-22T06:58:01.280731299Z","frequency":868.3,"modulation":"LORA","data_rate":"SF11BW125","airtime":659456000,"coding_rate":"4/5","gateways":[{"gtw_id":"eui-ac1f09fffe013d9b","timestamp":2674798644,"time":"","channel":1,"rssi":-120,"snr":-9.5,"rf_chain":0}]}}
Timestamp is microseconds, gateway unique, rolls over about hourly and resets any time the concentrator chip does, so you can’t really use that.
Realistically packet archiving is incompatible with the LoRaWan spec prohibition of out of sequence frame counts, especially in a public network with gateways run by others. If you want to do that you probably need custom software to send the archived packets somewhere other than TTN for independent validation and decryption.
Or you can guess at archived packet times based on frame count and assuming a somewhat consistent interval between packet attempts.
Indeed, but in some situations there is no other way to prevent data loss during power outages.
A couple of gateway manufacturers implemented this feature (Kerlink).
Using Lacuna’s satellite system, which stores and forwards data to a LoRaWAN network, the same issue will happen. Therefore The Things Stack V3 will be adding a special channel via which buffered packets can be uploaded, without affecting the mac state.
See Support downstream resending traffic out-of-order · Issue #2708 · TheThingsNetwork/lorawan-stack · GitHub
Regarding the RAK gateway, it’s a workaround that is needed until TTS implements the new api.
@yutao can we get the gateway’s time added into the time field? If the gateway is NTP synced, this time is accurate enough.
Hi @jpmeijers . The time have already exist in the GWMP protocol. But it only be reported when GPS was fixed.
It means that the Indoor gateway with out GPS will not report time.
/* Packet RX time (GPS based), 37 useful chars */
if (ref_ok == true) {
/* convert packet timestamp to UTC absolute time */
j = lgw_cnt2utc(local_ref, p->count_us, &pkt_utc_time);
if (j == LGW_GPS_SUCCESS) {
/* split the UNIX timestamp to its calendar components */
x = gmtime(&(pkt_utc_time.tv_sec));
j = snprintf((char *)(buff_up + buff_index), TX_BUFF_SIZE-buff_index, ",\"time\":\"%04i-%02i-%02iT%02i:%02i:%02i.%06liZ\"", (x->tm_year)+1900, (x->tm_mon)+1, x->tm_mday, x->tm_hour, x->tm_min, x->tm_sec, (pkt_utc_time.tv_nsec)/1000); /* ISO 8601 format */
if (j > 0) {
buff_index += j;
} else {
MSG("ERROR: [up] snprintf failed line %u\n", (__LINE__ - 4));
exit(EXIT_FAILURE);
}
}
/* convert packet timestamp to GPS absolute time */
j = lgw_cnt2gps(local_ref, p->count_us, &pkt_gps_time);
if (j == LGW_GPS_SUCCESS) {
pkt_gps_time_ms = pkt_gps_time.tv_sec * 1E3 + pkt_gps_time.tv_nsec / 1E6;
j = snprintf((char *)(buff_up + buff_index), TX_BUFF_SIZE-buff_index, ",\"tmms\":%" PRIu64 "", pkt_gps_time_ms); /* GPS time in milliseconds since 06.Jan.1980 */
if (j > 0) {
buff_index += j;
} else {
MSG("ERROR: [up] snprintf failed line %u\n", (__LINE__ - 4));
exit(EXIT_FAILURE);
}
}
}
This is the code of packet forwarder
Indeed, what the Semtech code makes clearer than the description is that in the Semtech protocol that’s a GPS referenced time so shouldn’t be used for an approximate when feeding a public server which expects GPS accuracy or nothing.
Doing this usefully would need a new scheme, with new software on both ends - which if the infrastructure end is TTN cannot happen until they have a spec to target.
What could work right now is for jpmeijers to point the packet forwarder at their own local archiving code running on or next to the gateway that would also pass it to TTN when connected, add approximate timestamps there, and then when possible ship archived packets to their own decoding logic server outside of TTN.
Or to work out approximate times of the archived packets based on the increment of their frame count relative to packets timely processed through TTN.
Of course there are workarounds, but none solves the problem elegantly.
Currently the data recovery feature of the RAK7258 does not work. I bought these gateways for this feature. If RAK’s gateways’ data recovery feature doesn’t work, there is no reason for me to buy their gateways, and I’d be better off with a cheaper brand like Mikrotik.
I tried enabling fake gps in the hopes that the packet forwarder will then add the time field. Unfortunately that does not seem to work.
I filed an issue on the Semtech repo: time field empty when no GPS present · Issue #141 · Lora-net/packet_forwarder · GitHub
Fake GPS off:
{
"gtw_id":"eui-60c5a8fffe761551",
"timestamp":11257844,
"time":"",
"channel":5,
"rssi":-38,
"snr":9.5,
"rf_chain":0
}
Fake GPS on:
{
"gtw_id":"eui-60c5a8fffe761551",
"timestamp":600364156,
"time":"",
"channel":5,
"rssi":-38,
"snr":9,
"rf_chain":0,
"latitude":"<REDACTED>",
"longitude":"<REDACTED>",
"altitude":190
}
Emm. Fake GPS will not work. Because the packet forwarder need to sync the time with GPS, not only the location.
Anyway. I will provide you a firmware version which include time field without GPS.
May I have your E-mail ? You can send it to [email protected].
Thanks you.
1 Like
The code does what it is supposed to, so that won’t go far with Semtech.
The only mistake is in your expectation.
To repeat, packet archiving is fundamentally incompatible with the design of LoRaWan.
It’s only really workable in the case of a custom network that starts from LoRaWan, but then diverges in allowing this.