No Ack on Command on LoraServer 3.X (Protobuf)

Issue: Downlink command doesn’t Ack since the last Release

Setup: Firmware in last release 1.1.0062 R202

Older version firmware :
[email protected]:~# cat /etc/openwrt_version
1.1.0056_Release
[email protected]:~# cat /etc/openwrt_release
DISTRIB_VERSION=“1.1.0056”
DISTRIB_RELEASE=“1.1.0056_Release”
DISTRIB_REVISION=“193”
DISTRIB_TARGET=“ramips”
DISTRIB_DATE=“20191112”
DISTRIB_TIME=“181531”
DISTRIB_OEM=""
DISTRIB_PRODUCT=""

Updated Firmware
[email protected]:/# cat /etc/openwrt_release
DISTRIB_VERSION=“1.1.0062”
DISTRIB_RELEASE=“1.1.0062_Release”
DISTRIB_REVISION=“202”
DISTRIB_TARGET=“ramips”
DISTRIB_DATE=“20200313”
DISTRIB_TIME=“163707”
DISTRIB_OEM=""
DISTRIB_PRODUCT=""

LoRa® Server: OFF

Details:
I’m using 2 RAK7249 gateways. I sent IMMEDIATE message over MQTT to the gateway using the LoraServer3.X (Protobuf).
All was ok until I made an update of the firmware on one of my gateway. Since this update, the gateway doesn’t respond with Ack anymore.

I paste here the log that concerne the problem on R202 revision
The connection is reset after the donwlink command reception.
Note that the same downlink commande receives a Ack on R193.

Thu Apr 9 08:57:28 2020 daemon.info gwBridge[1606]: mosqMsgCB topic : gateway/60c5a8fffe76f8a4/command/down
Thu Apr 9 08:57:28 2020 daemon.info gwBridge[1606]: gwBridgeParseMosqDownlink mqtt downlink packet - size 123
Thu Apr 9 08:57:28 2020 daemon.info loraSrv[1604]: child process 1606 exited due to signal 11
Thu Apr 9 08:57:28 2020 daemon.info evloop[2838]: evloopInit Done.
Thu Apr 9 08:57:28 2020 daemon.info md[2838]: mdInit Done.
Thu Apr 9 08:57:28 2020 daemon.info mqttEv[2838]: mqttEvInit Done.
Thu Apr 9 08:57:28 2020 daemon.info gwBridge[2838]: Enter gwBridgeInit
Thu Apr 9 08:57:28 2020 daemon.info unknown[2838]: gwManagerInit: Init gateway Manager
Thu Apr 9 08:57:28 2020 daemon.info bufrd[2838]: bufrdInit Done.
Thu Apr 9 08:57:28 2020 daemon.info bufrd[2838]: ENTER bufrdCreate `gwbridge-rd’
Thu Apr 9 08:57:28 2020 daemon.info gwBridge[2838]: MQTT Init Start
Thu Apr 9 08:57:28 2020 daemon.info gwBridge[2838]: Mqtt - version 3.1
Thu Apr 9 08:57:28 2020 daemon.info gwBridge[2838]: Mqtt - Client ID gwbridge-60c5a8fffe76f8a4
Thu Apr 9 08:57:28 2020 daemon.info gwBridge[2838]: Mqtt - Retain 0
Thu Apr 9 08:57:28 2020 daemon.info gwBridge[2838]: Mqtt - Clean Session 1
Thu Apr 9 08:57:28 2020 daemon.info gwBridge[2838]: Mqtt - Qos 1
Thu Apr 9 08:57:28 2020 daemon.info gwBridge[2838]: Mqtt - keepalive 10
Thu Apr 9 08:57:28 2020 daemon.info mqttEv[2838]: mqttEvRegister : Register mosquitto event [gwBridgeS Mqtt Client]
Thu Apr 9 08:57:28 2020 daemon.info gwBridge[2838]: MQTT Init Done
Thu Apr 9 08:57:28 2020 daemon.info loraSrv[2838]: Entering evloopRun
Thu Apr 9 08:57:28 2020 daemon.info evloop[2838]: ENTER evloopRun
Thu Apr 9 08:57:28 2020 daemon.info evloop[2838]: evloopOnce: initial time; assume 0 elapsed time
Thu Apr 9 08:57:28 2020 daemon.info gwBridge[2838]: mosqConnectCB: Mqtt Connect Success …
Thu Apr 9 08:57:28 2020 daemon.info mqttEv[2838]: [gwBridgeS Mqtt Client] subscribe gateway/60c5a8fffe76f8a4/command/down - QOS 1
Thu Apr 9 08:57:31 2020 user.info lora_pkt_fwd[1584]:
INFO: Disabling GPS mode for concentrator’s counter…
Thu Apr 9 08:57:31 2020 user.info lora_pkt_fwd[1584]: INFO: host/sx1301 time offset=(1586422457s:237430µs) - drift=-2709µs
Thu Apr 9 08:57:31 2020 user.info lora_pkt_fwd[1584]: INFO: Enabling GPS mode for concentrator’s counter.

@yutao Please help here.

This seems to be a bug. I am trying to solve it

Ok, thanks for the reply. I’ll continue my work with the previous firmware version.

Note that loraserver 3.x specified a “gateway/+/event/exec” which is the result of the execution of a given command.
It could be usefull to publish it too.

Hi:
I tried to send IMMEDIATE downlink message and TIMESTAMPED downlink message.Everything looks OK.
Can you share me your mqtt bridge configuration of gateway?

Fri Apr 10 07:09:17 2020 daemon.info gwBridge[3458]: mosqMsgCB topic : gateway/60c5a8fffe76125c/command/down
Fri Apr 10 07:09:17 2020 daemon.info gwBridge[3458]: gwBridgeParseMosqDownlink mqtt downlink packet - size 101
Fri Apr 10 07:09:17 2020 daemon.info gwBridge[3458]: [TxAck] - Protobuf size 31
Fri Apr 10 07:09:17 2020 daemon.info mqttEv[3458]: mqttEvPublish : [[gwBridgeS Mqtt Client]] publish
Fri Apr 10 07:09:19 2020 daemon.info gwBridge[3458]: mosqMsgCB topic : gateway/60c5a8fffe76125c/command/down
Fri Apr 10 07:09:19 2020 daemon.info gwBridge[3458]: gwBridgeParseMosqDownlink mqtt downlink packet - size 102
Fri Apr 10 07:09:19 2020 daemon.info gwBridge[3458]: [TxAck] - Protobuf size 32
Fri Apr 10 07:09:19 2020 daemon.info mqttEv[3458]: mqttEvPublish : [[gwBridgeS Mqtt Client]] publish
Fri Apr 10 07:09:21 2020 daemon.info gwBridge[3458]: mosqMsgCB topic : gateway/60c5a8fffe76125c/command/down
Fri Apr 10 07:09:21 2020 daemon.info gwBridge[3458]: gwBridgeParseMosqDownlink mqtt downlink packet - size 102
Fri Apr 10 07:09:21 2020 daemon.info gwBridge[3458]: [TxAck] - Protobuf size 32
Fri Apr 10 07:09:21 2020 daemon.info mqttEv[3458]: mqttEvPublish : [[gwBridgeS Mqtt Client]] publish
Fri Apr 10 07:09:23 2020 daemon.info gwBridge[3458]: [Uplink] - Protobuf size 153
Fri Apr 10 07:09:23 2020 daemon.info mqttEv[3458]: mqttEvPublish : [[gwBridgeS Mqtt Client]] publish
Fri Apr 10 07:09:42 2020 daemon.info gwBridge[3458]: Stats : Protobuf size 97
Fri Apr 10 07:09:42 2020 daemon.info mqttEv[3458]: mqttEvPublish : [[gwBridgeS Mqtt Client]] publish
Fri Apr 10 07:09:57 2020 daemon.info gwBridge[3458]: mosqMsgCB topic : gateway/60c5a8fffe76125c/command/down
Fri Apr 10 07:09:57 2020 daemon.info gwBridge[3458]: gwBridgeParseMosqDownlink mqtt downlink packet - size 102
Fri Apr 10 07:09:57 2020 daemon.info gwBridge[3458]: [TxAck] - Protobuf size 32
Fri Apr 10 07:09:57 2020 daemon.info mqttEv[3458]: mqttEvPublish : [[gwBridgeS Mqtt Client]] publish
Fri Apr 10 07:10:01 2020 daemon.info gwBridge[3458]: [Uplink] - Protobuf size 151
Fri Apr 10 07:10:01 2020 daemon.info mqttEv[3458]: mqttEvPublish : [[gwBridgeS Mqtt Client]] publish
Fri Apr 10 07:10:12 2020 daemon.info gwBridge[3458]: Stats : Protobuf size 97
Fri Apr 10 07:10:12 2020 daemon.info mqttEv[3458]: mqttEvPublish : [[gwBridgeS Mqtt Client]] publish
Fri Apr 10 07:10:20 2020 daemon.info db[3428]: DataBase Backup
Fri Apr 10 07:10:42 2020 daemon.info gwBridge[3458]: Stats : Protobuf size 91
Fri Apr 10 07:10:42 2020 daemon.info mqttEv[3458]: mqttEvPublish : [[gwBridgeS Mqtt Client]] publish

Configuration MQTT bridge

Packet forwarder conf

I use the ProtoBuf definition for the gateway messages from Chirpstack

I code in GoLang and use the standard Golang protoBuf Marshaller

The build-in loraserver is stopped

Why the MQTT Broker Address is localhost (127.0.0.1)? Did you use the MQTT Broker inside the gateway to communicate with Chripstack?

I use the following architecture :
Internal Gateway -> Internal MQTT broker -> External Application Connected as MQTT client
In this configuration, we have no internal or external LoraWan Server.

Our application catches the Lora Message from the gateway, stores messages and sent it later to another Main Gateway over Lora.

Note that , in order to enable the gateway to gateway communication, we use a small hack (as specified in pure Lora Layer), and the gateway sent the message with polarization inversion=false.

Interesting. So, the External Application sub / pub the MQTT Messages.
This the proto files of MQTT Bridge.
gw.proto
syntax = “proto3”;

package gw;

option go_package = "github.com/brocaar/chirpstack-network-server/api/gw";

import "common.proto";
import "google/protobuf/timestamp.proto";
import "google/protobuf/duration.proto";


enum DownlinkTiming {
    // Send the downlink immediately.
    IMMEDIATELY = 0;

    // Send downlink at the given delay (based on provided context).
    DELAY = 1;

    // Send at given GPS epoch value.
    GPS_EPOCH = 2;
}


enum FineTimestampType {
    // No fine-timestamp available.
    NONE = 0;

    // Encrypted fine-timestamp.
    ENCRYPTED = 1;

    // Plain fine-timestamp.
    PLAIN = 2;
}

message UplinkTXInfo {
	// Frequency (Hz).
	uint32 frequency = 1;

    // Modulation.
    common.Modulation modulation = 2;

    oneof modulation_info {
        // LoRa modulation information.
        LoRaModulationInfo lora_modulation_info = 3 [json_name = "loRaModulationInfo"];

        // FSK modulation information.
        FSKModulationInfo fsk_modulation_info = 4;
    }
}

message LoRaModulationInfo {
    // Bandwidth.
    uint32 bandwidth = 1;

    // Speading-factor.
    uint32 spreading_factor = 2;

    // Code-rate.
    string code_rate = 3;

    // Polarization inversion.
    bool polarization_inversion = 4;
}

message FSKModulationInfo {
    // Bandwidth.
    uint32 bandwidth = 1;

    // Bitrate.
    uint32 bitrate = 2;
}

message EncryptedFineTimestamp {
    // AES key index used for encrypting the fine timestamp.
    uint32 aes_key_index = 1;

    // Encrypted 'main' fine-timestamp (ns precision part of the timestamp).
    bytes encrypted_ns = 2 [json_name = "encryptedNS"];

    // FPGA ID.
    bytes fpga_id = 3 [json_name = "fpgaID"];
}

message PlainFineTimestamp {
    // Full timestamp.
    google.protobuf.Timestamp time = 1;
}

message GatewayStats {
    // Gateway ID.
    bytes gateway_id = 1 [json_name = "gatewayID"];

    // Gateway IP.
    string ip = 9;

    // Gateway time.
    google.protobuf.Timestamp time = 2;

    // Gateway location.
    common.Location location = 3;

    // Gateway configuration version (this maps to the config_version sent
    // by ChirpStack Network Server to the gateway).
    string config_version = 4;

    // Number of radio packets received.
    uint32 rx_packets_received = 5;

    // Number of radio packets received with valid PHY CRC.
    uint32 rx_packets_received_ok = 6 [json_name = "rxPacketsReceivedOK"];

    // Number of downlink packets received for transmission.
    uint32 tx_packets_received = 7;

    // Number of downlink packets emitted.
    uint32 tx_packets_emitted = 8;

    // Additional gateway meta-data.
    map<string, string> meta_data = 10;

    // Stats ID (UUID).
    // Unique identifier for the gateway stats.
    bytes stats_id = 11 [json_name = "statsID"];
}

message UplinkRXInfo {
    // Gateway ID.
    bytes gateway_id = 1 [json_name = "gatewayID"];

    // RX time (only set when the gateway has a GPS module).
    google.protobuf.Timestamp time = 2;

    // RX time since GPS epoch (only set when the gateway has a GPS module).
    google.protobuf.Duration time_since_gps_epoch = 3 [json_name = "timeSinceGPSEpoch"];

    // RSSI.
    int32 rssi = 5;

    // LoRa SNR.
    double lora_snr = 6 [json_name = "loRaSNR"];

    // Channel.
    uint32 channel = 7;

    // RF Chain.
    uint32 rf_chain = 8;

    // Board.
    uint32 board = 9;

    // Antenna.
    uint32 antenna = 10;

    // Location.
    common.Location location = 11;

    // Fine-timestamp type.
    FineTimestampType fine_timestamp_type = 12;

    // Fine-timestamp data.
    oneof fine_timestamp {
        // Encrypted fine-timestamp data.
        EncryptedFineTimestamp encrypted_fine_timestamp = 13;

        // Plain fine-timestamp data.
        PlainFineTimestamp plain_fine_timestamp = 14;
    }

    // Gateway specific context.
    bytes context = 15;

    // Uplink ID (UUID bytes).
    // Unique and random ID which can be used to correlate the uplink across multiple logs.
    bytes uplink_id = 16 [json_name = "uplinkID"];
}

message DownlinkTXInfo {
    // Gateway ID.
    bytes gateway_id = 1 [json_name = "gatewayID"];

    // TX frequency (in Hz).
    uint32 frequency = 5;

    // TX power (in dBm).
    int32 power = 6;

    // Modulation.
    common.Modulation modulation = 7;

    oneof modulation_info {
        // LoRa modulation information.
        LoRaModulationInfo lora_modulation_info = 8 [json_name = "loRaModulationInfo"];

        // FSK modulation information.
        FSKModulationInfo fsk_modulation_info = 9;
    }

    // The board identifier for emitting the frame.
    uint32 board = 10;

    // The antenna identifier for emitting the frame.
    uint32 antenna = 11;

    // Timing defines the downlink timing to use.
    DownlinkTiming timing = 12;

    oneof timing_info {
        // Immediately timing information.
        ImmediatelyTimingInfo immediately_timing_info = 13;

        // Context based delay timing information.
        DelayTimingInfo delay_timing_info = 14;

        // GPS Epoch timing information.
        GPSEpochTimingInfo gps_epoch_timing_info = 15;
    }

    // Gateway specific context.
    // In case of a Class-A downlink, this contains a copy of the uplink context.
    bytes context = 16;
}

message ImmediatelyTimingInfo {
    // Not implemented yet.
}


message DelayTimingInfo {
    // Delay (duration).
    // The delay will be added to the gateway internal timing, provided by the context object.
    google.protobuf.Duration delay = 1;
}

message GPSEpochTimingInfo {
    // Duration since GPS Epoch.
    google.protobuf.Duration time_since_gps_epoch = 1 [json_name = "timeSinceGPSEpoch"];
}

message UplinkFrame {
    // PHYPayload.
    bytes phy_payload = 1;

    // TX meta-data.
    UplinkTXInfo tx_info = 2;

    // RX meta-data.
    UplinkRXInfo rx_info = 3;
}

message UplinkFrameSet {
    // PHYPayload.
    bytes phy_payload = 1;

    // TX meta-data.
    UplinkTXInfo tx_info = 2;

    // RX meta-data set.
    repeated UplinkRXInfo rx_info = 3;
}

message DownlinkFrame {
    // PHYPayload.
    bytes phy_payload = 1;

    // TX meta-data.
    DownlinkTXInfo tx_info = 2;

    // Token (uint16 value).
    // Deprecated: will be replaced by downlink_id.
    uint32 token = 3;

    // Downlink ID (UUID).
    bytes downlink_id = 4 [json_name = "downlinkID"];
}

message DownlinkTXAck {
    // Gateway ID.
    bytes gateway_id = 1 [json_name = "gatewayID"];

    // Token (uint16 value).
    // Deprecated: will be replaced by downlink_id.
    uint32 token = 2;

    // Error.
    string error = 3;

    // Downlink ID (UUID).
    bytes downlink_id = 4 [json_name = "downlinkID"];
}

message GatewayConfiguration {
    // Gateway ID.
    bytes gateway_id = 1 [json_name = "gatewayID"];

    // Configuration version.
    string version = 2;

    // Channels.
    repeated ChannelConfiguration channels = 3;
}

message ChannelConfiguration {
    // Frequency (Hz).
    uint32 frequency = 1;

    // Channel modulation.
    common.Modulation modulation = 2;

    oneof modulation_config {
        // LoRa modulation config.
        LoRaModulationConfig lora_modulation_config = 3 [json_name = "loRaModulationConfig"];

        // FSK modulation config.
        FSKModulationConfig fsk_modulation_config = 4;
    }

    // Board index.
    uint32 board = 5;

    // Demodulator index (of the given board).
    uint32 demodulator = 6;
}

message LoRaModulationConfig {
    // Bandwidth.
    uint32 bandwidth = 1;

    // Spreading-factors.
    repeated uint32 spreading_factors = 2;
}

message FSKModulationConfig {
    // Bandwidth.
    uint32 bandwidth = 1;

    // Bitrate.
    uint32 bitrate = 2;
}

message GatewayCommandExecRequest {
    // Gateway ID.
    bytes gateway_id = 1 [json_name = "gatewayID"];

    // Command to execute.
    // This command must be pre-configured in the ChirpStack Gateway Bridge configuration.
    string command = 2;

    // Execution request ID (UUID).
    // The same token will be returned when the execution of the command has
    // completed.
    bytes ExecId = 3 [json_name = "execID"];

    // Standard input.
    bytes stdin = 4;

    // Environment variables.
    map<string, string> environment = 5;
}

message GatewayCommandExecResponse {
    // Gateway ID.
    bytes gateway_id = 1 [json_name = "gatewayID"];

    // Execution request ID (UUID).
    bytes exec_id = 2 [json_name = "execID"];

    // Standard output.
    bytes stdout = 3;

    // Standard error.
    bytes stderr = 4;

    // Error message.
    string error = 5;
}

common.proto
syntax = “proto3”;

package common;

option go_package = "github.com/brocaar/chirpstack-network-server/api/common";


enum Modulation {
    // LoRa
    LORA = 0;

    // FSK
    FSK = 1;
}

enum Region {
    // EU868
    EU868 = 0;

    // US915
    US915 = 2;

    // CN779
    CN779 = 3;

    // EU433
    EU433 = 4;

    // AU915
    AU915 = 5;

    // CN470
    CN470 = 6;

    // AS923
    AS923 = 7;

    // KR920
    KR920 = 8;

    // IN865
    IN865 = 9;

    // RU864
    RU864 = 10;
}


message KeyEnvelope {
    // KEK label.
    string kek_label = 1;

    // AES key (when the kek_label is set, this key is encrypted using a key
    // known to the join-server and application-server.
    // For more information please refer to the LoRaWAN Backend Interface
    // 'Key Transport Security' section.
    bytes aes_key = 2;
}

enum LocationSource {
    // Unknown.
    UNKNOWN = 0;

    // GPS.
    GPS = 1;

    // Manually configured.
    CONFIG = 2;

    // Geo resolver.
    GEO_RESOLVER = 3;
}

message Location {
    // Latitude.
    double latitude = 1;

    // Longitude.
    double longitude = 2;

    // Altitude.
    double altitude = 3;

    // Location source.
    LocationSource source = 4;

    // Accuracy (in meters).
    uint32 accuracy = 5;
}

message ContextDetails {
    // Context ID (UUID).
    bytes context_id = 1;
}

You can share your “External Application” to me so that I can reproduce the problem?
[email protected]