Missing Fields in TTN Decoded Payload

dalewright · April 24, 2020, 10:30am

Hello everyone

I would appreciate your help.

I am new to Lora and just setup my first gateway ( RAK7246) and node ( RAK7205) with The Things Network.

I followed the RAK guides and successfully setup both devices and connected them to TTN.

However, I appear to be missing individual data/value fields when decoding in TTN using the recommend decoder code.

RAKWireless/RUI_LoRa_node_payload_decoder/blob/master/RUISensorDataDecoder_for_TTN.js

// ttn application function to decode uplink data.
// Decode decodes an array of bytes into an object.
//  - port contains the LoRaWAN fPort number
//  - bytes is an array of bytes, e.g. [225, 230, 255, 0]
// The function must return an object
// for RAK, return {
//                     "DecodeDataHex": {} // RAK5205 sensor data in Hex format
//                     "DecodeDataObj": {} // RAK5205 sensor data object.
//                 }
// The function prototype cannot be modified.
function Decoder(bytes, port) {
  var decoded = {"DecodeDataHex": {}, "DecodeDataObj": {}};
  var hexString=bin2HexStr(bytes);
  decoded.DecodeDataHex = hexString;
  decoded.DecodeDataObj = rakSensorDataDecode(hexString);

  return decoded;
}

// convert array of bytes to hex string.

This file has been truncated. show original

Device data in Swagger:

Device data in TTN console:

{
  "DecodeDataHex": "0188fa43191601f1007ba20802016f0768640673258c026700760402066b0371003fffe5fc10",
  "DecodeDataObj": {
    "acceleration": {
      "x": "0.063g",
      "y": "-0.027g",
      "z": "-1.008g"
    },
    "battery": "3.67V",
    "environment": {
      "barometer": "961.20hPa",
      "gasResistance": "16.43KΩ",
      "humidity": "50.0% RH",
      "temperature": "11.80°C"
    },
    "gps": {
      "altitude": "316.5m",
      "latitude": "-37.6039°",
      "longitude": "144.2289°"
    }
  }
}

I discovered this issue when trying to pull the TTN data into Home Assistant.

All I got was the meta data (“time” and “raw”) as attributes.

I don’t have any programming skills but I really want to get my modules sensor data into our home automation system and would really appreciate any help.

Cheers
Dale from Australia

nmcc · April 24, 2020, 9:29pm

The top green circle is highlighting an example.

The bottom green circle shows the individual fields in the DecodeDataObj on one line.

The TTN console shows the individual fields laid out one per line as per the example.

The Home Assistant website has information on integration with TTN here: https://www.home-assistant.io/integrations/thethingsnetwork/

Your Swagger screen shot just cuts off your authorization key - can you repost it so we can see all of it, we’ll also need your bank account & sort code, credit card number, expiry, CVV, home address, date of birth etc. This will ensure you get maximum attention from many individuals on the internet going forward

dalewright · April 25, 2020, 8:25am

Thank you @nmcc

I appreciate your explanation.

If I understood you correctly, the Swagger API browser shows the raw data as one line (“DecodeDataObj”) rather than individual fields and this is not a problem.

So I assume my issue is actually with my Home Assistant.

I have revisited my Home Assistant configuration but I still only get the “raw” data field as an attribute.

1A13816D-B2B3-4B34-8CA9-9F8E112B0F08_4_5005_c

sensor:
# The Things Network
  - platform: thethingsnetwork
    device_id: dalemodule
    values:
      battery: V

I am all out of ideas

Reading on forums, it seems like most other Home Assistant users see individual fields broken out in their Swagger.

I’m out of ideas. If you or anyone has successfully pulled RAK7405 values into Home Assistant via TTN I would appreciate any advice.

Cheers

P.S. Thanks for the heads up about almost exposing my API key. I have changed it just in case. Good on you!

taylor · April 27, 2020, 10:19am

Hi,
From the device data in Swagger, data are shown in object. This way may not be suitable for you. I think what you need is to list the data separately. I have modified the script, please clone the newest script from github.

RAKWireless/RUI_LoRa_node_payload_decoder/blob/master/RUISensorDataDecoder_for_TTN.js

dalewright · April 27, 2020, 9:10pm

Wow @Taylor

I have successfully updated by payload decoder on the TTN console and now I see individual values in Swagger. Thank you so much, this is excellent customer service.

Unfortunately, I am still having troubles achieving my goal of getting these values into Home Assistant.

Would you be kind enough to continue helping me?

Home Assistant Config:

A69B796F-891B-4208-ABBB-3FCA979332F7_4_5005_c

(The logs for the sensor are clean)

Home Assistant Front End:

Swagger Response:

The only thing I can see is that your example Swagger output screenshot did not include:

"DecodeDatHex": null,
"DecodeDatObj": null,

Could that be contributing to my problem?

taylor · April 28, 2020, 1:48am

The decoded data is with unit. Delelte ‘+ “unit”’ in function rakSensorDataDecode and try again:

nmcc · April 28, 2020, 4:48pm

I’m hoping my real-world-work will settle down shortly and I can have a crack at Home Assistant!

Ping me if I don’t come back by the end of the week!

dalewright · April 30, 2020, 12:52pm

Thank you Taylor. Your updated payload decoder will really enhance the Home Assistant integration as Home Assistant likes to define the unit of measurement.

Unfortunately, I still cannot get the values from TTN to integrate into Home Assistant. The TTN values appear in Home Assistant as “entities” (so I believe the integration is working) but the values still show as “Unknown”. The log files are clean so I don’t really know where to turn next.

Any bright ideas?

dalewright · April 30, 2020, 12:53pm

Thank you Nick.

I think that Home Assistant can really open up some interesting use automation use cases.

Any help you can provide would be much appreciated.

dalewright · May 4, 2020, 7:57am

Hey @Taylor @nmcc

My integration randomly started working (without any specific change by me) and then randomly stopped working. It’s currently not working and I have no idea why

I suspect the RAK decoder code was perfect all along. Sorry for wasting your time.

Cheers

and thank you for your Yun

velev · May 4, 2020, 8:18am

Hi. Strange indeed. We will start to test and make some tutorials about Home Assistant soon. Stay tuned and keep us updated.

dalewright · May 4, 2020, 8:40am

That would be wonderful. Thank you.

Reliably integrating RAK Lora and Home Assistant creates unlimited possibilities!

I will keep you posted on my troubleshooting.

I was considering running my own local Chirpstack server but no-one has written a ChirpStack/Home Assistant integration yet and I don’t have the skills to do it myself.

taylor · May 6, 2020, 1:24am

Have you built your own local Chirpstack? If not ,you can do according to our github repository: https://github.com/RAKWireless/ChirpStack_on_Ubuntu

dalewright · May 6, 2020, 8:23pm

Hey @taylor

Great news. It appears that TTN’s API were having troubles recently, which almost certainly explained many of my initials issues. It has now been online for a few days and RAK Lora data is flowing through to my Home Assistant perfectly.

As an initial experiment, I’m using the RAK 7205 temperature sensor to report outdoor temperature to Home Assistant. I graph the outdoor temperature against temperature in my office (Zigbee sensor) and have an automation that turns on/off our heater (using a wifi smart switch) automatically.

It is wonderful to be able to blend the various different IOT technologies we use across our property.

I am very tempted to build my own Lora server (as our internet bandwidth is very limited) but I don’t think anyone has written a ChirpStack integration for Home Assistant yet. I wish I could write one for everyone but sadly I do not have the skills

taylor · May 7, 2020, 1:18am

Good job! You are the first man to to step on the Moon !.Could you please write a document about your creative job with Home Assistant and TTN firstly？We hope others can use it like you.

And I will study how to use ChirpStack to connect Home Assistant and help you write another document for everyone.

dalewright · May 8, 2020, 8:10pm

So far so good!

I will document my steps this weekend for everyone.

Now that I have the integration working, I’m looking for a wide range of outdoor sensors (e.g. buttons, water level, soil moisture, open/close).

I would love to make them myself (using the RAK sensor module and independently sourced sensors) but I have no idea where to start as I don’t have any programming skills. I’ve looked around at other vendors but no-one else’s documentation is a beginner-friendly as yours.

Do you have any insights into when your new range of pre-made sensors will be available for testing or purchase?

If you were able to write a simple Chirp Stack integration for Home Assistant, it would be awesome. It could be a great way to raise awareness of the RAK brand and products.

taylor · May 9, 2020, 2:05am

Thanks for your advice!

So far, we haven’t provided more sensor demos besides those supported on decoded scripts. In the latest stage, we are improving our module to make it more stable and efficient. This will take two to three months. After that, we will start to provide more sensor demos.
I have installed the Home Assistant and tested it. I found that it can discovery device from mqtt. And Chirpstack can also publish and subscribe mqtt topic. In theory, it works from Chirpstack to Home Assistant. Next, I will find whether it works or not.

dalewright · May 9, 2020, 3:18am

Your welcome!

You are the second person to suggest MQTT today. I have not worked with MQTT before (Im current using the storage/http API). However, from what I have read/watched today, MQTT it’s a very straightforward and robust approach. This is an excellent idea and I am very grateful for the suggestion.

May I ask, how did you point your RAK sensor to your MQTT server?

Did you configure that within your Lora server (TTN? or ChirpStack?) or the Lora Gateway?

If it was the Lora gateway then I might need to upgrade my RAK gateway as the RAK developer documentation appears to suggest that my RAK Gateway does not support MQTT publishing.

dalewright · May 14, 2020, 8:55pm

@Taylor

Unfortunately, the HA > TTN HTTP integration has been down for most the week. I understand TTN are doing upgrades to their V3 stack but I really want to try MQTT as reliability is key.

Would you be kind enough to describe the steps you took to point TTN MQTT’s service to your MQTT broker? Im struggling to figure it out on my own

Cheers

taylor · May 19, 2020, 6:34am

The MQTT is used on Chirpstack, I haven’t passed the test yet.