A short story of the migration process for The-Things-Network v2 to v3

13 Nov 2021 - tsp
Last update 13 Nov 2021
Reading time 8 mins

The community driven The things network made a major overhaul of their backend services while also integrating some commercial grade paid services. The TTN is a LoRA WAN backend routing network that gathers data using many distributed community operated LoRA WAN gateways. This allows one to build a worldwide long range low bandwidth network that’s freely usable by anyone in the covered area. Since LoRA covers large areas - due to it’s nearly 110 dB link budget - this provides a network that’s reachable nearly everywhere at least with large spreading factors. In areas with better gateway coverage higher data rates and device densities can be supported - and in addition one can also use multilateration to estimate the location of a device.

If you want to know more about LoRA in general and not only about the TTN migration process take a look at my other LoRA related blog entries.

During the major overhaul the TTN has been moved to a new software stack that allows it to scale nearly infinite - most of the services are now hosted on Amazon AWS. Unfortunately this also meant that one had to migrate gateways, devices and also perform major modifications on ones applications. This has to be done until the 1st of December in 2021 - then the old v2 stack will be shut down finally. Note that the new stack supports many interesting new features that might allow one to build even more modular applications - for example one could build an on demand near infinitly scaling solution when backhauling data using webhooks into a cloud environment such as AWS and using serverless lambdas for example - this might be interesting from two points of view. On one side a hobby project might only use a few messages per time period and thus running a public machine just to subscribe to the MQTT topic (especially if you’re renting this machine in some compute center) might be economic and resource based overkill - on the other hand the MQTT subscriptions don’t scale so if you saturate on the the maximum message rate your broker can handle this approach might also be more interesting.

In my opinion one should perform the migration in the following order:

An RAK Wireless LoRA gateway board

Preparing the applications

This is the most work that one will have to do. Of course one has to register the application again in the v3 Console. One just adds the new application and assigns an application-id - it might be helpful to assign the same name as for the v2 application but this is not necessary.

Then one has to change basic connection information:

Now the application can connect to the MQTT broker again. Then one has to account for a change in the MQTT topics. Basically they work the same as before but now they include the v3 prefix:

This is usually also simple to solve. Now comes the most tedious part. The message format has radically changed. There is no raw_payload field any more - this has been renamed; Metadata is encoded entirely different as well as the device and application information inside the JSON structure. One will have a hard time just adjusting existing code instead of writing the mappings in a custom way. A typical new message looks like the following (some information has been stripped from this dump):

{
  "end_device_ids": {
    "device_id": "XXXXXXXXXXXXXXXX",
    "application_ids": {
      "application_id": "XXXXXXXXXXXXXXXX"
    },
    "dev_eui": "XXXXXXXXXXXXXXXX",
    "join_eui": "XXXXXXXXXXXXXXXX",
    "dev_addr": "XXXXXXXXXXXXXXXX"
  },
  "correlation_ids": [
    "as:up:XXXXXXXXXXXXXXXX",
    "ns:uplink:XXXXXXXXXXXXXXXX",
    "pba:conn:up:XXXXXXXXXXXXXXXX",
    "pba:uplink:XXXXXXXXXXXXXXXX",
    "rpc:/ttn.lorawan.v3.GsNs/HandleUplink:XXXXXXXXXXXXXXXX",
    "rpc:/ttn.lorawan.v3.NsAs/HandleUplink:XXXXXXXXXXXXXXXX"
  ],
  "received_at": "YYYY-MM-DDTHH:MM:SS.888102695Z",
  "uplink_message": {
    "session_key_id": "XXXXXXXXXXXXXXXX",
    "f_port": 1,
    "f_cnt": 99,
    "frm_payload": "sPWW",
    "rx_metadata": [
      {
        "gateway_ids": {
          "gateway_id": "packetbroker"
        },
        "packet_broker": {
          "message_id": "XXXXXXXXXXXXXXXX",
          "forwarder_net_id": "000013",
          "forwarder_tenant_id": "ttnv2",
          "forwarder_cluster_id": "ttn-v2-eu-4",
          "forwarder_gateway_eui": "XXXXXXXXXXXXXXXX",
          "forwarder_gateway_id": "eui-XXXXXXXXXXXXXXXX",
          "home_network_net_id": "000013",
          "home_network_tenant_id": "ttn",
          "home_network_cluster_id": "eu1.cloud.thethings.network"
        },
        "time": "YYYY-MM-DDTHH:MM:SS.888102695Z",
        "rssi": -34,
        "channel_rssi": -34,
        "snr": 8.8,
        "location": {
          "latitude": 00.00000000,
          "longitude": 00.00000000,
          "altitude": 000
        },
        "uplink_token": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
      },
      {
        "gateway_ids": {
          "gateway_id": "XXXXXXXXXXXXXXXX",
          "eui": "XXXXXXXXXXXXXXXX"
        },
        "time": "YYYY-MM-DDTHH:MM:SS.888102695Z",
        "timestamp": 0000000000,
        "rssi": -34,
        "channel_rssi": -34,
        "snr": 8.8,
        "uplink_token": "XXXXXXXXXXXXXXXXXXXXXXXX",
        "channel_index": 1
      }
    ],
    "settings": {
      "data_rate": {
        "lora": {
          "bandwidth": 125000,
          "spreading_factor": 7
        }
      },
      "coding_rate": "4/5",
      "frequency": "868300000"
    },
    "received_at": "YYYY-MM-DDTHH:MM:SS.888102695Z",
    "consumed_airtime": "0.051456s",
    "network_ids": {
      "net_id": "000013",
      "tenant_id": "ttn",
      "cluster_id": "ttn-eu1"
    }
  }
}

As one can see the raw information is now found in uplink_message.frm_payload. The fields for the device and application EUI have been renamed and moved into other substructures and the metadata that had previously been embedded directly into the message format has been distributed in a variety of different nested objects.

In the best case then connect the application with a second MQTT connection to the v3 application. Be prepared to handle duplicate messages via v2 and v3 though.

Moving devices

This is pretty simple. If one does this by hand one just has to create a new entry for each device. One has to select a few properties per device when adding manually:

As soon as the device has been added to the v3 application one can simply change the application key in the v2 console and wait till the device tires the next join procedure that will then be rejected by the v2 stack and thus be forwarded to the v3 stack transparently where it succeeds. Depending on the device there are also some implementations that require one to power cycle the device to re-join.

This is everything that’s required for the devices.

Moving gateways

This is by far the most simple part but one should only move gateways after all devices have been moved to v3. Gateway that operate on the v2 endpoint forward messages also to v3 applications to reduce the migration impact on the network coverage. v3 gateways do not forward towards v2 applications.

Simply add the gateway in the v3 console. One might use the same gateway EUI in case one later on simply wants to copy the configuration. For gateways operating on the UDP packet forwarders one should also enable shedule downlink late. One should also configure the correct frequency plan and copy the gateway server address for gateway configuration.

As soon as the gateway has been entered into the console one might simply add the v3 backend to the /opt/ttn-gateway/bin/local_conf.json when using the poly packet forwarder. In this case one just has to add the new server endpoint to gateway_conf.servers (in case you’re also operating your own local network you might already deliver packets to many different endpoints anyways):

"servers": [
		{ "server_address": "router.eu.thethings.network", "serv_port_up": 1700, "serv_port_down": 1700, "serv_enabled": true },
		{ "server_address" : "eu1.cloud.thethings.network", "serv_port_up": 1700, "serv_port_down": 1700, "serv_enabled" : true }
],

As soon as migration has been fully done one might remove the old router.eu.thethings.network entry or at least disable it.

This article is tagged:


Data protection policy

Dipl.-Ing. Thomas Spielauer, Wien (webcomplains389t48957@tspi.at)

This webpage is also available via TOR at http://jugujbrirx3irwyx.onion/

Valid HTML 4.01 Strict Powered by FreeBSD IPv6 support