Add software update capability #709

agners · 2024-05-18T00:05:20Z

This adds software update capability to the Python Matter Server.

As it stands, the implementation does the following things:

Check the DCL for software updates for a particular device
If an update is available, it downloads the firmware locally
Prepares the OTA Provider on the Matter fabric (using the Linux OTA Provider example app)
3a. On first use, the OTA Provider App is first commissioned to the fabric
Notifies the node about the availability of the OTA Provider and the update

The node then communicates directly with the OTA Provider. Progress of the update can be observed in the OTA Software Update Requestor Cluster of the device being updated.

There are a few things which still need to be sorted out:

Lifecycle management of the OTA Provider App needs improvement (termination not working/adding more updates)
Potential candidates need to be better checked if they are indeed compatible (minApplicableSoftwareVersion/maxApplicableSoftwareVersion)
Checksum of the update files needs to be checked after download
Split between update check and update apply
Handle multiple update calls for the same node
Support updates from testnet DCL

olavt · 2024-05-18T08:28:51Z

Will there be some new API-commands available to control this?

olavt · 2024-05-19T06:39:54Z

It would be nice if these steps where totally independent:

Check the DCL for software updates for a particular device
If an update is available, it downloads the firmware locally

I would like them to be:

Check the DCL for software updates for a particular device
If an update is available go to step 3
Download a firmware file from a given location (either one from the one specified by the DCL or any other location)

The reason I want this is to be able to test firmware updates for devices in development, which is not yet published to the DCL. Then step 1 and 2 would be bypassed.

It would be nice with a separate API-function for each of the steps.

marcelveldt · 2024-05-22T13:12:34Z

TODO:

add separate api command to check for updates
periodic check for node updates - set flag on matterNodeData
Show "update available" in Matter UI
HA update entity attached to the flag + manual service call to check for updates
The check for updates should implement throttling (HA has helpers for that)

agners · 2024-05-24T12:04:03Z

Will there be some new API-commands available to control this?

Yes,

It would be nice if these steps where totally independent:
1. Check the DCL for software updates for a particular device

2. If an update is available, it downloads the firmware locally

Yes, I intend to split this two operations. It is a bit a question of what the reference will be to apply the update. I intend to use the software version number (in integer format) as reference, as this should be unique. So the commands will be:

  {
    "command": "check_node_update",
    "args": {
      "node_id": 50
    }
  }

This will return information about the software, the exact format is tbd (currently a raw dump of the JSON data returned by the DCL).

Then with this command the update will get applied:

  {
    "command": "update_node",
    "args": {
      "node_id": 50,
      "software_version": 123,
    }
  }

Will then start the node update.

I would like them to be:
1. Check the DCL for software updates for a particular device

2. If an update is available go to step 3

3. Download a firmware file from a given location (either one from the one specified by the DCL or any other location)
The reason I want this is to be able to test firmware updates for devices in development, which is not yet published to the DCL. Then step 1 and 2 would be bypassed.

Agreed, we'll need to have a way to support non-DCL updates. In fact, for testing I already need this (I've now added hardcoded updates for the OTA Requestor example).

There are various use cases, from just not in DCL but online, to just an update file from somewhere. Each of these cases need some metadata to match the update file with the right devices. The metadata should probably closely follow the DCL format.

For online ones, we could probably just pass it through the WebSocket API as well, essentially how you suggest. But I don't think this will cover all use cases. 🤔

agners · 2024-06-05T17:27:03Z

Check if there is a software update available using DCL software update information.

The OTA provider downloads the updates and prepares them so Matter devices can consume them.

Use the OTA Provider example app to implement a OTA provider. The example app supports a JSON update descriptor file to manage update metadata. Tested with the OTA Requestor app.

Start and commission OTA Provider App when necessary. Use random discriminator and passcode. Store the Node ID of the OTA Provider App once setup for fast re-use.

Verify that the DCL software update is indeed applicable to the currently software running on the device. Add test coverage as well.

Add global check for update where we can insert hardcoded updates or updates from other sources in the future.

Make check_node_update a separate WebSocket command which only checks for updates. The update_node command then will download and actually apply the update.

Add Update specific exceptions and raise them where appropriate.

Add capability to verify the checksum of the OTA file while downloading it.

Add two new client commands to check for updates and trigger the update.

Create log files for each OTA Provider run. Improve setup and commissioning of the OTA Provider.

Most update logic is related to the external OTA provider (like commissioning and configuring it). This commit moves most of the update logic into the ExternalOtaProvider class.

Make use of the new ChipDeviceControllerWrapper to communicate with the device directly. This avoids unnecessary node interviewing and showing up on the controller side.

Support updating to a specific software version by string. This requires fetching all versions one by one, but is much more user friendly. The code still checks that all restrictions are met (min/max version, etc).

Instead of trying to reuse the same OTA Provider instance for multiple OTA requests, create a new instance for each request. This is easier to implement and also allows parallel updates. Because the OTA Provider is now ephemeral, we need to commission it on every update. But this is quick and reliable, so not a big deal. To support multiple updates at once, we need to make sure the OTA Providers use a distinct Matter port (hence passing 0) and distinct node ids. The current implementation simply uses the target node id plus a fixed offset. Since a single node can only run one update at a time, this is sufficient. Furthermore, some updates seem to have a difference in reported versionNumberString value in the DCL vs. what is actually in the OTA metadata. Specifically Eve updates from the Testnet DCL are such updates (e.g. 3.2.0 vs 3.2.6705). When using the OTA Provider with the --otaImageList option, this discrepancy is an issue and causes OTA Provider to abort. Using the single OTA update per OTA Provider instance allows us to use --filepath, which doesn't check the versionNumberString.

When the node's UpdateStateEnum changes from Querying to Idle it means the update file did not get processed. This could be due to temporary network issues or the update file not being honored by the target node.

Instead of using an Event use a Future to mark completion of the update. This allows to set an Exception in case we see update state transitions which are unexpected (specifically kQuerying -> kIdle).

Running multiple updates on the same node doesn't make sense. This should be mostly handled by the client/UX. But we do check on server side for the critical path.

Don't raise an exception when there is no software version info on DCL. It is perfectly fine to operate such nodes. An informational message is good enough for this case.

Use a new model MatterSoftwareVersion to store the software version information typically fetched from DCL for the Matter nodes.

We've added new commands, so a schema version bump is needed.

Dockerfile

marcelveldt · 2024-06-29T09:13:07Z

Dockerfile

@@ -25,6 +26,21 @@ RUN \

 ARG PYTHON_MATTER_SERVER

+ENV chip_example_url "https://github.com/agners/matter-linux-example-apps/releases/download/v1.3.0.0"


did you manage to move it to the libs org ?

marcelveldt · 2024-06-29T09:14:47Z

matter_server/server/device_controller.py

 DESCRIPTOR_PARTS_LIST_ATTRIBUTE_PATH = create_attribute_path_from_attribute(
    0, Clusters.Descriptor.Attributes.PartsList
 )
+BASIC_INFORMATION_VENDOR_ID_ATTRIBUTE_PATH = create_attribute_path_from_attribute(
+    0, Clusters.BasicInformation.Attributes.VendorID
+)
+BASIC_INFORMATION_PRODUCT_ID_ATTRIBUTE_PATH = create_attribute_path_from_attribute(
+    0, Clusters.BasicInformation.Attributes.ProductID
+)
 BASIC_INFORMATION_SOFTWARE_VERSION_ATTRIBUTE_PATH = (
    create_attribute_path_from_attribute(
        0, Clusters.BasicInformation.Attributes.SoftwareVersion
    )
 )
+BASIC_INFORMATION_SOFTWARE_VERSION_STRING_ATTRIBUTE_PATH = (
+    create_attribute_path_from_attribute(
+        0, Clusters.BasicInformation.Attributes.SoftwareVersionString
+    )
+)
+


In a follow-up iteration we should maybe start moving all these constants into the constants module

well, hmm maybe not as these are not plain/simple constants but created from a util function.

agners marked this pull request as draft May 18, 2024 07:54

agners force-pushed the add-software-update-capability branch from 745ff7d to b43b024 Compare May 24, 2024 09:25

agners added the new-feature New feature or request label May 24, 2024

agners force-pushed the add-software-update-capability branch 2 times, most recently from c6e19e2 to bb700de Compare June 5, 2024 14:01

agners mentioned this pull request Jun 6, 2024

Use secrets to generate the discriminator #736

Merged

agners force-pushed the add-software-update-capability branch from b649d5a to 21ea899 Compare June 12, 2024 07:09

agners mentioned this pull request Jun 24, 2024

Add Matter update entities for devices with OTA requestor home-assistant/core#120304

Draft

20 tasks

agners added 17 commits June 24, 2024 23:57

Implement Update check using DCL software information

30cfc3a

Check if there is a software update available using DCL software update information.

Initial implementation of OTA provider

da78a30

The OTA provider downloads the updates and prepares them so Matter devices can consume them.

Implement update using OTA Provider app

8781e84

Use the OTA Provider example app to implement a OTA provider. The example app supports a JSON update descriptor file to manage update metadata. Tested with the OTA Requestor app.

Setup OTA Provider App automatically when necessary

40e9b3d

Start and commission OTA Provider App when necessary. Use random discriminator and passcode. Store the Node ID of the OTA Provider App once setup for fast re-use.

Deploy chip-ota-provider-app in container

a0821a3

Check if DCL software updates are indeed applicable

b8ed3c9

Verify that the DCL software update is indeed applicable to the currently software running on the device. Add test coverage as well.

Introduce hardcoded updates

73c79d6

Add global check for update where we can insert hardcoded updates or updates from other sources in the future.

Split update WebSocket command into two commands

c8b748b

Make check_node_update a separate WebSocket command which only checks for updates. The update_node command then will download and actually apply the update.

Introduce Update logic specific exceptions

ff30959

Add Update specific exceptions and raise them where appropriate.

Implement OTA checksum verification

8af3881

Add capability to verify the checksum of the OTA file while downloading it.

Add client commands for updates

e9010b7

Add two new client commands to check for updates and trigger the update.

Improve DCL error message when download fails

b8b29dd

Improve OTA Provider handling

5d89e09

Create log files for each OTA Provider run. Improve setup and commissioning of the OTA Provider.

Move almost all update logic into ExternalOtaProvider

cc0a6dd

Most update logic is related to the external OTA provider (like commissioning and configuring it). This commit moves most of the update logic into the ExternalOtaProvider class.

Update implementation to work with latest refactoring

c7c587b

Make use of the new ChipDeviceControllerWrapper to communicate with the device directly. This avoids unnecessary node interviewing and showing up on the controller side.

Simplify ExternalOtaProvider

5b11fd7

Support specific version by string

fb4528c

Support updating to a specific software version by string. This requires fetching all versions one by one, but is much more user friendly. The code still checks that all restrictions are met (min/max version, etc).

agners added 6 commits June 24, 2024 23:57

Raise update error if the node moves from querying to idle

8328cea

When the node's UpdateStateEnum changes from Querying to Idle it means the update file did not get processed. This could be due to temporary network issues or the update file not being honored by the target node.

Improve logging and use Future to mark completion

e14cc04

Instead of using an Event use a Future to mark completion of the update. This allows to set an Exception in case we see update state transitions which are unexpected (specifically kQuerying -> kIdle).

Make sure that only one updates is running at a time

8717c0b

Running multiple updates on the same node doesn't make sense. This should be mostly handled by the client/UX. But we do check on server side for the critical path.

Use new commissioning API

3332de3

Ignore when there is no software version info on DCL

0eefa43

Don't raise an exception when there is no software version info on DCL. It is perfectly fine to operate such nodes. An informational message is good enough for this case.

agners force-pushed the add-software-update-capability branch from 21ea899 to 0bf4bc5 Compare June 24, 2024 21:58

Add MatterSoftwareVersion model for check_node_update

7f5460b

Use a new model MatterSoftwareVersion to store the software version information typically fetched from DCL for the Matter nodes.

agners force-pushed the add-software-update-capability branch from 0bf4bc5 to 7f5460b Compare June 24, 2024 21:58

agners changed the title ~~WIP: Add software update capability~~ Add software update capability Jun 24, 2024

agners marked this pull request as ready for review June 24, 2024 21:59

Bump Server schema

0764463

We've added new commands, so a schema version bump is needed.

agners force-pushed the add-software-update-capability branch from 8c22aa5 to 0764463 Compare June 25, 2024 07:53

marcelveldt reviewed Jun 25, 2024

View reviewed changes

Dockerfile Show resolved Hide resolved

marcelveldt reviewed Jun 25, 2024

View reviewed changes

Dockerfile Show resolved Hide resolved

marcelveldt reviewed Jun 29, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add software update capability #709

Add software update capability #709

agners commented May 18, 2024 •

edited

Loading

olavt commented May 18, 2024

olavt commented May 19, 2024

marcelveldt commented May 22, 2024

agners commented May 24, 2024

agners commented Jun 5, 2024

marcelveldt Jun 29, 2024

marcelveldt Jun 29, 2024

marcelveldt Jun 29, 2024

		@@ -25,6 +26,21 @@ RUN \

		ARG PYTHON_MATTER_SERVER

		ENV chip_example_url "https://github.com/agners/matter-linux-example-apps/releases/download/v1.3.0.0"

Add software update capability #709

Are you sure you want to change the base?

Add software update capability #709

Conversation

agners commented May 18, 2024 • edited Loading

olavt commented May 18, 2024

olavt commented May 19, 2024

marcelveldt commented May 22, 2024

agners commented May 24, 2024

agners commented Jun 5, 2024

marcelveldt Jun 29, 2024

Choose a reason for hiding this comment

marcelveldt Jun 29, 2024

Choose a reason for hiding this comment

marcelveldt Jun 29, 2024

Choose a reason for hiding this comment

agners commented May 18, 2024 •

edited

Loading