From 3dc657ea60e5aa55e87ebef7b9ae6def59535b39 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Mon, 15 Mar 2021 02:44:08 -0700 Subject: [PATCH 01/44] Add RFC for pluggable discovery --- RFCs/0000-pluggable-discovery.md | 653 +++++++++++++++++++++++++++++++ 1 file changed, 653 insertions(+) create mode 100644 RFCs/0000-pluggable-discovery.md diff --git a/RFCs/0000-pluggable-discovery.md b/RFCs/0000-pluggable-discovery.md new file mode 100644 index 0000000..2c5fa8b --- /dev/null +++ b/RFCs/0000-pluggable-discovery.md @@ -0,0 +1,653 @@ +# Pluggable Discovery + +## Overview + +This document describes how the Pluggable Discovery works and how it +should integrate with monitors and uploaders. + +## Problem + +When a sketch is uploaded to an Arduino board the only means for +transferring the binary data to the microcontroller is the serial port. +Currently: + +- the subroutines to enumerate the serial ports and to possibly + identify a board are “hardcoded” in the IDE/CLI + +- the only way to identify a board is via USB VID/PID identifiers + +- the serial monitor, to communicate with the board after the upload, + is “hardcoded” in the IDE + +The current structure does not allow to use different kind of “ports” to +communicate with the microcontroller. For example it would be +interesting to have the possibility to: + +- discover a board (MDNS) and upload via network (OTA) + +- discover a board and upload via a communication port that is not the + serial port (for example via USB-HID as implemented by Teensy or via + any other kind of port) + +- allow a third party to implement his own upload protocol / port + +Let’s see how such a system may be implemented. + +## Constraints + +- A communication port may use any kind of **protocol** + +- Software support for a new **protocol** must be added to the system + as a **plugin** + +- Each port must be **enumerated/discovered** + +- Each port may provide metadata to identify the board model + (**identification properites**) + +- Each port may provide metadata to identify a specific instance of + the board (**serial number / MAC address**) + +- Each port must have an **unique address** + +- A single device may expose multiple ports + +## Pluggable discoveries + +The proposed solution is to provide a tool that can enumerate and +discover the ports for a **single specific protocol**. There will be a +tool for each supported protocol so in the end we will have a “Serial +ports discovery”, a “Network port discovery” and so on. + +These tools must be in the form of executables that can be launched as a +subprocess using a `platform.txt` command line recipe. They will +communicate to the parent process via stdin/stdout, in particular a +discovery will accept commands as plain text strings from stdin and will +send answers back in JSON format on stdout. Each tool will implement the +commands to list and enumerate ports for a specific protocol as +specified in this document. + +### Pluggable discovery API via stdin/stdout + +After startup, the tool will just stay idle waiting for commands. +The available commands are: `START`, `STOP`, `QUIT`, `LIST` and +`START_SYNC`. + +#### START command + +The `START` command initializes and start the discovery internal +subroutines. This command must be called before `LIST` or `START_SYNC`. +The response to the start command is: + +```JSON +{ + "eventType": "start", + "message": "OK" +} +``` + +#### STOP command + +The `STOP` command stops the discovery internal subroutines and possibly +free the internally used resources. This command should be called if the +client wants to pause the discovery for a while. The response to the +command is: + +```JSON +{ + "eventType": "stop", + "message": "OK" +} +``` + +#### QUIT command + +The `QUIT` command terminates the discovery. The response to `QUIT` is: + +```JSON +{ + "eventType": "quit", + "message": "OK" +} +``` + +after this output the discovery exits. + +#### LIST command + +The `LIST` command executes an enumeration of the ports and returns a +list of the available ports at the moment of the call. The format of the +response is the following: + +```JSON +{ + "eventType": "list", + "ports": [ + { + "address": <-- THE ADDRESS OF THE PORT + "label": <-- HOW THE PORT IS DISPLAYED ON THE GUI + "protocol": <-- THE PROTOCOL USED BY THE BOARD + "protocolLabel": <-- HOW THE PROTOCOL IS DISPLAYED ON THE GUI + "prefs": { + <-- A LIST OF PROPERTIES FOR THAT SPECIFIC BOARD + }, + "identificationPrefs": { + <-- A LIST OF PROPERTIES TO IDENTIFY THE BOARD MODEL + } + }, { + ... <-- OTHER PORTS... + } + ] +} +``` + +The `ports` field contains a list of the available ports. + +Each port has: + +- an `address` (for example `/dev/ttyACM0` for serial ports or + `192.168.10.100` for network ports) + +- a `label` that is the human readable form of the `address` (it may + be for example `ttyACM0` or `SSH on 192.168.10.100`) + +- `protocol` is the protocol identifier (such as `serial` or `dfu` or + `ssh`) + +- `protocolLabel` is the `protocol` in human readable form (for + example `Serial port` or `DFU USB` or `Network (ssh)`) + +- `prefs` is a list of key/value pairs that represent information + relative to the specific instance of the board + +- `identificationPrefs` is a list of key value pairs that represent + information to identify the board **model** + +To make the above more clear let’s show an example with the +`serial_discovery`: + +```JSON +{ + "eventType": "list", + "ports": [ + { + "address": "/dev/ttyACM0", + "label": "ttyACM0", + "protocol": "serial", + "protocolLabel": "Serial Port (USB)", + "prefs": { + "pid": "0x804e", + "vid": "0x2341", + "serialNumber": "EBEABFD6514D32364E202020FF10181E" + }, + "identificationPrefs": { + "pid": "0x804e", + "vid": "0x2341" + } + } + ] +} +``` + +In this case the serial port metadata comes from an USB serial +converter. The USB VID/PID and USB SERIAL NUMBER properties are also +reported inside `prefs`. Inside the `identificationPrefs` instead we +have only the properties useful for product identification (in this case +only USB VID/PID is useful to identify the board model). + +The `LIST` command performs a one-shot polling of the ports. If you need +continuous monitoring of ports you should use the `START_SYNC` command. + +#### START_SYNC command + +The `START_SYNC` command puts the tool in "events" mode: the discovery +will send `add` and `remove` events each time a new port is detected or +removed respectively. + +If the CLI is used in command-line mode (`arduino-cli board list` +command) then we need a one-shot output and we can run the discoveries +with the `LIST` command instead. Note that some discoveries may not be +able to `LIST` ports immediately after the launch (in particular network +protocols like MDNS, Bluetooth, etc. requires some seconds to receive +the broadcasts from all available clients). + +If the CLI is running in daemon mode, ideally, all the installed +discoveries should be started simultaneously in “event mode” (with +`START_SYNC`) and the list of available ports should be cached inside +the CLI daemon. + +The `add` event looks like the following: + +```JSON +{ + "eventType": "add", + "port": { + "address": "/dev/ttyACM0", + "label": "ttyACM0", + "prefs": { ... }, + "identificationPrefs": { ... }, + "protocol": "serial", + "protocolLabel": "Serial Port (USB)" + } +} +``` + +it basically provides the same information as the `list` event but for a +single port. After calling `START_SYNC` an initial burst of add events +may be generated in sequence to report all the ports available at the +moment of the start. + +The `remove` event looks like the following: + +```JSON +{ + "eventType": "remove", + "port": { + "address": "/dev/ttyACM0" + } +} +``` + +the content is straightforward, in this case only the `address` field is +reported. + +A demo tool is available here: +[https://github.com/arduino/serial-discovery](https://github.com/arduino/serial-discovery#example-of-usage) + +A typical usage scenario is here: +https://github.com/arduino/serial-discovery#example-of-usage + +## Integration with CLI and core platforms + +### Discovery tool install and launch + +Discovery tools should be build natively for each OS and the CLI should +run the correct tool for the running OS. This infrastracture is already +available for platforms tools so the most natural way forward is to +distribute the discoveries as tools within core platforms (in the same +way we do for `gcc` or `avrdude`). + +Some discovery like the `serial-discovery` must be always available, so +they will be part of the `builtin` platform package and installed +without the need to be part of a real platform (`builtin` is a dummy +package that we use to install tools that are not part of any platforms +like `ctags` for example). + +The CLI will run `serial-discovery`, and other “builtin“ discoveries +that we may want, by default. + +3rd party platforms may add other discoveries by providing them as tools +dependencies for their platform and by adding a directive to their +`platform.txt` that informs the CLI that a new discovery is available. + +``` +discovery.DISCOVERY_ID.pattern=DISCOVERY_RECIPE +``` + +The CLI will look for directives matching the above pattern. +`DISCOVERY_ID` must be replaced by a unique identifier for the +particular discovery and `DISCOVERY_RECIPE` must be replaced by the +command line to run to launch the discovery. An example could be: + +``` +## Teensy Ports Discovery +discovery.teensy.pattern="{runtime.tools.teensy_ports.path}/hardware/tools/teensy_ports" -J2 +``` + +in this case the platform provides a new `teensy` discovery and the +command line tool named `teensy_ports` is launched with the `-J2` +parameter to start the discovery tool. + +### Duplicate discoveries + +It may happen that different 3rd party platforms provides the same +discovery or different versions of the same discovery or, worse, +different version of the same discovery launched with different +parameters. + +We can partially handle this if the `DISCOVERY_ID` field in +`platform.txt` is well defined: from the CLI we could group together the +platforms that requires the same discovery and launch the latest version +available just once. How the different 3rd party will agree on the +`DISCOVERY_ID` value population is TBD. + +### Board identification + +The metadata `identificationPrefs` associated to a port can be used to +identify the board attached to that port. The algorithm is very simple: +if a board listed in the platform file `boards.txt` match the same +`identificationPrefs` coming from the discovery then the board is a +“candidate” board attached to that port. Some identification properties +may be imperfect and not uniquely identify a board, in that case more +boards can match the same `identificationPrefs`, that’s why we called it +“candidate”. + +Let’s see an example to clarify things a bit, let suppose that we have +the following `identificationPrefs` coming from the serial discovery: + +```JSON +[...CUT...] + "port": { + "address": "/dev/ttyACM0", + "identificationPrefs": { + "pid": "0x804e", + "vid": "0x2341" + } +[...CUT...] +``` + +so we got `pid=0x804e, vid=0x2341`. Let’s suppose we have the following +`boards.txt`: + +``` +# Arduino Zero (Prorgamming Port) +# --------------------------------------- +arduino_zero_edbg.name=Arduino Zero (Programming Port) +arduino_zero_edbg.vid.0=0x03eb +arduino_zero_edbg.pid.0=0x2157 +arduino_zero_edbg.debug.tool=gdb +arduino_zero_edbg.upload.tool=openocd +arduino_zero_edbg.upload.protocol=sam-ba +[...CUT...] +# Arduino Zero (Native USB Port) +# -------------------------------------- +arduino_zero_native.name=Arduino Zero (Native USB Port) +arduino_zero_native.vid.0=0x2341 +arduino_zero_native.pid.0=0x804d +arduino_zero_native.vid.1=0x2341 +arduino_zero_native.pid.1=0x004d +arduino_zero_native.vid.2=0x2341 +arduino_zero_native.pid.2=0x824d +arduino_zero_native.vid.3=0x2341 +arduino_zero_native.pid.3=0x024d +arduino_zero_native.upload.tool=bossac +arduino_zero_native.upload.protocol=sam-ba +[...CUT...] +# Arduino MKR1000 +# ----------------------- +mkr1000.name=Arduino MKR1000 +mkr1000.vid.0=0x2341 <------- MATCHING IDs +mkr1000.pid.0=0x804e <------- MATCHING IDs +mkr1000.vid.1=0x2341 +mkr1000.pid.1=0x004e +mkr1000.vid.2=0x2341 +mkr1000.pid.2=0x824e +mkr1000.vid.3=0x2341 +mkr1000.pid.3=0x024e +mkr1000.debug.tool=gdb +mkr1000.upload.tool=bossac +mkr1000.upload.protocol=sam-ba +[...CUT...] +``` + +As we can see the only board that has the two properties matching is the +`mkr1000`, in this case the CLI knows that the board is 100% an MKR1000. + +Note that `vid` and `pid` properties are just free text key/value pairs: +the discovery may returns basically anything, the board just needs to +have the same properties defined in `boards.txt` to be identified. + +We can also specify multiple identification properties for the same +board using the `.N` suffix, for example: + +``` +myboard.name=My Wonderful Arduino Compatible Board +myboard.pears=20 +myboard.apples=30 +``` + +will match on `pears=20, apples=30` but: + +``` +myboard.name=My Wonderful Arduino Compatible Board +myboard.pears.0=20 +myboard.apples.0=30 +myboard.pears.1=30 +myboard.apples.1=40 +``` + +will match on both `pears=20, apples=30` and `pears=30, apples=40`. + +## Upload (state of the art) + +In this section we will discuss the current status of the upload +business logic in the IDE/CLI. + +### Serial upload (CLI and Java IDE) + +Currently, to upload we use the `tools.UPLOAD_RECIPE_ID.upload.pattern` +recipe in `platform.txt`, where `UPLOAD_RECIPE_ID` is defined as the +value of `upload.tool` property in `boards.txt` for the currently +user-selected board. + +The recipe contains the variables `{serial.port}` and +`{serial.port.file}` that are replaced with the user selected serial +port (in particular on unix systems `{serial.port.file}` contains only +the file name `ttyACM0` while `{serial.port}` contains the full path to +the serial port `/dev/ttyACM0`). A typical serial upload recipe is +something like: + +`tools.bossac.upload.pattern="{runtime.tools.bossac-1.7.0-arduino3.path}/bossac" --port={serial.port.file} -U true -i -e -w -v "{build.path}/{build.project_name}.bin" -R` + +### Network upload (Java IDE only) + +The Java IDE has an integrated MDNS client that listen for network +boards. If a network board is selected the +`tools.UPLOAD_RECIPE_ID.upload.network_pattern` is used for upload. +`UPLOAD_RECIPE_ID` is obtained in the same way as for the serial upload. +A typical recipe is: + +`tools.bossac.upload.network_pattern="{runtime.tools.arduinoOTA.path}/bin/arduinoOTA" -address {serial.port} -port 65280 -username arduino -password "{network.password}" -sketch "{build.path}/{build.project_name}.bin" -upload /sketch -b` + +please note that: + +- the property is called `network_pattern` instead of `pattern` and, + yes, this is hardcoded + +- the port address is still called `{serial.port}` even if it is a + network address + +- there is a `{network.password}` that is a user entered field and, + yes, this is hardcoded + +## Upload with Pluggable discovery + +Given the above, it is clear that the pluggable discovery should provide +features to allow: + +1. selection of different upload recipes based on the port `protocol` +1. provide extra port metadata to use on the command line recipe +1. provide eventual extra info that should be user-supplied (by + entering them in a dialog or provided in other ways) +1. should be backward compatible with the current legacy + +### Recipe selection based on protocol + +To allow the above the `UPLOAD_RECIPE_ID` selection is now **dependent +on protocol**. The `boards.txt` now must specify a recipe id for each +type of upload protocol supported by the board. Previously we used the +property `upload.tool=UPLOAD_RECIPE_ID` now this property should be +changed to `upload.tool.UPLOAD_PROTOCOL=UPLOAD_RECIPE_ID` so we can +choose the correct tool for each protocol. We should also keep the old +property for backward compatibility + +**Before:** + +``` +myboard.upload.tool=bossac +``` + +**After:** + +``` +# keep for backward compatibility +myboard.upload.tool=avrdude +# Upload recipes +myboard.upload.tool.serial=bossac +myboard.upload.tool.network=arduino_ota +``` + +The selected port address will be provided in the variable +`{upload.address}`. Other metadata provided by the discover in the +`prefs` section will be provided in the `{upload.port.*}` variables. + +For backward compatibility we will keep a copy of the address also in +`{serial.port}` and in the specific case of a `protocol=serial` we will +populate also `{serial.port.file}`. + +For example, the following port metadata coming from a pluggable +discovery: + +```JSON +{ + "eventType": "add", + "port": { + "address": "/dev/ttyACM0", + "label": "ttyACM0", + "protocol": "serial", + "protocolLabel": "Serial Port (USB)" + "prefs": { + "pid": "0x804e", + "vid": "0x2341", + "serialNumber": "EBEABFD6514D32364E202020FF10181E" + }, + "identificationPrefs": { + "pid": "0x804e", + "vid": "0x2341" + } + } +} +``` + +will be available on the recipe as the variables: + +``` +{upload.address} = /dev/ttyACM0 +{upload.protocol} = serial +{upload.port.pid} = 0x8043 +{upload.port.vid} = 0x2341 +{upload.port.serialNumber} = EBEABFD6514D32364E202020FF10181E +{serial.port} = ttyACM0 # for backward compatibility +{serial.port.file} = /dev/ttyACM0 # only because protocol=serial +``` + +Here another example: + +```JSON +{ + "eventType": "add", + "port": { + "address": "192.168.1.232", + "label": "SSH on my-board ()192.168.1.232)", + "protocol": "ssh", + "protocolLabel": "SSH Network port" + "prefs": { + "macprefix": "AA:BB:CC", + "macaddress": "AA:BB:CC:DD:EE:FF" + }, + "identificationPrefs": { + "macprefix": "AA:BB:CC", + } + } +} +``` + +that is translated to: + +``` +{upload.address} = 192.168.1.232 +{upload.protocol} = ssh +{upload.port.macprefix} = AA:BB:CC +{upload.port.macaddress} = AA:BB:CC:DD:EE:FF +{serial.port} = 192.168.1.232 # for backward compatibility +``` + +This configuration, together with protocol selection, allows to remove +the hardcoded `network_pattern`, now we can replace the legacy recipe +(split into multiple lines for clarity): + +``` +tools.bossac.upload.network_pattern="{runtime.tools.arduinoOTA.path}/bin/arduinoOTA" + -address {serial.port} -port 65280 + -sketch "{build.path}/{build.project_name}.bin" +``` + +with: + +``` +tools.arduino_ota.upload.pattern="{runtime.tools.arduinoOTA.path}/bin/arduinoOTA" + -address {upload.address} -port 65280 + -sketch "{build.path}/{build.project_name}.bin" +``` + +### User provided fields + +The recipe may require some user-entered fields (like username or +password). In this case the recipe must use the special placeholder +`{upload.user.FIELD_NAME}` where `FIELD_NAME` must be declared +separately in the recipe using the following format: + +``` +tools.UPLOAD_RECIPE_ID.upload.field.FIELD_NAME=FIELD_LABEL +tools.UPLOAD_RECIPE_ID.upload.field.FIELD_NAME.secret=true +``` + +`FIELD_LABEL` is the label shown in the dialog to ask the user to enter +the value of the field. The `secret` property is optional and it should +be set to `true` if the field is a secret (like passwords or token). + +Let’s see how a complete example will look like: + +``` +tools.arduino_ota.upload.field.username=Username +tools.arduino_ota.upload.field.password=Password +tools.arduino_ota.upload.field.password.secret=true +tools.arduino_ota.upload.pattern="{runtime.tools.arduinoOTA.path}/bin/arduinoOTA" + -address {upload.address} -port 65280 + -username "{upload.user.username} + -password "{upload.user.password}" + -sketch "{build.path}/{build.project_name}.bin" +``` + +## CLI command line UX considerations + +Currently only serial ports are allowed on the CLI, so we don’t need to +specify a protocol when we select a port, if we write +`arduino-cli upload --port /dev/ttyACM0` or +`arduino-cli upload --port COM1` we are sure that this is a serial port. + +Things get a bit more complex with Pluggable Discovery: `--port COM1` +could be a serial port or the hostname of a network board that is called +exactly `COM1`. To solve this amiguity we may add a flag to the `upload` +command to specify the protocol so it becomes +`arduino-cli upload --protocol serial --port COM1`. + +Obviously using `--protocol` is ugly and makes the command line +overcrowded. To avoid the `--protocol` flag we may adopt some +strategies: + +- we could check the current overall port list and see if the + specified port address match **only one** of the available ports + +- we may check if the address is a valid address for **only one** of + the supported protocol. For example if we have support for `serial` + and `network` only and if the entered address is `1.2.3.4` we know + that this is a `network` address. This strategy requires that the + pluggable discoveries have a command to check if the address is a + valid address for their specific protocol. + +- we may save the protocol as part of the `board attach` command (as + we do already for the port address) + +- we may use a URL format to specify protocol and address at the same + time: `--port serial:///dev/ttyACM0` or `--port ssh://1.2.3.4` or + `--port xyz://12-34/32:23/442/212`. BTW It may be problematic to + express an address in form of an URL. + +## Appendix + +We have a POC implementation of a serial discovery: +https://github.com/arduino/serial-discovery + +and a network discovery: +https://github.com/arduino/mdns-discovery From 2f7f65da0666b19d2fa081029d28746f096cbff9 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Tue, 16 Mar 2021 16:04:07 +0100 Subject: [PATCH 02/44] Fixed some typos and made RFC compliant to RFC template --- RFCs/0000-pluggable-discovery.md | 326 +++++++++---------------------- 1 file changed, 93 insertions(+), 233 deletions(-) diff --git a/RFCs/0000-pluggable-discovery.md b/RFCs/0000-pluggable-discovery.md index 2c5fa8b..81966e1 100644 --- a/RFCs/0000-pluggable-discovery.md +++ b/RFCs/0000-pluggable-discovery.md @@ -2,82 +2,55 @@ ## Overview -This document describes how the Pluggable Discovery works and how it -should integrate with monitors and uploaders. +This document describes how the Pluggable Discovery works and how it should integrate with monitors and uploaders. ## Problem -When a sketch is uploaded to an Arduino board the only means for -transferring the binary data to the microcontroller is the serial port. -Currently: - -- the subroutines to enumerate the serial ports and to possibly - identify a board are “hardcoded” in the IDE/CLI +When a sketch is uploaded to an Arduino board the only way for transferring the binary executable to the microcontroller is through the serial port. Currently: +- the subroutines to enumerate the serial ports and to possibly identify a board are “hardcoded” in the IDE/CLI - the only way to identify a board is via USB VID/PID identifiers +- the "serial monitor", to communicate with the board after the upload, is “hardcoded” in the IDE -- the serial monitor, to communicate with the board after the upload, - is “hardcoded” in the IDE - -The current structure does not allow to use different kind of “ports” to -communicate with the microcontroller. For example it would be -interesting to have the possibility to: +The current structure does not allow to use different kind of “ports” to communicate with the microcontroller. For example it would be interesting to have the possibility to: - discover a board (MDNS) and upload via network (OTA) -- discover a board and upload via a communication port that is not the - serial port (for example via USB-HID as implemented by Teensy or via - any other kind of port) +- discover a board and upload via a communication port that is not the serial port (for example via USB-HID as implemented by Teensy or via any other kind of port) - allow a third party to implement his own upload protocol / port -Let’s see how such a system may be implemented. +The pluggable discovery aims to provide a solution to these problems. ## Constraints - A communication port may use any kind of **protocol** -- Software support for a new **protocol** must be added to the system - as a **plugin** +- Software support for a new **protocol** must be added to the system as a **plugin** - Each port must be **enumerated/discovered** -- Each port may provide metadata to identify the board model - (**identification properites**) +- Each port may provide metadata to identify the board model (**identification properites**) -- Each port may provide metadata to identify a specific instance of - the board (**serial number / MAC address**) +- Each port may provide metadata to identify a specific instance of the board (**serial number / MAC address**) - Each port must have an **unique address** - A single device may expose multiple ports -## Pluggable discoveries +## Recommended Solution -The proposed solution is to provide a tool that can enumerate and -discover the ports for a **single specific protocol**. There will be a -tool for each supported protocol so in the end we will have a “Serial -ports discovery”, a “Network port discovery” and so on. +The proposed solution is to provide a tool that can enumerate and discover the ports for a **single specific protocol**. There will be a tool for each supported protocol so, in the end, we will have a “Serial ports discovery”, a “Network port discovery” and so on. -These tools must be in the form of executables that can be launched as a -subprocess using a `platform.txt` command line recipe. They will -communicate to the parent process via stdin/stdout, in particular a -discovery will accept commands as plain text strings from stdin and will -send answers back in JSON format on stdout. Each tool will implement the -commands to list and enumerate ports for a specific protocol as -specified in this document. +These tools must be in the form of executables that can be launched as a subprocess using a `platform.txt` command line recipe. They will communicate to the parent process via stdin/stdout, in particular a discovery will accept commands as plain text strings from stdin and will send answers back in JSON format on stdout. Each tool will implement the commands to list and enumerate ports for a specific protocol as specified in this document. ### Pluggable discovery API via stdin/stdout -After startup, the tool will just stay idle waiting for commands. -The available commands are: `START`, `STOP`, `QUIT`, `LIST` and -`START_SYNC`. +After startup, the tool will just stay idle waiting for commands. The available commands are: `START`, `STOP`, `QUIT`, `LIST` and `START_SYNC`. #### START command -The `START` command initializes and start the discovery internal -subroutines. This command must be called before `LIST` or `START_SYNC`. -The response to the start command is: +The `START` command initializes and start the discovery internal subroutines. This command must be called before `LIST` or `START_SYNC`. The response to the start command is: ```JSON { @@ -88,10 +61,7 @@ The response to the start command is: #### STOP command -The `STOP` command stops the discovery internal subroutines and possibly -free the internally used resources. This command should be called if the -client wants to pause the discovery for a while. The response to the -command is: +The `STOP` command stops the discovery internal subroutines and possibly free the internally used resources. This command should be called if the client wants to pause the discovery for a while. The response to the command is: ```JSON { @@ -115,11 +85,9 @@ after this output the discovery exits. #### LIST command -The `LIST` command executes an enumeration of the ports and returns a -list of the available ports at the moment of the call. The format of the -response is the following: +The `LIST` command executes an enumeration of the ports and returns a list of the available ports at the moment of the call. The format of the response is the following: -```JSON +``` { "eventType": "list", "ports": [ @@ -145,26 +113,19 @@ The `ports` field contains a list of the available ports. Each port has: -- an `address` (for example `/dev/ttyACM0` for serial ports or - `192.168.10.100` for network ports) +- an `address` (for example `/dev/ttyACM0` for serial ports or `192.168.10.100` for network ports) -- a `label` that is the human readable form of the `address` (it may - be for example `ttyACM0` or `SSH on 192.168.10.100`) +- a `label` that is the human readable form of the `address` (it may be for example `ttyACM0` or `SSH on 192.168.10.100`) -- `protocol` is the protocol identifier (such as `serial` or `dfu` or - `ssh`) +- `protocol` is the protocol identifier (such as `serial` or `dfu` or `ssh`) -- `protocolLabel` is the `protocol` in human readable form (for - example `Serial port` or `DFU USB` or `Network (ssh)`) +- `protocolLabel` is the `protocol` in human readable form (for example `Serial port` or `DFU USB` or `Network (ssh)`) -- `prefs` is a list of key/value pairs that represent information - relative to the specific instance of the board +- `prefs` is a list of key/value pairs that represent information relative to the specific instance of the board -- `identificationPrefs` is a list of key value pairs that represent - information to identify the board **model** +- `identificationPrefs` is a list of key value pairs that represent information to identify the board **model** -To make the above more clear let’s show an example with the -`serial_discovery`: +To make the above more clear let’s show an example with the `serial_discovery`: ```JSON { @@ -189,32 +150,17 @@ To make the above more clear let’s show an example with the } ``` -In this case the serial port metadata comes from an USB serial -converter. The USB VID/PID and USB SERIAL NUMBER properties are also -reported inside `prefs`. Inside the `identificationPrefs` instead we -have only the properties useful for product identification (in this case -only USB VID/PID is useful to identify the board model). +In this case the serial port metadata comes from an USB serial converter. The USB VID/PID and USB SERIAL NUMBER properties are also reported inside `prefs`. Inside the `identificationPrefs` instead we have only the properties useful for product identification (in this case only USB VID/PID is useful to identify the board model). -The `LIST` command performs a one-shot polling of the ports. If you need -continuous monitoring of ports you should use the `START_SYNC` command. +The `LIST` command performs a one-shot polling of the ports. If you need continuous monitoring of ports you should use the `START_SYNC` command. #### START_SYNC command -The `START_SYNC` command puts the tool in "events" mode: the discovery -will send `add` and `remove` events each time a new port is detected or -removed respectively. +The `START_SYNC` command puts the tool in "events" mode: the discovery will send `add` and `remove` events each time a new port is detected or removed respectively. -If the CLI is used in command-line mode (`arduino-cli board list` -command) then we need a one-shot output and we can run the discoveries -with the `LIST` command instead. Note that some discoveries may not be -able to `LIST` ports immediately after the launch (in particular network -protocols like MDNS, Bluetooth, etc. requires some seconds to receive -the broadcasts from all available clients). +If the CLI is used in command-line mode (`arduino-cli board list` command) then we need a one-shot output and we can run the discoveries with the `LIST` command instead. Note that some discoveries may not be able to `LIST` ports immediately after the launch (in particular network protocols like MDNS, Bluetooth, etc. requires some seconds to receive the broadcasts from all available clients). -If the CLI is running in daemon mode, ideally, all the installed -discoveries should be started simultaneously in “event mode” (with -`START_SYNC`) and the list of available ports should be cached inside -the CLI daemon. +If the CLI is running in daemon mode, ideally, all the installed discoveries should be started simultaneously in “event mode” (with `START_SYNC`) and the list of available ports should be cached inside the CLI daemon. The `add` event looks like the following: @@ -224,18 +170,22 @@ The `add` event looks like the following: "port": { "address": "/dev/ttyACM0", "label": "ttyACM0", - "prefs": { ... }, - "identificationPrefs": { ... }, + "prefs": { + "pid": "0x804e", + "vid": "0x2341", + "serialNumber": "EBEABFD6514D32364E202020FF10181E" + }, + "identificationPrefs": { + "pid": "0x804e", + "vid": "0x2341" + }, "protocol": "serial", "protocolLabel": "Serial Port (USB)" } } ``` -it basically provides the same information as the `list` event but for a -single port. After calling `START_SYNC` an initial burst of add events -may be generated in sequence to report all the ports available at the -moment of the start. +it basically provides the same information as the `list` event but for a single port. After calling `START_SYNC` an initial burst of add events may be generated in sequence to report all the ports available at the moment of the start. The `remove` event looks like the following: @@ -248,8 +198,7 @@ The `remove` event looks like the following: } ``` -the content is straightforward, in this case only the `address` field is -reported. +the content is straightforward, in this case only the `address` field is reported. A demo tool is available here: [https://github.com/arduino/serial-discovery](https://github.com/arduino/serial-discovery#example-of-usage) @@ -257,73 +206,42 @@ A demo tool is available here: A typical usage scenario is here: https://github.com/arduino/serial-discovery#example-of-usage -## Integration with CLI and core platforms +### Integration with CLI and core platforms -### Discovery tool install and launch +#### Discovery tool install and launch -Discovery tools should be build natively for each OS and the CLI should -run the correct tool for the running OS. This infrastracture is already -available for platforms tools so the most natural way forward is to -distribute the discoveries as tools within core platforms (in the same -way we do for `gcc` or `avrdude`). +Discovery tools should be build natively for each OS and the CLI should run the correct tool for the running OS. This infrastracture is already available for platforms tools so the most natural way forward is to distribute the discoveries as tools within core platforms (in the same way we do for `gcc` or `avrdude`). -Some discovery like the `serial-discovery` must be always available, so -they will be part of the `builtin` platform package and installed -without the need to be part of a real platform (`builtin` is a dummy -package that we use to install tools that are not part of any platforms -like `ctags` for example). +Some discovery like the `serial-discovery` must be always available, so they will be part of the `builtin` platform package and installed without the need to be part of a real platform (`builtin` is a dummy package that we use to install tools that are not part of any platforms like `ctags` for example). -The CLI will run `serial-discovery`, and other “builtin“ discoveries -that we may want, by default. +The CLI will run `serial-discovery`, and other “builtin“ discoveries that we may want, by default. -3rd party platforms may add other discoveries by providing them as tools -dependencies for their platform and by adding a directive to their -`platform.txt` that informs the CLI that a new discovery is available. +3rd party platforms may add other discoveries by providing them as tools dependencies for their platform and by adding a directive to their `platform.txt` that informs the CLI that a new discovery is available. ``` discovery.DISCOVERY_ID.pattern=DISCOVERY_RECIPE ``` -The CLI will look for directives matching the above pattern. -`DISCOVERY_ID` must be replaced by a unique identifier for the -particular discovery and `DISCOVERY_RECIPE` must be replaced by the -command line to run to launch the discovery. An example could be: +The CLI will look for directives matching the above pattern. `DISCOVERY_ID` must be replaced by a unique identifier for the particular discovery and `DISCOVERY_RECIPE` must be replaced by the command line to run to launch the discovery. An example could be: ``` ## Teensy Ports Discovery discovery.teensy.pattern="{runtime.tools.teensy_ports.path}/hardware/tools/teensy_ports" -J2 ``` -in this case the platform provides a new `teensy` discovery and the -command line tool named `teensy_ports` is launched with the `-J2` -parameter to start the discovery tool. +in this case the platform provides a new `teensy` discovery and the command line tool named `teensy_ports` is launched with the `-J2` parameter to start the discovery tool. -### Duplicate discoveries +#### Duplicate discoveries -It may happen that different 3rd party platforms provides the same -discovery or different versions of the same discovery or, worse, -different version of the same discovery launched with different -parameters. +It may happen that different 3rd party platforms provides the same discovery or different versions of the same discovery or, worse, different version of the same discovery launched with different parameters. -We can partially handle this if the `DISCOVERY_ID` field in -`platform.txt` is well defined: from the CLI we could group together the -platforms that requires the same discovery and launch the latest version -available just once. How the different 3rd party will agree on the -`DISCOVERY_ID` value population is TBD. +We can partially handle this if the `DISCOVERY_ID` field in `platform.txt` is well defined: from the CLI we could group together the platforms that requires the same discovery and launch the latest version available just once. How the different 3rd party will agree on the `DISCOVERY_ID` value population is TBD. -### Board identification +#### Board identification -The metadata `identificationPrefs` associated to a port can be used to -identify the board attached to that port. The algorithm is very simple: -if a board listed in the platform file `boards.txt` match the same -`identificationPrefs` coming from the discovery then the board is a -“candidate” board attached to that port. Some identification properties -may be imperfect and not uniquely identify a board, in that case more -boards can match the same `identificationPrefs`, that’s why we called it -“candidate”. +The metadata `identificationPrefs` associated to a port can be used to identify the board attached to that port. The algorithm is very simple: if a board listed in the platform file `boards.txt` match the same `identificationPrefs` coming from the discovery then the board is a “candidate” board attached to that port. Some identification properties may be imperfect and not uniquely identify a board, in that case more boards can match the same `identificationPrefs`, that’s why we called it “candidate”. -Let’s see an example to clarify things a bit, let suppose that we have -the following `identificationPrefs` coming from the serial discovery: +Let’s see an example to clarify things a bit, let suppose that we have the following `identificationPrefs` coming from the serial discovery: ```JSON [...CUT...] @@ -336,8 +254,7 @@ the following `identificationPrefs` coming from the serial discovery: [...CUT...] ``` -so we got `pid=0x804e, vid=0x2341`. Let’s suppose we have the following -`boards.txt`: +so we got `pid=0x804e, vid=0x2341`. Let’s suppose we have the following `boards.txt`: ``` # Arduino Zero (Prorgamming Port) @@ -380,15 +297,11 @@ mkr1000.upload.protocol=sam-ba [...CUT...] ``` -As we can see the only board that has the two properties matching is the -`mkr1000`, in this case the CLI knows that the board is 100% an MKR1000. +As we can see the only board that has the two properties matching is the `mkr1000`, in this case the CLI knows that the board is surely an MKR1000. -Note that `vid` and `pid` properties are just free text key/value pairs: -the discovery may returns basically anything, the board just needs to -have the same properties defined in `boards.txt` to be identified. +Note that `vid` and `pid` properties are just free text key/value pairs: the discovery may returns basically anything, the board just needs to have the same properties defined in `boards.txt` to be identified. -We can also specify multiple identification properties for the same -board using the `.N` suffix, for example: +We can also specify multiple identification properties for the same board using the `.N` suffix, for example: ``` myboard.name=My Wonderful Arduino Compatible Board @@ -408,68 +321,44 @@ myboard.apples.1=40 will match on both `pears=20, apples=30` and `pears=30, apples=40`. -## Upload (state of the art) +### Upload (state of the art) -In this section we will discuss the current status of the upload -business logic in the IDE/CLI. +In this section we will discuss the current status of the upload business logic in the IDE/CLI. -### Serial upload (CLI and Java IDE) +#### Serial upload (CLI and Java IDE) -Currently, to upload we use the `tools.UPLOAD_RECIPE_ID.upload.pattern` -recipe in `platform.txt`, where `UPLOAD_RECIPE_ID` is defined as the -value of `upload.tool` property in `boards.txt` for the currently -user-selected board. +Currently, to upload we use the `tools.UPLOAD_RECIPE_ID.upload.pattern` recipe in `platform.txt`, where `UPLOAD_RECIPE_ID` is defined as the value of `upload.tool` property in `boards.txt` for the currently user-selected board. -The recipe contains the variables `{serial.port}` and -`{serial.port.file}` that are replaced with the user selected serial -port (in particular on unix systems `{serial.port.file}` contains only -the file name `ttyACM0` while `{serial.port}` contains the full path to -the serial port `/dev/ttyACM0`). A typical serial upload recipe is -something like: +The recipe contains the variables `{serial.port}` and`{serial.port.file}` that are replaced with the user selected serial port (in particular on unix systems `{serial.port.file}` contains only the file name `ttyACM0` while `{serial.port}` contains the full path to the serial port `/dev/ttyACM0`). A typical serial upload recipe is something like: `tools.bossac.upload.pattern="{runtime.tools.bossac-1.7.0-arduino3.path}/bossac" --port={serial.port.file} -U true -i -e -w -v "{build.path}/{build.project_name}.bin" -R` -### Network upload (Java IDE only) +#### Network upload (Java IDE only) -The Java IDE has an integrated MDNS client that listen for network -boards. If a network board is selected the -`tools.UPLOAD_RECIPE_ID.upload.network_pattern` is used for upload. -`UPLOAD_RECIPE_ID` is obtained in the same way as for the serial upload. -A typical recipe is: +The Java IDE has an integrated MDNS client that listen for network boards. If a network board is selected the`tools.UPLOAD_RECIPE_ID.upload.network_pattern` is used for upload. `UPLOAD_RECIPE_ID` is obtained in the same way as for the serial upload. A typical recipe is: `tools.bossac.upload.network_pattern="{runtime.tools.arduinoOTA.path}/bin/arduinoOTA" -address {serial.port} -port 65280 -username arduino -password "{network.password}" -sketch "{build.path}/{build.project_name}.bin" -upload /sketch -b` please note that: -- the property is called `network_pattern` instead of `pattern` and, - yes, this is hardcoded +- the property is called `network_pattern` instead of `pattern`, this is hardcoded -- the port address is still called `{serial.port}` even if it is a - network address +- the port address is still called `{serial.port}` even if it is a network address -- there is a `{network.password}` that is a user entered field and, - yes, this is hardcoded +- there is a `{network.password}` that is a user entered field, this is hardcoded -## Upload with Pluggable discovery +### Upload with Pluggable discovery -Given the above, it is clear that the pluggable discovery should provide -features to allow: +Given the above, it is clear that the pluggable discovery should provide features to allow: 1. selection of different upload recipes based on the port `protocol` 1. provide extra port metadata to use on the command line recipe -1. provide eventual extra info that should be user-supplied (by - entering them in a dialog or provided in other ways) +1. provide eventual extra info that should be user-supplied (by entering them in a dialog or provided in other ways) 1. should be backward compatible with the current legacy -### Recipe selection based on protocol +#### Recipe selection based on protocol -To allow the above the `UPLOAD_RECIPE_ID` selection is now **dependent -on protocol**. The `boards.txt` now must specify a recipe id for each -type of upload protocol supported by the board. Previously we used the -property `upload.tool=UPLOAD_RECIPE_ID` now this property should be -changed to `upload.tool.UPLOAD_PROTOCOL=UPLOAD_RECIPE_ID` so we can -choose the correct tool for each protocol. We should also keep the old -property for backward compatibility +To allow the above the `UPLOAD_RECIPE_ID` selection is now **dependent on protocol**. The `boards.txt` now must specify a recipe id for each type of upload protocol supported by the board. Previously we used the property `upload.tool=UPLOAD_RECIPE_ID` now this property should be changed to `upload.tool.UPLOAD_PROTOCOL=UPLOAD_RECIPE_ID` so we can choose the correct tool for each protocol. We should also keep the old property for backward compatibility **Before:** @@ -481,22 +370,17 @@ myboard.upload.tool=bossac ``` # keep for backward compatibility -myboard.upload.tool=avrdude +myboard.upload.tool=bossac # Upload recipes myboard.upload.tool.serial=bossac myboard.upload.tool.network=arduino_ota ``` -The selected port address will be provided in the variable -`{upload.address}`. Other metadata provided by the discover in the -`prefs` section will be provided in the `{upload.port.*}` variables. +The selected port address will be provided in the variable `{upload.address}`. Other metadata provided by the discover in the `prefs` section will be provided in the `{upload.port.*}` variables. -For backward compatibility we will keep a copy of the address also in -`{serial.port}` and in the specific case of a `protocol=serial` we will -populate also `{serial.port.file}`. +For backward compatibility we will keep a copy of the address also in `{serial.port}` and in the specific case of a `protocol=serial` we will populate also `{serial.port.file}`. -For example, the following port metadata coming from a pluggable -discovery: +For example, the following port metadata coming from a pluggable discovery: ```JSON { @@ -546,7 +430,7 @@ Here another example: "macaddress": "AA:BB:CC:DD:EE:FF" }, "identificationPrefs": { - "macprefix": "AA:BB:CC", + "macprefix": "AA:BB:CC" } } } @@ -562,9 +446,7 @@ that is translated to: {serial.port} = 192.168.1.232 # for backward compatibility ``` -This configuration, together with protocol selection, allows to remove -the hardcoded `network_pattern`, now we can replace the legacy recipe -(split into multiple lines for clarity): +This configuration, together with protocol selection, allows to remove the hardcoded `network_pattern`, now we can replace the legacy recipe (split into multiple lines for clarity): ``` tools.bossac.upload.network_pattern="{runtime.tools.arduinoOTA.path}/bin/arduinoOTA" @@ -580,21 +462,16 @@ tools.arduino_ota.upload.pattern="{runtime.tools.arduinoOTA.path}/bin/arduinoOTA -sketch "{build.path}/{build.project_name}.bin" ``` -### User provided fields +#### User provided fields -The recipe may require some user-entered fields (like username or -password). In this case the recipe must use the special placeholder -`{upload.user.FIELD_NAME}` where `FIELD_NAME` must be declared -separately in the recipe using the following format: +The recipe may require some user-entered fields (like username or password). In this case the recipe must use the special placeholder `{upload.user.FIELD_NAME}` where `FIELD_NAME` must be declared separately in the recipe using the following format: ``` tools.UPLOAD_RECIPE_ID.upload.field.FIELD_NAME=FIELD_LABEL tools.UPLOAD_RECIPE_ID.upload.field.FIELD_NAME.secret=true ``` -`FIELD_LABEL` is the label shown in the dialog to ask the user to enter -the value of the field. The `secret` property is optional and it should -be set to `true` if the field is a secret (like passwords or token). +`FIELD_LABEL` is the label shown in the dialog to ask the user to enter the value of the field. The `secret` property is optional and it should be set to `true` if the field is a secret (like passwords or token). Let’s see how a complete example will look like: @@ -609,40 +486,23 @@ tools.arduino_ota.upload.pattern="{runtime.tools.arduinoOTA.path}/bin/arduinoOTA -sketch "{build.path}/{build.project_name}.bin" ``` -## CLI command line UX considerations +## Open Questions + +### CLI command line UX considerations -Currently only serial ports are allowed on the CLI, so we don’t need to -specify a protocol when we select a port, if we write -`arduino-cli upload --port /dev/ttyACM0` or -`arduino-cli upload --port COM1` we are sure that this is a serial port. +Currently only serial ports are allowed on the CLI, so we don’t need to specify a protocol when we select a port, if we write `arduino-cli upload --port /dev/ttyACM0` or `arduino-cli upload --port COM1` we are sure that this is a serial port. -Things get a bit more complex with Pluggable Discovery: `--port COM1` -could be a serial port or the hostname of a network board that is called -exactly `COM1`. To solve this amiguity we may add a flag to the `upload` -command to specify the protocol so it becomes -`arduino-cli upload --protocol serial --port COM1`. +Things get a bit more complex with Pluggable Discovery: `--port COM1` could be a serial port or the hostname of a network board that is called exactly `COM1`. To solve this amiguity we may add a flag to the `upload` command to specify the protocol so it becomes `arduino-cli upload --protocol serial --port COM1`. -Obviously using `--protocol` is ugly and makes the command line -overcrowded. To avoid the `--protocol` flag we may adopt some -strategies: +Obviously using `--protocol` is ugly and makes the command line overcrowded. To avoid the `--protocol` flag we may adopt some strategies: -- we could check the current overall port list and see if the - specified port address match **only one** of the available ports +- we could check the current overall port list and see if the specified port address match **only one** of the available ports -- we may check if the address is a valid address for **only one** of - the supported protocol. For example if we have support for `serial` - and `network` only and if the entered address is `1.2.3.4` we know - that this is a `network` address. This strategy requires that the - pluggable discoveries have a command to check if the address is a - valid address for their specific protocol. +- we may check if the address is a valid address for **only one** of the supported protocol. For example if we have support for `serial` and `network` only and if the entered address is `1.2.3.4` we know that this is a `network` address. This strategy requires that the pluggable discoveries have a command to check if the address is a valid address for their specific protocol. -- we may save the protocol as part of the `board attach` command (as - we do already for the port address) +- we may save the protocol as part of the `board attach` command (as we do already for the port address) -- we may use a URL format to specify protocol and address at the same - time: `--port serial:///dev/ttyACM0` or `--port ssh://1.2.3.4` or - `--port xyz://12-34/32:23/442/212`. BTW It may be problematic to - express an address in form of an URL. +- we may use a URL format to specify protocol and address at the same time: `--port serial:///dev/ttyACM0` or `--port ssh://1.2.3.4` or `--port xyz://12-34/32:23/442/212`. BTW It may be problematic to express an address in form of an URL. ## Appendix From 3254493dc326cde713f15cba89958c1ba1559b9e Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Wed, 31 Mar 2021 17:18:28 +0200 Subject: [PATCH 03/44] Assign RFC number --- RFCs/{0000-pluggable-discovery.md => 0002-pluggable-discovery.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename RFCs/{0000-pluggable-discovery.md => 0002-pluggable-discovery.md} (100%) diff --git a/RFCs/0000-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md similarity index 100% rename from RFCs/0000-pluggable-discovery.md rename to RFCs/0002-pluggable-discovery.md From 78ee413dd84c3f96e2686bc2ef03516dd09f7c13 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Thu, 1 Apr 2021 17:47:16 +0200 Subject: [PATCH 04/44] Added a note about multiple 'add' events --- RFCs/0002-pluggable-discovery.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index 81966e1..02bfd92 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -200,6 +200,8 @@ The `remove` event looks like the following: the content is straightforward, in this case only the `address` field is reported. +If the information about a port needs to be updated the discovery may send a new `add` message for the same port address without sending a `remove` first: this means that all the previous information about the port must be discarded and replaced with the new one. + A demo tool is available here: [https://github.com/arduino/serial-discovery](https://github.com/arduino/serial-discovery#example-of-usage) From bfcb83df0346ce83e88a645bfb0a164d6a38ae62 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Thu, 1 Apr 2021 18:12:54 +0200 Subject: [PATCH 05/44] Added a note about START_SYNC --- RFCs/0002-pluggable-discovery.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index 02bfd92..4ce5425 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -46,6 +46,8 @@ These tools must be in the form of executables that can be launched as a subproc ### Pluggable discovery API via stdin/stdout +All the commands listed in this specification must be implemented in the discovery. + After startup, the tool will just stay idle waiting for commands. The available commands are: `START`, `STOP`, `QUIT`, `LIST` and `START_SYNC`. #### START command From 16a2687456907628982d1a43d01f0b7855691bfb Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Thu, 1 Apr 2021 22:43:12 +0200 Subject: [PATCH 06/44] Added discovery priority based on the currently selected board --- RFCs/0002-pluggable-discovery.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index 4ce5425..5e5f754 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -241,6 +241,8 @@ It may happen that different 3rd party platforms provides the same discovery or We can partially handle this if the `DISCOVERY_ID` field in `platform.txt` is well defined: from the CLI we could group together the platforms that requires the same discovery and launch the latest version available just once. How the different 3rd party will agree on the `DISCOVERY_ID` value population is TBD. +In case different discoveries provide conflicting information (for example if two discoveries provide different information for the same port address) we could partially mitigate the issue by giving priority to the discovery that is bundled with the package of the selected board. + #### Board identification The metadata `identificationPrefs` associated to a port can be used to identify the board attached to that port. The algorithm is very simple: if a board listed in the platform file `boards.txt` match the same `identificationPrefs` coming from the discovery then the board is a “candidate” board attached to that port. Some identification properties may be imperfect and not uniquely identify a board, in that case more boards can match the same `identificationPrefs`, that’s why we called it “candidate”. From afac1ccf1f8eac9460c08d3ebe8b6a4f31e1ea21 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Fri, 30 Apr 2021 18:10:07 +0200 Subject: [PATCH 07/44] fix: Address and Protocol must be unique --- RFCs/0002-pluggable-discovery.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index 5e5f754..56d0e70 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -34,7 +34,7 @@ The pluggable discovery aims to provide a solution to these problems. - Each port may provide metadata to identify a specific instance of the board (**serial number / MAC address**) -- Each port must have an **unique address** +- Each port must have an **unique address and protocol pair** - A single device may expose multiple ports @@ -195,14 +195,15 @@ The `remove` event looks like the following: { "eventType": "remove", "port": { - "address": "/dev/ttyACM0" + "address": "/dev/ttyACM0", + "protocol": "serial" } } ``` -the content is straightforward, in this case only the `address` field is reported. +the content is straightforward, in this case only the `address` and `protocol` fields are reported. -If the information about a port needs to be updated the discovery may send a new `add` message for the same port address without sending a `remove` first: this means that all the previous information about the port must be discarded and replaced with the new one. +If the information about a port needs to be updated the discovery may send a new `add` message for the same port address and protocol without sending a `remove` first: this means that all the previous information about the port must be discarded and replaced with the new one. A demo tool is available here: [https://github.com/arduino/serial-discovery](https://github.com/arduino/serial-discovery#example-of-usage) From 2a1cc89580a7b8207d166fcb8764f17fbd958506 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Fri, 30 Apr 2021 18:11:15 +0200 Subject: [PATCH 08/44] Renamed 'prefs' field to 'properties' --- RFCs/0002-pluggable-discovery.md | 38 ++++++++++++++++---------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index 56d0e70..e482510 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -32,7 +32,7 @@ The pluggable discovery aims to provide a solution to these problems. - Each port may provide metadata to identify the board model (**identification properites**) -- Each port may provide metadata to identify a specific instance of the board (**serial number / MAC address**) +- Each port may provide **properties relative to the port** (USB serial number / MAC address) - Each port must have an **unique address and protocol pair** @@ -98,8 +98,8 @@ The `LIST` command executes an enumeration of the ports and returns a list of th "label": <-- HOW THE PORT IS DISPLAYED ON THE GUI "protocol": <-- THE PROTOCOL USED BY THE BOARD "protocolLabel": <-- HOW THE PROTOCOL IS DISPLAYED ON THE GUI - "prefs": { - <-- A LIST OF PROPERTIES FOR THAT SPECIFIC BOARD + "properties": { + <-- A LIST OF PROPERTIES OF THE PORT }, "identificationPrefs": { <-- A LIST OF PROPERTIES TO IDENTIFY THE BOARD MODEL @@ -123,7 +123,7 @@ Each port has: - `protocolLabel` is the `protocol` in human readable form (for example `Serial port` or `DFU USB` or `Network (ssh)`) -- `prefs` is a list of key/value pairs that represent information relative to the specific instance of the board +- `properties` is a list of key/value pairs that represent information relative to the specific port - `identificationPrefs` is a list of key value pairs that represent information to identify the board **model** @@ -138,7 +138,7 @@ To make the above more clear let’s show an example with the `serial_discovery` "label": "ttyACM0", "protocol": "serial", "protocolLabel": "Serial Port (USB)", - "prefs": { + "properties": { "pid": "0x804e", "vid": "0x2341", "serialNumber": "EBEABFD6514D32364E202020FF10181E" @@ -152,7 +152,7 @@ To make the above more clear let’s show an example with the `serial_discovery` } ``` -In this case the serial port metadata comes from an USB serial converter. The USB VID/PID and USB SERIAL NUMBER properties are also reported inside `prefs`. Inside the `identificationPrefs` instead we have only the properties useful for product identification (in this case only USB VID/PID is useful to identify the board model). +In this case the serial port metadata comes from an USB serial converter. The USB VID/PID and USB SERIAL NUMBER properties are also reported inside `properties`. Inside the `identificationPrefs` instead we have only the properties useful for product identification (in this case only USB VID/PID is useful to identify the board model). The `LIST` command performs a one-shot polling of the ports. If you need continuous monitoring of ports you should use the `START_SYNC` command. @@ -172,7 +172,7 @@ The `add` event looks like the following: "port": { "address": "/dev/ttyACM0", "label": "ttyACM0", - "prefs": { + "properties": { "pid": "0x804e", "vid": "0x2341", "serialNumber": "EBEABFD6514D32364E202020FF10181E" @@ -396,17 +396,17 @@ For example, the following port metadata coming from a pluggable discovery: "address": "/dev/ttyACM0", "label": "ttyACM0", "protocol": "serial", - "protocolLabel": "Serial Port (USB)" - "prefs": { - "pid": "0x804e", - "vid": "0x2341", - "serialNumber": "EBEABFD6514D32364E202020FF10181E" - }, - "identificationPrefs": { - "pid": "0x804e", - "vid": "0x2341" - } - } + "protocolLabel": "Serial Port (USB)", + "properties": { + "pid": "0x804e", + "vid": "0x2341", + "serialNumber": "EBEABFD6514D32364E202020FF10181E" + }, + "identificationPrefs": { + "pid": "0x804e", + "vid": "0x2341" + } + } } ``` @@ -432,7 +432,7 @@ Here another example: "label": "SSH on my-board ()192.168.1.232)", "protocol": "ssh", "protocolLabel": "SSH Network port" - "prefs": { + "properties": { "macprefix": "AA:BB:CC", "macaddress": "AA:BB:CC:DD:EE:FF" }, From 2cab0a9ed0a8317572006e850bece5b4598955f7 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Fri, 30 Apr 2021 18:12:02 +0200 Subject: [PATCH 09/44] Made matching algorithm more explicit about mixing identification props --- RFCs/0002-pluggable-discovery.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index e482510..e8b2fa4 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -326,7 +326,7 @@ myboard.pears.1=30 myboard.apples.1=40 ``` -will match on both `pears=20, apples=30` and `pears=30, apples=40`. +will match on both `pears=20, apples=30` and `pears=30, apples=40` but not `pears=20, apples=40`, in that sense each "set" of identification properties is indepentent from each other and cannot be mixed for port matching. ### Upload (state of the art) From e17982cf8cb05f01d95b0675c01021daefb3b180 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Fri, 30 Apr 2021 18:48:32 +0200 Subject: [PATCH 10/44] Removed identificationPrefs It has been replaced by the upload_port.* prefix in the boards.txt. --- RFCs/0002-pluggable-discovery.md | 98 ++++++++++++++------------------ 1 file changed, 43 insertions(+), 55 deletions(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index e8b2fa4..c8b73f7 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -30,8 +30,6 @@ The pluggable discovery aims to provide a solution to these problems. - Each port must be **enumerated/discovered** -- Each port may provide metadata to identify the board model (**identification properites**) - - Each port may provide **properties relative to the port** (USB serial number / MAC address) - Each port must have an **unique address and protocol pair** @@ -100,9 +98,6 @@ The `LIST` command executes an enumeration of the ports and returns a list of th "protocolLabel": <-- HOW THE PROTOCOL IS DISPLAYED ON THE GUI "properties": { <-- A LIST OF PROPERTIES OF THE PORT - }, - "identificationPrefs": { - <-- A LIST OF PROPERTIES TO IDENTIFY THE BOARD MODEL } }, { ... <-- OTHER PORTS... @@ -125,8 +120,6 @@ Each port has: - `properties` is a list of key/value pairs that represent information relative to the specific port -- `identificationPrefs` is a list of key value pairs that represent information to identify the board **model** - To make the above more clear let’s show an example with the `serial_discovery`: ```JSON @@ -142,17 +135,13 @@ To make the above more clear let’s show an example with the `serial_discovery` "pid": "0x804e", "vid": "0x2341", "serialNumber": "EBEABFD6514D32364E202020FF10181E" - }, - "identificationPrefs": { - "pid": "0x804e", - "vid": "0x2341" } } ] } ``` -In this case the serial port metadata comes from an USB serial converter. The USB VID/PID and USB SERIAL NUMBER properties are also reported inside `properties`. Inside the `identificationPrefs` instead we have only the properties useful for product identification (in this case only USB VID/PID is useful to identify the board model). +In this case the serial port metadata comes from an USB serial converter. Inside the `properties` we have all the properties of the port, and some of them may be useful for product identification (in this case only USB VID/PID is useful to identify the board model). The `LIST` command performs a one-shot polling of the ports. If you need continuous monitoring of ports you should use the `START_SYNC` command. @@ -177,10 +166,6 @@ The `add` event looks like the following: "vid": "0x2341", "serialNumber": "EBEABFD6514D32364E202020FF10181E" }, - "identificationPrefs": { - "pid": "0x804e", - "vid": "0x2341" - }, "protocol": "serial", "protocolLabel": "Serial Port (USB)" } @@ -246,29 +231,35 @@ In case different discoveries provide conflicting information (for example if tw #### Board identification -The metadata `identificationPrefs` associated to a port can be used to identify the board attached to that port. The algorithm is very simple: if a board listed in the platform file `boards.txt` match the same `identificationPrefs` coming from the discovery then the board is a “candidate” board attached to that port. Some identification properties may be imperfect and not uniquely identify a board, in that case more boards can match the same `identificationPrefs`, that’s why we called it “candidate”. +The `properties` associated to a port can be used to identify the board attached to that port. The algorithm is simple: + +- each board listed in the platform file `boards.txt` may declare a set of `upload_port.*` properties +- if each `upload_port.*` property has a match in the `properties` set coming from the discovery then the board is a “candidate” board attached to that port. -Let’s see an example to clarify things a bit, let suppose that we have the following `identificationPrefs` coming from the serial discovery: +Some port `properties` may not be precise enough to uniquely identify a board, in that case more boards may match the same set of `properties`, that’s why we called it “candidate”. + +Let’s see an example to clarify things a bit, let's suppose that we have the following `properties` coming from the serial discovery: ```JSON [...CUT...] "port": { "address": "/dev/ttyACM0", - "identificationPrefs": { + "properties": { "pid": "0x804e", - "vid": "0x2341" + "vid": "0x2341", + "serialNumber": "EBEABFD6514D32364E202020FF10181E" } [...CUT...] ``` -so we got `pid=0x804e, vid=0x2341`. Let’s suppose we have the following `boards.txt`: +in this case we can use `vid` and `pid` to identify the board. The `serialNumber`, instead, is unique for that specific instance of the board so it can't be used to identify the board model. Let’s suppose we have the following `boards.txt`: ``` # Arduino Zero (Prorgamming Port) # --------------------------------------- arduino_zero_edbg.name=Arduino Zero (Programming Port) -arduino_zero_edbg.vid.0=0x03eb -arduino_zero_edbg.pid.0=0x2157 +arduino_zero_edbg.upload_port.vid=0x03eb +arduino_zero_edbg.upload_port.pid=0x2157 arduino_zero_edbg.debug.tool=gdb arduino_zero_edbg.upload.tool=openocd arduino_zero_edbg.upload.protocol=sam-ba @@ -276,28 +267,28 @@ arduino_zero_edbg.upload.protocol=sam-ba # Arduino Zero (Native USB Port) # -------------------------------------- arduino_zero_native.name=Arduino Zero (Native USB Port) -arduino_zero_native.vid.0=0x2341 -arduino_zero_native.pid.0=0x804d -arduino_zero_native.vid.1=0x2341 -arduino_zero_native.pid.1=0x004d -arduino_zero_native.vid.2=0x2341 -arduino_zero_native.pid.2=0x824d -arduino_zero_native.vid.3=0x2341 -arduino_zero_native.pid.3=0x024d +arduino_zero_native.upload_port.0.vid=0x2341 +arduino_zero_native.upload_port.0.pid=0x804d +arduino_zero_native.upload_port.1.vid=0x2341 +arduino_zero_native.upload_port.1.pid=0x004d +arduino_zero_native.upload_port.2.vid=0x2341 +arduino_zero_native.upload_port.2.pid=0x824d +arduino_zero_native.upload_port.3.vid=0x2341 +arduino_zero_native.upload_port.3.pid=0x024d arduino_zero_native.upload.tool=bossac arduino_zero_native.upload.protocol=sam-ba [...CUT...] # Arduino MKR1000 # ----------------------- mkr1000.name=Arduino MKR1000 -mkr1000.vid.0=0x2341 <------- MATCHING IDs -mkr1000.pid.0=0x804e <------- MATCHING IDs -mkr1000.vid.1=0x2341 -mkr1000.pid.1=0x004e -mkr1000.vid.2=0x2341 -mkr1000.pid.2=0x824e -mkr1000.vid.3=0x2341 -mkr1000.pid.3=0x024e +mkr1000.upload_port.0.vid=0x2341 <------- MATCHING IDs +mkr1000.upload_port.0.pid=0x804e <------- MATCHING IDs +mkr1000.upload_port.1.vid=0x2341 +mkr1000.upload_port.1.pid=0x004e +mkr1000.upload_port.2.vid=0x2341 +mkr1000.upload_port.2.pid=0x824e +mkr1000.upload_port.3.vid=0x2341 +mkr1000.upload_port.3.pid=0x024e mkr1000.debug.tool=gdb mkr1000.upload.tool=bossac mkr1000.upload.protocol=sam-ba @@ -306,24 +297,24 @@ mkr1000.upload.protocol=sam-ba As we can see the only board that has the two properties matching is the `mkr1000`, in this case the CLI knows that the board is surely an MKR1000. -Note that `vid` and `pid` properties are just free text key/value pairs: the discovery may returns basically anything, the board just needs to have the same properties defined in `boards.txt` to be identified. +Note that `vid` and `pid` properties are just free text key/value pairs: the discovery may return basically anything, the board just needs to have the same properties defined in `boards.txt` as `upload_port.*` to be identified. We can also specify multiple identification properties for the same board using the `.N` suffix, for example: ``` myboard.name=My Wonderful Arduino Compatible Board -myboard.pears=20 -myboard.apples=30 +myboard.upload_port.pears=20 +myboard.upload_port.apples=30 ``` will match on `pears=20, apples=30` but: ``` myboard.name=My Wonderful Arduino Compatible Board -myboard.pears.0=20 -myboard.apples.0=30 -myboard.pears.1=30 -myboard.apples.1=40 +myboard.upload_port.0.pears=20 +myboard.upload_port.0.apples=30 +myboard.upload_port.1.pears=30 +myboard.upload_port.1.apples=40 ``` will match on both `pears=20, apples=30` and `pears=30, apples=40` but not `pears=20, apples=40`, in that sense each "set" of identification properties is indepentent from each other and cannot be mixed for port matching. @@ -431,15 +422,12 @@ Here another example: "address": "192.168.1.232", "label": "SSH on my-board ()192.168.1.232)", "protocol": "ssh", - "protocolLabel": "SSH Network port" - "properties": { - "macprefix": "AA:BB:CC", - "macaddress": "AA:BB:CC:DD:EE:FF" - }, - "identificationPrefs": { - "macprefix": "AA:BB:CC" - } - } + "protocolLabel": "SSH Network port", + "properties": { + "macprefix": "AA:BB:CC", + "macaddress": "AA:BB:CC:DD:EE:FF" + } + } } ``` From 44e8d99dca2cd669cbc40189dcc3fb156a70dbe4 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Fri, 30 Apr 2021 18:55:00 +0200 Subject: [PATCH 11/44] Added some backward compatibility considerations --- RFCs/0002-pluggable-discovery.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index c8b73f7..ea52729 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -319,6 +319,22 @@ myboard.upload_port.1.apples=40 will match on both `pears=20, apples=30` and `pears=30, apples=40` but not `pears=20, apples=40`, in that sense each "set" of identification properties is indepentent from each other and cannot be mixed for port matching. +#### Backward compatibility considerations + +Many platforms predating the pluggable discovery have the following definitions for board identification: + +``` +myboard.vid=0x1234 +myboard.pid=0x4567 +``` + +to ensure backward compatibility we will transparently and automatically convert these definitions into the new format + +``` +myboard.upload_port.0.vid=0x1234 +myboard.upload_port.0.pid=0x4567 +``` + ### Upload (state of the art) In this section we will discuss the current status of the upload business logic in the IDE/CLI. From b10d52db074f74a2e39d39ae5617de6ff633f87e Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Fri, 30 Apr 2021 19:01:50 +0200 Subject: [PATCH 12/44] Added HELLO command --- RFCs/0002-pluggable-discovery.md | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index ea52729..b4bb435 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -1,4 +1,4 @@ -# Pluggable Discovery +# Pluggable Discovery protocol (Version 1.0.0) ## Overview @@ -46,7 +46,33 @@ These tools must be in the form of executables that can be launched as a subproc All the commands listed in this specification must be implemented in the discovery. -After startup, the tool will just stay idle waiting for commands. The available commands are: `START`, `STOP`, `QUIT`, `LIST` and `START_SYNC`. +After startup, the tool will just stay idle waiting for commands. The available commands are: `HELLO`, `START`, `STOP`, `QUIT`, `LIST` and `START_SYNC`. + +#### HELLO command + +`HELLO` is the **first command that must be sent** to the discovery to tell the name of the client and the version of the pluggable discovery protocol that the client supports. +The syntax of the command is: + +`HELLO ""` + +- `` is the protocol version (currently `1.0.0`) + +- `` is the name and version of the client (double-quotes `"` are not allowed) + +some examples: + +- `HELLO 1.0.0 "Arduino IDE 1.8.13"` + +- `HELLO 1.0.0 "arduino-cli 1.2.3"` + +the response to the command is: + +```JSON +{ + "eventType": "hello", + "message": "OK" +} +``` #### START command From a88bd92e28183475bc8b663004f323318d908fce Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Tue, 4 May 2021 18:19:07 +0200 Subject: [PATCH 13/44] Added possibility to use discovery from other platforms --- RFCs/0002-pluggable-discovery.md | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index b4bb435..1e27e49 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -247,13 +247,39 @@ discovery.teensy.pattern="{runtime.tools.teensy_ports.path}/hardware/tools/teens in this case the platform provides a new `teensy` discovery and the command line tool named `teensy_ports` is launched with the `-J2` parameter to start the discovery tool. +#### Using a discovery made by a 3rd party + +A platform may opt to depend on a discovery developed by a 3rd party instead of writing and maintaining his own. Since writing a good-quality cross-platform discovery is very hard and time consuming, we expect this option to be used by the majority of developers. + +To depend on a 3rd party discovery the platform must add the following directive in the `platform.txt` file: + +``` +discovery.required=PLATFORM:ARCHITECTURE:DISCOVERY_ID +``` + +or if the platform needs more discoveries: + +``` +discovery.required.0=PLATFORM:ARCHITECTURE:DISCOVERY_ID_1 +discovery.required.1=PLATFORM:ARCHITECTURE:DISCOVERY_ID_2 +... +``` + +The `PACKAGER:ARCHITECTURE:DISCOVERY_ID` field represents a unique identifier to a 3rd party discovery in particular the `PACKAGER:ARCHITECTURE:...` part is the same as in the FQBN for the boards. + +For example if a platform needs the `network` discovery from the Arduino AVR platform it may specify it with: + +``` +discovery.required=arduino:avr:network +``` + #### Duplicate discoveries It may happen that different 3rd party platforms provides the same discovery or different versions of the same discovery or, worse, different version of the same discovery launched with different parameters. We can partially handle this if the `DISCOVERY_ID` field in `platform.txt` is well defined: from the CLI we could group together the platforms that requires the same discovery and launch the latest version available just once. How the different 3rd party will agree on the `DISCOVERY_ID` value population is TBD. -In case different discoveries provide conflicting information (for example if two discoveries provide different information for the same port address) we could partially mitigate the issue by giving priority to the discovery that is bundled with the package of the selected board. +In case different discoveries provide conflicting information (for example if two discoveries provide different information for the same port address) we could partially mitigate the issue by giving priority to the discovery that is used by the package of the selected board. #### Board identification From 4c7f1bf056801ed042d40a7ab1cc805921294115 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Tue, 4 May 2021 18:19:28 +0200 Subject: [PATCH 14/44] cleaned up examples for board identification --- RFCs/0002-pluggable-discovery.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index 1e27e49..193da0c 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -312,9 +312,6 @@ in this case we can use `vid` and `pid` to identify the board. The `serialNumber arduino_zero_edbg.name=Arduino Zero (Programming Port) arduino_zero_edbg.upload_port.vid=0x03eb arduino_zero_edbg.upload_port.pid=0x2157 -arduino_zero_edbg.debug.tool=gdb -arduino_zero_edbg.upload.tool=openocd -arduino_zero_edbg.upload.protocol=sam-ba [...CUT...] # Arduino Zero (Native USB Port) # -------------------------------------- @@ -327,8 +324,6 @@ arduino_zero_native.upload_port.2.vid=0x2341 arduino_zero_native.upload_port.2.pid=0x824d arduino_zero_native.upload_port.3.vid=0x2341 arduino_zero_native.upload_port.3.pid=0x024d -arduino_zero_native.upload.tool=bossac -arduino_zero_native.upload.protocol=sam-ba [...CUT...] # Arduino MKR1000 # ----------------------- @@ -341,9 +336,6 @@ mkr1000.upload_port.2.vid=0x2341 mkr1000.upload_port.2.pid=0x824e mkr1000.upload_port.3.vid=0x2341 mkr1000.upload_port.3.pid=0x024e -mkr1000.debug.tool=gdb -mkr1000.upload.tool=bossac -mkr1000.upload.protocol=sam-ba [...CUT...] ``` From 972e15aef61b30df001da0638ddfeef6331fade3 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Thu, 13 May 2021 13:57:04 +0200 Subject: [PATCH 15/44] Update RFCs/0002-pluggable-discovery.md Co-authored-by: Matthijs Kooijman --- RFCs/0002-pluggable-discovery.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index 193da0c..98ea42b 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -36,7 +36,7 @@ The pluggable discovery aims to provide a solution to these problems. - A single device may expose multiple ports -## Recommended Solution +## Proposed Solution The proposed solution is to provide a tool that can enumerate and discover the ports for a **single specific protocol**. There will be a tool for each supported protocol so, in the end, we will have a “Serial ports discovery”, a “Network port discovery” and so on. From 8aa8f6e43360b8f042fee978004bf1be176d1bf8 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Thu, 13 May 2021 13:37:37 +0200 Subject: [PATCH 16/44] Made 'plugin' more explicit --- RFCs/0002-pluggable-discovery.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index 98ea42b..4e51161 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -26,7 +26,7 @@ The pluggable discovery aims to provide a solution to these problems. - A communication port may use any kind of **protocol** -- Software support for a new **protocol** must be added to the system as a **plugin** +- Software support for a new **protocol** must be added to the system **using an external command line tool** - Each port must be **enumerated/discovered** From 4b9d4b2e99afce2553e7332f2aa3a5a4601b3971 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Thu, 13 May 2021 14:21:15 +0200 Subject: [PATCH 17/44] Added HELLO negotiation --- RFCs/0002-pluggable-discovery.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index 4e51161..e70052c 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -50,12 +50,12 @@ After startup, the tool will just stay idle waiting for commands. The available #### HELLO command -`HELLO` is the **first command that must be sent** to the discovery to tell the name of the client and the version of the pluggable discovery protocol that the client supports. +`HELLO` is the **first command that must be sent** to the discovery to tell the name of the client/IDE and the version of the pluggable discovery protocol that the client/IDE supports. The syntax of the command is: `HELLO ""` -- `` is the protocol version (currently `1.0.0`) +- `` is the maximum protocol version supported by the client/IDE (currently `1.0.0`) - `` is the name and version of the client (double-quotes `"` are not allowed) @@ -70,10 +70,17 @@ the response to the command is: ```JSON { "eventType": "hello", + "protocolVersion": "1.0.0", "message": "OK" } ``` +The `protocolVersion` field represents the protocol version that will be used in the rest of the communication. There are three possible cases: + +- if the client/IDE supports the same or a more recent version of the protocol than the discovery, then the IDE should go into a compatibility mode and use the protocol level supported by the discovery. +- if the discovery supports a more recent version of the protocol than the client/IDE: the discovery should downgrade itself into compatibility mode and report a `protocolVersion` that is less than or equal to the one supported by the client/IDE. +- if the discovery cannot go into compatibility mode, it will report the protocol version supported (even if greater than the version supported by the client/IDE) and the client/IDE may decide to terminate the discovery or produce an error/warning. + #### START command The `START` command initializes and start the discovery internal subroutines. This command must be called before `LIST` or `START_SYNC`. The response to the start command is: From 0316cb966a21e0e50cefe3bd851adf5f604aad87 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Thu, 13 May 2021 14:25:05 +0200 Subject: [PATCH 18/44] Use a single number revision scheme --- RFCs/0002-pluggable-discovery.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index e70052c..85ce207 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -1,4 +1,4 @@ -# Pluggable Discovery protocol (Version 1.0.0) +# Pluggable Discovery protocol (Version 1) ## Overview @@ -55,22 +55,22 @@ The syntax of the command is: `HELLO ""` -- `` is the maximum protocol version supported by the client/IDE (currently `1.0.0`) +- `` is the maximum protocol version supported by the client/IDE (currently `1`) - `` is the name and version of the client (double-quotes `"` are not allowed) some examples: -- `HELLO 1.0.0 "Arduino IDE 1.8.13"` +- `HELLO 1 "Arduino IDE 1.8.13"` -- `HELLO 1.0.0 "arduino-cli 1.2.3"` +- `HELLO 1 "arduino-cli 1.2.3"` the response to the command is: ```JSON { "eventType": "hello", - "protocolVersion": "1.0.0", + "protocolVersion": "1", "message": "OK" } ``` From e051454c7c4ed1b38c3087a764984ce4d896849d Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Thu, 13 May 2021 14:43:21 +0200 Subject: [PATCH 19/44] Added 'name' field in hypotetical serial-discovery --- RFCs/0002-pluggable-discovery.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index 85ce207..ffce32d 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -167,7 +167,8 @@ To make the above more clear let’s show an example with the `serial_discovery` "properties": { "pid": "0x804e", "vid": "0x2341", - "serialNumber": "EBEABFD6514D32364E202020FF10181E" + "serialNumber": "EBEABFD6514D32364E202020FF10181E", + "name": "ttyACM0" } } ] @@ -197,7 +198,8 @@ The `add` event looks like the following: "properties": { "pid": "0x804e", "vid": "0x2341", - "serialNumber": "EBEABFD6514D32364E202020FF10181E" + "serialNumber": "EBEABFD6514D32364E202020FF10181E", + "name": "ttyACM0" }, "protocol": "serial", "protocolLabel": "Serial Port (USB)" @@ -306,7 +308,8 @@ Let’s see an example to clarify things a bit, let's suppose that we have the f "properties": { "pid": "0x804e", "vid": "0x2341", - "serialNumber": "EBEABFD6514D32364E202020FF10181E" + "serialNumber": "EBEABFD6514D32364E202020FF10181E", + "name": "ttyACM0" } [...CUT...] ``` @@ -458,7 +461,8 @@ For example, the following port metadata coming from a pluggable discovery: "properties": { "pid": "0x804e", "vid": "0x2341", - "serialNumber": "EBEABFD6514D32364E202020FF10181E" + "serialNumber": "EBEABFD6514D32364E202020FF10181E", + "name": "ttyACM0" }, "identificationPrefs": { "pid": "0x804e", @@ -476,6 +480,7 @@ will be available on the recipe as the variables: {upload.port.pid} = 0x8043 {upload.port.vid} = 0x2341 {upload.port.serialNumber} = EBEABFD6514D32364E202020FF10181E +{upload.port.name} = ttyACM0 {serial.port} = ttyACM0 # for backward compatibility {serial.port.file} = /dev/ttyACM0 # only because protocol=serial ``` From bad688c885ee20c5b779a43048cef459c668d935 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Thu, 13 May 2021 14:49:39 +0200 Subject: [PATCH 20/44] Update RFCs/0002-pluggable-discovery.md Co-authored-by: Matthijs Kooijman --- RFCs/0002-pluggable-discovery.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index ffce32d..bb11fa7 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -207,7 +207,7 @@ The `add` event looks like the following: } ``` -it basically provides the same information as the `list` event but for a single port. After calling `START_SYNC` an initial burst of add events may be generated in sequence to report all the ports available at the moment of the start. +It basically provides the same information as the `list` event but for a single port. After calling `START_SYNC` an initial burst of add events must be generated in sequence to report all the ports available at the moment of the start. The `remove` event looks like the following: From 3916891f620c4a427f3f3f9aad11c8ee60ecc8c7 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Thu, 13 May 2021 14:50:01 +0200 Subject: [PATCH 21/44] Update RFCs/0002-pluggable-discovery.md Co-authored-by: Matthijs Kooijman --- RFCs/0002-pluggable-discovery.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index bb11fa7..8a374c4 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -492,7 +492,7 @@ Here another example: "eventType": "add", "port": { "address": "192.168.1.232", - "label": "SSH on my-board ()192.168.1.232)", + "label": "SSH on my-board (192.168.1.232)", "protocol": "ssh", "protocolLabel": "SSH Network port", "properties": { From 26de7ce222163a3a43264d15cb53710acb509a3b Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Thu, 13 May 2021 14:50:18 +0200 Subject: [PATCH 22/44] Update RFCs/0002-pluggable-discovery.md Co-authored-by: Matthijs Kooijman --- RFCs/0002-pluggable-discovery.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index 8a374c4..6f52e97 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -221,7 +221,7 @@ The `remove` event looks like the following: } ``` -the content is straightforward, in this case only the `address` and `protocol` fields are reported. +The content is straightforward, in this case only the `address` and `protocol` fields are reported. If the information about a port needs to be updated the discovery may send a new `add` message for the same port address and protocol without sending a `remove` first: this means that all the previous information about the port must be discarded and replaced with the new one. From 8affbd7739ad9a9545e94226902535d8b588e5bb Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Thu, 13 May 2021 14:50:35 +0200 Subject: [PATCH 23/44] Update RFCs/0002-pluggable-discovery.md Co-authored-by: Matthijs Kooijman --- RFCs/0002-pluggable-discovery.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index 6f52e97..039990e 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -463,10 +463,6 @@ For example, the following port metadata coming from a pluggable discovery: "vid": "0x2341", "serialNumber": "EBEABFD6514D32364E202020FF10181E", "name": "ttyACM0" - }, - "identificationPrefs": { - "pid": "0x804e", - "vid": "0x2341" } } } From a4a7e639baa3878cdabc9920799be0508ae3ade3 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Thu, 13 May 2021 14:53:53 +0200 Subject: [PATCH 24/44] fixed link --- RFCs/0002-pluggable-discovery.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index 039990e..7ca41f2 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -226,7 +226,7 @@ The content is straightforward, in this case only the `address` and `protocol` f If the information about a port needs to be updated the discovery may send a new `add` message for the same port address and protocol without sending a `remove` first: this means that all the previous information about the port must be discarded and replaced with the new one. A demo tool is available here: -[https://github.com/arduino/serial-discovery](https://github.com/arduino/serial-discovery#example-of-usage) +https://github.com/arduino/serial-discovery A typical usage scenario is here: https://github.com/arduino/serial-discovery#example-of-usage From ec88d16a2fa79c4b4cc936dbb68beda0b40d8218 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Thu, 13 May 2021 14:54:52 +0200 Subject: [PATCH 25/44] Update RFCs/0002-pluggable-discovery.md Co-authored-by: Matthijs Kooijman --- RFCs/0002-pluggable-discovery.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index 7ca41f2..d8afcf4 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -235,7 +235,7 @@ https://github.com/arduino/serial-discovery#example-of-usage #### Discovery tool install and launch -Discovery tools should be build natively for each OS and the CLI should run the correct tool for the running OS. This infrastracture is already available for platforms tools so the most natural way forward is to distribute the discoveries as tools within core platforms (in the same way we do for `gcc` or `avrdude`). +Discovery tools should be built natively for each OS and the CLI should run the correct tool for the running OS. This infrastracture is already available for platform tools so the most natural way forward is to distribute the discoveries as tools within core platforms (in the same way we do for `gcc` or `avrdude`). Some discovery like the `serial-discovery` must be always available, so they will be part of the `builtin` platform package and installed without the need to be part of a real platform (`builtin` is a dummy package that we use to install tools that are not part of any platforms like `ctags` for example). From e50a9dce050283637b34854c9b3685ab366effed Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Thu, 13 May 2021 15:29:54 +0200 Subject: [PATCH 26/44] Removed weird cuts markers --- RFCs/0002-pluggable-discovery.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index d8afcf4..6f1784d 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -302,7 +302,6 @@ Some port `properties` may not be precise enough to uniquely identify a board, i Let’s see an example to clarify things a bit, let's suppose that we have the following `properties` coming from the serial discovery: ```JSON -[...CUT...] "port": { "address": "/dev/ttyACM0", "properties": { @@ -311,7 +310,6 @@ Let’s see an example to clarify things a bit, let's suppose that we have the f "serialNumber": "EBEABFD6514D32364E202020FF10181E", "name": "ttyACM0" } -[...CUT...] ``` in this case we can use `vid` and `pid` to identify the board. The `serialNumber`, instead, is unique for that specific instance of the board so it can't be used to identify the board model. Let’s suppose we have the following `boards.txt`: From 7f8c1942bca5f8a6edc4ce72fc92f0aca0c684c2 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Thu, 13 May 2021 15:33:37 +0200 Subject: [PATCH 27/44] Update RFCs/0002-pluggable-discovery.md Co-authored-by: Matthijs Kooijman --- RFCs/0002-pluggable-discovery.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index 6f1784d..3d0ad8b 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -475,8 +475,8 @@ will be available on the recipe as the variables: {upload.port.vid} = 0x2341 {upload.port.serialNumber} = EBEABFD6514D32364E202020FF10181E {upload.port.name} = ttyACM0 -{serial.port} = ttyACM0 # for backward compatibility -{serial.port.file} = /dev/ttyACM0 # only because protocol=serial +{serial.port} = /dev/ttyACM0 # for backward compatibility +{serial.port.file} = ttyACM0 # only because protocol=serial ``` Here another example: From 69c3f9f9cb681fe18ac1c0e769dc361ed9d406eb Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Mon, 17 May 2021 11:56:49 +0200 Subject: [PATCH 28/44] Added a note about LIST timings --- RFCs/0002-pluggable-discovery.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index 3d0ad8b..47b197a 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -177,7 +177,9 @@ To make the above more clear let’s show an example with the `serial_discovery` In this case the serial port metadata comes from an USB serial converter. Inside the `properties` we have all the properties of the port, and some of them may be useful for product identification (in this case only USB VID/PID is useful to identify the board model). -The `LIST` command performs a one-shot polling of the ports. If you need continuous monitoring of ports you should use the `START_SYNC` command. +The `LIST` command performs a one-shot polling of the ports. The discovery should answer as soon as reasonably possible, without any additional delay. + +Some discoveries may require some time to discover a new port (for example network protocols like MDNS, Bluetooth, etc. requires some seconds to receive the broadcasts from all available clients) in that case is fine to answer with an empty or incomplete list. #### START_SYNC command From c591ebbbe8e1b5e1d709b852e3410bf53d8ab7ce Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Mon, 17 May 2021 12:02:25 +0200 Subject: [PATCH 29/44] Specify that backward compatibliity rules apply only for VID/PID --- RFCs/0002-pluggable-discovery.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index 47b197a..e5865ac 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -375,7 +375,7 @@ will match on both `pears=20, apples=30` and `pears=30, apples=40` but not `pear #### Backward compatibility considerations -Many platforms predating the pluggable discovery have the following definitions for board identification: +Many platforms predating the pluggable discovery have the following definitions for board identification using USB Serial VID/PID: ``` myboard.vid=0x1234 From 5149d928614d7895e501a7f7f95da7b1d83cdb03 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Mon, 17 May 2021 12:06:12 +0200 Subject: [PATCH 30/44] Specify that backward compatibliity rules apply even for indexed rules --- RFCs/0002-pluggable-discovery.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index e5865ac..60136f5 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -382,6 +382,13 @@ myboard.vid=0x1234 myboard.pid=0x4567 ``` +or: + +``` +myboard.vid.0=0x1234 +myboard.pid.0=0x4567 +``` + to ensure backward compatibility we will transparently and automatically convert these definitions into the new format ``` From cc44f9b202d3c89432b2c5129f774102d849395e Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Mon, 17 May 2021 12:35:45 +0200 Subject: [PATCH 31/44] Builtin discoveries are used by default, unless specified --- RFCs/0002-pluggable-discovery.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index 60136f5..a4fcead 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -239,10 +239,6 @@ https://github.com/arduino/serial-discovery#example-of-usage Discovery tools should be built natively for each OS and the CLI should run the correct tool for the running OS. This infrastracture is already available for platform tools so the most natural way forward is to distribute the discoveries as tools within core platforms (in the same way we do for `gcc` or `avrdude`). -Some discovery like the `serial-discovery` must be always available, so they will be part of the `builtin` platform package and installed without the need to be part of a real platform (`builtin` is a dummy package that we use to install tools that are not part of any platforms like `ctags` for example). - -The CLI will run `serial-discovery`, and other “builtin“ discoveries that we may want, by default. - 3rd party platforms may add other discoveries by providing them as tools dependencies for their platform and by adding a directive to their `platform.txt` that informs the CLI that a new discovery is available. ``` @@ -284,6 +280,19 @@ For example if a platform needs the `network` discovery from the Arduino AVR pla discovery.required=arduino:avr:network ``` +#### built-in discoveries and backward compatibliity consideration + +Some discoveries like the Arduino `serial-discovery` or the Arduino `network-discovery` must be always available, so they will be part of the `builtin` platform package and installed without the need to be part of a real platform (`builtin:builtin` is a dummy package that we use to install tools that are not part of any platforms like `ctags` for example). + +If a platform requires the builtin discoveries it must declare the usage with: + +``` +discovery.required.0=builtin:builtin:serial_discovery +discovery.required.1=builtin:builtin:network_discovery +``` + +For backward compatiblity, if a platform does not declare any discovery (using the `discovery.*` properties in `platform.txt`) it will automatically use all the builtin discoveries. This will allow all legacy platforms to softly migrate to pluggable discovery. + #### Duplicate discoveries It may happen that different 3rd party platforms provides the same discovery or different versions of the same discovery or, worse, different version of the same discovery launched with different parameters. From 875ce9749719df9eef2aacb43edc5e5232992422 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Mon, 17 May 2021 15:03:55 +0200 Subject: [PATCH 32/44] fix typo --- RFCs/0002-pluggable-discovery.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index a4fcead..38af3eb 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -291,7 +291,7 @@ discovery.required.0=builtin:builtin:serial_discovery discovery.required.1=builtin:builtin:network_discovery ``` -For backward compatiblity, if a platform does not declare any discovery (using the `discovery.*` properties in `platform.txt`) it will automatically use all the builtin discoveries. This will allow all legacy platforms to softly migrate to pluggable discovery. +For backward compatibility, if a platform does not declare any discovery (using the `discovery.*` properties in `platform.txt`) it will automatically use all the builtin discoveries. This will allow all legacy platforms to softly migrate to pluggable discovery. #### Duplicate discoveries From c3242ad1563712e14e6d6192a7a3ae3a6b2a54e1 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Mon, 17 May 2021 15:17:00 +0200 Subject: [PATCH 33/44] Transfer ALL port metadata into upload preferences --- RFCs/0002-pluggable-discovery.md | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index 38af3eb..0a49d2c 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -460,7 +460,7 @@ myboard.upload.tool.serial=bossac myboard.upload.tool.network=arduino_ota ``` -The selected port address will be provided in the variable `{upload.address}`. Other metadata provided by the discover in the `prefs` section will be provided in the `{upload.port.*}` variables. +The selected port address will be provided in the variable `{upload.port.address}`. In general, all the metadata provided by the discovery in the `port` section will be provided under the `{upload.port.*}` variables. For backward compatibility we will keep a copy of the address also in `{serial.port}` and in the specific case of a `protocol=serial` we will populate also `{serial.port.file}`. @@ -487,12 +487,14 @@ For example, the following port metadata coming from a pluggable discovery: will be available on the recipe as the variables: ``` -{upload.address} = /dev/ttyACM0 -{upload.protocol} = serial -{upload.port.pid} = 0x8043 -{upload.port.vid} = 0x2341 -{upload.port.serialNumber} = EBEABFD6514D32364E202020FF10181E -{upload.port.name} = ttyACM0 +{upload.port.address} = /dev/ttyACM0 +{upload.port.label} = ttyACM0 +{upload.port.protocol} = serial +{upload.port.protocolLabel} = Serial Port (USB) +{upload.port.properties.pid} = 0x8043 +{upload.port.properties.vid} = 0x2341 +{upload.port.properties.serialNumber} = EBEABFD6514D32364E202020FF10181E +{upload.port.properties.name} = ttyACM0 {serial.port} = /dev/ttyACM0 # for backward compatibility {serial.port.file} = ttyACM0 # only because protocol=serial ``` @@ -518,10 +520,12 @@ Here another example: that is translated to: ``` -{upload.address} = 192.168.1.232 -{upload.protocol} = ssh -{upload.port.macprefix} = AA:BB:CC -{upload.port.macaddress} = AA:BB:CC:DD:EE:FF +{upload.port.address} = 192.168.1.232 +{upload.port.label} = SSH on my-board (192.168.1.232) +{upload.port.protocol} = ssh +{upload.port.protocolLabel} = SSH Network port +{upload.port.properties.macprefix} = AA:BB:CC +{upload.port.properties.macaddress} = AA:BB:CC:DD:EE:FF {serial.port} = 192.168.1.232 # for backward compatibility ``` @@ -537,7 +541,7 @@ with: ``` tools.arduino_ota.upload.pattern="{runtime.tools.arduinoOTA.path}/bin/arduinoOTA" - -address {upload.address} -port 65280 + -address {upload.port.address} -port 65280 -sketch "{build.path}/{build.project_name}.bin" ``` @@ -559,7 +563,7 @@ tools.arduino_ota.upload.field.username=Username tools.arduino_ota.upload.field.password=Password tools.arduino_ota.upload.field.password.secret=true tools.arduino_ota.upload.pattern="{runtime.tools.arduinoOTA.path}/bin/arduinoOTA" - -address {upload.address} -port 65280 + -address {upload.port.address} -port 65280 -username "{upload.user.username} -password "{upload.user.password}" -sketch "{build.path}/{build.project_name}.bin" From 5bdfe30bc0b2d9f9e5dbe0f70eb5a7b6c2c686f0 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Mon, 24 May 2021 17:33:15 +0200 Subject: [PATCH 34/44] Added no-port protocol support --- RFCs/0002-pluggable-discovery.md | 59 ++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index 0a49d2c..ff3d2ba 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -569,6 +569,65 @@ tools.arduino_ota.upload.pattern="{runtime.tools.arduinoOTA.path}/bin/arduinoOTA -sketch "{build.path}/{build.project_name}.bin" ``` +#### Support for upload "without a port" (`no-port` dummy protocol) + +Some upload tools already have the port detection builtin so there is no need to specify an upload port (for example the `openocd` tool is often able to autodetect the devices to upload by itself). + +To support this particular use case a dummy `no-port` protocol has been reserved: + +``` +myboard.upload.tool.no-port=openocd_without_port +``` + +The `no-port` upload recipe will be selected when: + +- The upload port is not specified + +or + +- The upload port is specified but the protocol of the selected port doesn't match any of the available upload protocols for a board + +Let's see some examples to clarify: + +``` +board1.upload.tool.no-port=openocd_without_port + +board2.upload.tool.serial=bossac +board2.upload.tool.no-port=openocd_without_port + +board3.upload.tool.serial=bossac +``` + +In the `board1` case: the `openocd_without_port` recipe will be always used, even if a port has been selected by the user. + +In the `board2` case: the `bossac` recipe will be used if the port selected is a `serial` port, otherwise the `openocd_without_port` will be used in all other cases (even if a port has been selected by the user). + +In the `board3` case: the `bossac` recipe will be used if the port selected is a `serial` port, otherwise the upload will fail. + +A lot of existing platforms already have recipes without an explicit port address, in this case the upload tool specified in the old (non-pluggable) way will be considered as a `no-port` upload, for example let's consider the Arduino Zero board: + +``` +# Arduino Zero (Prorgamming Port) +# --------------------------------------- +arduino_zero_edbg.name=Arduino Zero (Programming Port) +arduino_zero_edbg.vid.0=0x03eb +arduino_zero_edbg.pid.0=0x2157 + +arduino_zero_edbg.upload.tool=openocd <--- CONSIDERED AS no-port PROT. +arduino_zero_edbg.upload.protocol=sam-ba +arduino_zero_edbg.upload.maximum_size=262144 +arduino_zero_edbg.upload.maximum_data_size=32768 +arduino_zero_edbg.upload.use_1200bps_touch=false +arduino_zero_edbg.upload.wait_for_upload_port=false +arduino_zero_edbg.upload.native_usb=false +``` + +in this case the upload definition will be always considered as a `no-port` by default, in other words, it will be automatically converted into: + +``` +arduino_zero_edbg.upload.no-port.tool=openocd +``` + ## Open Questions ### CLI command line UX considerations From db340b1ff1d643cdd17f4717549949f5d1ebb658 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Mon, 24 May 2021 18:03:54 +0200 Subject: [PATCH 35/44] Rename 'user' custom field to 'field' --- RFCs/0002-pluggable-discovery.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index ff3d2ba..c9d834d 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -547,7 +547,7 @@ tools.arduino_ota.upload.pattern="{runtime.tools.arduinoOTA.path}/bin/arduinoOTA #### User provided fields -The recipe may require some user-entered fields (like username or password). In this case the recipe must use the special placeholder `{upload.user.FIELD_NAME}` where `FIELD_NAME` must be declared separately in the recipe using the following format: +The recipe may require some user-entered fields (like username or password). In this case the recipe must use the special placeholder `{upload.field.FIELD_NAME}` where `FIELD_NAME` must be declared separately in the recipe using the following format: ``` tools.UPLOAD_RECIPE_ID.upload.field.FIELD_NAME=FIELD_LABEL @@ -564,8 +564,8 @@ tools.arduino_ota.upload.field.password=Password tools.arduino_ota.upload.field.password.secret=true tools.arduino_ota.upload.pattern="{runtime.tools.arduinoOTA.path}/bin/arduinoOTA" -address {upload.port.address} -port 65280 - -username "{upload.user.username} - -password "{upload.user.password}" + -username "{upload.field.username} + -password "{upload.field.password}" -sketch "{build.path}/{build.project_name}.bin" ``` From bc3963d75f6a64e1b00cdb7b604e29f8897f2895 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Wed, 26 May 2021 09:01:45 +0200 Subject: [PATCH 36/44] Added error handling --- RFCs/0002-pluggable-discovery.md | 79 ++++++++++++++++++++++++++++++-- 1 file changed, 75 insertions(+), 4 deletions(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index c9d834d..4d8d92b 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -48,9 +48,11 @@ All the commands listed in this specification must be implemented in the discove After startup, the tool will just stay idle waiting for commands. The available commands are: `HELLO`, `START`, `STOP`, `QUIT`, `LIST` and `START_SYNC`. +The discovery must not introduce any delay in the protocol and must respond to all commands as fast as possible. + #### HELLO command -`HELLO` is the **first command that must be sent** to the discovery to tell the name of the client/IDE and the version of the pluggable discovery protocol that the client/IDE supports. +`HELLO` **must be the first command sent** to the discovery to tell the name of the client/IDE and the version of the pluggable discovery protocol that the client/IDE supports. The syntax of the command is: `HELLO ""` @@ -92,6 +94,18 @@ The `START` command initializes and start the discovery internal subroutines. Th } ``` +If the discovery could not start, for any reason, it must report the error with: + +```JSON +{ + "eventType": "start", + "error": true, + "message": "Permission error" +} +``` + +The `error` field must be set to `true` and the `message` field should contain a description of the error. + #### STOP command The `STOP` command stops the discovery internal subroutines and possibly free the internally used resources. This command should be called if the client wants to pause the discovery for a while. The response to the command is: @@ -103,6 +117,18 @@ The `STOP` command stops the discovery internal subroutines and possibly free th } ``` +If an error occurs: + +```JSON +{ + "eventType": "stop", + "error": true, + "message": "Resource busy" +} +``` + +The `error` field must be set to `true` and the `message` field should contain a description of the error. + #### QUIT command The `QUIT` command terminates the discovery. The response to `QUIT` is: @@ -114,7 +140,7 @@ The `QUIT` command terminates the discovery. The response to `QUIT` is: } ``` -after this output the discovery exits. +after this output the discovery exits. This command is supposed to always succeed. #### LIST command @@ -181,11 +207,42 @@ The `LIST` command performs a one-shot polling of the ports. The discovery shoul Some discoveries may require some time to discover a new port (for example network protocols like MDNS, Bluetooth, etc. requires some seconds to receive the broadcasts from all available clients) in that case is fine to answer with an empty or incomplete list. +If an error occurs and the discovery can't complete the enumeration is must report the error with: + +```JSON +{ + "eventType": "list", + "error": true, + "message": "Resource busy" +} +``` + +The `error` field must be set to `true` and the `message` field should contain a description of the error. + #### START_SYNC command -The `START_SYNC` command puts the tool in "events" mode: the discovery will send `add` and `remove` events each time a new port is detected or removed respectively. +The `START_SYNC` command puts the tool in "events" mode: the discovery will send `add` and `remove` events each time a new port is detected or removed respectively. If the discovery goes into "events" mode successfully the response to this command is: + +```JSON +{ + "eventType": "start_sync", + "message": "OK" +} +``` -If the CLI is used in command-line mode (`arduino-cli board list` command) then we need a one-shot output and we can run the discoveries with the `LIST` command instead. Note that some discoveries may not be able to `LIST` ports immediately after the launch (in particular network protocols like MDNS, Bluetooth, etc. requires some seconds to receive the broadcasts from all available clients). +After this message the discoery will send `add` and `remove` event asyncronoushly (more on that later). If an error occurs and the discovery can't go in "events" mode the error must be reported as: + +```JSON +{ + "eventType": "start_sync", + "error": true, + "message": "Resource busy" +} +``` + +The `error` field must be set to `true` and the `message` field should contain a description of the error. + +If the CLI is used in command-line mode (for example the `arduino-cli board list` command) then we need a one-shot output and we can run the discoveries with the `LIST` command instead of using `START_SYNC`. Note that some discoveries may not be able to `LIST` ports immediately after the launch (in particular network protocols like MDNS, Bluetooth, etc. requires some seconds to receive the broadcasts from all available clients). If the CLI is running in daemon mode, ideally, all the installed discoveries should be started simultaneously in “event mode” (with `START_SYNC`) and the list of available ports should be cached inside the CLI daemon. @@ -227,6 +284,20 @@ The content is straightforward, in this case only the `address` and `protocol` f If the information about a port needs to be updated the discovery may send a new `add` message for the same port address and protocol without sending a `remove` first: this means that all the previous information about the port must be discarded and replaced with the new one. +#### Invalid commands + +If the client sends an invalid or malformed command, the discovery should answer with: + +```JSON +{ + "eventType": "command_error", + "error": true, + "message": "Unknown command XXXX" +} +``` + +#### Reference implementations + A demo tool is available here: https://github.com/arduino/serial-discovery From c6cdecc86c80bbae1bcba5795460acc4204b4569 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Wed, 26 May 2021 09:10:09 +0200 Subject: [PATCH 37/44] Moved paragraph and cleared arduino-cli usage scenario --- RFCs/0002-pluggable-discovery.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index 4d8d92b..225a087 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -242,10 +242,6 @@ After this message the discoery will send `add` and `remove` event asyncronoushl The `error` field must be set to `true` and the `message` field should contain a description of the error. -If the CLI is used in command-line mode (for example the `arduino-cli board list` command) then we need a one-shot output and we can run the discoveries with the `LIST` command instead of using `START_SYNC`. Note that some discoveries may not be able to `LIST` ports immediately after the launch (in particular network protocols like MDNS, Bluetooth, etc. requires some seconds to receive the broadcasts from all available clients). - -If the CLI is running in daemon mode, ideally, all the installed discoveries should be started simultaneously in “event mode” (with `START_SYNC`) and the list of available ports should be cached inside the CLI daemon. - The `add` event looks like the following: ```JSON @@ -296,6 +292,14 @@ If the client sends an invalid or malformed command, the discovery should answer } ``` +#### arduino-cli usage scenario + +If `arduino-cli` is used in command-line mode (for example the `arduino-cli board list` command) then we need a one-shot output and we can run the discoveries with the `LIST` command instead of using `START_SYNC`. + +Note that some discoveries may not be able to `LIST` ports immediately after the launch (in particular network protocols like MDNS, Bluetooth, etc. requires some seconds, even minutes, to receive the broadcasts from all available clients). This is intrinsic in how the `LIST` command is defined and it's a duty of the client (CLI/IDE) to take it into consideration. + +If `arduino-cli` is running in daemon mode, ideally, it should start all the installed discoveries simultaneously in “event mode” (with `START_SYNC`) and the list of available ports should be cached and updated in realtime inside the `arduino-cli` daemon. + #### Reference implementations A demo tool is available here: From e756a44a51df3df9ee69e4535cd4a28b670de16e Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Wed, 26 May 2021 14:59:43 +0200 Subject: [PATCH 38/44] Added more details about message sequencing --- RFCs/0002-pluggable-discovery.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index 225a087..4646b26 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -48,7 +48,7 @@ All the commands listed in this specification must be implemented in the discove After startup, the tool will just stay idle waiting for commands. The available commands are: `HELLO`, `START`, `STOP`, `QUIT`, `LIST` and `START_SYNC`. -The discovery must not introduce any delay in the protocol and must respond to all commands as fast as possible. +After each command the client always expect a response from the discovery. The discovery must not introduce any delay and must respond to all commands as fast as possible. #### HELLO command @@ -242,6 +242,8 @@ After this message the discoery will send `add` and `remove` event asyncronoushl The `error` field must be set to `true` and the `message` field should contain a description of the error. +Once in "event" mode, the discovery is allowed to send `add` and `remove` messages asynchronously in realtime, this means that the client must be able to handle these incoming messages at any moment. + The `add` event looks like the following: ```JSON From 1c082fd45fb429ee3c355cd19819bcd4cfe45668 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Thu, 27 May 2021 17:53:47 +0200 Subject: [PATCH 39/44] Discovery install via package_index.json --- RFCs/0002-pluggable-discovery.md | 188 +++++++++++++++++++++++++------ 1 file changed, 155 insertions(+), 33 deletions(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index 4646b26..d9df510 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -310,73 +310,195 @@ https://github.com/arduino/serial-discovery A typical usage scenario is here: https://github.com/arduino/serial-discovery#example-of-usage -### Integration with CLI and core platforms +### Integration with `arduino-cli` and core platforms -#### Discovery tool install and launch +In this section we will see how discvoeries are distributed and integrated with Arduino platforms. -Discovery tools should be built natively for each OS and the CLI should run the correct tool for the running OS. This infrastracture is already available for platform tools so the most natural way forward is to distribute the discoveries as tools within core platforms (in the same way we do for `gcc` or `avrdude`). +#### Discovery tool distribution -3rd party platforms may add other discoveries by providing them as tools dependencies for their platform and by adding a directive to their `platform.txt` that informs the CLI that a new discovery is available. +The discovery tools must be built natively for each OS and the CLI should run the correct tool for the running OS. -``` -discovery.DISCOVERY_ID.pattern=DISCOVERY_RECIPE -``` +The distribution infrastracture is already available for platform tools, like compilers and uploaders, through `package_index.json` so, the most natural way forward is to distribute also the discoveries in the same way. +3rd party developers should provide their discovery tools by adding them as resources in the `tools` section of `package_index.json` (at the `packages` level). -The CLI will look for directives matching the above pattern. `DISCOVERY_ID` must be replaced by a unique identifier for the particular discovery and `DISCOVERY_RECIPE` must be replaced by the command line to run to launch the discovery. An example could be: +Let's see how this looks into a `package_index.json` example: -``` -## Teensy Ports Discovery -discovery.teensy.pattern="{runtime.tools.teensy_ports.path}/hardware/tools/teensy_ports" -J2 +```diff +{ + "packages": [ + { + "name": "arduino", + "maintainer": "Arduino", + "websiteURL": "http://www.arduino.cc/", + + "platforms": [ + ... + ], + "tools": [ + { + "name": "arm-none-eabi-gcc", + "version": "4.8.3-2014q1", + "systems": [ ... ] + }, ++ { ++ "name": "ble-discovery", <--- Discovery is distributed as a TOOL ++ "version": "1.0.0", ++ "systems": [ ++ { ++ "host": "x86_64-pc-linux-gnu", ++ "url": "http://example.com/ble-disc-1.0.0-linux64.tar.gz", ++ "archiveFileName": "ble-disc-1.0.0-linux64.tar.gz", ++ "checksum": +SHA-256:0123456789abcdef0123456789abcdef0123456789abcdef", ++ "size": "12345678" ++ }, ++ ... ++ ] ++ } + ], + } + } +} ``` -in this case the platform provides a new `teensy` discovery and the command line tool named `teensy_ports` is launched with the `-J2` parameter to start the discovery tool. +In this case we are adding an hypotetical `ble-discovery` version `1.0.0` to the toolset of the vendor `arduino`. From now on, we can uniquely refer to this discovery with the pair `PACKAGER` and `DISCOVERY_NAME`, in this case `arduino` and `ble-discovery` respectively. -#### Using a discovery made by a 3rd party +The compressed archive of the discovery must contain only a single executable file (the discovery itself) inside a single root folder. This is mandatory since the CLI will run this file automatically when the discovery is started. + +#### Discovery tools integration + +Each core platform must refer to the specific discovery tools they need by adding them a new `discoveryDependencies` field of the `package_index.json`. Let's look again at the previous example: + +```diff +{ + "packages": [ + { + "name": "arduino", + "maintainer": "Arduino", + "websiteURL": "http://www.arduino.cc/", + + "platforms": [ + { + "name": "Arduino AVR Boards", + "architecture": "avr", + "version": "1.6.2", + ... + "toolsDependencies": [ + { + "packager": "arduino", + "name": "arm-none-eabi-gcc", + "version": "4.8.3-2014q1" + }, + { + "packager": "arduino", + "name": "CMSIS", + "version": "4.5.0" + }, + ... + ], ++ "discoveryDependencies": [ <--- Discoveries used in the platform ++ { ++ "packager": "arduino", ++ "name": "ble-discovery" ++ <--- Version is not required! ++ } ++ ] + }, + + { + "name": "Arduino SAMD Boards", + "architecture": "samd", + "version": "1.6.18", + ... + "toolsDependencies": [ ... ], ++ "discoveryDependencies": [ ... ] + } + + ], + "tools": [ + { + "name": "arm-none-eabi-gcc", + "version": "4.8.3-2014q1", + "systems": [ ... ] + }, + { + "name": "ble-discovery", + "version": "1.0.0", + "systems": [ ... ] + } + ], + } + } +} +``` -A platform may opt to depend on a discovery developed by a 3rd party instead of writing and maintaining his own. Since writing a good-quality cross-platform discovery is very hard and time consuming, we expect this option to be used by the majority of developers. +Adding the needed discoveries in the `discoveryDependencies` allows the CLI to install them together with the platform. Also, differently from the other `toolsDependencies`, the version is not required since it will always be used the latest version available. -To depend on a 3rd party discovery the platform must add the following directive in the `platform.txt` file: +Finally, to actually bind a discovery to a platform, we must also declare in the `platform.txt` that we want to use that specific discovery with the direcive: ``` -discovery.required=PLATFORM:ARCHITECTURE:DISCOVERY_ID +discovery.required=PLATFORM:DISCOVERY_NAME ``` -or if the platform needs more discoveries: +or if the platform needs more discoveries we can use the indexed version: ``` -discovery.required.0=PLATFORM:ARCHITECTURE:DISCOVERY_ID_1 -discovery.required.1=PLATFORM:ARCHITECTURE:DISCOVERY_ID_2 +discovery.required.0=PLATFORM:DISCOVERY_ID_1 +discovery.required.1=PLATFORM:DISCOVERY_ID_2 ... ``` -The `PACKAGER:ARCHITECTURE:DISCOVERY_ID` field represents a unique identifier to a 3rd party discovery in particular the `PACKAGER:ARCHITECTURE:...` part is the same as in the FQBN for the boards. - -For example if a platform needs the `network` discovery from the Arduino AVR platform it may specify it with: +in our specific example the directive should be: ``` -discovery.required=arduino:avr:network +discovery.required=arduino:ble-discovery ``` -#### built-in discoveries and backward compatibliity consideration +#### Using a discovery made by a 3rd party + +A platform developer may opt to depend on a discovery developed by a 3rd party instead of writing and maintaining his own. + +Since writing a good-quality cross-platform discovery is very hard and time consuming, we expect this option to be the one used by the majority of the developers. -Some discoveries like the Arduino `serial-discovery` or the Arduino `network-discovery` must be always available, so they will be part of the `builtin` platform package and installed without the need to be part of a real platform (`builtin:builtin` is a dummy package that we use to install tools that are not part of any platforms like `ctags` for example). +#### Direct discovery integration (not recommended) + +A discovery may be directly added to a platform, without passing through the `discoveryDependencies` in `package_index.json`, using the following directive in the `platform.txt`: + +``` +discovery.DISCOVERY_ID.pattern=DISCOVERY_RECIPE +``` -If a platform requires the builtin discoveries it must declare the usage with: +`DISCOVERY_ID` must be replaced by a unique identifier for the particular discovery and `DISCOVERY_RECIPE` must be replaced by the command line to launch the discovery. An example could be: ``` -discovery.required.0=builtin:builtin:serial_discovery -discovery.required.1=builtin:builtin:network_discovery +## Teensy Ports Discovery +discovery.teensy.pattern="{runtime.tools.teensy_ports.path}/hardware/tools/teensy_ports" -J2 ``` -For backward compatibility, if a platform does not declare any discovery (using the `discovery.*` properties in `platform.txt`) it will automatically use all the builtin discoveries. This will allow all legacy platforms to softly migrate to pluggable discovery. +in this case the platform provides a new `teensy` discovery and the command line tool named `teensy_ports` is launched with the `-J2` parameter to start the discovery tool. In this case the command line pattern may contain any extra parameter in the formula: this is different from the discoveries installed through the `discoveryDependencies` field that are run automatically without any command line parameter. + +This kind of integration may turn out useful: + +- during the development of a platform (because providing a full `package_index.json` may be cumbersome) +- if the discovery is specific for a platform and can not be used by 3rd party + +Anyway, since this kind of integration does not allow reusing a discovery between different platforms, we do not recommend its use. -#### Duplicate discoveries +#### built-in discoveries and backward compatibliity considerations + +Some discoveries like the Arduino `serial-discovery` or the Arduino `network-discovery` must be always available, so they will be part of the `builtin` package and installed without the need to be part of a real package (`builtin` is a dummy package that we use to install tools that are not part of any platforms like `ctags` for example). + +If a platform requires the builtin discoveries it must declare it with: + +``` +discovery.required.0=builtin:serial_discovery +discovery.required.1=builtin:network_discovery +``` -It may happen that different 3rd party platforms provides the same discovery or different versions of the same discovery or, worse, different version of the same discovery launched with different parameters. +For backward compatibility, if a platform does not declare any discovery (using the `discovery.*` properties in `platform.txt`) it will automatically inherits all the builtin discoveries. This will allow all legacy non-pluggable platforms to migrate to pluggable discovery without disruption. -We can partially handle this if the `DISCOVERY_ID` field in `platform.txt` is well defined: from the CLI we could group together the platforms that requires the same discovery and launch the latest version available just once. How the different 3rd party will agree on the `DISCOVERY_ID` value population is TBD. +#### Conflicting discoveries -In case different discoveries provide conflicting information (for example if two discoveries provide different information for the same port address) we could partially mitigate the issue by giving priority to the discovery that is used by the package of the selected board. +In case different discoveries provide conflicting information (for example if two discoveries provide different information for the same port) we could partially mitigate the issue by giving priority to the discovery that is used by the package of the selected board. #### Board identification From 6ed74a2090fabd386ef373dbaec985db1f1e2d0d Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Thu, 27 May 2021 17:54:04 +0200 Subject: [PATCH 40/44] Give to 'Board Identification' its own chapter --- RFCs/0002-pluggable-discovery.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index d9df510..29058c0 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -500,7 +500,7 @@ For backward compatibility, if a platform does not declare any discovery (using In case different discoveries provide conflicting information (for example if two discoveries provide different information for the same port) we could partially mitigate the issue by giving priority to the discovery that is used by the package of the selected board. -#### Board identification +### Board identification The `properties` associated to a port can be used to identify the board attached to that port. The algorithm is simple: From 2e078d4779576651ede61bc5f7ebef2db36cd0d1 Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Mon, 31 May 2021 12:11:01 +0200 Subject: [PATCH 41/44] Only serial-discovery and network-discovery are automatically added to legacy platforms --- RFCs/0002-pluggable-discovery.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index 29058c0..928ea54 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -490,11 +490,11 @@ Some discoveries like the Arduino `serial-discovery` or the Arduino `network-dis If a platform requires the builtin discoveries it must declare it with: ``` -discovery.required.0=builtin:serial_discovery -discovery.required.1=builtin:network_discovery +discovery.required.0=builtin:serial-discovery +discovery.required.1=builtin:network-discovery ``` -For backward compatibility, if a platform does not declare any discovery (using the `discovery.*` properties in `platform.txt`) it will automatically inherits all the builtin discoveries. This will allow all legacy non-pluggable platforms to migrate to pluggable discovery without disruption. +For backward compatibility, if a platform does not declare any discovery (using the `discovery.*` properties in `platform.txt`) it will automatically inherits `builtin:serial-discovery` and `builtin:network-discovery` (but not other `builtin` discoveries that may be possibly added in the future). This will allow all legacy non-pluggable platforms to migrate to pluggable discovery without disruption. #### Conflicting discoveries From 188510293ca2790f5f02bf4df608c2a221e8fdcd Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Mon, 31 May 2021 12:28:30 +0200 Subject: [PATCH 42/44] Renamed 'no-port' protocol to 'default' --- RFCs/0002-pluggable-discovery.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index 928ea54..9d2f627 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -768,17 +768,17 @@ tools.arduino_ota.upload.pattern="{runtime.tools.arduinoOTA.path}/bin/arduinoOTA -sketch "{build.path}/{build.project_name}.bin" ``` -#### Support for upload "without a port" (`no-port` dummy protocol) +#### Support for `default` fallback (allows upload "without a port") Some upload tools already have the port detection builtin so there is no need to specify an upload port (for example the `openocd` tool is often able to autodetect the devices to upload by itself). -To support this particular use case a dummy `no-port` protocol has been reserved: +To support this particular use case a dummy `default` protocol has been reserved: ``` -myboard.upload.tool.no-port=openocd_without_port +myboard.upload.tool.default=openocd_without_port ``` -The `no-port` upload recipe will be selected when: +The `default` upload protocol is a kind of wildcard-protocol, it will be selected when: - The upload port is not specified @@ -789,21 +789,21 @@ or Let's see some examples to clarify: ``` -board1.upload.tool.no-port=openocd_without_port +board1.upload.tool.default=openocd_without_port board2.upload.tool.serial=bossac -board2.upload.tool.no-port=openocd_without_port +board2.upload.tool.default=openocd_without_port board3.upload.tool.serial=bossac ``` -In the `board1` case: the `openocd_without_port` recipe will be always used, even if a port has been selected by the user. +In the `board1` case: the `openocd_without_port` recipe will be always used, whatever the user's port selection. -In the `board2` case: the `bossac` recipe will be used if the port selected is a `serial` port, otherwise the `openocd_without_port` will be used in all other cases (even if a port has been selected by the user). +In the `board2` case: the `bossac` recipe will be used if the port selected is a `serial` port, otherwise the `openocd_without_port` will be used in all other cases (whatever the user's port selection). In the `board3` case: the `bossac` recipe will be used if the port selected is a `serial` port, otherwise the upload will fail. -A lot of existing platforms already have recipes without an explicit port address, in this case the upload tool specified in the old (non-pluggable) way will be considered as a `no-port` upload, for example let's consider the Arduino Zero board: +A lot of existing platforms already have recipes without an explicit port address, in this case the upload tool specified in the old (non-pluggable) way will be considered as a `default` protocol upload, for example let's consider the Arduino Zero board: ``` # Arduino Zero (Prorgamming Port) @@ -812,7 +812,7 @@ arduino_zero_edbg.name=Arduino Zero (Programming Port) arduino_zero_edbg.vid.0=0x03eb arduino_zero_edbg.pid.0=0x2157 -arduino_zero_edbg.upload.tool=openocd <--- CONSIDERED AS no-port PROT. +arduino_zero_edbg.upload.tool=openocd <--- CONSIDERED AS default PROTOCOL arduino_zero_edbg.upload.protocol=sam-ba arduino_zero_edbg.upload.maximum_size=262144 arduino_zero_edbg.upload.maximum_data_size=32768 @@ -821,10 +821,10 @@ arduino_zero_edbg.upload.wait_for_upload_port=false arduino_zero_edbg.upload.native_usb=false ``` -in this case the upload definition will be always considered as a `no-port` by default, in other words, it will be automatically converted into: +in this case the upload definition will be always considered as a `default` protocol, in other words, it will be automatically converted into: ``` -arduino_zero_edbg.upload.no-port.tool=openocd +arduino_zero_edbg.upload.default.tool=openocd ``` ## Open Questions From 68300300231f6f5cb1c5416fe78fb1fd6aa7e70d Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Mon, 31 May 2021 12:37:44 +0200 Subject: [PATCH 43/44] Made backward compatibility mode more explicit for default upload --- RFCs/0002-pluggable-discovery.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index 9d2f627..e0bd0e6 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -803,7 +803,7 @@ In the `board2` case: the `bossac` recipe will be used if the port selected is a In the `board3` case: the `bossac` recipe will be used if the port selected is a `serial` port, otherwise the upload will fail. -A lot of existing platforms already have recipes without an explicit port address, in this case the upload tool specified in the old (non-pluggable) way will be considered as a `default` protocol upload, for example let's consider the Arduino Zero board: +A lot of legacy platforms already have recipes without an explicit port address, for example let's consider the Arduino Zero board: ``` # Arduino Zero (Prorgamming Port) @@ -812,7 +812,7 @@ arduino_zero_edbg.name=Arduino Zero (Programming Port) arduino_zero_edbg.vid.0=0x03eb arduino_zero_edbg.pid.0=0x2157 -arduino_zero_edbg.upload.tool=openocd <--- CONSIDERED AS default PROTOCOL +arduino_zero_edbg.upload.tool=openocd <--- arduino_zero_edbg.upload.protocol=sam-ba arduino_zero_edbg.upload.maximum_size=262144 arduino_zero_edbg.upload.maximum_data_size=32768 @@ -821,12 +821,14 @@ arduino_zero_edbg.upload.wait_for_upload_port=false arduino_zero_edbg.upload.native_usb=false ``` -in this case the upload definition will be always considered as a `default` protocol, in other words, it will be automatically converted into: +in this case, to ensure backward compatibility, the upload tool specified in the old (non-pluggable) way will be considered as a `default` protocol upload, and it will be automatically converted into: ``` arduino_zero_edbg.upload.default.tool=openocd ``` +Please note that the transformation above is intended only as a backward compatibility helper and it will be applied only on platforms that does not support Pluggable Discovery at all: if any other `*.upload.*.tool` or `discovery.*` definition is found in the platform, the transformation above will **not** be automatically applied. + ## Open Questions ### CLI command line UX considerations From ef193317ac8e9a4d4102e0887459fcb668420fdd Mon Sep 17 00:00:00 2001 From: Cristian Maglie Date: Mon, 21 Jun 2021 16:17:16 +0200 Subject: [PATCH 44/44] Changed field to integer --- RFCs/0002-pluggable-discovery.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RFCs/0002-pluggable-discovery.md b/RFCs/0002-pluggable-discovery.md index e0bd0e6..1f16327 100644 --- a/RFCs/0002-pluggable-discovery.md +++ b/RFCs/0002-pluggable-discovery.md @@ -72,7 +72,7 @@ the response to the command is: ```JSON { "eventType": "hello", - "protocolVersion": "1", + "protocolVersion": 1, "message": "OK" } ```