Library manager: Library dependencies

[Avamander]

Is there a possibility that in the future versions of Arduino IDE library could define it's dependencies thus making sure it will work.

Something like this:

1. Library has a line something similar to requires=<other library's name>:<optional minimum version> and/or requiresIDE=<version> in it's library.properties
2. User presses download but it does not have a required library installed, there is one installed but without library.properties or the library is outdated (with requiresIDE the IDE's version incompatibility is also warned about)
3. User is prompted to download or update all the required libraries

This would definetly for example improve how people use libraries like RF24, RF24Network and RF24Mesh (I am sure there are more) making sure both the library and it's dependencies are up-to-date when any of those are downloaded.

[carlosperate]

It would also be quite useful if a library could have an Arduino core dependency (ideally a list with all compatible Arduino cores), as some libraries might be processor-specific.

[s-celles]

Coming from #7567 (comment)

What about using a dependency manager?

There's currently:

name	url	repository
Buckaroo	http://www.buckaroo.pm/	https://github.com/LoopPerfect/buckaroo
Conan	https://conan.io/	https://github.com/conan-io/conan
Hunter		https://github.com/ruslo/hunter
cpm	http://www.cpm.rocks/	https://github.com/iauns/cpm
cget	https://cget.readthedocs.io/	https://github.com/pfultz2/cget
sw (previously known as cppan)	https://software-network.org/	https://github.com/SoftwareNetwork/sw
vcpkg		https://github.com/Microsoft/vcpkg
conda	https://conda.io/	https://github.com/conda/conda
build2	https://build2.org	https://github.com/build2/build2

and probably more.

They are more or less coupled to a build system.

See also
https://www.reddit.com/r/cpp/comments/8t0ufu/what_is_a_good_package_manager_for_c/
http://pfultz2.com/blog/2017/10/27/universal-package-manager/

[s-celles]

Library has a line something similar to requires=<other library's name>: and/or requiresIDE= in it's library.properties

@Avamander specifying optionally minimum version may not be enough because if API of a library have some breaking changes you will also need to restrict library version with a maximum version to use latest previous major version

We could need to specify for example:

ArduinoJson<6.0.0,>=5.0.0

But I'm wondering if all registered Arduino libraries uses semantic versioning. Because we need to be able to order version numbering without error / uncertainty.

Related arduino/arduino-cli#105

[per1234]

I'm wondering if all registered Arduino libraries uses semantic versioning.

They don't. There is a required version format, which includes semver, but also allows non-semver compliant version numbers.

From https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification#libraryproperties-file-format

version - version of the library. Version should be semver compliant. 1.2.0 is correct; 1.2 is accepted; r5, 003, 1.1c are invalid

I think this is a programmatic definition of the requirement:
https://github.com/bugst/relaxed-semver

[njlr]

Hi! 👋

One of the authors of Buckaroo here.

I had a look at the library.json format and I think it will be feasible to add native support for this in Buckaroo. From the looks of things, these libraries will build very cleanly! 👍

However, I could not see any dependency information in the library files that I found.

Is the dependency information encoded anywhere?

I am looking for something like:

This library depends on package X, which can be found at github.com/a/x

[s-celles]

Thanks @njlr for coming here to help us.

I wonder if this kind of information shouldn't be available (first) in library.properties (or somewhere else). It's currently not the case.

I think this file should contain dependencies for compiling a sketch ie you need for example LibraryXYZ latest registered version.

But there is differents cases to manage.

Is LibraryXYZ a "registered" library or not?

If it's a registered library library_index.json could (partially) help (to know where to download it)

Version number restriction could also be required for installing some libraries (because sometimes, installing latest version of a library doesn't work because API of a library changed)

Being able to install an (unregistered) library from a Git repository at a given commit, or a given branch should also be considered.

I opened some days ago issues to manage this with arduino-cli

• Installing unregistered library arduino-cli#128
• Version restriction for installing library arduino-cli#105

but the main problem here is to first define new data file to define dependencies.

Is adding new fields to library.properties a good idea? should we have a separate file (a YAML or a JSON file) to manage only dependencies?

I don't really know where the Arduino team want to go with this but from my perspective I will use a separate human readable file for this (and I think that YAML file is great for this).

We should be aware that some examples of a library could need more dependencies that a library itself so we should have a mechanism to override this config file for a given directory. (this problem is quite comparable to https://github.com/ianfixes/arduino_ci/issues/109 )

[njlr]

Hi @scls19fr thanks for the response.

If it's a registered library library.json could (partially) help (to know where to download it)

By "registered" I assume you mean registered on https://www.arduinolibraries.info?

Being able to install an (unregistered) library from a Git repository at a given commit, or a given branch should also be considered.

Buckaroo already works directly on GitHub (or any Git) repositories, so there are no issues there 👍

but the main problem here is to first define new data file to define dependencies.

Is adding new fields to library.properties a good idea? should we have a separate file (a YAML or a JSON file) to manage only dependencies?

Buckaroo has a TOML file (similar benefits to YAML) to define this.

Here is what it might look like:

[[dependency]]
package = "github.com/bblanchon/ArduinoJson"
version = "6.8.0"

[[dependency]]
package = "github.com/arduino-libraries/ArduinoHttpClient"
version = "revision=b6424e430db57fa0c67f82333e80d58b1521a97b"

Buckaroo supports sem-ver ranges, branches, tags and even commits.

It will examine each dependency it finds and recursively fetch its dependencies, assuming there is a TOML file.

The big question is how to integrate this with the Arduino workflow. Currently, Buckaroo puts dependencies into ./buckaroo and wires them together using Buck (Buck is a build system similar to Make, but more powerful). We could add Arduino as an additional back-end option.

I saw in the IDE that packages are put into ./libraries. However, using ino at the command-line, libraries seem to go inside ./lib.

• Do you know how most users are building their Arduino projects?

• Can you point to any large open-source Arduino projects that I can examine?

[s-celles]

By registered I mean that is available in library_index.json (sorry about my mistake of library.json ... library.json is a PlatformIO config file)

ArduinoJson https://github.com/bblanchon/ArduinoJson is a quite large open source project

[njlr]

So library.properties would be the Arduino package file?

For example: https://github.com/bblanchon/ArduinoJson/blob/0685a36f0e51ba71b96c441697e0432ed2a6b176/library.properties

I see that the file does not contain any header or source-file lists. Does this mean that there is a convention about the layout of the files?

[s-celles]

This article https://www.arduino.cc/en/guide/libraries describes how libraries are installed and this one deals with creating Arduino lib https://www.arduino.cc/en/hacking/libraries

You can also have a look at some wiki articles such as https://github.com/arduino/Arduino/wiki/Library-Manager-FAQ

Code is in a src directory
Examples are in an examples directory and are automatically shown in IDE
Unit tests can be in a test directory (when using arduino_ci but that's a quite new project currently not widely used).

When a user wants to build a project he must ensure (manually) that all libraries are installed correctly (and with the correct version). I personally think the current situation can be problematic as the concept of environment doesn't exist arduino/arduino-cli#108 (but that's just my own opinion).

[sandeepmistry]

This was added in #8600.

[ladyada]

yaaaaaaaaaaaay

[s-celles]

Good news but I'm wondering if dependencies version are taken into account.

I only see

      "requires": [
        { "name": "WiFi101" },
        { "name": "ArduinoSound" }
      ],

nothing about a possible version restriction of a library (see arduino/arduino-cli#105 )

[cmaglie]

Good news but I'm wondering if dependencies version are taken into account.

This is being implemented, the general dependency-resolution (with version restrictions) requires a much more complex backtracking algorithm, for now we decided to deploy the "light" version of lib dependecy (that should cover 99% of the cases anyway).

[ArminJo]

Great work,
a feature I waited for so long...

What must I do to describe the library dependencies in my library.properties file?
I cannot find any reference in https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification