What is the logic for overwriting parameters included in a property?

[randaz81]

Well this is a bit complicated. I investigated a little the problem and this is my report on the current status.

This is the scenario: we have a .ini file that includes another .ini file.
The included .ini file, is, let's say, a generic configuration file which describes some properties.
The parent .ini file, instead, wants to overwrite some parameters for a specific scenario/robot etc.
What will happen?

First fact:

The parameters and the sections at the uppermost level are internally rearranged and ordered alphabetically.
This means that a file that contains the parameters:

param2 10
param1 5
param3 4

[aaaa]
param4 3

Is internally stored as:

(aaaa (param4 3)) (param1 5) (param2 10) (param 3)

Why this?
Well, looking at the class class Property::Private in Property.cpp we have that data is sorted as a key-value map.
std::map<std::string, PropertyItem> data;
The rationale for using the std::map container is to retrieve data quickly (I suppose)

Changing this behavior to have data ordered by insertion is not straightforward.
If we use, for example, the std::unordered_map container, data will be neither order alphabetically nor by insertion.
Indeed they will be ordered following some obscure logic of the container.
Googling the problem, I found:

• this link that describes exactly our problem: https://stackoverflow.com/questions/1098175/a-stdmap-that-keep-track-of-the-order-of-insertion
• This solution: a custom container which stores data by insertion, but uses and hash function for fast data retrieval, as in the map. https://github.com/Tessil/ordered-map

But.. let's ignore for now this fact. The real problems are coming below.

Second fact:

The parameters located in the subsection are ordered by insertion.
This means that a file that contains the parameters:

[mysection]
param2 10
param1 5
param3 4

Is internally stored as: (mysection (param2 10) (param1 5) (param3 4))
This is fine, it is the behavior that we were expecting
This happens because sections are internally stored as bottles:

class PropertyItem
{
public:
    Bottle bot;

So far, everything is fine...

Third fact:

i.e. how parameters are overwritten if the included file contains some parameters which have the same name of those contained in the parent file.
Let's first consider the following example, in which first we define a parameter, then we import a file:
main.ini

param1 10
[import context1 included.ini]

included.ini

param1 5

In this case the final value of param1 is 5: the parameter has been succesfully overwritten.

Now, let's consider the opposite case in which we first include a file, then we overrite the parameter:
main.ini

[import context1 included.ini]
param1 10

included.ini

param1 5

In this case the final value of param1 is 10: Also in this case everything is ok. The last parameter wins, as expected.

However, we already know that parameters inside sections are handled in a different way respect to the parameteres belonging to the upper most level.
Let's consider the example:
main.ini

[section foo]
param1 1

[import context1 included.ini]

included.ini

[section foo]
param1 1000000
param2 5

The resulting prop.toString() will containing the following data:

(foo (param1 1) (param1 100000) (param2 5))

so the second param1 is appended at the end of the bottle and it does not overrite the previous value of param1.
Thus, if we use:

p.findGroup("foo").find("param1").asInt32();

The bottle is scanned from left to right, resulting in param=1 (the value of the parameter has not been overwritten)

Conslusions:

The current implementation does not really allows to overrwrite parameters, but a workaround is possible.
If the parent file is somehow a general configuration file, while the included file is a more specific one, we may include it before the parameter we want to overwrite.
The last example can be thus re-arranged as follows:
main.ini

[import context1 included.ini]

[section foo]
param1 1

included.ini

[section foo]
param1 1000000
param2 5

Of course this is not very intuitive, since we generally expect that the last parameter overwrites all the others.
Additionally ,the workaround just mentioned is just an example, and it is not recommended, because it may not work anymore if in the future we fix this issue.
To obtain the correct behavior, something needs to be changed in the yarp::os::Property implementation.

[PeterBowman]

This behavior is also inconsistent with command line arguments. I reported a similar issue at: #1889.

[drdanz]

Thanks for the detailed explaination.

Changing the internal logic in Property and ResourceFinder is not going to be easy.
First of all by changing anything it will be very difficult to know if we are breaking something in users configuration files.
Moreover the whole architecture is messed up (.ini file parsing is inside Property, the ResourceFinder does not just "find resources", but also reads files and command line...).

I believe that this part should be completely reconsidered and refactored. In my opinion, it does not makes too much sense to have all of this in YARP, one option could be to use some external tool for this, for example Elektra.