[ For discussion / Feature Request ] Dump Inventory back to its native representation via Inventory Plugins

[matman26]

Please do not close issues before giving the one posting the opportunity to reply, as Github does not allow non-contributors to reopen issues closed by contributors (see here). I'm also pretty sure that contributors are not notified by messages on closed issues so there's the possibility discussion threads will go unanswered; this also leads to issues getting duplicated.

Anyway, this refers back to #764 with clarifications/corrections to my last reply. My idea was to have Inventory plugins be "read-write" in the sense that Nornir could use them to serialize the current runtime inventory back to its original format. Let's suppose I begin a network discovery workflow with a single host (using SimpleInventory as an example):

# hosts.yaml file
R1:
  hostname: 192.168.0.1
  username: admin
  password: password

I run some CDP discovery on R1 and end up discovering R2. After some business logic I add R2 to my current inventory during runtime, using something such as:

from nornir import InitNornir
from nornir.core.inventory import Host

# Initialize Nornir
nr = InitNornir(config_file='config.yaml')

# ... Run some tasks, here, grab data necessary to identify R2

# Add R2 to current inventory
nr.inventory.hosts['R2'] = Host(name='R2', username='cisco', password='developer', hostname='192.168.0.2')

I want some sort of "inventory dumping" mechanism that delegates the task of writing my current runtime inventory back to its original data format. Something such as:

nr.dump_inventory()

Would call some logic from SimpleInventory, which would result in the original hosts.yaml being modified to:

# hosts.yaml file
R1:
  hostname: 192.168.0.1
  username: admin
  password: password
R2:
  hostname: 192.168.0.2
  username: cisco
  password: developer

Of course, it's important that this "serialization" be delegated to Inventory plugins because how this serialization must be done depends on the inventory source; updating devices on Netbox when using nornir_netbox is different from updating the yaml files for SimpleInventory.

I believe this could be achieved by implementing some sort of dump_inventory() method on the Nornir class, this method would call back to an Inventory Plugins' method responsible for writing to the data source. This would require inventory plugins to implement the business logic to update their respective data source, but would make nornir much more dynamic in how it handles inventories as these could then be updated automatically after some runtime manipulation. Maintainers of inventory plugins could choose not to implement this feature, of course, in which case the feature wouldn't be supported for that plugin but everything else would still work as expected. Users would still always have the possibility of implementing this inventory dump themselves if they so need or desire, but to me it makes sense that nornir could provide this as a feature.

Just thought I'd give my 10 cents and bring up the discussion, since it could simplify some cool network as code and version control scenarios where inventory sources are updated automagically after detecting some changes to the network.

[dbarrosop]

Please, use issues for issues and discussions for discussions. I have migrated the issue to a discussion myself.

[dbarrosop]

So what you describe is already supported, however, you are proposing a standardized interface for this. I have a few concerns/questions:

1. How would this work with different types of plugins (i.e. SimpleInventory vs NetboxInventory)? While in one type you "dump" the inventory once, on the other you have to make several calls, one for device discovered.
2. How would this feature's implementation look like? Roughly speaking.
3. What would happen with existing inventories that don't have support for this? (existing and new ones)
4. Given this is already supported via functions or tasks, what would be the benefit for this new method?
5. More gernally, is this a feature the overall community is interested in?

Thanks for taking the time explaining the feature, I think we need to delve a bit into the details though and also see if the overall community thinks this is useful. Adding features have a cost in terms of testing and maintaining so the returns need to outweigh the costs.

[matman26]

Thanks for the explanation on the Discussions tab, I haven't used it previously so didn't know that was an option, I'll post any other ideas I have over here.

1. The in-and-outs will depend on what the inventory plugin reads from. I agree this could be convoluted for SoT's such as Netbox as you would need to run several GET requests and potentially POSTs for each host to update, but I feel this is something that people will want to do with Nornir one way or another leading to people reinventing the wheel several times over around the community.
2. Inventory plugins currently implement classes that are required to expose a load() method which returns an Inventory-type object. To enable dumping one could add a new write() method that takes an Inventory-type object as argument and dumps it back to its original source. This would require plugin developers to implement a new public method to be called from Nornir.
   From the Nornir core standpoint, I'm not sure where to best fit it, but we would need either a method belonging to the Nornir class or a function that takes the InventoryPlugin as an argument and calls its write() attribute, also giving it in an Invetory-type or Nornir-type object to dump. Something as barebones as this could do the trick: (let's suppose a function is added to nornir/init_nornir.py just for simplicity in the coming example)

# Adapted from the load_inventory function in nornir/init_nornir.py
def write_inventory(config: Config, inv : Inventory):
    InventoryPluginRegister.auto_register()
    inventory_plugin = InventoryPluginRegister.get_plugin(config.inventory.plugin)
    inventory_writer = getattr(inventory_plugin, 'write', False)
    if inventory_writer:
        inventory_writer(inv)
    else:
        logger.error('Write method is not supported for the current inventory plugin.')
        raise Exception("NotImplementedMethod")

This would be called during during runtime (going back to my initial example workflow):

from nornir import InitNornir, write_inventory
from nornir.core.inventory import Host

# Initialize Nornir
nr = InitNornir(config_file='config.yaml')

# ... Run some tasks, here, grab data necessary to identify R2

# Add R2 to current inventory
nr.inventory.hosts['R2'] = Host(name='R2', username='cisco', password='developer', hostname='192.168.0.2')

write_inventory(nr.config, nr.inventory)

3. This change introduces non-breaking changes to currently running inventory plugins. The high-level method that delegates the dumping will check whether or not the inventory plugin exposes the write() method. If it does not, an NotImplemented or similar exception will be raised. There could be a table somewhere in the documentation listing inventory plugins that conform to this so that users know in advance whether or not they can use the feature for their inventory plugin of choice.
4. This can be implemented via tasks and functions but as I mentioned in my first point it gives room for developers reinventing the wheel several times over on their own. It makes more sense to have inventory plugins, which already interact with their specific data source, be the ones to decide how to better write updated data to it. This would make nornir inventories truly dynamic out-of-the-box, without additional effort on the nornir users part.
5. This is going to come down to what the rest of the community think on the issue, but I made my points on what I think would be the advantages on my previous points. Advantages I see:
   • Any 'writeable inventory plugins', including text-based ones, instantly become dynamic by design as they can be updated according to network state collected during runtime
   • Easier to use for nornir users than developing this dumping manually (especially if we're talking about updating SoTs)
   • Writing hosts and updating existing hosts on an inventory in a programmatic manner is something people will want to do often enough that will lead to people re-implementing the same procedure for different projects/inventory types