Backup is not done automatically

[SerhiiZahuba]

Plugin version

1.2.1

netbox-plugin-extensions version

1.0.4

NetBox version

3.2

Python version

3.9

Steps to Reproduce

I used your config example. But the config did not automatically become. Please let me know what else to check.
? I ask you to clarify where you can see the reservation logs?

Expected Behavior

successful backup

Observed Behavior

not see successful backup

[DanSheps]

What exactly did you do to setup the backups?

[SerhiiZahuba]

1. add task in netbox
   

2. in netbox config i add 'frequency': 3600
   'repository': '/opt/netbox-config-backup',

3. systemctl enable ncb-rq@{1..10}

4. systemctl start ncb-rq@{1..10}

[SerhiiZahuba]

Apr  9 09:30:39 -Inventory python3: [2022-04-09 10:30:39,742][ERROR] - Traceback (most recent call last):
Apr  9 09:30:39 -Inventory python3: File "/opt/netbox/venv/lib/python3.9/site-packages/netbox_config_backup/tasks.py", line 114, in backup_job
Apr  9 09:30:39 -Inventory python3: commit = backup_config(backup)
Apr  9 09:30:39 -Inventory python3: File "/opt/netbox/venv/lib/python3.9/site-packages/netbox_config_backup/tasks.py", line 92, in backup_config
Apr  9 09:30:39 -Inventory python3: commit = backup.set_config(configs)
Apr  9 09:30:39 -Inventory python3: File "/opt/netbox/venv/lib/python3.9/site-packages/netbox_config_backup/models/backups.py", line 100, in set_config
Apr  9 09:30:39 -Inventory python3: current = self.get_config()
Apr  9 09:30:39 -Inventory python3: File "/opt/netbox/venv/lib/python3.9/site-packages/netbox_config_backup/models/backups.py", line 91, in get_config
Apr  9 09:30:39 -Inventory python3: running = repository.read(f'{self.uuid}.running')
Apr  9 09:30:39 -Inventory python3: File "/opt/netbox/venv/lib/python3.9/site-packages/netbox_config_backup/git.py", line 91, in read
Apr  9 09:30:39 -Inventory python3: mode, sha = object_store.tree_lookup_path(self.repository.__getitem__, tree, path)
Apr  9 09:30:39 -Inventory python3: File "/opt/netbox/venv/lib/python3.9/site-packages/dulwich/object_store.py", line 1175, in tree_lookup_path
Apr  9 09:30:39 -Inventory python3: return tree.lookup_path(lookup_obj, path)
Apr  9 09:30:39 -Inventory python3: File "/opt/netbox/venv/lib/python3.9/site-packages/dulwich/objects.py", line 1201, in lookup_path
Apr  9 09:30:39 -Inventory python3: mode, sha = obj[p]
Apr  9 09:30:39 -Inventory python3: File "/opt/netbox/venv/lib/python3.9/site-packages/dulwich/objects.py", line 1066, in __getitem__
Apr  9 09:30:39 -Inventory python3: return self._entries[name]
Apr  9 09:30:39 -Inventory python3: KeyError: b'667b3011-f5d9-4c35-8e8d-76600f36497e.running'
Apr  9 09:30:39 -Inventory python3: [2022-04-09 10:30:39,743][ERROR] - b'667b3011-f5d9-4c35-8e8d-76600f36497e.running'

after update module to 1.2.2 i see in column [Next attempt] time to next backup, but if i enter to backup task i dont see backup

[DanSheps]

Thanks, I think I see the problem, I will have a new version up soon.

[SerhiiZahuba]

Apr 11 19:39:40 Inventory python3: [2022-04-11 20:39:40,081][ERROR] - Traceback (most recent call last):
Apr 11 19:39:40 Inventory python3: File "/opt/netbox/venv/lib/python3.9/site-packages/netbox_config_backup/tasks.py", line 114, in backup_job
Apr 11 19:39:40 Inventory python3: commit = backup_config(backup)
Apr 11 19:39:40 Inventory python3: File "/opt/netbox/venv/lib/python3.9/site-packages/netbox_config_backup/tasks.py", line 92, in backup_config
Apr 11 19:39:40 Inventory python3: commit = backup.set_config(configs)
Apr 11 19:39:40 Inventory python3: File "/opt/netbox/venv/lib/python3.9/site-packages/netbox_config_backup/models/backups.py", line 103, in set_config
Apr 11 19:39:40 Inventory python3: if Differ().is_diff(current.get(file), configs.get(file)):
Apr 11 19:39:40 Inventory python3: File "/opt/netbox/venv/lib/python3.9/site-packages/netbox_config_backup/utils/git.py", line 33, in is_diff
Apr 11 19:39:40 Inventory python3: diff = list(self.plain_compare(a, b))
Apr 11 19:39:40 Inventory python3: File "/opt/netbox/venv/lib/python3.9/site-packages/netbox_config_backup/utils/git.py", line 18, in plain_compare
Apr 11 19:39:40 Inventory python3: for tag, alo, ahi, blo, bhi in cruncher.get_opcodes():
Apr 11 19:39:40 Inventory python3: File "/usr/local/lib/python3.9/difflib.py", line 525, in get_opcodes
Apr 11 19:39:40 Inventory python3: for ai, bj, size in self.get_matching_blocks():
Apr 11 19:39:40 Inventory python3: File "/usr/local/lib/python3.9/difflib.py", line 442, in get_matching_blocks
Apr 11 19:39:40 Inventory python3: la, lb = len(self.a), len(self.b)
Apr 11 19:39:40 Inventory python3: TypeError: object of type 'NoneType' has no len()
Apr 11 19:39:40 Inventory python3: [2022-04-11 20:39:40,081][ERROR] - object of type 'NoneType' has no len()

this is with version 1.2.3

[SerhiiZahuba]

Apr 13 07:04:41 Inventory python3: [2022-04-13 08:04:41,345][ERROR] - Traceback (most recent call last):
Apr 13 07:04:41 Inventory python3: File "/opt/netbox/venv/lib/python3.9/site-packages/netbox_config_backup/tasks.py", line 114, in backup_job
Apr 13 07:04:41 Inventory python3: commit = backup_config(backup)
Apr 13 07:04:41 Inventory python3: File "/opt/netbox/venv/lib/python3.9/site-packages/netbox_config_backup/tasks.py", line 92, in backup_config
Apr 13 07:04:41 Inventory python3: commit = backup.set_config(configs)
Apr 13 07:04:41 Inventory python3: File "/opt/netbox/venv/lib/python3.9/site-packages/netbox_config_backup/models/backups.py", line 107, in set_config
Apr 13 07:04:41 Inventory python3: output = repository.write(f'{self.uuid}.{file}', current)
Apr 13 07:04:41 Inventory python3: File "/opt/netbox/venv/lib/python3.9/site-packages/netbox_config_backup/git.py", line 56, in write
Apr 13 07:04:41 Inventory python3: porcelain.add(self.repository, path)
Apr 13 07:04:41 Inventory python3: File "/opt/netbox/venv/lib/python3.9/site-packages/dulwich/porcelain.py", line 493, in add
Apr 13 07:04:41 Inventory python3: r.stage(relpaths)
Apr 13 07:04:41 Inventory python3: File "/opt/netbox/venv/lib/python3.9/site-packages/dulwich/repo.py", line 1294, in stage
Apr 13 07:04:41 Inventory python3: index = self.open_index()
Apr 13 07:04:41 Inventory python3: File "/opt/netbox/venv/lib/python3.9/site-packages/dulwich/repo.py", line 1265, in open_index
Apr 13 07:04:41 Inventory python3: raise NoIndexPresent()
Apr 13 07:04:41 Inventory python3: dulwich.errors.NoIndexPresent
Apr 13 07:04:41 Inventory python3: [2022-04-13 08:04:41,345][ERROR] -

this is with version 1.2.4

[DanSheps]

Just curious, did you initialize the git repository yet?

[DanSheps]

Unfortunately, I went back to a clean version and I am unable to reproduce this.

[SerhiiZahuba]

I tried to make a return to the netbox version without this module and then added it. The error remains. I describe how I installed the module, maybe something is wrong.

source /opt/netbox/venv/bin/activate
pip install netbox-config-backup
vim /opt/netbox/netbox/netbox/configuration.py
cd /opt/
git clone https://github.com/DanSheps/netbox-config-backup.git
cd /opt/netbox/netbox/
python3 netbox/manage.py migrate
cd /etc/systemd/system/
cp netbox-rq.service [email protected]
vim [email protected]
systemctl daemon-reload
systemctl enable ncb-rq@{1..10}
systemctl start ncb-rq@{1..10}
systemctl restart netbox netbox-rq

[DanSheps]

Where are you seeing this error? Is this just in the syslogs?

[DanSheps]

Can you try a git log in the git repository directory?

[SerhiiZahuba]

Where are you seeing this error? Is this just in the syslogs?
Yes

My fault, I corrected the permissions on the repository folder. Now some files have started to appear.
There are three backups.
There are several questions

1. There is a physical file on the server (running config) and there is this config in the web. But there is no config in the Startup web, but it is physically on the server.
2. On the screen, the second config. There is no physical config on the server and the web is also empty.
3. On the screen, the third config. There is a physical config on the server (running +Startup ) but on the web, when I go inside the backup, there are no backup tasks.
4. on the screen, I pointed out the error. When I move the mouse over the picture, it shows Edit what is wrong. Since this is a view and not an edit.

[DanSheps]

• There is a physical file on the server (running config) and there is this config in the web. But there is no config in the Startup web, but it is physically on the server.
• On the screen, the second config. There is no physical config on the server and the web is also empty.
• On the screen, the third config. There is a physical config on the server (running +Startup ) but on the web, when I go inside the backup, there are no backup tasks.

This is likely due to your permissions errors. Please first correct all your permissions (chown (netboxuser):(netboxgroup) /path/to/repository -R). You can then try using the management command "rebuild" which will rebuild the git database internally.

on the screen, I pointed out the error. When I move the mouse over the picture, it shows Edit what is wrong. Since this is a view and not an edit.

I have opened a issue for this

[SerhiiZahuba]

5. Found another problem. If you click on the Diff icon next to the created backup, then there will be an error 'BackupCommit' object has no attribute 'encode' on the web.

Apr 14 09:17:38 -Inventory gunicorn: File "/opt/netbox/venv/lib/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner
Apr 14 09:17:38 -Inventory gunicorn: response = get_response(request)
Apr 14 09:17:38 -Inventory gunicorn: File "/opt/netbox/venv/lib/python3.9/site-packages/django/core/handlers/base.py", line 197, in _get_response
Apr 14 09:17:38 -Inventory gunicorn: response = wrapped_callback(request, *callback_args, **callback_kwargs)
Apr 14 09:17:38 -Inventory gunicorn: File "/opt/netbox/venv/lib/python3.9/site-packages/django/views/generic/base.py", line 84, in view
Apr 14 09:17:38 -Inventory gunicorn: return self.dispatch(request, *args, **kwargs)
Apr 14 09:17:38 -Inventory gunicorn: File "/opt/netbox/venv/lib/python3.9/site-packages/django/views/generic/base.py", line 119, in dispatch
Apr 14 09:17:38 -Inventory gunicorn: return handler(request, *args, **kwargs)
Apr 14 09:17:38 -Inventory gunicorn: File "/opt/netbox/venv/lib/python3.9/site-packages/netbox_config_backup/views.py", line 122, in get
Apr 14 09:17:38 -Inventory gunicorn: diff = list(repo.diff(path, prevcommit, current.commit.sha))
Apr 14 09:17:38 -Inventory gunicorn: File "/opt/netbox/venv/lib/python3.9/site-packages/netbox_config_backup/git.py", line 106, in diff
Apr 14 09:17:38 -Inventory gunicorn: data.append(self.read(file, commit))
Apr 14 09:17:38 -Inventory gunicorn: File "/opt/netbox/venv/lib/python3.9/site-packages/netbox_config_backup/git.py", line 91, in read
Apr 14 09:17:38 -Inventory gunicorn: tree = self.repository[index.encode('ascii')].tree
Apr 14 09:17:38 -Inventory gunicorn: AttributeError: 'BackupCommit' object has no attribute 'encode'

[SerhiiZahuba]

Sorry, I have a couple more questions, problems.

1. I suggest adding pagination to the backup page. Very inconvenient when there are 20-40 backups, the page will be very large.
   

2. If you go to the backup task and click delete on top, then after deletion, the transition goes to the main page of the site. It would be logical that after deleting the task, the redirect would be to the main page of the module.

3. If possible, it would be convenient if on the main page of the module you can see the IP address of the device from which the backup will be. Now you can only specify the name of the device\backup.

4. Add the ability to exclude lines from the backup. This is necessary, for example, when there is a backup creation time in the backup and then each new backup will be changed, but in fact only the time has changed in the backup. For example, you can exclude the first lines by the line named clock-period (for cisco)

[DanSheps]

1. I suggest adding pagination to the backup page. Very inconvenient when there are 20-40 backups, the page will be very large.

It is being looked into, but if you could open a issue that would be great.

2. If you go to the backup task and click delete on top, then after deletion, the transition goes to the main page of the site. It would be logical that after deleting the task, the redirect would be to the main page of the module.

Please go ahead and open a issue for that.

3. If possible, it would be convenient if on the main page of the module you can see the IP address of the device from which the backup will be. Now you can only specify the name of the device\backup.

There will be a change in the next version, that will include the ability to set the IP. This isn't exactly what you were after but I can look at doing something like this possibly. Again, please open a issue.

4. Add the ability to exclude lines from the backup. This is necessary, for example, when there is a backup creation time in the backup and then each new backup will be changed, but in fact only the time has changed in the backup. For example, you can exclude the first lines by the line named clock-period (for cisco)

This is technically probably do-able, however I don't think I am going to put the development effort into it as it would require parsing the entire config to exclude that line. That said, I am going to reduce the individual commits (currently a commit is generated even if there is no change) being generated to only where there is a diff generated.