Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature] Added support for ZeroTier parser #135 #204

Merged
merged 10 commits into from
Sep 22, 2023
4 changes: 4 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,11 @@ on:
push:
branches:
- master
- gsoc23
pull_request:
branches:
- master
- gsoc23

jobs:
build:
Expand Down Expand Up @@ -64,6 +66,8 @@ jobs:
- name: Install test dependencies
run: |
pip install -U -r requirements-test.txt
pip install --force-reinstall \
git+https://github.com/openwisp/openwisp-controller/@issue-606/zerotier-member-auth-ip-assign

- name: Install openwisp-network-topology
run: |
Expand Down
91 changes: 89 additions & 2 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,7 @@ Available features
- CNML 1.0
- OpenVPN
- Wireguard
- ZeroTier
- additional formats can be added by
`writing custom netdiff parsers <https://github.com/openwisp/netdiff#parsers>`_
* **network topology visualizer** based on
Expand Down Expand Up @@ -340,6 +341,92 @@ Sending data for topology with RECEIVE strategy
or, alternatively, a non-admin visualizer page is also available at
the URL ``/topology/topology/<TOPOLOGY-UUID>/``.

Sending data for ZeroTier topology with RECEIVE strategy
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Follow the procedure described below to setup ZeroTier topology with RECEIVE strategy.

**Note:** In this example, the **Shared systemwide (no organization)**
option is used for the ZeroTier topology organization. You are free to
opt for any organization, as long as both the topology and the device share
the same organization, assuming the `OpenWISP controller integration
<#integration-with-openwisp-controller-and-openwisp-monitoring>`_ feature is enabled.

1. Create topology for ZeroTier
###############################

1. Visit ``admin/topology/topology/add`` to add a new topology.

2. We will set the **Label** of this topology to ``ZeroTier`` and
select the topology **Format** from the dropdown as ``ZeroTier``.

3. Select the strategy as ``RECEIVE`` from the dropdown.

.. image:: https://raw.githubusercontent.com/openwisp/openwisp-network-topology/docs/docs/zerotier-tutorial/topology-1.png
:alt: ZeroTier topology configuration example 1

4. Let use default **Expiration time** ``0`` and make sure **Published** option is checked.

5. After clicking on the **Save and continue editing** button, a topology receive URL is generated.
Make sure you copy that URL for later use in the topology script.

.. image:: https://raw.githubusercontent.com/openwisp/openwisp-network-topology/docs/docs/zerotier-tutorial/topology-2.png
:alt: ZeroTier topology configuration example 2

2. Create a script for sending ZeroTier topology data
#####################################################

1. Now, create a script (e.g: ``/opt/send-zt-topology.sh``) that sends
the ZeroTier topology data using a POST request. In the example script below,
we are sending the ZeroTier self-hosted controller peers data:

.. code-block:: shell

#!/bin/bash
# command to fetch zerotier controller peers data in json format
COMMAND="zerotier-cli peers -j"
UUID="<TOPOLOGY-UUID-HERE>"
KEY="<TOPOLOGY-KEY-HERE>"
OPENWISP_URL="https://<OPENWISP_DOMAIN_HERE>"
$COMMAND |
# Upload the topology data to OpenWISP
curl -X POST \
--data-binary @- \
--header "Content-Type: text/plain" \
$OPENWISP_URL/api/v1/network-topology/topology/$UUID/receive/?key=$KEY

2. Add the ``/opt/send-zt-topology.sh`` script created in the previous step
to the root crontab, here's an example which sends the topology data every **5 minutes**:

.. code-block:: shell

# flag script as executable
chmod +x /opt/send-zt-topology.sh

.. code-block:: shell

# open rootcrontab
sudo crontab -e

## Add the following line and save

echo */5 * * * * /opt/send-zt-topology.sh

**Note:** When using the **ZeroTier** topology, ensure that
you use ``sudo crontab -e`` to edit the **root crontab**. This step
is essential because the ``zerotier-cli peers -j`` command requires **root privileges**
for kernel interaction, without which the command will not function correctly.

3. Once the steps above are completed, you should see nodes and links
being created automatically, you can see the network topology graph
from the admin page of the topology change page (you have to click on
the **View topology graph** button in the upper right part of the page)
or, alternatively, a non-admin visualizer page is also available at
the URL ``/topology/topology/<TOPOLOGY-UUID>/``.

.. image:: https://raw.githubusercontent.com/openwisp/openwisp-network-topology/docs/docs/zerotier-tutorial/topology-graph.png
:alt: ZeroTier topology graph example 1

Management Commands
-------------------

Expand Down Expand Up @@ -457,8 +544,8 @@ Integration with OpenWISP Controller and OpenWISP Monitoring

If you use `OpenWISP Controller <https://github.com/openwisp/openwisp-controller>`_
or `OpenWISP Monitoring <https://github.com/openwisp/openwisp-monitoring>`_
and you use OpenVPN or Wireguard for the management VPN, you can use the integration
available in ``openwisp_network_topology.integrations.device``.
and you use OpenVPN, Wireguard or ZeroTier for the management VPN, you can use
the integration available in ``openwisp_network_topology.integrations.device``.

This additional and optional module provides the following features:

Expand Down
29 changes: 29 additions & 0 deletions openwisp_network_topology/integrations/device/base/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,9 @@ class AbstractDeviceNode(UUIDModel):
'netdiff.WireguardParser': {
'auto_create': 'auto_create_wireguard',
},
'netdiff.ZeroTierParser': {
'auto_create': 'auto_create_zerotier',
},
'netdiff.NetJsonParser': {
'auto_create': 'auto_create_netjsongraph',
},
Expand Down Expand Up @@ -127,6 +130,32 @@ def auto_create_wireguard(cls, node):
return
return cls.save_device_node(device, node)

@classmethod
def auto_create_zerotier(cls, node):
"""
Implementation of the integration between
controller and network-topology modules
when using ZeroTier (using the `member_id`)
"""
member_id = node.properties.get('address')
if not member_id:
return

Device = load_model('config', 'Device')
device_filter = models.Q(config__vpnclient__secret__startswith=member_id)
if node.organization_id:
device_filter &= models.Q(organization_id=node.organization_id)
device = (
Device.objects.only(
'id', 'name', 'last_ip', 'management_ip', 'organization_id'
)
.filter(device_filter)
.first()
)
if not device:
return
return cls.save_device_node(device, node)

@classmethod
def auto_create_netjsongraph(cls, node):
if len(node.addresses) < 2:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ def handle(self, *args, **kwargs):
queryset = Node.objects.select_related('topology').filter(
Q(topology__parser='netdiff.OpenvpnParser')
| Q(topology__parser='netdiff.WireguardParser')
| Q(topology__parser='netdiff.ZeroTierParser')
)
for node in queryset.iterator():
DeviceNode.auto_create(node)
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
CreateConfigTemplateMixin,
TestVpnX509Mixin,
TestWireguardVpnMixin,
TestZeroTierVpnMixin,
)
from openwisp_ipam.tests import CreateModelsMixin as SubnetIpamMixin

Expand All @@ -38,13 +39,16 @@
class Base(
TestVpnX509Mixin,
TestWireguardVpnMixin,
TestZeroTierVpnMixin,
SubnetIpamMixin,
CreateConfigTemplateMixin,
CreateGraphObjectsMixin,
TestOrganizationMixin,
):
topology_model = Topology
node_model = Node
_ZT_SERVICE_REQUESTS = 'openwisp_controller.config.api.zerotier_service.requests'
_ZT_GENERATE_IDENTITY_SUBPROCESS = 'openwisp_controller.config.base.vpn.subprocess'

def _init_test_node(
self,
Expand Down Expand Up @@ -94,13 +98,58 @@ def _init_wireguard_test_node(self, topology, addresses=[], create=True, **kwarg
node.save()
return node

def _init_zerotier_test_node(
self, topology, addresses=None, label='test', member_id=None, create=True
):
if not addresses:
addresses = [self._TEST_ZT_MEMBER_CONFIG['address']]
node = Node(
organization=topology.organization,
topology=topology,
label=label,
addresses=addresses,
# zt peer address is `memeber_id`
properties={'address': member_id},
)
if create:
node.full_clean()
node.save()
return node

def _create_wireguard_test_env(self, parser):
org = self._get_org()
device, _, _ = self._create_wireguard_vpn_template()
device.organization = org
topology = self._create_topology(organization=org, parser=parser)
return topology, device

@mock.patch(_ZT_GENERATE_IDENTITY_SUBPROCESS)
@mock.patch(_ZT_SERVICE_REQUESTS)
def _create_zerotier_test_env(self, mock_requests, mock_subprocess, parser):
mock_requests.get.side_effect = [
# For node status
self._get_mock_response(200, response=self._TEST_ZT_NODE_CONFIG)
]
mock_requests.post.side_effect = [
# For create network
self._get_mock_response(200),
# For controller network join
self._get_mock_response(200),
# For controller auth and ip assignment
self._get_mock_response(200),
# For member auth and ip assignment
self._get_mock_response(200),
]
mock_stdout = mock.MagicMock()
mock_stdout.stdout.decode.return_value = self._TEST_ZT_MEMBER_CONFIG['identity']
mock_subprocess.run.return_value = mock_stdout
org = self._get_org()
device, _, _ = self._create_zerotier_vpn_template()
device.organization = org
topology = self._create_topology(organization=org, parser=parser)
member_id = device.config.vpnclient_set.first().member_id
return topology, device, member_id

def _create_test_env(self, parser):
organization = self._get_org()
vpn = self._create_vpn(name='test VPN', organization=organization)
Expand Down Expand Up @@ -198,6 +247,47 @@ def test_auto_create_wireguard(self):
except KeyError:
self.fail('KeyError raised')

def test_auto_create_zerotier(self):
topology, device, member_id = self._create_zerotier_test_env(
parser='netdiff.ZeroTierParser'
)
self.assertEqual(DeviceNode.objects.count(), 0)
with self.subTest('assert number of queries'):
with self.assertNumQueries(15):
node = self._init_zerotier_test_node(topology, member_id=member_id)
self.assertEqual(DeviceNode.objects.count(), 1)
device_node = DeviceNode.objects.first()
self.assertEqual(device_node.device, device)
self.assertEqual(device_node.node, node)

with self.subTest('not run on save'):
with mock.patch.object(transaction, 'on_commit') as on_commit:
node.save()
on_commit.assert_not_called()

def test_auto_create_zerotier_failures(self):
topology, device, member_id = self._create_zerotier_test_env(
parser='netdiff.ZeroTierParser'
)

with self.subTest('member_id not present'):
self._init_zerotier_test_node(topology)
self.assertFalse(DeviceNode.objects.exists())

with self.subTest('member_id does not exist'):
self._init_zerotier_test_node(topology, member_id='non_existent_id')
self.assertFalse(DeviceNode.objects.exists())

with self.subTest('exception during save'):
with mock.patch.object(
DeviceNode, 'save', side_effect=Exception('test')
) as save:
with mock.patch.object(models_logger, 'exception') as logger_exception:
self._init_zerotier_test_node(topology, member_id=member_id)
save.assert_called_once()
logger_exception.assert_called_once()
self.assertEqual(DeviceNode.objects.count(), 0)

def test_filter_by_link(self):
topology, device, cert = self._create_test_env(parser='netdiff.OpenvpnParser')

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ class Migration(migrations.Migration):
('netdiff.CnmlParser', 'CNML 1.0'),
('netdiff.OpenvpnParser', 'OpenVPN'),
('netdiff.WireguardParser', 'Wireguard'),
('netdiff.ZeroTierParser', 'ZeroTier'),
],
help_text='Select topology format',
max_length=128,
Expand Down
1 change: 1 addition & 0 deletions openwisp_network_topology/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ def get_settings_value(option, default):
('netdiff.CnmlParser', 'CNML 1.0'),
('netdiff.OpenvpnParser', 'OpenVPN'),
('netdiff.WireguardParser', 'Wireguard'),
('netdiff.ZeroTierParser', 'ZeroTier'),
]

PARSERS = DEFAULT_PARSERS + get_settings_value('PARSERS', [])
Expand Down
2 changes: 1 addition & 1 deletion requirements.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
openwisp-users @ https://github.com/openwisp/openwisp-users/tarball/master
netdiff @ https://github.com/openwisp/netdiff/tarball/master
netdiff @ https://github.com/openwisp/netdiff/tarball/issue-106/add-zt-parser
jsonfield~=3.1.0
django-flat-json-widget @ https://github.com/openwisp/django-flat-json-widget/tarball/master
openwisp-utils[celery] @ https://github.com/openwisp/openwisp-utils/tarball/master
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ class Migration(migrations.Migration):
('netdiff.CnmlParser', 'CNML 1.0'),
('netdiff.OpenvpnParser', 'OpenVPN'),
('netdiff.WireguardParser', 'Wireguard'),
('netdiff.ZeroTierParser', 'ZeroTier'),
],
help_text='Select topology format',
max_length=128,
Expand Down
Loading