Linux network plugin: NetworkManager & systemd-networkd #932

twiggler · 2024-11-05T08:28:33Z

Implement the NetworkPlugin for Linux.

Initial support is for:

NetworkManager
systemd-networkd

The ips method of the Linux OS plugin is implemented using the new parser; other methods such as dns and dhcp still use the legacy solution until we implemented parsers for the remaining network configuration systems. Let me know if this is wrong.

Update: I retained the old ips. The new implementation can be invoked by prefixing with the network namespace, i.e. -f network.interfaces

While doing research I discovered that systemd supports "drop-in" configuration directories (see https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html) . Since this feature applicable to all systemd domains and not only networking, I propose we extend the systemd config parser with support for this (#933). I checked this with Stefan de Reuver.

Keep in mind that unlike the OSX and Windows implementations, which report actual values, the Linux implementation reports configuration of interfaces, or "potential" interfaces, as there is often no way to determine which configuration was active. I think this results in some friction with the interface records; for example, the mac address needed to be extended to a list.

Closes #776

codecov · 2024-11-05T09:48:30Z

Codecov Report

Attention: Patch coverage is 92.53731% with 15 lines in your changes missing coverage. Please review.

Project coverage is 77.14%. Comparing base (015c33f) to head (e3982a7).
Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
dissect/target/plugins/os/unix/linux/network.py	92.63%	14 Missing ⚠️
dissect/target/plugins/os/unix/bsd/osx/network.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #932      +/-   ##
==========================================
+ Coverage   76.98%   77.14%   +0.15%     
==========================================
  Files         322      323       +1     
  Lines       27583    27801     +218     
==========================================
+ Hits        21236    21446     +210     
- Misses       6347     6355       +8

Flag	Coverage Δ
unittests	`77.14% <92.53%> (+0.15%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Miauwkeru · 2024-11-07T09:26:32Z

dissect/target/plugins/os/unix/linux/network.py

+                dhcp_ipv4, dhcp_ipv6 = self._parse_dhcp(network_section.get("DHCP"))
+                if link_mac := link_section.get("MACAddress"):
+                    mac_addresses.add(link_mac)
+                if match_macs := match_section.get("MACAddress"):
+                    mac_addresses.update(match_macs.split(" "))
+                if permanent_macs := match_section.get("PermanentMACAddress"):
+                    mac_addresses.update(permanent_macs.split(" "))
+
+                if dns_value := network_section.get("DNS"):
+                    if isinstance(dns_value, str):
+                        dns_value = [dns_value]
+                    dns.update({self._parse_dns_ip(dns_ip) for dns_ip in dns_value})
+
+                if address_value := network_section.get("Address"):
+                    if isinstance(address_value, str):
+                        address_value = [address_value]
+                    ip_interfaces.update({ip_interface(addr) for addr in address_value})
+
+                if gateway_value := network_section.get("Gateway"):
+                    if isinstance(gateway_value, str):
+                        gateway_value = [gateway_value]
+                    gateways.update({ip_address(gateway) for gateway in gateway_value})


Might be an idea to create some helper functions and change it to something like this:

def _splitter(value: str | None, seperator=" ") -> list[str]: if not value: return [] return value.split(seperator) def _create_list(func, value: str | list | None): if not value: return [] if isinstance(value, str): value = [value] return value

then for the code itself:

mac_addresses.update(_splitter(match_section.get("MACAddress")) ip_interfaces.update(map(ip_address, create_list(network_section.get("Address")))

Or create a function with a map inside of it, cause this is still a bit lengthy ofc

With regard to _splitter

I re-read the documentation of str.split() and if you don´t specify a separator, it will separate on white space, and return an empty list if the string is empty.

With regard to _create_list

Wrote general purpose to_list in dissect/target/helpers/utils.py.
Did not opt for the map in this case, one line for value retrieval and one line for the update is fine imo

Miauwkeru · 2024-11-07T09:27:40Z

dissect/target/plugins/os/unix/linux/network.py

+                vlan_values = network_section.get("VLAN", [])
+                vlan_ids = {
+                    virtual_networks[vlan_name]
+                    for vlan_name in ([vlan_values] if isinstance(vlan_values, str) else vlan_values)


I would put this above the comprehension statement. It becomes a bit more readable then. Currently a lot is going on imho

Simplified with to_list

Miauwkeru · 2024-11-07T09:31:12Z

dissect/target/plugins/os/unix/linux/network.py

+                    configurator="systemd-networkd",
+                )
+            except Exception as e:
+                self._target.log.warning("Error parsing network config file %s: %s", config_file, e)


its better to put e inside a self._target.log.debug("", exc_info=e) there are some examples for that everywhere. such as dissect/target/container.py

I think a warning is better; that way, the analyst can respond by analyzing the potentially broken configuration file manually.

There are apparently also some wrong examples; I took this from

dissect.target/dissect/target/plugins/os/unix/bsd/osx/network.py

Line 92 in 759392f

self.target.log.warning("Error reading configuration for network device %s: %s", name, e)

Will fix it there too.

Miauwkeru · 2024-11-07T09:32:02Z

dissect/target/plugins/os/unix/linux/network.py

+
+    # Can be enclosed in brackets for IPv6. Can also have port, iface name, and SNI, which we ignore.
+    # Example: [1111:2222::3333]:9953%ifname#example.com
+    dns_ip_patttern = re.compile(r"((?:\d{1,3}\.){3}\d{1,3})|\[(\[?[0-9a-fA-F:]+\]?)\]")


Wouldn't it be better to use capture groups for this, instead of using indexes of 1 and 2 on L276?
Would make it a bit more readable.

Sure, although Imo it is still unreadable.

Alternative could be parser combinators, but we don't use them yet. I would have to search for a suitable python implementation.

Miauwkeru · 2024-11-07T09:37:52Z

dissect/target/plugins/os/unix/linux/network.py

+        match = self.dns_ip_patttern.search(address)
+        if match:
+            return ip_address(match.group(1) or match.group(2))
+        else:
+            raise ValueError(f"Invalid DNS address format: {address}")


Suggested change

match = self.dns_ip_patttern.search(address)

if match:

return ip_address(match.group(1) or match.group(2))

else:

raise ValueError(f"Invalid DNS address format: {address}")

if match:= self.dns_ip_patttern.search(address):

return ip_address(match.group(1) or match.group(2))

raise ValueError(f"Invalid DNS address format: {address}")

Miauwkeru · 2024-11-07T10:14:24Z

dissect/target/plugins/os/unix/linux/network.py

+    def _parse_lastconnected(self, value: str) -> datetime | None:
+        """Parse last connected timestamp."""
+        if not value:
+            return None
+
+        timestamp_int = int(value)
+        return datetime.fromtimestamp(timestamp_int, timezone.utc)


Suggested change

def _parse_lastconnected(self, value: str) -> datetime | None:

"""Parse last connected timestamp."""

if not value:

return None

timestamp_int = int(value)

return datetime.fromtimestamp(timestamp_int, timezone.utc)

def _parse_lastconnected(self, last_connected: str) -> datetime | None:

"""Parse last connected timestamp."""

if not last_connected:

return None

return datetime.fromtimestamp(int(last_connected), timezone.utc)

Miauwkeru · 2024-11-07T10:20:14Z

dissect/target/plugins/os/unix/linux/network.py

+                        vlan_id_by_interface[parent_interface] = int(vlan_id)
+                    continue
+
+                dns = set[ip_address]()


Suggested change

dns = set[ip_address]()

dns: set[ip_address] = set()

Miauwkeru · 2024-11-07T10:22:37Z

dissect/target/plugins/os/unix/linux/network.py

+
+
+class LinuxConfigParser:
+    VlanIdByName = dict[str, int]


You could put this type definition outside the class. Then it can just be used by any class without needing to first call LinuxConfigParser

Yeah, I want to prevent polluting the module scope. But moved it, since the requirement to fully qualify decreases readability

Miauwkeru · 2024-11-07T10:23:05Z

dissect/target/plugins/os/unix/linux/network.py

+            yield from manager.interfaces()
+
+
+class LinuxConfigParser:


Suggested change

class LinuxConfigParser:

class LinuxNetworkParser:

wouldn't this be more accurate?

Changed into LinuxNetworkConfigParser, which is perhaps even better

Miauwkeru · 2024-11-07T10:25:57Z

dissect/target/plugins/os/unix/linux/network.py

+                for ip_version in ["ipv4", "ipv6"]:
+                    ip_section: dict[str, str] = config.get(ip_version, {})
+                    for key, value in ip_section.items():
+                        # nmcli inserts a trailling semicolon
+                        if key == "dns" and (stripped := value.rstrip(";")):
+                            dns.update({ip_address(addr) for addr in stripped.split(";")})
+                        elif key.startswith("address"):
+                            # Undocumented: single gateway on address line. Observed when running:


This is my indentation allergy talking. Is it an idea to do this in a seperate method that returns a dict of the stuff we need? then use **output for the record descriptor.

I feel you.

Usually, I do things like that in the second pass, when we have learned more.
I will indulge you, but with a dataclass though for added type safety

The base class uses this field so we cannot make it a singleton/collection depending on os.

- connections with same iface name caused conflicts. - support vlan matching on uuid.

Miauwkeru · 2024-11-12T08:34:55Z

dissect/target/plugins/os/unix/linux/network.py

+        """Parse DHCP value from systemd network configuration file to a named tuple (ipv4, ipv6)."""
+
+        if value is None or value == "no":
+            return self.DhcpConfig(ipv4=False, ipv6=False)


a named tuple feels a bit like overkill, as you don't use it like a named tuple elsewhere.

I think most of it can be replaced with adding this to the docstring:

Raises: ValueError: if an unknown DHCP value was passed through Returns: a tuple of booleans (ipv4, ipv6) telling how DHCP was configured

The named tuple is scoped to the class, reflecting its specificity.

Arguably making the code explicit is favorable to writing a comment explaining it.
Although these are not mutually exclusive, since this is a private method a single line docstring should be adequate.

Miauwkeru · 2024-11-12T08:37:42Z

dissect/target/helpers/utils.py

+T = TypeVar("T")
+
+
+def to_list(value: T | list[T]) -> list[T]:
+    """Convert a single value or a list of values to a list.
+
+    Args:
+        value: The value to convert.
+
+    Returns:
+        A list of values.
+    """
+    if not isinstance(value, list):
+        return [value]
+    return value
+
+


Maybe a bit overkill to put here as no other class uses it yet, no?

I feel the method is too general to put anywhere else, although the module names helpers and utils are a bit nondescript.

Besides, if we keep it in for example the SystemdNetworkConfigParser, then it is likely that no one is going to use it.

until we find everyone is making it again and move it then :P but i get your point

Miauwkeru · 2024-11-12T08:38:41Z

dissect/target/plugins/os/unix/linux/network.py

+                source=self.source,
+                last_connected=self.last_connected,
+                name=self.name,
+                mac=[self.mac_address] if self.mac_address else [],


not going to use to_list here?

This has different semantics than to_list: it maps None to an empty list, which to_list does not do.

Ah, I was remembering an implementation where None was transformed into an empty list. So that is fair

Miauwkeru · 2024-11-12T08:40:09Z

dissect/target/plugins/os/unix/linux/network.py

+            uuid = vlan_id_by_interface.get(context.uuid) if context.uuid else None
+            name = vlan_id_by_interface.get(connection.name) if connection.name else None


no need to do the if conditions here, the get on a dictionary returns a None by default. if context.uuid is not, it will try to index it using a None which returns a None cause it does not exist.

I am defending against lookup of None in vlan_id_by_interface, which would be a logic error.
But perhaps overly cautious, removed.

However, there is an edge case where there are multiple vlans bound to the same interface, where the first vlan is bound by iface name, and the second by uuid.

Rewrote and adjusted unit test

Miauwkeru · 2024-11-12T08:52:16Z

dissect/target/plugins/os/unix/linux/network.py

+        elif key.startswith("route"):
+            if gateway := self._parse_route(value):
+                context.gateways.add(gateway)


Suggested change

elif key.startswith("route"):

if gateway := self._parse_route(value):

context.gateways.add(gateway)

elif key.startswith("route") and (gateway := self._parse_route(value)):

context.gateways.add(gateway)

?

Looking at this again, I actually prefer the nested if because they inner and outer if have different purposes.

The outer if switches on the key type, and the inner if is part of the action of handling a certain configuration value. This way, we can scan more easily over the different key types.

I made all other outer clauses consistent. This leads to more indentation but I think it is for the better

Miauwkeru · 2024-11-12T08:55:53Z

dissect/target/plugins/os/unix/linux/network.py

+        elif key == "method" and ip_version == "ipv4":
+            context.dhcp_ipv4 = value == "auto"
+        elif key == "method" and ip_version == "ipv6":
+            context.dhcp_ipv6 = value == "auto"


Suggested change

elif key == "method" and ip_version == "ipv4":

context.dhcp_ipv4 = value == "auto"

elif key == "method" and ip_version == "ipv6":

context.dhcp_ipv6 = value == "auto"

elif key == "method":

setattr(context, f"dhcp_{ip_version}", value == "auto")

tho maybe this is a bit too much magic :P

Clever, but I think the original is more readable.

Miauwkeru · 2024-11-12T09:00:00Z

dissect/target/plugins/os/unix/linux/network.py

+        if key == "dns" and (stripped := value.rstrip(";")):
+            context.dns.update({ip_address(addr) for addr in stripped.split(";")})


Suggested change

if key == "dns" and (stripped := value.rstrip(";")):

context.dns.update({ip_address(addr) for addr in stripped.split(";")})

if key == "dns":

context.dns.update({ip_address(addr) for addr in value.split(";") if addr})

wouldn't this also work?

It would also work, but I think the original more clearly reflects that we are guarding against a trailing ;.

Then again, maybe empty values are somehow allowed so your suggestion is slightly more robust.

Applied suggestion

the empty values get ignored due to the if addr if addr ='' for example. that is some truethy magic in python. same with an empty list and such

With maybe empty values are somehow allowed

I meant according to the Network Manager spec. (I applied your suggestion)

Miauwkeru · 2024-11-12T09:02:52Z

dissect/target/plugins/os/unix/linux/network.py

+        parent_interface = sub_type.get("parent", None)
+        vlan_id = sub_type.get("id", None)


get() doesn't need an explicit None as it is the default for dicts :)

Miauwkeru · 2024-11-12T09:07:22Z

dissect/target/plugins/os/unix/linux/network.py

+                    vlan_id
+                    for vlan_name in to_list(vlan_names)
+                    if (vlan_id := virtual_networks.get(vlan_name)) is not None


Suggested change

vlan_id

for vlan_name in to_list(vlan_names)

if (vlan_id := virtual_networks.get(vlan_name)) is not None

virtual_networks.get(_name) for _name in to_list(vlan_names) if _name in virtual_networks

isn't this a bit more clear?

What I don't like is that the lookup will be performed twice.

In any case, rewrote as an explicit loop. Also fixed a type-inconsistency.

Also adjusted unit test to check for multiple vlans.

dissect/target/plugins/os/unix/linux/network.py

twiggler marked this pull request as draft November 5, 2024 08:47

twiggler force-pushed the linux-network-manager branch 4 times, most recently from f2e7ce1 to b91d450 Compare November 5, 2024 09:37

twiggler marked this pull request as ready for review November 5, 2024 09:49

twiggler requested a review from cecinestpasunepipe November 5, 2024 09:49

Miauwkeru requested changes Nov 7, 2024

View reviewed changes

twiggler force-pushed the linux-network-manager branch 2 times, most recently from 0573816 to 759392f Compare November 8, 2024 08:36

twiggler added 6 commits November 8, 2024 11:16

Add linux network manager

e861202

fix: make mac address a list.

b7c0376

The base class uses this field so we cannot make it a singleton/collection depending on os.

fix: convert last connected timestamp to utc

21d4551

fix: silly use of Iterator

c421286

fix: return all NetworkManager connections.

bb6d332

- connections with same iface name caused conflicts. - support vlan matching on uuid.

Add docs

304e959

twiggler force-pushed the linux-network-manager branch 3 times, most recently from 5046d28 to 83258be Compare November 8, 2024 14:23

Process review comments

3b56d58

twiggler force-pushed the linux-network-manager branch from 83258be to 3b56d58 Compare November 8, 2024 16:45

twiggler requested a review from Miauwkeru November 11, 2024 08:12

Miauwkeru requested changes Nov 12, 2024

View reviewed changes

twiggler force-pushed the linux-network-manager branch from a9689fc to 99bb458 Compare November 12, 2024 13:03

fix: edge case multiple vlans bound to same iface

e3982a7

twiggler force-pushed the linux-network-manager branch from 99bb458 to e3982a7 Compare November 12, 2024 22:08

twiggler requested a review from Miauwkeru November 12, 2024 22:13

		uuid = vlan_id_by_interface.get(context.uuid) if context.uuid else None
		name = vlan_id_by_interface.get(connection.name) if connection.name else None

		if key == "dns" and (stripped := value.rstrip(";")):
		context.dns.update({ip_address(addr) for addr in stripped.split(";")})

		parent_interface = sub_type.get("parent", None)
		vlan_id = sub_type.get("id", None)

Linux network plugin: NetworkManager & systemd-networkd #932

Are you sure you want to change the base?

Linux network plugin: NetworkManager & systemd-networkd #932

Conversation

twiggler commented Nov 5, 2024 • edited Loading

codecov bot commented Nov 5, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

twiggler Nov 8, 2024 • edited Loading

Choose a reason for hiding this comment

With regard to _splitter

With regard to _create_list

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

twiggler Nov 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

twiggler Nov 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

twiggler Nov 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

twiggler Nov 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

twiggler Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

twiggler Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

twiggler Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

twiggler Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

twiggler Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

twiggler commented Nov 5, 2024 •

edited

Loading

codecov bot commented Nov 5, 2024 •

edited

Loading

twiggler Nov 8, 2024 •

edited

Loading

With regard to `_splitter`

twiggler Nov 7, 2024 •

edited

Loading

twiggler Nov 7, 2024 •

edited

Loading

twiggler Nov 7, 2024 •

edited

Loading

twiggler Nov 7, 2024 •

edited

Loading

twiggler Nov 12, 2024 •

edited

Loading

twiggler Nov 12, 2024 •

edited

Loading

twiggler Nov 12, 2024 •

edited

Loading

twiggler Nov 12, 2024 •

edited

Loading

twiggler Nov 12, 2024 •

edited

Loading