Add target-diff #664

JSCU-CNI · 2024-04-04T16:55:14Z

This PR adds the command target-diff, which can be used to compare two or more targets against one another:

$ target-diff --help

target-diff

positional arguments:
  {shell,fs,query}      Mode for differentiating targets
    shell               Open an interactive shell to compare two or more targets.
    fs                  Yield records about differences between target filesystems.
    query               Differentiate plugin outputs between two or more targets.

options:
  -d, --deep            Compare file contents even if metadata suggests they have been left unchanged (default: False)
  -l LIMIT, --limit LIMIT
                        How many bytes to compare before assuming a file is left unchanged (0 for no limit) (default:
                        32768)

fs mode outputs records denoting filesystem changes from one target to the other:

$ target-diff --deep fs src.tar dst.tar

<differential/file/created hostname=None domain=None src_target='src.tar' dst_target='dst.tar' path='/changes/only_on_dst'>
<differential/file/deleted hostname=None domain=None src_target='src.tar' dst_target='dst.tar' path='/changes/only_on_src'>
<differential/file/modified hostname=None domain=None src_target='src.tar' dst_target='dst.tar' path='/changes/changed' diff=[b'--- \n', b'+++ \n', b'@@ -1 +1 @@\n', b'-SRC', b'+DST']>

Using query mode, you can compare plugin outputs from one target to the other:

$ target-diff query -f users src.tar dst.tar

<differential/record/unchanged hostname=None domain=None src_target='src.tar' dst_target='dst.tar' record=<unix/user hostname='dst_target' domain=None name='root' passwd='x' uid=0 gid=0 gecos='root' home='/root' shell='/bin/bash' source='/etc/passwd'>>
<differential/record/unchanged hostname=None domain=None src_target='src.tar' dst_target='dst.tar' record=<unix/user hostname='dst_target' domain=None name='user' passwd='x' uid=1000 gid=1000 gecos='user' home='/home/user' shell='/bin/bash' source='/etc/passwd'>>
<differential/record/created hostname=None domain=None src_target='src.tar' dst_target='dst.tar' record=<unix/user hostname='dst_target' domain=None name='dst_user' passwd='x' uid=1001 gid=1001 gecos='dst_user' home='/home/dst_user' shell='/bin/bash' source='/etc/passwd'>>
<differential/record/deleted hostname=None domain=None src_target='src.tar' dst_target='dst.tar' record=<unix/user hostname='src_target' domain=None name='src_user' passwd='x' uid=1001 gid=1001 gecos='src_user' home='/home/src_user' shell='/bin/bash' source='/etc/passwd'>>

In shell mode, you can browse the target filesystems like in target-shell, where directory listings will show which files / directories have been changed, added or deleted. Using the plugin command, plugin outputs can be compared from within the shell context.

$ target-diff shell src.tar dst.tar

(dst_target/src_target)/diff />help

Target Diff
==========


Documented commands (type help <topic>):
=================================================================
cat  clear  diff   exit  help  ls    plugin  previous  set
cd   cyber  enter  find  list  next  prev    python  

(dst_target/src_target)/diff />cd changes
(dst_target/src_target)/diff /changes>ls
changed
only_on_dst (deleted)
only_on_src (created)
subdirectory_both
subdirectory_dst (deleted)
subdirectory_src (created)
unchanged

target-diff depends on fox-it/flow.record#107. To allow tests to run for this PR we've temporarily bumped flow.record to 3.15.dev10 in pyproject.toml

When three or more targets are provided, you can choose between treating every target as a 'delta' or compare every target against one 'absolute' target. Treating targets as 'deltas' is useful if you have multiple snapshots of the same target from different points in time. Treating targets as 'absolutes' can be useful in situations where you have a 'golden image' that you want to compare different targets against.

To keep code duplication low between tools/diff.py and tools/shell.py, this PR adds a superclass ExtendedCmd to shell.py that contains most of the functionality that is shared between the two. Both TargetCmd and DifferentialCli inherit from this class.

yunzheng

I did not review the whole PR, just added some small flow.record quality of life suggestions since the merge of fox-it/flow.record#115

Available in flow.record==3.15.

pyproject.toml

dissect/target/tools/diff.py

JSCU-CNI · 2024-06-12T10:34:32Z

Thanks for your suggestions on the flow.record context manager addition. We were wondering if there is an ETA on a review of the PR as a whole, considering it's been over 2 months.

JSCU-CNI · 2024-08-01T09:26:38Z

We understand this PR will take some time to review, but we do think it might be worthwhile to incorporate the changes to shell.py (the ExtendedCmd class, some fixes to target-shell, moving some functions outside of Cmd classes so they can be re-used) into main earlier. In its current state, this PR being left open for a long time causes it to increasingly divert from target-shell.

What we could do is move the shell.py changes into a seperate PR that you can then review first. We don't mind making this PR, and while we're at it we can also pick up some open issues along the way, such as #623, #624, #625 and #585. These improvements can be used for target-fs as well, as it only makes sense to re-use as much as possible between target-shell and target-fs when it comes to outputting target filesystem information. This will likely create a merge conflict with #716 if that is not yet merged, but if that ends up happening we'll incorporate those changes as well.

Do you agree with this approach?

Schamper · 2024-08-01T12:38:09Z

I am working slowly through the backlog of long outstanding PRs but had not gotten to this yet. Your proposal of splitting it up sounds like a good idea though, it's always nice if we can split of large PRs into separate smaller ones. I won't complain either if you pick up some of the mentioned improvements along the way 😄.

I've just merged #716 for your convenience so feel free to do with that as you please 😉.

* `--hex` can be used to diff binary files in a readable way. * `--only-changed` can be used to omit unchanged records when comparing plugin outputs

Fix a test and update ls and completedefault to how target-shell does it.

JSCU-CNI · 2024-09-16T09:02:42Z

Target-diff should be more or less on par with the changes made in #812 and should be ready for review @Schamper .

yunzheng · 2024-10-07T10:38:44Z

I will check if someone from the dissect team can review this.

cecinestpasunepipe · 2024-10-28T10:38:53Z

dissect/target/tools/diff.py

+        type=int,
+        help="How many bytes to compare before assuming a file is left unchanged (0 for no limit)",
+    )
+    subparsers = parser.add_subparsers(help="Mode for differentiating targets", dest="mode")


Suggested change

subparsers = parser.add_subparsers(help="Mode for differentiating targets", dest="mode")

subparsers = parser.add_subparsers(help="Mode for differentiating targets", dest="mode", required=True)

cecinestpasunepipe · 2024-10-28T10:39:59Z

dissect/target/tools/diff.py

+    shell_mode.add_argument("targets", metavar="TARGETS", nargs="*", help="Targets to differentiate between")
+
+    fs_mode = subparsers.add_parser("fs", help="Yield records about differences between target filesystems.")
+    fs_mode.add_argument("targets", metavar="TARGETS", nargs="*", help="Targets to differentiate between")


Suggested change

fs_mode.add_argument("targets", metavar="TARGETS", nargs="*", help="Targets to differentiate between")

fs_mode.add_argument("targets", metavar="TARGETS", nargs="+", help="Targets to differentiate between")

cecinestpasunepipe · 2024-10-28T10:40:13Z

dissect/target/tools/diff.py

+    subparsers = parser.add_subparsers(help="Mode for differentiating targets", dest="mode")
+
+    shell_mode = subparsers.add_parser("shell", help="Open an interactive shell to compare two or more targets.")
+    shell_mode.add_argument("targets", metavar="TARGETS", nargs="*", help="Targets to differentiate between")


Suggested change

shell_mode.add_argument("targets", metavar="TARGETS", nargs="*", help="Targets to differentiate between")

shell_mode.add_argument("targets", metavar="TARGETS", nargs="+", help="Targets to differentiate between")

cecinestpasunepipe · 2024-10-28T10:40:31Z

dissect/target/tools/diff.py

+    )
+
+    query_mode = subparsers.add_parser("query", help="Differentiate plugin outputs between two or more targets.")
+    query_mode.add_argument("targets", metavar="TARGETS", nargs="*", help="Targets to differentiate between")


Suggested change

query_mode.add_argument("targets", metavar="TARGETS", nargs="*", help="Targets to differentiate between")

query_mode.add_argument("targets", metavar="TARGETS", nargs="+", help="Targets to differentiate between")

to avoid errors when invoking target-diff without targets or mode

cecinestpasunepipe · 2024-10-28T10:45:35Z

dissect/target/tools/diff.py

+
+    args = parser.parse_args()
+    process_generic_arguments(args)
+


Suggested change

if len(args.targets) < 2:

print("At least two targets are required for diff.")

exit(1)

codecov · 2024-10-28T10:54:31Z

Codecov Report

Attention: Patch coverage is 70.28986% with 164 lines in your changes missing coverage. Please review.

Project coverage is 76.63%. Comparing base (6c672fa) to head (5436273).

Files with missing lines	Patch %	Lines
dissect/target/tools/diff.py	70.28%	164 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #664      +/-   ##
==========================================
- Coverage   76.75%   76.63%   -0.13%     
==========================================
  Files         315      316       +1     
  Lines       27126    27678     +552     
==========================================
+ Hits        20820    21210     +390     
- Misses       6306     6468     +162

Flag	Coverage Δ
unittests	`76.63% <70.28%> (-0.13%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

cecinestpasunepipe · 2024-10-28T14:43:12Z

dissect/target/tools/diff.py

+                exclude=args.exclude,
+            )
+        elif args.mode == "query":
+            iterator = differentiate_target_plugin_outputs(


Suggested change

iterator = differentiate_target_plugin_outputs(

if args.deep:

log.warning("--deep parameter ignored for query mode.")

iterator = differentiate_target_plugin_outputs(

cecinestpasunepipe · 2024-10-28T18:25:51Z

dissect/target/tools/diff.py

+        self._select_source_and_dest(0, 1)
+        if len(self.targets) > 2:
+            # Some help may be nice if you are diffing more than 2 targets at once
+            self.do_help(arg=None)


I think this line should be moved after the super().__init__() otherwise you get an error:
AttributeError: 'DifferentialCli' object has no attribute 'stdout'

cecinestpasunepipe · 2024-10-29T13:59:15Z

dissect/target/tools/diff.py

+                return self._write_entry_contents_to_stdout(entry.dst_target_entry, stdout)
+        print(f"File {name} not found.")
+        return False
+


Suggested change

@arg("path", nargs="?")

@alias("xxd")

def cmd_hexdump(self, args: argparse.Namespace, stdout: TextIO) -> bool:

setattr(args, "hex", True)

return self.cmd_diff(args, stdout)

cecinestpasunepipe · 2024-10-29T14:47:43Z

dissect/target/tools/diff.py

+    """Given a list of targets, compare targets against one another and yield File[Created|Modified|Deleted]Records
+    indicating the differences between them."""
+    if len(targets) < 2:
+        raise ValueError("Provide two or more targets to differentiate between.")


this is true for all modes right? maybe move it to a more generic place? - see my relevant corresponding suggestion...

cecinestpasunepipe · 2024-10-31T10:20:48Z

dissect/target/tools/diff.py

+        type=int,
+        help="How many bytes to compare before assuming a file is left unchanged (0 for no limit)",
+    )
+    subparsers = parser.add_subparsers(help="Mode for differentiating targets", dest="mode")


might be an interesting idea to split the subparsers into multiple different functions for readability...

yunzheng requested changes May 16, 2024

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

dissect/target/tools/diff.py Outdated Show resolved Hide resolved

dissect/target/tools/diff.py Outdated Show resolved Hide resolved

JSCU-CNI force-pushed the feature/add-target-diff branch 2 times, most recently from 131e6b5 to aac31af Compare June 12, 2024 09:02

JSCU-CNI mentioned this pull request Aug 5, 2024

Add target-shell history #786

Merged

EinatFox mentioned this pull request Aug 6, 2024

Add target-diff PR#664 #795

Open

EinatFox linked an issue Aug 6, 2024 that may be closed by this pull request

Add target-diff PR#664 #795

Open

JSCU-CNI mentioned this pull request Aug 8, 2024

Refactor target-shell #812

Merged

JSCU-CNI added 4 commits August 20, 2024 11:32

Add target-diff

bb810f4

Use ignore_fields_for_comparison context manager

283490e

Add --hex and --only-changed

a06b16e

* `--hex` can be used to diff binary files in a readable way. * `--only-changed` can be used to omit unchanged records when comparing plugin outputs

fix target-diff with changes from main

8a0c6af

JSCU-CNI force-pushed the feature/add-target-diff branch from 7f1b27d to 8a0c6af Compare August 20, 2024 10:17

JSCU-CNI added 3 commits August 20, 2024 12:18

Merge branch 'main' into feature/add-target-diff

f51572b

Merge branch 'main' into feature/add-target-diff

22396aa

Small fixes

59c9652

Fix a test and update ls and completedefault to how target-shell does it.

Merge branch 'main' into feature/add-target-diff

2c2a759

JSCU-CNI requested a review from yunzheng October 7, 2024 09:56

yunzheng removed their request for review October 7, 2024 10:38

JSCU-CNI added 2 commits October 21, 2024 10:43

Merge branch 'main' into feature/add-target-diff

8342523

Merge branch 'main' into feature/add-target-diff

5436273

cecinestpasunepipe self-requested a review October 24, 2024 14:40

cecinestpasunepipe self-assigned this Oct 24, 2024

cecinestpasunepipe requested changes Oct 28, 2024

View reviewed changes

cecinestpasunepipe requested changes Oct 29, 2024

View reviewed changes

cecinestpasunepipe reviewed Oct 29, 2024

View reviewed changes

cecinestpasunepipe reviewed Oct 31, 2024

View reviewed changes

cecinestpasunepipe assigned JSCU-CNI and unassigned cecinestpasunepipe Oct 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add target-diff #664

Add target-diff #664

JSCU-CNI commented Apr 4, 2024

yunzheng left a comment

JSCU-CNI commented Jun 12, 2024

JSCU-CNI commented Aug 1, 2024

Schamper commented Aug 1, 2024

JSCU-CNI commented Sep 16, 2024

yunzheng commented Oct 7, 2024

cecinestpasunepipe Oct 28, 2024

cecinestpasunepipe Oct 28, 2024

cecinestpasunepipe Oct 28, 2024

cecinestpasunepipe Oct 28, 2024

cecinestpasunepipe Oct 28, 2024

cecinestpasunepipe Oct 28, 2024

codecov bot commented Oct 28, 2024 •

edited

Loading

cecinestpasunepipe Oct 28, 2024

cecinestpasunepipe Oct 28, 2024

cecinestpasunepipe Oct 29, 2024

cecinestpasunepipe Oct 29, 2024

cecinestpasunepipe Oct 31, 2024

	subparsers = parser.add_subparsers(help="Mode for differentiating targets", dest="mode")
	subparsers = parser.add_subparsers(help="Mode for differentiating targets", dest="mode", required=True)

	fs_mode.add_argument("targets", metavar="TARGETS", nargs="*", help="Targets to differentiate between")
	fs_mode.add_argument("targets", metavar="TARGETS", nargs="+", help="Targets to differentiate between")

	shell_mode.add_argument("targets", metavar="TARGETS", nargs="*", help="Targets to differentiate between")
	shell_mode.add_argument("targets", metavar="TARGETS", nargs="+", help="Targets to differentiate between")

	query_mode.add_argument("targets", metavar="TARGETS", nargs="*", help="Targets to differentiate between")
	query_mode.add_argument("targets", metavar="TARGETS", nargs="+", help="Targets to differentiate between")

+if len(args.targets) < 2:
+        print("At least two targets are required for diff.")
+        exit(1)

+    @arg("path", nargs="?")
+    @alias("xxd")
+    def cmd_hexdump(self, args: argparse.Namespace, stdout: TextIO) -> bool:
+        setattr(args, "hex", True)
+        return self.cmd_diff(args, stdout)

Add target-diff #664

Are you sure you want to change the base?

Add target-diff #664

Conversation

JSCU-CNI commented Apr 4, 2024

yunzheng left a comment

Choose a reason for hiding this comment

JSCU-CNI commented Jun 12, 2024

JSCU-CNI commented Aug 1, 2024

Schamper commented Aug 1, 2024

JSCU-CNI commented Sep 16, 2024

yunzheng commented Oct 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Oct 28, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Oct 28, 2024 •

edited

Loading