-
Notifications
You must be signed in to change notification settings - Fork 723
Use file content heuristics to decide file reader. #1962
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Dimi1010
wants to merge
69
commits into
seladb:dev
Choose a base branch
from
Dimi1010:feature/heuristic-file-selection
base: dev
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 58 commits
Commits
Show all changes
69 commits
Select commit
Hold shift + click to select a range
02de760
Added heuristics file content detector that determines the content ba…
Dimi1010 d2b6339
Merge remote-tracking branch 'upstream/dev' into feature/heuristic-fi…
Dimi1010 685dd9f
Moved stream checkpoint outside format detector as it is not directly…
Dimi1010 40dee69
Added a new factory function `createReader` that uses the new heurist…
Dimi1010 f1e3e18
Add <algorithm> include.
Dimi1010 8da1790
Added unit tests.
Dimi1010 3ad51e2
Deprecated old factory function.
Dimi1010 15c2000
Add byte-swapped zstd magic number.
Dimi1010 17af8d4
Lint
Dimi1010 46418ec
Move enum closer to first usage.
Dimi1010 3d713ab
Added unit tests for file reader device factory.
Dimi1010 a2391ec
Revert indentation.
Dimi1010 ea328d7
Fixed StreamCheckpoint to restore original stream state.
Dimi1010 db86c3e
Merge branch 'dev' into feature/heuristic-file-selection
Dimi1010 4aed9bd
Merge branch 'dev' into feature/heuristic-file-selection
Dimi1010 a83ae2b
Moved isStreamSeekable helper to inside `CaptureFileFormatDetector`.
Dimi1010 916e872
Added pcap magic number for Alexey Kuznetzov's modified pcap format.
Dimi1010 022529f
Merge remote-tracking branch 'upstream/dev' into feature/heuristic-fi…
Dimi1010 169fcd2
Split the unit test into multiple smaller tests.
Dimi1010 db8c848
Merge branch 'dev' into feature/heuristic-file-selection
Dimi1010 3e74912
Merge remote-tracking branch 'upstream/dev' into feature/heuristic-fi…
Dimi1010 f1613c4
Added helper to indicate if ZstSupport is enabled for PcapNg devices.
Dimi1010 bc2bacd
Split pcap microsecond and nanosecond file heuristics tests.
Dimi1010 58ac45d
Skipping Zst test case if zst is not supported.
Dimi1010 3b4b5ad
Due to file heuristics returning PcapNG format on Zstd archive, if Zs…
Dimi1010 18379b4
Lint
Dimi1010 8a4f6f8
Added invalid device factory to pcap tag.
Dimi1010 7776e0e
Updated static zst archives to be actual archives.
Dimi1010 4f52f59
Centralized PTF test name width under a macro.
Dimi1010 88ebfff
Add Pcap++Test header files to test sources for IDE tooling.
Dimi1010 41fe188
Fixed test output formatting.
Dimi1010 c8ae4f8
Lint
Dimi1010 c7cab2b
Typo fix.
Dimi1010 6d55077
Merge remote-tracking branch 'upstream/dev' into feature/heuristic-fi…
Dimi1010 682eeac
Shortened test names.
Dimi1010 07804da
Simplified invalid file test.
Dimi1010 9c4fc08
Simplified ZST tests.
Dimi1010 d975157
Added snoop test.
Dimi1010 40530df
Expanded granularity of file format detection.
Dimi1010 96a61b2
Marked `checkSupport` functions as constexpr to enable compile time o…
Dimi1010 55a6b7a
Exclude json from pre-commit cppcheck as it is slow due to many defin…
Dimi1010 3ab14e7
Lint
Dimi1010 5dd9a30
Fix runtime side effects inside constexpr function.
Dimi1010 45ad769
Added a secondary factory function to separate mixed error handling m…
Dimi1010 d24a9ad
Revert deprecation message, as doxygen is unhappy.
Dimi1010 f5ff879
Update tests.
Dimi1010 2c1b2c4
Update deprecation warning to point to the function closer to the sig…
Dimi1010 8d1ed1d
Catch general exception instead of runtime error.
Dimi1010 0ea2da9
Shortened deprecation message due to pre-commit warnings when its is …
Dimi1010 c209c90
Fix braces.
Dimi1010 8d77aa0
Simplfy test.
Dimi1010 af12d2f
Added tests for createReader failures.
Dimi1010 b357087
Merge branch 'dev' into feature/heuristic-file-selection
Dimi1010 443c883
Simplified pcap detection to not require to read the entire pcap header.
Dimi1010 202d5cc
Added const qualifiers to detector methods.
Dimi1010 e6b2aa9
Added dedicated unit tests for CaptureFileFormatDetector.
Dimi1010 b8fb635
Added more tests for `createReader`.
Dimi1010 c6c7720
Add static assert for array indice checks.
Dimi1010 181a8b4
Updated detectPcap selection.
Dimi1010 76aa850
Merge branch 'dev' into feature/heuristic-file-selection
Dimi1010 d8f7419
Extracted capture format detector to remove it from publicly availabl…
Dimi1010 91a7a0a
Fix includes.
Dimi1010 e275950
Removed duplicate files from tracking.
Dimi1010 e7a42b5
Lint
Dimi1010 54f7bae
Trimmed pcapng sample.
Dimi1010 93cba3d
Merge branch 'dev' into feature/heuristic-file-selection
Dimi1010 3643fac
Change PcapNGZst to ZstArchive. Zst to PcapNG branch folding is done …
Dimi1010 ec5980f
Added separate format value for "modified" pcap to separate the forma…
Dimi1010 b3639a9
Docs fix.
Dimi1010 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If I'm not mistaken, this used to be in the
.cppfile, right? Is the reason we moved it to the.hfile is to make it easier to test?If yes, I think we can test it using
createReader()- create a temporary fake file with the data we want to test, and delete it when the test is doneThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried that suggestion initially, but it would have been an extremely fragile unit test. The "pass" conditions would have been checked indirectly.
Also,
createReaderhas multiple return paths for Nano / Zst file formats, which would have caused complications since the format test would have needed to care about the environment it runs at, which it doesn't have to as a standalone.Any additional changes to
createReadercould also break the test, which they really shouldn't. For example, I am thinking of maybe adding additional logic for Zst archive to check if the compressed data is actually a pcapng, and not a random file. This would be a nightmare to make compatible with the "spoofed files" test due to assumptions on the test thatcreateReaderdoesn't do anything more complicated than check the initial magic number.So, in the end, you end up with a more compilcated unit test to read through that:
createReaderfactory, too.createReaderas it uses its behavior to testdetectFormat.Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I understand it's better to test
CaptureFileFormatDetectoras a standalone class, but it requires exposing it in the.hfile which is not great (even though it's in theinternalnamespace). TestingcreateReaderis a bit more fragile, but I don't think the difference is that big. Of course, if we add logic to detect more file types or update the existing detection logic some tests might break, but we easily fix them as needed.I usually try to avoid the
internalnamespace where possible because it's still in the.hfile and is exposed to users, and we'd like to keep our API as clean as possibleThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is a big difference and it's not always an easy fix. I plan to add the aforementioned Zst checks in another PR after this one, and that would make zst spoofing in
createReaderimpossible, due to zst format automatically being checked for PcapNg or Unknown contents. Therefor you can't rely on the return ofcreateReaderto find out what the return ofdetectFormatwas, becausenullptrcan be returned from several paths fromdetectFormatreturn value (Unknown, Nano + unsupported, Zst + unsupported). We have already had issues with tests being silently broken (#1977 comes to mind), so I would prefer to avoid fragile tests if we can.Fair, it is exposed, but the that is the entire reason of having the
internalnamespace. It is a common convention that external users shouldn't really touch it. If you want to keep the primary public header files clean there are a couple options:internal/detailin their public include folder, where they keep all their internal code headers that need to be exposed. That keeps the "internal" code separate from the "public" code, if users want to read through the headers. This is a common convention used in Boost libraries. "public" headers that depend on internal headers include them from theinternalsubfolder.CaptureFileFormatDetectoris only needed in thecpppart and not in the header part, we can extract it to a fully internal header, kept with the source files. This would prevent it from being exposed in the public API, but the Test project can be manually set to search for headers from "Pcap++/src" too, to allow it to link in the tests.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure I understand... if we create fake files we know which type to expect, so all the test needs to do is verify the created file device is of the expected type 🤔
I guess we can do that, but I still don't understand why we can't test it with
createReaderortryCreateReaderUh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With what the current iteration of the detector is, either works, tbh. The initial implementation, which was added to the
.cppbecause a lot of the business logic of the factory was intermixed with the detector. (e.g. Zst archive -> PcapNG folding,PcapandPcapNanobeing justPcap). That is no longer the case.But sure, it was extracted due to the requirements for unit tests for every magic number, which would be much easier to maintain in the long run if done directly on the format detector. For one, it avoids the filesystem, which IMO is always more trouble than its worth if it can be avoided relatively trivially. For another, it is a technical debt on expanding
createReadervalidation logic.The good reason I have is that this unit test through
createReaderwill need to be changed literally in the next PR I plan to make after this one , to keep it running even though I don't plan to touch the format detector code.The planned changes in validation being:
createReaderand checking the format of the archived file. This will essentially brick any spoofed ZST file, as it will not be able to be unpacked, fail factory validation and returnnullptr.open()/close()be called inside the factory prior to device return to run secondary validation that the reader can actually be opened. File devices can't be retargeted so no point in returning a reader that will just fail to open when the user tries, IMO. This will essentially brick all spoofed files since they can't be opened by the device by definition.If you insist on having it done through
createReaderthen fine, but that solution opens up more work for future changes that I have planned around with the current one.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think this piece of logic will change much because I don't expect more file types to be added (and even if we will, it only happens rarely), so maintenance in the long run shouldn't be an issue. For the same reason I don't think we'll expand
createReadermuchI don't think we want to unpack the Zstd archive just to see if it's valid. We have the pacpng library that does that. The logic in
createReaderdoes an educated guess, not a bullet-proof validation. Otherwise we can argue that checking the magic numbers is not enough - why not validate the entire pcap / pcapng file? Of course we don't want to do that because libpcap is doing it for us. The same should apply for ZstdAs mentioned earlier, I don't think it's the approach we want.
createReadershould do an educated guess, nothing moreAgain, I don't think this logic will change much after this refactoring. Even if it will, I don't think it'll be a huge refactoring
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Curious why wouldn't you want that?
I would reason that having a better validation sequence inside the factory would make for cleaner UX due to less boilerplate needed by the user?
Having the integrated open / validation would allow single line, before use. I don't see much use cases where you would want to create a device reader and not open to read from it, no? It also fits with the RAII methodology of avoiding a 2-stage init where possible.
The live devices need
open()because they are created by the runtime at startup.File devices don't need to have that limitation since they are entirely created by the user.
Yes, but I am unsure if it gives a precise error message of what went wrong or just a generic failure error.
Which is as simple as calling
open()inside the factory function, no? As you said, the backend already does validation, so why not reuse it for the factory validation?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Extracting the archived file just to verify the format is wasteful and might take a long time, especially if it's called on large files. Also - for these large files it'd mean the file is extracted twice - once in
createReaderand again when actually reading the fileIf this is indeed the case, maybe we need to fix the LightPcapNg code?
Not necessarily - as far as I know
open()checks mostly the header and doesn't go over the rest of the file, so a user can open a file with a correct header but corrupted data and reading the file will failUh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is no need to extract the entire file. ZST compression works on independent frames allowing frame-by-frame (streaming) decompression. We only need to decompress the first frame to read the magic number to validate that the archive contents appear to be PcapNG. How large the total file is is irrelevant.
Incidentally, frame-by-frame is also how LightPcapNG reads the ZST archive. It decompresses a frame, reads the fully decompressed PcapNG records in it, and decompresses the next frame if needed.
Which is C code, and makes it much harder to output a readable error. Not to mention we need to deal with passing that error up the stack.
But it will still have passed
open(). My idea isn't thatcreateReadershould validate that everything is correct. It is that it should validate just enough to guarantee that the returned device can successfully pass anopen()call. The device might even be returned already opened and ready for reading, reducing the user side boilerplate.There is no reason to return a device that can't even be opened, since the user can't do anything with it. It just adds more boilerplate as the user has to do the error handling twice.
If the records afterwards are corrupted at some point, the read should fail when the corrupted data is reached.