v0.0.0rc16

This release is the first to include the newly released CyTRICS Schema v1.0.1, previously only alluded to by the output Surfactant produced. It also includes fixes for a silent crash on files with permission errors, is able to handle more complex file content patterns from retire.js and emba data sources, and updates documentation in a few places.

What's Changed

• feat: add CyTRICS Schema v1.0.1 by @nightlark in #500
• feat: Enable more complex regexes to be parsed by @nightlark in #539
• feat: add SBOM vis context menu to pin and delete nodes by @JosephHAtWork in #516
• fix: silent crash on files with permission errors by @JosephHAtWork in #513
• docs: add replace_dst function docstring by @CoWeAtWork in #530
• docs: Update elf_relationship.py docstrings by @CoWeAtWork in #530
• docs: update dapper dataset install instructions by @nightlark in #521
• ci: Add workflow for publishing plugins to PyPI with trusted publisher authentication by @Copilot in #540
• Add Copilot instructions for repository configuration by @Copilot in #533
• build(deps): bump cyclonedx-python-lib from 11.1.0 to 11.2.0 by @dependabot[bot] in #518
• build(deps): bump cyclonedx-python-lib from 11.2.0 to 11.4.0 by @dependabot[bot] in #524
• build(deps): bump cyclonedx-python-lib from 11.4.0 to 11.5.0 by @dependabot[bot] in #528
• build(deps): bump python-msi from 0.0.0a2 to 0.0.0a3 by @dependabot[bot] in #525
• build(deps): bump lief from 0.17.0 to 0.17.1 by @dependabot[bot] in #522

New Contributors

• @CoWeAtWork made their first contribution in #520
• @Copilot made their first contribution in #533

Full Changelog: v0.0.0rc15...v0.0.0rc16

v0.0.0rc15

This release fixes several bugs including an issue with Java class files being misidentified as Mach-O files, parsing pe files with bad headers, and a crash due to lief changing the format of a data type they return. Several new features of note are the addition of a tab in the TUI for changing plugin settings, and the initial versions of the SBOM visualization plugin, and the DAPper plugin which leverages the DAPper datasets to identify potential packages that could have installed a given file.

What's Changed

• refactor(test_grype): simplify install_grype() to use Anchore endpoint by @willis89pr in #475
• fix: magic byte file type id for macho also being identified as a javaclass file by @nightlark in #486
• fix: store mach-o signature content as hex string instead of lief memoryview by @nightlark in #487
• fix: pe_file parsing error with missing headers by @wangmot in #485
• feat: Add ContextEntry to Identifying File Types by @matthewkelley22 in #482
• feat: TUI Plugin Settings by @KendallHarterAtWork in #451
• feat: SBOM Visualization Plugin by @JosephHAtWork in #478
• fix: Error handling for msi files with missing .cab files and some other OLE-specific errors by @wangmot in #501
• feat: SBOMVis UI improvements by @JosephHAtWork in #490
• testing: Add TUI tests by @KendallHarterAtWork in #495
• fix: add check to ensure input_sbom for generate command has "read" attribute by @nightlark in #504
• feat: Add DAPper plugin by @monwen in #499

New Contributors

• @monwen made their first contribution in #499

Full Changelog: v0.0.0rc14...v0.0.0rc15

BlackHat USA 2025 Demo Bugfix Release

13 was an unlucky number! But actually, there were two pretty major bugs in the previous release (one of which was uncovered by fixing the first bug). The grype plugin issue was just blocking CI tests.

What's Changed

• fix: missing section in grype plugin pyproject.toml by @nightlark in #473
• fix: id magic returning empty list blocked other id plugins by @nightlark in #472
• fix: ensure infoextractor plugins get an empty list for the ftype instead of None by @nightlark in #472
• build(deps): update textual requirement from ==4.* to ==5.* by @dependabot[bot] in #465

Full Changelog: v0.0.0rc13...v0.0.0rc14

BlackHat USA 2025 Demo Release

This release updates the minimum required Python version to 3.9, fixes some crashes, adds a number of new features including support for extracting metadata from RPM packages, along with the following BREAKING CHANGES:

• dropped support for Python 3.8
• the config_tui subcommand has been removed; its functionality has been included as part of the tui subcommand
• the createconfig subcommand has been removed; the tui provides the ability to create and modify context files in a more user friendly way
• specimen config has been renamed to specimen context: the special config: prefix has been changed to context:, and the terminology has been updated in several other places

What's Changed

• chore!: drop support for obsolete Python 3.8 by @nightlark in #441
• fix!: remove deprecated setup.py support for old pip versions by @nightlark in #440
• ci: add concurrency groups to GitHub Actions workflows by @willis89pr in #442
• feat: Added RPM Package Recognition by @matthewkelley22 in #443
• refactor: relationships to use an internal graph representation by @willis89pr in #433
• fix: add check before old SBOM relationship UUID node deletion by @JosephHAtWork in #449
• fix: TUI context config file output install_prefix instead of installPrefix by @KendallHarterAtWork in #458
• ci: add regression test for all samples in tests/data by @WorkingRobot in #445
• feat: change filetype from type id plugins from str to list[str] for polyglot detection by @wangmot in #434
• fix: crash when dnfile object returned is missing expected attributes by @wangmot in #463
• feat: RPM Package Metadata Extraction by @matthewkelley22 in #456
• ci: fix regression test non-deterministic output and commenting on PRs from forks by @WorkingRobot in #466
• refactor!: rename specimen config to specimen context by @nightlark in #467
• feat!: remove config_tui subcommand, replaced by tui subcommand by @nightlark in #468
• feat!: remove createconfig subcommand, superceded by TUI for modifying context files by @nightlark in #469
• feat: allow extraction persistence and cache extractions of unsuccessful runs by @WorkingRobot in #457
• build(deps): bump cyclonedx-python-lib from 10.3.0 to 10.4.1 by @dependabot[bot] in #453
• build(deps): bump cyclonedx-python-lib from 10.4.1 to 11.0.0 by @dependabot[bot] in #461
• build(deps): update textual requirement from ==3.* to >=3,<5 by @dependabot[bot] in #454
• build(deps): update dnfile requirement from ==0.15.* to ==0.16.* by @dependabot[bot] in #455

New Contributors

• @matthewkelley22 made their first contribution in #443

Full Changelog: v0.0.0rc12...v0.0.0rc13

Final release supporting Python 3.8

This is the final release on PyPI that will support Python 3.8 -- after this, the minimum version of Python will be bumped up to 3.9+

Other notable changes in this release include:

• Pattern databases can be automatically updated independent of Surfactant releases (run surfactant plugin update-db --all)
• Support for decompressing several file types including various tar archives, rar files (via unrar), and MSI installers
• Support for getting info from U-Boot/uImage file headers
• Improved testing of optional plugins
• Fixed several crashes
• Made the core Surfactant package pure Python by making a dependency optional that had resulted in a transitive dependency on a compiled library

BREAKING CHANGES

• The web ui subcommand has been removed; use the tui (terminal UI) instead, which provides a better user experience than the web UI was able to
• jar file info gathering support now requires installing the "java" extra dependencies (pip install surfactant[java])

What's Changed

• add parens in ftype condition by @wangmot in #357
• Bump cyclonedx-python-lib from 8.8.0 to 9.1.0 by @dependabot in #358
• Update textual requirement from ==1.* to 2.* by @dependabot in #352
• Refactor: Centralize Database Management with BaseDatabaseManager and Migrate Derived Managers by @willis89pr in #341
• Add pytest ignore option for plugins directory in pyproject.toml by @willis89pr in #362
• File decompression by @wangmot in #360
• docs: update specimen configuration and ContextEntry options by @nightlark in #370
• ci: set default workflow permissions by @nightlark in #372
• refactor: rename config.py to context.py by @nightlark in #371
• feat: Add support for (legacy) U-Boot/uImage file headers by @nightlark in #363
• feats: remove limitation on gzip file type recognition, and support decompressing non-tar gzip, bzip2, and xz files by @nightlark in #373
• feat: support plugin hints for software entry fields by @nightlark in #374
• add current_context variable to extract_file_info hook by @wangmot in #376
• CI Workflow for 3rd-Party Plugins by @willis89pr in #353
• Port web-based UI to a Textual TUI by @KendallHarterAtWork in #332
• add context entry for archives by @wangmot in #379
• add filetype identification to archive files by @wangmot in #381
• build(deps): update textual requirement from ==2.* to >=2,<4 by @dependabot in #380
• More error checking for existing paths by @KendallHarterAtWork in #383
• Container Prefix support by @KendallHarterAtWork in #384
• feat: recognize native lib patterns in U-Boot files by @nightlark in #386
• build(deps): bump lief from 0.16.4 to 0.16.5 by @dependabot in #390
• TUI improvements by @KendallHarterAtWork in #392
• feat: Add fast pattern matching using AhoCorasick searching by @nightlark in #388
• fix: vendor field being set to a list of list of strings (vendors) by @willis89pr in #402
• Add more recognized file formats by @KendallHarterAtWork in #397
• Gzip garbage bytes by @wangmot in #405
• Database URL Hosting and Overriding by @willis89pr in #378
• build(deps): bump requests from 2.32.3 to 2.32.4 in /docs by @dependabot in #418
• Fix crash when ID'ing small files by @JosephHAtWork in #414
• Update: angrimportfinder plugin fixes by @T0pAc3 in #416
• docs: remove LC link in Usage section in the README by @lfquintaz in #424
• add RAR filetype id and extraction by @wangmot in #387
• Add .msi extraction support by @WorkingRobot in #412
• fix: bump minimum requires-python to 3.8.1 by @nightlark in #429
• chore: update pyproject.toml to install pymsi from PyPI by @nightlark in #431
• fix!: remove transitive dependency on compiled ct3 module by making javatools optional by @nightlark in #430
• chore!: remove web ui command by @nightlark in #432
• build(deps): bump python-msi from 0.0.0a0 to 0.0.0a2 by @dependabot in #436
• Add support for ReadTheDocs-hosted database_sources.toml by @willis89pr in #419
• fix: use project.license SPDX expression instead of deprecated license classifiers by @nightlark in #439

New Contributors

• @T0pAc3 made their first contribution in #416
• @lfquintaz made their first contribution in #424
• @WorkingRobot made their first contribution in #412

Full Changelog: v0.0.0rc11...v0.0.0rc12

Getting ready for v0!

If you encounter issues updating Surfactant due to dependencies, try upgrading the dependencies in your virtual environment with the --upgrade --force-reinstall pip options.

BREAKING CHANGE

• Including all files (for the given extractPaths) in the SBOM is now the default behavior, even if Surfactant doesn't recognize the file type. The include all files options have therefore been removed, and replaced with an omit unrecognized types option. To revert to the old behavior the --omit_unrecognized_types command line argument for a single run of Surfactant, or set the core.omit_unrecognized_types surfactant config option to true to use the old behavior for all Surfactant runs. In specimen configuration files, entries now support omitUnrecognizedTypes to apply the old behavior to only a particular group of extractPaths.

What's Changed

• Generate API pages for documentation website by @Moses-Mk in #275
• Added plugin list subcommand functionality by @willis89pr in #279
• Added enable and disable to plugins command by @willis89pr in #281
• Add plugin install and uninstall commands by @willis89pr in #283
• Update README.md plugins section by @willis89pr in #286
• Add dependency groups for dev, test, and docs tooling by @nightlark in #292
• plugin update-db by @willis89pr in #288
• Add include/exclude extensions options by @KendallHarterAtWork in #262
• Update configmanager.py to expand user directory in returned paths by @willis89pr in #304
• Native library detection plugin by @wangmot in #267
• Updatejsdb by @willis89pr in #298
• Remove ref name entry from .git_archival.txt by @nightlark in #308
• Add zlib file identification by @KendallHarterAtWork in #313
• Update PE subsystem type and machine type enums by @nightlark in #297
• Recognize Perl Scripts by @Eriken79 in #316
• Add initialization hook to plugin system by @willis89pr in #318
• Add options to merge command for controlling creation of system entry and type of relationships by @nightlark in #201
• Add information to docs showing overview of how Surfactant works by @nightlark in #321
• Refactor and Relocate Native Library Pattern Loading by @willis89pr in #320
• Refactor and Enhance Docker Scout Integration by @willis89pr in #322
• Add check for section header string table in ELF infoextractor by @JosephHAtWork in #317
• fix: crash when checking zlib magic bytes on empty or 1-byte files by @nightlark in #328
• feat: Add configuration option to enable/disable Docker Scout by @willis89pr in #326
• Add TUI for creating/modifying config files. by @KendallHarterAtWork in #287
• feat: Enhance JS database management with hashing and timestamp functionality by @willis89pr in #327
• Adding basic framework for the CLI, no changes to old cli. by @shaynakapadia in #261
• Refactor TOML file handling and improve code clarity by @willis89pr in #339
• Native Database Version Tracking by @willis89pr in #340
• feat: Generate SBOM for a single file by @nightlark in #330
• docs: Add core.include_all_files to list of settings by @nightlark in #346
• fix: Specimen config includeAllFiles option being ignored by @nightlark in #347
• Rename include all files to omit unrecognized types by @nightlark in #348
• fix: SPDX validation failure when no packages are present by @nightlark in #349
• Bump cyclonedx-python-lib from 7.6.2 to 8.5.0 by @dependabot in #284
• Bump cyclonedx-python-lib from 8.5.0 to 8.5.1 by @dependabot in #334
• Bump cyclonedx-python-lib from 8.5.1 to 8.7.0 by @dependabot in #344
• Bump cyclonedx-python-lib from 8.7.0 to 8.8.0 by @dependabot in #351
• Bump lief from 0.15.1 to 0.16.2 by @dependabot in #314
• Bump lief from 0.16.2 to 0.16.3 by @dependabot in #333
• Bump lief from 0.16.3 to 0.16.4 by @dependabot in #355
• Update textual requirement from ==0.88.* to >=0.88,<1.1 by @dependabot in #335
• Update pyelftools requirement from ==0.31.* to >=0.31,<0.33 by @dependabot in #354

New Contributors

• @Moses-Mk made their first contribution in #275
• @willis89pr made their first contribution in #279
• @JosephHAtWork made their first contribution in #317

Full Changelog: v0.0.0rc10...v0.0.0rc11

Update for SecTor 2024

Updated release for SecTor 2024 demo, with a list of the more interesting changes since BHUSA release below. Mach-O file info extraction is the big one -- it's an optional feature that can be installed with pip install surfactant[macho] (note: uses lief as a dependency, which only releases binary wheels for currently supported Python versions... older versions of lief don't retroactively get compiled wheels for new Python versions either).

What's Changed

• Sourcetrail output by @KendallHarterAtWork in #236
• added .bat extension by @Eriken79 in #252
• Bump pefile from 2023.2.7 to 2024.8.26 by @dependabot in #253
• Add option to include all files by @KendallHarterAtWork in #246
• Bump cyclonedx-python-lib from 7.6.1 to 7.6.2 by @dependabot in #263
• CYT 341 Add basic Mach-O file info extractor by @Czatar in #184
• Add support for extract paths being files by @nightlark in #244
• Fix issues identified by mypy by @nightlark in #265

New Contributors

• @Eriken79 made their first contribution in #252

Full Changelog: v0.0.0rc8...v0.0.0rc10

Updated release for BH USA demo

v0.0.0rc8

Add support for user config file for plugin options (#231)