SBOM generation is missing a few Python packages listed in the requirements.txt file

[gobiltd]

What happened:

SBOM generated for python is missing few packages listed in the requirements.txt file

What you expected to happen:

SBOM should list all packages mentioned in the requirements.txt file

Steps to reproduce the issue:

1. Create requirements.txt with below content and run the syft scan for generate SBOM.

PySocks==1.7.1
coloredlogs>=7.3.1
click>=7.0
verboselogs>=1.7
dateparser>=0.7.0

urllib3==1.26.17
requests==2.25.1
retrying==1.3.3
strk-identity-auth==2.49
nwauto-python-commons==0.1.93
flask==2.0.3


wheel>=0.38.1

nose>=1.0
pytz>=2016.10
pytest==3.0.1
computec>=1.0.169
Werkzeug==2.2.2

2. Run the Syft

   /usr/local/bin/syft /scratch/app/gobgovin/jars/python-requirements/ -o syft-json=syft_python_bom_test.json
   ✔ Indexed file system /scratch/app/gobgovin/jars/python-requirements
   ✔ Cataloged contents beb33833a4b4aaabf7dde249b8713668b3b495f08e9619455f643cabe6be8c42
   ├── ✔ Packages [9 packages]
   └── ✔ Executables [0 executables]

Identified only 9 packages, 7 packages are missing.
ex : verboselog:1.7, dateparser:0.7.0, etc (These are some of missing packages to identify by sfyt)

Anything else we need to know?:
Syft is parsing only {package-name}=={version} format only, remaining all other formats are not parsing and listing in SBOM from requirements.txt . (ex: {package-name}>={version})
Please refer page, few other formats also supported by python requirements.txt file - https://packaging.python.org/en/latest/specifications/version-specifiers/#version-specifiers

Environment:

• Output of syft version:
  Application: syft
  Version: 1.4.1
  BuildDate: 2024-05-09T19:45:46Z
  GitCommit: c200896
  GitDescription: v1.4.1
  Platform: linux/amd64
  GoVersion: go1.21.9
  Compiler: gc

• OS (e.g: cat /etc/os-release or similar):
  NAME="Oracle Linux Server"
  VERSION="7.9"

[gliptak]

https://github.com/anchore/syft/blob/main/syft/pkg/cataloger/python/parse_requirements.go#L198 specifically drops ranges

[kzantow]

Hi @gobiltd, because these requirements are not exact, but only minimums Syft by default is not including these. There is an option to enable Syft to "guess" version ranges: SYFT_PYTHON_GUESS_UNPINNED_REQUIREMENTS=true, e.g.

SYFT_PYTHON_GUESS_UNPINNED_REQUIREMENTS=true syft requirements.txt

We can revisit whether this should be enabled by default.

[wagoodman]

I think the most surprising thing here (from a user perspective) is that there are package names missing. We chatted about this in this weeks gardening live stream and came up with a few options (all variants include all packages from the requirements.txt, but vary in how the version field is handled):

• allow for showing ranges as the version ( >= 4.3). This would work but would most likely confuse downstream tooling entirely, so isn't something we'd like to do first.
• Always show UNKNOWN letting the soon to be merged Add compliance policy for empty name and version #3257 handle the case
• Show NOASSERTION

But another aspect from this is whatever we decide here, we should identify other ecosystem parsers that also have version ranges specified and add the same configuration and behavior (such as package.json for npm).

[willmurphyscode]

Notes for implementation:

The Python cataloger reading a requirements.txt is dropping lines that don't pin to a specific version because, historically, without an exact name and version Syft would drop the package. Now the Python cataloger ought to raise up packages with "" version and let the compliance config (e.g. SYFT_COMPLIANCE_MISSING_VERSION=stub) deal with the packages.

@gobiltd as a workaround, if you set SYFT_PYTHON_GUESS_UNPINNED_REQUIREMENTS=true in the environment, Syft will guess a version for requirements.txt entries that don't have an exact version specified, and then these packages will show up normally.