Skip to content

Latest commit

 

History

History
1067 lines (722 loc) · 38.5 KB

CHANGELOG.md

File metadata and controls

1067 lines (722 loc) · 38.5 KB
Raw

What's New

Thanks to all our contributors, users, and the many people that make detect-secrets possible! :heart:

If you love detect-secrets, please star our project on GitHub to show your support! ⭐

v1.5.0

May 6th, 2024

We apologise for the extreme delay in publishing a new release for our beloved detect-secrets. We at Yelp appreciate your continued support and your contributions to this valuable project!

📰 News

  • We're adding support for Python 3.10, 3.11 and 3.12 and we dropped support for Python 3.6 and 3.7! We hope this won't be too disruptive for you all. Be aware that in a next release, we'll remove support for Python 3.8 too, as it'll reach EOL in October 2024.

📣 Release Highlights

  • Added support for OS-agnostic baseline files (#586)

🎉 New Features

  • Added a detector for IP addresses (#692)
  • Added a detector for GitLab tokens (#782)
  • Added a detector for Telegram tokens (#808)
  • Added a detector for Pypi and TestPypi tokens (#819)
  • Added a detector for OpenAI tokens (#823)

✨ Usability

  • Added filenames in errors thrown when a plugin file specified in the .secrets.baseline is not found. (#719)
  • Changed the wording of the audit prompt (#738)

🔭 Accuracy

  • Improved DiscordBotTokenDetector to reduce false negatives (#628)
  • Improved KeywordDetector to reduce false positive for Golang (#675)
  • Improved AWSKeyDetector by adding more access key formats (#796)

🐛 Bugfixes

  • Fixed NotImplementedError in StatisticsAggregator (#678)
  • Fixed bug in YAMLTransformer related to parsing YAML files with achors and tags (#679)
  • Fixed IndexError in is_prefixed_with_dollar_sign caused by passing empty strings (#712)

🐍 Miscellaneous

  • Dropped support for Python 3.6 (#672)
  • Dropped support for Python 3.7 (#724)
  • Added support for Python 3.10 (#724)
  • Added support for Python 3.11 (#730)
  • Added support for Python 3.12 (#810)
  • Multiple dependency updates

v1.4.0

October 4th, 2022

📰 News

  • We're dropping support for Python 3.6 starting v1.5.0! Python 3.6 reached EOL on December 23, 2021 and, therefore, is currently unsupported. We hope this announcement gives you plenty of time to upgrade your project, if needed.

📣 Release Highlights

  • Improved filtering by excluding secrets that have already been detected by a regex-based detector (#612)

🎉 New Features

  • Added a detector for Discord bot tokens (#614)

✨ Usability

  • Improved the audit report to make it easier to parse programmatically (#619)

🔭 Accuracy

  • Improve ArtifactoryDetector plugin to reduce false positives (#499)

🐛 Bugfixes

  • Fixed the verify flow in audit report by adding the code snippet of the verified secret (#620)
  • Fixed deploy process to be environment configuration independent (#625)

🐍 Miscellaneous

  • Added support for .NET packages.lock.json files in the heuristic filter (#593)
  • Multiple dependency updates

v1.3.0

July 22nd, 2022

📣 Release Highlights

  • Add Windows operating system to Github CI Action (#528)
  • Enable dependabot for automated dependency updates built into GitHub (#531)
  • Improve performance for array slice (#555)

🎉 New Features

  • Improve keyword plugin to detect arrow key assignment (#567)
  • Add command line argument for detect-secrets-hook to return output as json (#569)

🐛 Bugfixes

  • Fix regex matching for npm plugin (#551)
  • Fix audit crashing when secret is not found on specified line (#568)
  • Fix # pragma: allowlist nextline secret secrets not filtered out of result set (#575)
  • Fix is_verified flag not stored in PotentialSecret (#578)

🐍 Miscellaneous

  • Only use ANSI color code in environments that support it (#523)
  • Multiple dependency updates
  • Make is_likely_id_string heuristic filter more strict to avoid eliminating true positives (#526)
  • Refactor AWS access key regex to minimize false positives (#571)
  • Correct spelling errors in code repository (#574)
  • Add py.typed to enable type hints for package consumers (#579)

v1.2.0

February 16th, 2022

📣 Release Highlights

  • Continuous integration github action added (#506)
  • Release pipeline github action added (#513)

🎉 New Features

  • New GitHub token plugin added (#465)
  • New SendGrid plugin added (#463)
  • More new ignored file extensions

🐛 Bugfixes

  • Fixes catastrophic backtracking for indirect reference heuristic (#509)
  • Fixes pre-commit hook secret equality checking causing updates to baseline with no real changes - only a timestamp update (#507)
  • Fixes python 3.8 failing to load plugins on windows and macos (#505)
  • Fixes yaml transformer inline dictionary index out of bounds exceptions (#501)
  • Fixes regex for slack url (#477)
  • Fixes AttributeError: 'PotentialSecret' object has no attribute 'line_number' by safely falling back to 0 if line_number isn't present. (#476)(#472)
  • Fixes gibberish-detector current version
  • Fixes filtering ordering in .secrets.baseline

🐍 Miscellaneous

  • Updated README due hook failing to interpret filenames with spaces (#470)
  • Add CI github action badge to README
  • Development dependency bumps (#519)

v1.1.0

April 14th, 2021

📣 Release Highlights

  • New gibberish filter added (#416)
  • Multiprocessing support, for faster scans! (#441)
  • Support for scanning different directories (rather than the current directory) (#440)

🎉 New Features

  • KeywordDetector supports whitespace secrets (#414)
  • KeywordDetector now supports prefix/suffixed keywords, and accuracy updates
  • Adding alphanumerical filter to ensure secrets have at least one letter/number in them (#428)
  • New filter added for ignoring common lock files (#417)
  • More new ignored file extensions
  • Adding filter to ignore swagger files
  • Added audit --report to extract secret values with a baseline (#387, thanks @pablosantiagolopez, @syn-4ck)

🔭 Accuracy

  • KeywordDetector now defaults to requiring quotes around secrets (#448)
  • KeywordDetector now searches for more keywords (#430)

🐛 Bugfixes

  • Filter caches are cleared when swapping between different Settings objects (#444)
  • Upgrading baselines from <0.12 migrates exclude to exclude-files rather than exclude-lines (#446)

🐍 Miscellaneous

  • More verbose logging, to help with debugging issues (#432)
  • YAMLTransformer handles binary entries differently

v1.0.3

February 26th, 2021

🐛 Bugfixes

  • Fixes SecretsCollection subtraction method, to handle non-overlapping files.
  • Fixes installation for Windows environments (#412, thanks @pablosantiagolopez)

v1.0.2

February 25th, 2021

🐛 Bugfixes

  • KeywordDetector is no longer case-sensitive.

v1.0.1

February 25th, 2021

🐛 Bugfixes

  • Fixes recursive loop with installation (#408, thanks @cbows)

v1.0.0

February 24th, 2021

📣 Release Highlights

  • Added a concept of "filters", to weed out false positives
  • Introduce the concept of "transformers", to standardize file parsing across plugins
  • Designed an upgrade system for easy migrations of older baseline versions
  • Core engine redesigned to support module usage (rather than just interacting with it through the command line)
  • Added a global Settings object for repeatable, serializable, configurations
  • Introduced dependency injection framework for easy-to-design filters.

💥 Breaking Changes

Honestly, too many to list out. Check out the original pull request (#355) for more details. It's safe to assume that if you interacted with detect-secrets as a module (rather than solely a pre-commit hook or CLI tool), the APIs have changed (for the better).

However, with the new upgrade infrastructure in place, the baseline files will auto upgrade by themselves. Users that have used it solely as a pre-commit hook or CLI tool may need to consult the "User Facing Changes" for flag renaming.

🎉 New Features

  • Added NpmDetector (#347, thanks @ninoseki)
  • Added AzureStorageKeyDetector (#359, thanks @DariuszPorowski)
  • Added SquareOauthDetector (#398, thanks @pablosantiagolopez)
  • Added --only-allowlisted flag to scan for inline ignores
  • Added --list-all-plugins to show a list of all plugins available to the engine
  • Added --exclude-secrets flag to ignore secrets that match specific regexes (#391, thanks @pablosantiagolopez)
  • Added --slim flag to generate baselines that minimize git diffs
  • Added --disable-filter to disable specific filters
  • Added --disable-plugin to disable specific plugins
  • Added support for # pragma: allowlist nextline secret to ignore the following line (#367, thanks @nickiaconis)

🔭 Accuracy

🎓 Walkthrough / Help

  • The README now includes examples of common usages, features, and an FAQ section for the common questions we often receive as GitHub issues.
  • So much better technical documentation!
  • Type support added

🐛 Bugfixes

  • Inline allowlisting is respected by regular scans, rather than only pre-commit hook
  • audit functionality improved on Windows machines
  • git operations now handle file paths with spaces
  • fix KeywordDetector hanging on very long lines (#373, thanks @gpflaum)

v0.14.3

August 27th, 2020

🔭 Accuracy

  • Verify Slack secrets more accurately (#325, thanks @dryoni)

🐛 Bugfixes

  • Fix a TypeError exception in adhoc string scanning (#336)

🐍 Miscellaneous

  • Fix an XML comment in documentation (#322, thanks @cilefen)

v0.14.2

July 25th, 2020

🐛 Bugfixes

  • Fixed an AttributeError exception in the pre-commit hook, when on Windows (#321, thanks @JohnNeville)

v0.14.1

July 13th, 2020

🐛 Bugfixes

  • Add missing tuple() conversion that raised a TypeError when using scan --update (#317, thanks @shaikmanu797)

v0.14.0

July 9th, 2020

📣 Release Highlights

  • Remove support for Python 2 (#292, big thanks to [@KevinHock]!)

🎉 New Features

  • Add support for custom plugins (#308, big thanks to [@KevinHock]!)

🎭 Performance

  • Check the allowlist after a secret is found, instead of before (#293, #244)

🔭 Accuracy

  • Make IBM plugins less noisy (#289, thanks to [@killuazhu])

🐛 Bugfixes

  • Display helpful error message when scanning a baseline from a newer detect-secrets version (#293, #269)

🐍 Miscellaneous

  • Pin coverage version used in testing (#290)

v0.13.1

March 26th, 2020

🎉 New Features

  • Adding plugin for IBM's Cloudant (#261, thanks [@killuazhu])
  • Adding plugin for IBM Cloud Object Storage HMAC (#263, thanks [@killuazhu])
  • Adding Twilio plugin (#267, thanks [@EdOverflow])

✨ Usability

  • Support for DETECT_SECRETS_SECURITY_TEAM environment variable to customize the pre-commit hook error message (#283, thanks [@0atman])

🐛 Bugfixes

  • Adhoc HighEntropyString scanning supports multiple words (#287)

v0.13.0

October 28th, 2019

📰 News

  • Rationale for the minor version bump:
    • Some accuracy changes that might change baselines significantly
    • @OiCMudkips' first release increases spookiness
    • It being almost Halloween increases spookiness

🎉 New Features

  • Added a Softlayer plugin (#254, thanks [@killuazhu] and [@justineyster])
  • Support URL-safe base64 strings in the base64 plugin (#245)

✨ Usability

  • Make it easier to add new plugins to detect-secrets (#248)

🔭 Accuracy

  • Exclude NOPASSWD from the keyword detector (#247, thanks [@security-architecture])
  • Ignore lines with id in them in the high-entropy plugins (#245)
  • Ignore UUIDs detected by the base64 plugin (#245)

🐛 Bugfixes

  • Fix the signal metric in the audit results view (#251)

v0.12.7

September 23rd, 2019

🎉 New Features

🔭 Accuracy

  • Added a --word-list option for filtering secrets with words in them (#241, do pip install detect-secrets[word_list] to use this feature)

🐛 Bugfixes

v0.12.6

September 16th, 2019

🎉 New Features

  • Added a MailchimpDetector plugin (#217, thanks [@dgzlopes])
  • Added verification for Slack webhooks (#233, thanks [@Patil2099])

🔭 Accuracy

  • Added handling of binary secrets in YAML files (#223)
  • Added various accuracy improvements to the KeywordDetector plugin (#229)

🐛 Bugfixes

  • Fixed a bug in the audit functionality where we crashed when the highlighter failed (#228)
  • Fixed a bug in the audit functionality where there was no (b)ack audit functionality when a secret was not found (#215, thanks [@dgzlopes])
  • Fixed a bug where we were not excluding SVG files (#219)

🐍 Miscellaneous

  • Added a unique exit code to identify baseline changes (#214, thanks [@lirantal])
  • Updated and ran our pre-commit hooks (#221, thanks [@killuazhu])

v0.12.5

July 23rd, 2019

🎉 New Features

  • Added webhook detection to our SlackDetector plugin (#195, thanks [@adrianbn])
  • Added support for scanning multiple files (#188, thanks [@dgzlopes])
  • Added support for scanning multiple repositories (#193)
  • Added verification for AWS access keys and Slack tokens (#194)
  • Added an audit --display-results feature to aid plugin development (#205)

🔭 Accuracy

  • Improved our Artifactory regex (#195, thanks [@adrianbn])
  • Improved sequential string detection to catch the Base64 character set (#207)
  • Moved our sequential string detection so it is used by all plugins (#196)

🎭 Performance

  • Added performance testing benchmarks (#181, #186, #187, thanks [@dgzlopes])

v0.12.4

May 22nd, 2019

📰 News

  • whitelist/blacklist have been replaced with allowlist/denylist (#178, thanks [@richo]). This includes using # pragma: allowlist secret now for inline allowlisting. # pragma: whitelist secret compatibility will be removed in a later major version bump.

🎉 New Features

  • Added a StripeDetector plugin (#169, thanks [@dgzlopes])
  • Improved handling of un-scannable files (#176, thanks [@dgzlopes])

🐍 Miscellaneous

  • Improved documentation of regex based detector's in the README (#177, thanks [@dgzlopes])

v0.12.3

May 13th, 2019

🎉 New Features

  • Added an ArtifactoryDetector plugin (#157 and #163, thanks [@justineyster])
  • Added support for Golang string assignments in the KeywordDetector plugin (#162, thanks [@baboateng])
  • Added support for XML inline whitelisting comments (#152, thanks [@killuazhu])
  • Added support for text after inline whitelisting comments (#168, thanks [@dgzlopes])

🐛 Bugfixes

  • Fixed a bug where filetype detection failed due to an inconsistent configparser import (#155, thanks [@Namburgesas])

🐍 Miscellaneous

  • Greatly improved the readability of regular expressions in the KeywordDetector plugin, and the maintainability of the corresponding test (#160 and #161, thanks [@baboateng])
  • Added a contribution guide (#166, thanks [@zioalex])
  • Documented all of our inline whitelisting directives (#165 and #172, thanks [@dgzlopes])

v0.12.2

March 21st, 2019

🐛 Bugfixes

  • Fixed a bug where the improved performance for high-entropy strings (#144) did not work on Python 2 (#147)

v0.12.1

March 21st, 2019

🎉 New Features

  • Added a --keyword-exclude argument to scan (#132, thanks [@hpandeycodeit])

🔭 Accuracy

  • For the KeywordDetector plugin: made quotes required for secrets in .cls and .java files, and skipped {{secrets like this}} in YAML files (#133/#145)

🎭 Performance

  • Improved performance when scanning for high-entropy strings (#144, thanks [@killuazhu])

🐛 Bugfixes

  • Fixed an uncaught UnicodeEncodeError exception in our ini file parser, when using Python 2 (#143)

🐍 Miscellaneous

  • Fixed the example pre-commit configuration in the README (#135, thanks [@nymous]) (#138, thanks [@neunkasulle])
  • Refactored some audit code into CodeSnippet and CodeSnippetHighlighter classes (#137)

v0.12.0

February 11th, 2019

🎉 New Features

  • Added a SlackDetector plugin (#122, thanks [@killuazhu])
  • Added a --use-all-plugins argument to --update that adds all plugins to the baseline (#124, thanks [@killuazhu])
  • Added --exclude-files and --exclude-lines arguments to scan (#127)

💥 Breaking Changes

  • Removed the --exclude CLI scan argument (#127)

🔭 Accuracy

  • Reduced false-positives by excluding more characters (!$&\';) in the BasicAuthDetector regex (#126, #123, thanks [@killuazhu])
  • Added more to the FALSE_POSITIVES dict for the KeywordDetector plugin, including password (#118)

🐛 Bugfixes

  • Fixed a bug where --update was adding all plugins to the baseline, instead of respecting the plugins used in the baseline (#124, thanks [@killuazhu])
  • Fixed an uncaught UnicodeEncodeError exception when scanning non-ini files (e.g. markdown) containing unicode, when using Python 2 (#128, thanks [@killuazhu])
  • Fixed a bug where non-ini files (e.g. markdown) containing unicode caused a UnicodeEncodeError exception in the audit functionality, when using Python 2 (#129, thanks [@killuazhu])
  • Fixed a bug where non-posix end of line characters caused a "Secret not found on line...." error in the audit functionality (#120, thanks [@killuazhu])
  • Fixed a bug where scan_diff, called by detect-secrets-server, was ignoring inline pragma: whitelist secret comments (#127)

🐍 Miscellaneous

v0.11.4

January 7th, 2019

🐛 Bugfixes

  • Fixed a TypeError bug introduced in #111 (#116)

v0.11.3

January 4th, 2019

🐛 Bugfixes

  • Fixed a bug where we were adding an extra-newline in detect-secrets scan output (#111)

🐍 Miscellaneous

  • Reorganized the code, mainly creating a common/ directory (#113)

v0.11.2

January 4th, 2019

🔭 Accuracy

v0.11.1

January 4th, 2019

🎉 New Features

  • Turned the KeywordDetector plugin back on, with new regexes and accuracy improvements (#86)
  • Added an AWSAccessKeyDetector plugin (#100)
  • Added the ability to scan .ini types files that don't have a header (#106)

🔭 Accuracy

  • Add blacklisting of PGP private key headers in PrivateKeyDetector plugin (#104)
  • Reduced false-positives by improving BasicAuthDetector plugin regex (#98)

🐛 Bugfixes

  • Fixed a bug where we were not showing removed lines in the audit functionality (#98)

🐍 Miscellaneous

  • Added whitelist directive regexes to match against inline comment syntaxes in more languages (#105)
  • Refactored various detectors to use RegexBasedDetector (#103)
  • Refactored the BashColor singleton into the colorize function (#109)
  • Small improvements to existing file parsers (#107)
  • Refactored the BasePlugin to use the WHITELIST_REGEX (#99)
  • Removed unidiff from standard dependencies (#101)

v0.11.0

November 26th, 2018

🎉 New Features

  • Made the pre-commit hook automatically update the baseline (#96)
  • Added the audit --diff functionality (#95)

🎨 Display Changes

  • Added display of secret type in audit functionality (#94)

v0.10.5

October 30th, 2018

🎨 Display Changes

  • Added a "Please git add the baseline" message (#89)
  • Improved the "Unable to open baseline file" message (#91)

🐛 Bugfixes

  • Update scan --update results to only propagate is_secret of new secrets (#90)

0.10.4

October 23rd, 2018

💥 Breaking Changes

  • Disabled KeywordDetector plugin temporarily (#89)

🎨 Display Changes

  • Ordered baseline hashes, for better diffs (#84)
  • Added a "Please git add the baseline" message (#89)
  • Improved error messages for pre-commit hook (#85)

🐛 Bugfixes

  • Fixed a couple bugs in the audit functionality, one for small files and the other case-sensitivity in the KeywordDetector plugin (#83, thanks [@jkozera])

0.10.3

October 4th, 2018

🎉 New Features

  • Added a KeywordDetector plugin, that was horrible and regretful (#76)

🐛 Bugfixes

  • Fixed a bug in scan --update where we would append the baseline exclude regex to itself (#78)
  • Fixed the regular expression in the BasicAuthDetector plugin so that it didn't run forever (#80)
  • Removed trailing whitespace from scan output (#78)

🐍 Miscellaneous

  • Added command line hints and baseline clarification in the README (#81, thanks [@JoshuaRLi])

0.10.2

September 12th, 2018

🎉 New Features

  • Added a (b)ack option to 'Is this a valid secret?' (#72, thanks [@cleborys])
  • Added a BasicAuthDetector plugin (#74)
  • Added CLI functionality to check strings in an adhoc manner (#73)

🐛 Bugfixes

  • Added a check to only load json from stdin if it exists (#69, thanks [@guykisel])

🐍 Miscellaneous

  • Fixed a typo in the README (#68, thanks [@whathejoe])

0.10.1

August 1st, 2018

🐛 Bugfixes

  • Fixed a bug where we didn't skip sequential strings when we should have (#67)

0.10.0

August 1st, 2018

🎉 New Features

  • Scan --all-files option (#57)
  • YAML inline whitelisting support (#50)

💥 Breaking Changes

  • Changed --audit and --scan to audit and scan (#51)
  • Changed scan --import <baseline> to scan --update <baseline> (#58)

🔭 Accuracy

  • Reduced false-positives caused by sequential strings, e.g. ABCDEF (#64)

🐛 Bugfixes

  • Fixed a bug where the pre-commit code would remove the is_secret attribute from audited baselines (#65)
  • Fixed an audit bug where we would crash if a file in the baseline did not exist (#56)
  • Improved the audit functionality to handle short files better (#48)

0.9.1

June 28th, 2018

🐛 Bugfixes

  • Fixed numbering system with interactive audit
  • Fixed "leapfrog" edge case for audit functionality (#47)

0.9.0

June 27th, 2018

🎉 New Features

  • Added ability to migrate baselines from an older version to a newer version
  • Added functionality to audit baseline, to distinguish difference between false and true positives in the baseline file (#44)
  • Upgraded PrivateKeyPlugin: more search parameters, more lines searched, and secret hash created using payload (rather than the entire line content)

💥 Breaking Changes

  • Differentiate between Base64HighEntropyStrings and HexHighEntropyStrings through secret_type (#26)
  • Got rid of SensitivityValues as a means to store plugin configs

🔭 Accuracy

  • Improved the heuristic for HexHighEntropyStrings, reducing the false positive rates for large numbers identified in code

🐛 Bugfixes

  • Baseline always outputs in sorted order now, to prevent unnecessary diffs (#25)
  • Escape exclude regex statements before compilation (#39)
  • Fixed case where details of plugins used were not included in the baseline, when the pre-commit hook updated it (#40)

🐍 Miscellaneous

  • Simplified logging by removing CustomLog (#46)

Before 0.9.0

🎉 New Features

  • Allow scanning of non-git files (#18)

🔭 Accuracy

  • Improved scanning of INI config files with HighEntropyString (#13 and #17)
  • Improved scanning of YAML files with HighEntropyString (#16)

🐛 Bugfixes

  • Fixed PrivateKeyDetector plugin analyze results' representation (#15)