v1.27.0

This release requires some system upgrades:

• Valkey 8.0+
• Kvrocks 2.10+
• Python 3.9+

Changes

• Improve documentation on capture page
• Speedup on-demand stop of archiver script
• Make all remaining indexes paginated
• Use new indexes to speedup rendering of hostnode popups and all views relying on indexes
• Maintenance and bugfixes

For Lacus, LacusCore and PlaywrightCapture changes, see Lacus release notes.

Full Changelog: v1.26.1...v1.27.0

v1.26.1

New features

• Major improvements in indexes, allows to paginate hits based on capture timestamp
• Add index for TLDs
• Transparent migration of urls, hostnames, ressources, HTTP Headers Hashes,and cookies indexes to new format
• Run many more DNS requests (MX, SOA, NS)
• Use new indexes on web interface, speedup rendering
• Optionally disable JavaScript during capture

What's Changed

• build(deps): bump docker/build-push-action from 5 to 6 by @dependabot in #939

Full Changelog: v1.26.0...v1.26.1

v1.26.0

Lacus, LacusCore, and PlaywrightCapture changes

• See Lacus v 1.11.0 release notes for details
• Many bugfixes and improvements
• Use more recent browsers

Har2Tree

Full Changelog: Lookyloo/har2tree@v1.25.0...v1.26.0

• Improve handling of embedded content
• Properly generate docs

Lookyloo

Full Changelog: v1.25.0...v1.26.0

• Improve typing for capture settings (Pydantic)
• Make it possible to categorize captures
• Improve error handling, logging
• Index categorized captures
• Get Captures via UUID in the API
• Add controller to Start/Stop scripts individually
• Add CSP HTTP headers whenever possible

What's Changed

• putting the login message in the right block by @adrima01 in #924
• Categories by @AntoniaBK in #925
• new: get uuids by category via API by @adrima01 in #926
• add new endpoint to remove capture by @jeroengui in #929

New Contributors

• @jeroengui made their first contribution in #929

v1.25.0 - Pass the Salt 2024

New features

• Endpoint to remove capture from the index (safely copies it in a directory)
• Configurable way to gather contact information for takedown of malicious websites
• FuzzImy hash based on HTML structure of the rendered page (algorithm of CERT PL)
• New simple capture page without any of the options
• Optional user config to overwrite capture settings and/or have default parameters
• Download tree as PNG (as long as the tree isn't too big)
• Optional auto-report on capture page (admin only)
• Optionally change the index page to the capture page, useful when the index is very big and takes a long time to load

Changes

• Many improvements when rendering panels on tree
• New menus
• Vast amount of bugfixes and improvements everywhere in the project by @AntoniaBK and @adrima01, see below.
• Support for valkey, new kvrocks, Ubuntu 24.04
• Optionally disable SRI validation while developing
• Partial removal of jQuery
• Many improvements in loading index, uses caching much more efficiently
• Restart webserver more often, avoids memory leaks
• Improve logging, reduces noise

What's Changed

• Lookup of abuse-c by @AntoniaBK in #904
• Remove redundant clause by @AntoniaBK in #906
• adding uwhoisd installation by @adrima01 in #907
• Fix: DataTable rename to treeHashesTable by @AntoniaBK in #909
• adding 3rd party report to mail by @adrima01 in #908
• Menu by @AntoniaBK in #910
• changes so that you can ignore the sri while developing by @adrima01 in #911
• Simple interface by @adrima01 in #912
• fix: removing unnecessary script and jQuery by @adrima01 in #913
• new: downloadable tree as png by @adrima01 in #915
• New: admin-only checkbox for auto-report by @AntoniaBK in #914
• fix: [modules] Gracefully accept no hashlookup fixes #916 by @cvandeplas in #917
• Fixing typo by @adrima01 in #918
• New: upload a capture via the API by @AntoniaBK in #919
• Recent captures by @adrima01 in #921
• Update generic.json.sample by @adrima01 in #923

New Contributors

• @cvandeplas made their first contribution in #917

Full Changelog: v1.24.0...v1.25.0

v1.24.0

New features

• Optionally attempt to allow tracking on capture, see lacus v1.9.0 release for details.
• [Admin Only] Index all captures, not only the public ones (in kvrocks instead of redis)
• Multiple improvements in correlation pages (Favicons, ressources
• Index favicons
• Compute favicons MM3H and like it to Shodan
• Search and index captcha IDs (reCaptcha, hCaptcha and Cloudflare)

What's Changed

• Changed misp_url by @adrima01 in #894
• Proxy with VT module by @DocArmoryTech in #897
• Module response added by @adrima01 in #898
• Made send_email available from the API by @AntoniaBK in #899
• Speedup async capture checks when the backlog in long
• Improve favicon rendering on tree
• Split capture building and indexing in two different scripts
• Reduce memory usage for indexing scripts

New Contributors

• @adrima01 made their first contribution in #894
• @DocArmoryTech made their first contribution in #897
• @AntoniaBK made their first contribution in #899

Full Changelog: v1.23.0...v1.24.0

v1.23.0

What's Changed

• New: autoreport to investigation team from API

• New: Archiving old captures to a S3 bucket (well over 2M on the production instance)

• New CIRCL Passive DNS module

• Many perf improvements

• Refactor indexing module

• Display IDNA domain when possible

• Improve statistics

• Improve lacus monitoring

• Fix docker image

• Python 3.12 support

• Strict typing

• Maintenance, dependencies updates, bugfixes

• PlaywrightCapture changelog

• LacusCore changelog

Full Changelog: v1.22.0...v1.23.0

Hack.lu 2023

If you haven't been upgrading lookyloo since the last tagged release, this release contains a lot of changes.

New features

• Support for HTTP Headers Hashing
• Support for archiving capture on S3FS, this is not completely implemented, but we use it on the public instance. Expect it to be usable for anyone in next release.
• Store HARs dumps in gzip archives
• MISP export when the capture downloads a file
• Handle captures where we have a rendered HTML and a no-click download is triggered in Javascript (TODO: support multiple downloads triggered in a single capture)
• Get downloaded file via the API
• Fetch favicons using default URL, and HTML content
• Support multiple MISP instances for submission and lookup

Changes

• Better rendering of the capture time
• Support re-processing captures that were mistakenly considered broken
• Improve logging
• Improve caching
• Auto-restart webservice to avoid memory leak
• Strip URL to capture (space and new line)
• Update dependencies, new browsers

Bugfixes

• Allow to run multiple backgroung indexing scripts
• Many related to the compression of the HAR and the pickles to reduce disk use
• Various encoding issues with rendered HTML

For more details regarding the captures, see Lacus v1.7.0 release notes.

Pass The Salt 2023

New Features

• Allow to pass a timezone, geolocation coordinates, locale, and color scheme to a capture
• Add a global proxy option in the settings
• Improve SMTP auth for notifications

Changes

• Store the capture settings in order to reuse them later (like for re-capture)
• Avoid failing if Lacus isn't available ant retry a few times

Bugfixes

• Properly handle captures with errors, improve logging accordingly
• Resubmit captures if they were deleted on Lacus without storing a response (generally if something crashed)

Spring release

This release is the outcome of a good two months of work on Lookyloo itself but also Lacus and its dependencies leading to the v1.5.0 release.

It also improves the support for the monitoring interface (still to be considered beta).

New features

• Compare captures via the API
• Submit any for to Pandora (if available)
• Allow automatic reporting via the API
• Can set an email to notify in the monitoring form

Changes

• Improve handling of long running processes,
• Improve logging all over the place
• Changes related to Lacus/LacusCore/PyLacus changes
• Easy way to check if two captures are different or not
• Store capture settings in the capture directory for potential later use
• Show proxy in UI if one was given
• Improve response when comparing captures

Bugfixes

• Avoid issues when the pickle requires too much recursivity
• Cloudflare services was always flagging URLs as their own
• The usual batch of bugfixes all over the place

March release

New features

• The email notification now attaches the contacts, making takedown requests easier.
• (WiP) Add settings for comparing captures. It is now possible to ignore domains and/or a substring in a resource URL loaded from the landing page.
• Update PyLookyloo to pass the settings when comparing captures
• [Admin users only] Modal to trigger admin-only tasks on a specific capture (hide/rebuild)

Bugfixes

• Fix docker compose (thanks to @bib0x)
• Avoid exception at multiple places when a capture is invalid for any reason

Changes

• Force protocol 5 for pickles (requires python 3.8, but lookyloo already required it anyway)
• Optimize pickle before storing, and archive them to reduce diskspace
• Bump dependencies (js & python)
• Improve logging (add capture UUID when possible, makes debugging easier)
• Always use LookylooException instead of Exception
• Update Playwright in PlaywrightCapture
• Improve logging in har2tree