Skip to content
This repository was archived by the owner on Jul 20, 2026. It is now read-only.

fix: make Scrapy framework logs visible through structlog #87

Merged
damo-da merged 1 commit into from
Jun 23, 2026
Merged

fix: make Scrapy framework logs visible through structlog #87

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 9 additions & 17 deletions ngm/logging.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,28 +53,20 @@ def configure_logging():

handler = logging.StreamHandler()
handler.setFormatter(formatter)
# Enforce the level on the handler too: Scrapy resets the root logger level
# to NOTSET when it initialises, so the handler is what actually filters.
handler.setLevel(log_level)

root_logger = logging.getLogger()
root_logger.handlers = [handler]
root_logger.setLevel(log_level)

for _log_name in [
"scrapy",
"scrapy.core.engine",
"scrapy.downloadermiddlewares",
"scrapy.extensions",
"scrapy.spidermiddlewares",
"scrapy.utils.signal",
"protego._protego",
"sqlalchemy",
"sqlalchemy.engine",
"boto3",
"botocore",
"urllib3",
]:
logger = logging.getLogger(_log_name)
logger.handlers = [handler]
logger.propagate = False
# Route everything through the single root handler above. We deliberately do
# NOT pin per-library handlers with propagate=False: Scrapy's startup runs
# dictConfig(DEFAULT_LOGGING), which strips handlers off the `scrapy` logger.
# With propagate=False that left `scrapy` (and its children's) records with
# nowhere to go, silently dropping all of Scrapy's framework logs (spider
# open/close, stats, errors). Letting them propagate to root keeps them.
Comment on lines +64 to +69

@gemini-code-assist gemini-code-assist Bot Jun 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

By removing the explicit logger configuration loop, third-party loggers like sqlalchemy.engine, boto3, botocore, and urllib3 will now propagate to the root logger and inherit its log level. When DEBUG mode is enabled, these libraries will flood the console with extremely verbose logs (such as every SQL query, AWS API request/response, and HTTP connection pool state), making local debugging very difficult and potentially leaking sensitive data in non-production environments.

We should explicitly set these noisy third-party loggers to WARNING level to keep the logs clean.

Suggested change
# Route everything through the single root handler above. We deliberately do
# NOT pin per-library handlers with propagate=False: Scrapy's startup runs
# dictConfig(DEFAULT_LOGGING), which strips handlers off the `scrapy` logger.
# With propagate=False that left `scrapy` (and its children's) records with
# nowhere to go, silently dropping all of Scrapy's framework logs (spider
# open/close, stats, errors). Letting them propagate to root keeps them.
# Route everything through the single root handler above. We deliberately do
# NOT pin per-library handlers with propagate=False: Scrapy's startup runs
# dictConfig(DEFAULT_LOGGING), which strips handlers off the `scrapy` logger.
# With propagate=False that left `scrapy` (and its children's) records with
# nowhere to go, silently dropping all of Scrapy's framework logs (spider
# open/close, stats, errors). Letting them propagate to root keeps them.
# Set sensible default levels for noisy third-party loggers
for noisy_logger in ["sqlalchemy.engine", "boto3", "botocore", "urllib3"]:
logging.getLogger(noisy_logger).setLevel(logging.WARNING)



def init_sentry():
Expand Down
6 changes: 6 additions & 0 deletions ngm/ngscrape/settings.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,12 @@
# Set log level
LOG_LEVEL = "INFO"

# Logging is owned by ngm.logging.setup() (structlog). Disable Scrapy's own
# logging install so it doesn't add a competing root handler; combined with
# routing scrapy loggers through the root structlog handler (see ngm/logging.py)
# this keeps Scrapy's framework logs (spider open/close, stats, errors) visible.
LOG_ENABLED = False
Comment on lines +32 to +36

@gemini-code-assist gemini-code-assist Bot Jun 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Since LOG_ENABLED is set to False, Scrapy's internal logging configuration is completely bypassed. As a result, the LOG_LEVEL = "INFO" setting (on line 30) is now ignored by Scrapy. To avoid developer confusion, we should document this behavior in the comment so it's clear that log levels must be configured via environment variables (like LOG_LEVEL or DEBUG) instead of the Scrapy settings file.

Suggested change
# Logging is owned by ngm.logging.setup() (structlog). Disable Scrapy's own
# logging install so it doesn't add a competing root handler; combined with
# routing scrapy loggers through the root structlog handler (see ngm/logging.py)
# this keeps Scrapy's framework logs (spider open/close, stats, errors) visible.
LOG_ENABLED = False
# Logging is owned by ngm.logging.setup() (structlog). Disable Scrapy's own
# logging install so it doesn't add a competing root handler; combined with
# routing scrapy loggers through the root structlog handler (see ngm/logging.py)
# this keeps Scrapy's framework logs (spider open/close, stats, errors) visible.
# Note: This makes the Scrapy `LOG_LEVEL` setting ignored; log levels are instead
# controlled via the `LOG_LEVEL` or `DEBUG` environment variables.
LOG_ENABLED = False


SPIDER_MODULES = ["ngm.ngscrape.spiders"]
NEWSPIDER_MODULE = "ngm.ngscrape.spiders"

Expand Down
Loading