Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#2] Add option to log request response body #4

Merged
merged 9 commits into from
Jun 8, 2023

Conversation

pi-sigma
Copy link
Contributor

@pi-sigma pi-sigma commented May 10, 2023

Closes open-formulieren/open-forms#2762
Closes #2 (partly)

  • added option to log request & response body
  • added configuration in admin to override settings

@pi-sigma pi-sigma force-pushed the feature/log-request-response-body branch from f596dfe to 5008ad8 Compare May 10, 2023 06:44
@codecov-commenter
Copy link

codecov-commenter commented May 10, 2023

Codecov Report

Merging #4 (0f7befb) into main (32d109c) will increase coverage by 3.74%.
The diff coverage is 91.72%.

❗ Current head 0f7befb differs from pull request most recent head 62ea45b. Consider uploading reports for the commit 62ea45b to get more accurate results

@@            Coverage Diff             @@
##             main       #4      +/-   ##
==========================================
+ Coverage   88.46%   92.20%   +3.74%     
==========================================
  Files           7        9       +2     
  Lines         104      218     +114     
  Branches        9       25      +16     
==========================================
+ Hits           92      201     +109     
- Misses          7       11       +4     
- Partials        5        6       +1     
Impacted Files Coverage Δ
log_outgoing_requests/formatters.py 75.00% <50.00%> (-15.00%) ⬇️
log_outgoing_requests/log_requests.py 81.25% <66.66%> (ø)
log_outgoing_requests/handlers.py 83.33% <77.27%> (+3.33%) ⬆️
log_outgoing_requests/admin.py 88.88% <94.73%> (+4.67%) ⬆️
...rations/0002_outgoingrequestslogconfig_and_more.py 100.00% <100.00%> (ø)
log_outgoing_requests/models.py 98.71% <100.00%> (+2.28%) ⬆️
log_outgoing_requests/validators.py 100.00% <100.00%> (ø)

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@pi-sigma pi-sigma changed the title Feature/log request response body [#2] Add option to log request response body May 10, 2023
Copy link
Member

@sergei-maertens sergei-maertens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also check the test coverage? This PR makes it drop, instead the coverage for a PR should be 100% to make sure you don't add untested code.

README.rst Outdated Show resolved Hide resolved
log_outgoing_requests/models.py Outdated Show resolved Hide resolved
log_outgoing_requests/models.py Outdated Show resolved Hide resolved
log_outgoing_requests/models.py Outdated Show resolved Hide resolved
log_outgoing_requests/models.py Outdated Show resolved Hide resolved
log_outgoing_requests/formatters.py Outdated Show resolved Hide resolved
log_outgoing_requests/formatters.py Outdated Show resolved Hide resolved
log_outgoing_requests/handlers.py Outdated Show resolved Hide resolved
log_outgoing_requests/handlers.py Outdated Show resolved Hide resolved
log_outgoing_requests/handlers.py Outdated Show resolved Hide resolved
@pi-sigma pi-sigma marked this pull request as draft May 12, 2023 07:16
@pi-sigma pi-sigma force-pushed the feature/log-request-response-body branch 8 times, most recently from c05af12 to fcf7741 Compare May 16, 2023 10:56
@pi-sigma pi-sigma marked this pull request as ready for review May 16, 2023 10:59
@pi-sigma pi-sigma force-pushed the feature/log-request-response-body branch 2 times, most recently from 0f7befb to 62ea45b Compare May 16, 2023 11:58
@pi-sigma pi-sigma marked this pull request as draft May 17, 2023 09:04
@pi-sigma pi-sigma force-pushed the feature/log-request-response-body branch from 62ea45b to f8a6b30 Compare May 17, 2023 14:30
@pi-sigma pi-sigma marked this pull request as ready for review May 17, 2023 14:34
@pi-sigma pi-sigma marked this pull request as draft May 22, 2023 07:58
@pi-sigma pi-sigma force-pushed the feature/log-request-response-body branch from f8a6b30 to 2688188 Compare May 22, 2023 09:25
@pi-sigma pi-sigma marked this pull request as ready for review May 22, 2023 09:47
Copy link
Member

@sergei-maertens sergei-maertens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still seeing many new tests that are not in pytest-style, what's the reasoning behind that?

.editorconfig Outdated Show resolved Hide resolved
README.rst Outdated Show resolved Hide resolved
README.rst Outdated Show resolved Hide resolved
log_outgoing_requests/admin.py Outdated Show resolved Hide resolved
log_outgoing_requests/admin.py Outdated Show resolved Hide resolved
log_outgoing_requests/validators.py Outdated Show resolved Hide resolved
log_outgoing_requests/validators.py Outdated Show resolved Hide resolved
log_outgoing_requests/validators.py Outdated Show resolved Hide resolved
tests/test_validators.py Outdated Show resolved Hide resolved
tests/test_validators.py Outdated Show resolved Hide resolved
@pi-sigma pi-sigma marked this pull request as draft May 23, 2023 12:11
@pi-sigma pi-sigma force-pushed the feature/log-request-response-body branch 2 times, most recently from d59bd02 to fbf3db0 Compare May 30, 2023 14:43
@pi-sigma pi-sigma force-pushed the feature/log-request-response-body branch from fbf3db0 to 6d1ef18 Compare May 31, 2023 06:42
@pi-sigma pi-sigma marked this pull request as ready for review May 31, 2023 06:43
@pi-sigma pi-sigma force-pushed the feature/log-request-response-body branch 2 times, most recently from 02df661 to 7b84da0 Compare May 31, 2023 12:48
@alextreme alextreme requested a review from vaszig June 1, 2023 08:05
@alextreme
Copy link
Member

@vaszig feel free to try this out and do a double-check of Paul's changes

Copy link
Member

@sergei-maertens sergei-maertens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't looked too much at the tests yet - let's get the implementation sorted out and approved first before I dive deeper into the tests.

Comment on lines +102 to +107
LOG_OUTGOING_REQUESTS_CONTENT_TYPES = [
"text/*",
"application/json",
"application/xml",
"application/soap+xml",
] # save request/response bodies with matching content type
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as discussed in the office - this would probably benefit from a dataclass so you can do things like:

LOG_OUTGOING_REQUESTS_CONTENT_TYPES = [
    ContentType(pattern="text/xml", default_encoding="iso-8859-3"),
    ContentType(pattern="text/*", default_encoding="utf-8"),
    ContentType(pattern="application/json", default_encoding="utf-8"),
    ...
]

then algorithm-wise you:

  1. extract the content type of the body (request or response)
  2. for every item in settings.LOG_OUTGOING_REQUESTS_CONTENT_TYPES
    1. check if there's an exact match
    2. if not, check if there's a pattern match (using the wildcard)
  3. if there's no match -> don't log it
  4. if there's a match
    1. try to get the charset from the content-type header
    2. if there's no charset, fall back to the ContentType.default_encoding
  5. Log the body

Somewhere in between there's also the check if settings/configuration and the body size

We also discussed that you might provide a setting/escape hatch to derive the encoding, something along the lines of:

def extract_encoding_for_body(content_type: str, body: bytes) -> str:
    ...  # default implementation is part of the above algorithm


LOG_OUTGOING_REQUESTS_EXTRACT_ENCODING = "log_outgoing_requests.utils.extract_encoding_for_body"

so the default setting value should be the library function, but projects can provide their own function. In the case of XML for example, you'd want to parse the first line and get the value from the <?xml encoding="..."> part, but that's for projects to figure out.

Comment on lines +61 to +71
LOG_OUTGOING_REQUESTS_DB_SAVE_BODY = True # save request/response body
LOG_OUTGOING_REQUESTS_EMIT_BODY = True # log request/response body
LOG_OUTGOING_REQUESTS_CONTENT_TYPES = [
"text/*",
"application/json",
"application/xml",
"application/soap+xml",
] # save request/response bodies with matching content type
LOG_OUTGOING_REQUESTS_MAX_CONTENT_LENGTH = 524_288 # maximal size (in bytes) for the request/response body
LOG_OUTGOING_REQUESTS_LOG_BODY_TO_STDOUT = True

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the library is currently missing a proper default settings mechanism, so I created #6 for this.

log_outgoing_requests/admin.py Outdated Show resolved Hide resolved
log_outgoing_requests/admin.py Outdated Show resolved Hide resolved
log_outgoing_requests/admin.py Outdated Show resolved Hide resolved
tests/conftest.py Outdated Show resolved Hide resolved
tests/test_config.py Outdated Show resolved Hide resolved
tox.ini Show resolved Hide resolved
tox.ini Show resolved Hide resolved
tox.ini Show resolved Hide resolved
@pi-sigma pi-sigma marked this pull request as draft June 2, 2023 07:16
@pi-sigma pi-sigma force-pushed the feature/log-request-response-body branch from 7b84da0 to 6bae3ef Compare June 4, 2023 19:35
@pi-sigma pi-sigma marked this pull request as ready for review June 5, 2023 07:47
Copy link
Collaborator

@vaszig vaszig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!This is getting a lot better!
After Sergei's remarks and because I am not so familiar with all the code which has been added, I don't have a lot to mention except some minor spelling mistakes and one remark-question concerning the model.

log_outgoing_requests/admin.py Outdated Show resolved Hide resolved
log_outgoing_requests/admin.py Outdated Show resolved Hide resolved
log_outgoing_requests/models.py Outdated Show resolved Hide resolved
Copy link
Member

@sergei-maertens sergei-maertens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's some follow up to be done, but that is going to happen in separate issues/PRs

@alextreme alextreme merged commit 8e769a2 into main Jun 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Request: Display body of request Add logging in the admin to see outgoing requests/responses
5 participants