Add logging buffering #5635

evgenyfedorov2 · 2024-11-13T15:09:37Z

Related to the #5123 proposal, this PR is focused on the logging buffering only. See also the sampling part #5574.

Microsoft Reviewers: Open in CodeFlow

src/Libraries/Microsoft.Extensions.Telemetry/Logging/ExtendedLogger.cs

...Libraries/Microsoft.AspNetCore.Diagnostics.Middleware/Buffering/HttpRequestBufferProvider.cs

src/Libraries/Microsoft.AspNetCore.Diagnostics.Middleware/Buffering/HttpRequestBufferOptions.cs

src/Libraries/Microsoft.Extensions.Telemetry/Buffering/GlobalBufferOptions.cs

...Core.Diagnostics.Middleware.Tests/Buffering/HttpRequestBufferLoggerBuilderExtensionsTests.cs

test/Libraries/Microsoft.AspNetCore.Diagnostics.Middleware.Tests/Logging/AcceptanceTests.cs

...s/Microsoft.Extensions.Telemetry.Tests/Buffering/GlobalBufferLoggerBuilderExtensionsTests.cs

test/Libraries/Microsoft.Extensions.Telemetry.Tests/Logging/ExtendedLoggerTests.cs

...Libraries/Microsoft.AspNetCore.Diagnostics.Middleware/Buffering/IHttpRequestBufferManager.cs

src/Libraries/Microsoft.Extensions.Telemetry/Buffering/BufferFilterRule.cs

...soft.AspNetCore.Diagnostics.Middleware/Buffering/HttpRequestBufferLoggerBuilderExtensions.cs

src/Libraries/Microsoft.Extensions.Telemetry/Buffering/GlobalBufferOptions.cs

src/Libraries/Microsoft.AspNetCore.Diagnostics.Middleware/Buffering/HttpRequestBufferManager.cs

Address PR comments Add .NET 8 support Add ExceptionJsonConverter

dotnet-comment-bot · 2024-12-16T15:24:35Z

‼️ Found issues ‼️

Project	Coverage Type	Expected	Actual
Microsoft.AspNetCore.Diagnostics.Middleware	Line	100	96.03 🔻
Microsoft.AspNetCore.Diagnostics.Middleware	Branch	100	98.51 🔻
Microsoft.Extensions.Telemetry	Line	93	86.2 🔻
Microsoft.Extensions.Telemetry	Branch	93	84.59 🔻
Microsoft.Extensions.Diagnostics.Testing	Line	99	85.78 🔻
Microsoft.Extensions.Diagnostics.Testing	Branch	99	79.57 🔻

🎉 Good job! The coverage increased 🎉
Update MinCodeCoverage in the project files.

Project	Expected	Actual
Microsoft.Extensions.Diagnostics.Probes	70	76
Microsoft.Extensions.Caching.Hybrid	75	86
Microsoft.Extensions.AI.OpenAI	72	77

Full code coverage report: https://dev.azure.com/dnceng-public/public/_build/results?buildId=896358&view=codecoverage-tab

src/Libraries/Microsoft.Extensions.Telemetry/Logging/ExtendedLogger.cs

src/Libraries/Microsoft.Extensions.Telemetry/Buffering/GlobalBuffer.cs

src/Libraries/Microsoft.Extensions.Telemetry/Buffering/ILoggingBuffer.cs

src/Libraries/Microsoft.Extensions.Telemetry/Buffering/GlobalBuffer.cs

src/Libraries/Microsoft.Extensions.Telemetry/Buffering/GlobalBufferManager.cs

src/Libraries/Microsoft.Extensions.Telemetry/ILoggerFilterRule.cs

KalleOlaviNiemitalo · 2024-12-17T16:13:44Z

src/Libraries/Microsoft.Extensions.Telemetry/Buffering/BufferSink.cs

+                        logger.Log(
+                            serializedRecord.LogLevel,
+                            serializedRecord.EventId,
+                            serializedRecord.Attributes,
+                            exception,
+                            (_, _) => serializedRecord.FormattedMessage ?? string.Empty);


This allocates a closure and a delegate for each ILogger.Log<TState> call. I can think of two ways to minimize the allocations:

Instead of serializedRecord.Attributes, pass serializedRecord itself as the state to ILogger.Log<TState>. Implement IReadOnlyList<KeyValuePair<string, object?>> in SerializedLogRecord by calling the corresponding methods of public IReadOnlyList<KeyValuePair<string, string>> Attributes.

Alternatively, keep using the same delegate across all iterations of both foreach loops; declare the serializedRecord variable and a delegate variable before the outer loop. That way, it would allocate only one closure and one delegate per BufferSink.LogRecords call, no matter how many log records are in IEnumerable<T> serializedRecords. This is safe because ILogger.Log<TState> is not allowed to save the formatter delegate for later calling.

makes sense for optimizations later, thank you

This code moved from BufferSink to BufferedLoggerProxy in d7661a6. The BufferedLogRecord instances are not pooled, so each BufferedLogRecord call costs at least one allocation per BufferedLogRecord; that may make the delegate and closure allocations less important.

src/Shared/ExceptionJsonConverter/ExceptionJsonConverter.cs

src/Libraries/Microsoft.Extensions.Telemetry/Buffering/PooledLogRecord.cs

src/Libraries/Microsoft.Extensions.Telemetry/Buffering/BufferFilterRule.cs

…dation

Buffering

988c709

dotnet-policy-service bot assigned evgenyfedorov2 Nov 13, 2024

noahfalk reviewed Nov 23, 2024

View reviewed changes

evgenyfedorov2 force-pushed the evgenyfedorov2/log_buffering branch 2 times, most recently from e40b7b6 to 9d61d12 Compare December 12, 2024 16:30

Major update

2f1a335

evgenyfedorov2 force-pushed the evgenyfedorov2/log_buffering branch from 9d61d12 to 2f1a335 Compare December 12, 2024 17:36

Remove Json exception converter

2d2412e

geeknoid reviewed Dec 15, 2024

View reviewed changes

...Libraries/Microsoft.AspNetCore.Diagnostics.Middleware/Buffering/IHttpRequestBufferManager.cs Outdated Show resolved Hide resolved

geeknoid reviewed Dec 15, 2024

View reviewed changes

src/Libraries/Microsoft.Extensions.Telemetry/Buffering/BufferFilterRule.cs Outdated Show resolved Hide resolved

geeknoid reviewed Dec 15, 2024

View reviewed changes

...soft.AspNetCore.Diagnostics.Middleware/Buffering/HttpRequestBufferLoggerBuilderExtensions.cs Outdated Show resolved Hide resolved

geeknoid reviewed Dec 15, 2024

View reviewed changes

src/Libraries/Microsoft.Extensions.Telemetry/Buffering/GlobalBufferOptions.cs Outdated Show resolved Hide resolved

geeknoid reviewed Dec 15, 2024

View reviewed changes

src/Libraries/Microsoft.Extensions.Telemetry/Buffering/GlobalBufferOptions.cs Outdated Show resolved Hide resolved

KalleOlaviNiemitalo reviewed Dec 16, 2024

View reviewed changes

src/Libraries/Microsoft.AspNetCore.Diagnostics.Middleware/Buffering/HttpRequestBufferManager.cs Outdated Show resolved Hide resolved

evgenyfedorov2 added 2 commits December 16, 2024 14:32

Fix namespaces

f7eaab1

Address PR comments Add .NET 8 support Add ExceptionJsonConverter

Merge branch 'main' into evgenyfedorov2/log_buffering

1f464df

evgenyfedorov2 force-pushed the evgenyfedorov2/log_buffering branch from 883334d to 022f00c Compare December 16, 2024 14:23

Fix build

a371d9c

evgenyfedorov2 force-pushed the evgenyfedorov2/log_buffering branch from 022f00c to a371d9c Compare December 16, 2024 14:51