Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dn loop bound box benchmark #2778

Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 10 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions Benchmarks/Benchmarks/NIOCoreBenchmarks/Benchmarks.swift
Original file line number Diff line number Diff line change
Expand Up @@ -41,4 +41,11 @@ let benchmarks = {
blackHole(asyncChanel)
}
}

Benchmark(
"NIOLoopBoundBox.value",
configuration: .init(metrics: defaultMetrics, scalingFactor: .kilo)
) { benchmark in
runNIOLoopBoundBoxInPlaceMutation(benchmark: benchmark)
}
}
37 changes: 37 additions & 0 deletions Benchmarks/Benchmarks/NIOCoreBenchmarks/NIOLoopBoundBox.swift
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
//===----------------------------------------------------------------------===//
//
// This source file is part of the SwiftNIO open source project
//
// Copyright (c) 2024 Apple Inc. and the SwiftNIO project authors
// Licensed under Apache License v2.0
//
// See LICENSE.txt for license information
// See CONTRIBUTORS.txt for the list of SwiftNIO project authors
//
// SPDX-License-Identifier: Apache-2.0
//
//===----------------------------------------------------------------------===//

import NIOCore
import NIOEmbedded
import Benchmark

func runNIOLoopBoundBoxInPlaceMutation(benchmark: Benchmark) {
let embeddedEventLoop = EmbeddedEventLoop()
let boundBox = NIOLoopBoundBox([Int](), eventLoop: embeddedEventLoop)
boundBox.value.reserveCapacity(1000)

benchmark.startMeasurement()

for _ in benchmark.scaledIterations {
boundBox.value.removeAll(keepingCapacity: true)
for i in 0..<1000 {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should only have the scaled iterations loop, not this inner one.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does that protect against runtime allocations?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, this is now no longer testing what I wanted to test: the performance of repeated mutation of value. CoW types like Array generally can have one off allocations but that isn't the real problem. The problem is if it happens on every iteration in a loop.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the future, the reason looping scaledIterations times helps is that the malloc values output are divided by scaledIterations. This means any allocation that doesn't happen every loop iteration is folded away, so spurious runtime allocations that may slightly differ between runs don't affect our thresholds.

boundBox.value.append(i)
}
}

benchmark.stopMeasurement()

precondition(boundBox.value.count == 1000)
precondition(boundBox.value.reduce(0, +) == 499500)
}
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
{
"mallocCountTotal" : 12
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"mallocCountTotal" : 0
}
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
{
"mallocCountTotal" : 108
}
"mallocCountTotal" : 90
}
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
{
"mallocCountTotal" : 165000
}
"mallocCountTotal" : 163578
}
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
{
"mallocCountTotal" : 12
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"mallocCountTotal" : 0
}
4 changes: 2 additions & 2 deletions Benchmarks/Thresholds/5.8/NIOPosixBenchmarks.TCPEcho.p90.json
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
{
"mallocCountTotal" : 110
}
"mallocCountTotal" : 92
}
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
{
"mallocCountTotal" : 12
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"mallocCountTotal" : 0
}
4 changes: 2 additions & 2 deletions Benchmarks/Thresholds/5.9/NIOPosixBenchmarks.TCPEcho.p90.json
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
{
"mallocCountTotal" : 110
}
"mallocCountTotal" : 92
}
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
{
"mallocCountTotal" : 164000
}
"mallocCountTotal" : 164224
}
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
{
"mallocCountTotal" : 12
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"mallocCountTotal" : 0
}
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
{
"mallocCountTotal" : 108
}
"mallocCountTotal" : 90
}
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
{
"mallocCountTotal" : 165000
}
"mallocCountTotal" : 164212
}
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
{
"mallocCountTotal" : 12
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"mallocCountTotal" : 0
}
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
{
"mallocCountTotal" : 108
}
"mallocCountTotal" : 90
}
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
{
"mallocCountTotal" : 165000
}
"mallocCountTotal" : 164204
}
4 changes: 2 additions & 2 deletions dev/update-benchmark-thresholds.sh
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,10 @@ set -o pipefail
here="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
target_repo=${2-"$here/.."}

for f in 57 58 59 510 -nightly; do
for f in 58 59 510 main next; do
echo "swift$f"

docker_file=$(if [[ "$f" == "-nightly" ]]; then f=main; fi && ls "$target_repo/docker/docker-compose."*"$f"*".yaml")
docker_file=$(ls "$target_repo/docker/docker-compose."*"$f"*".yaml")

docker-compose -f docker/docker-compose.yaml -f $docker_file run update-benchmark-baseline
done
4 changes: 2 additions & 2 deletions docker/docker-compose.2204.main.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ services:
test:
image: swift-nio:22.04-main
environment:
- SWIFT_VERSION=main
- SWIFT_VERSION=nightly-main
- MAX_ALLOCS_ALLOWED_10000000_asyncsequenceproducer=21
- MAX_ALLOCS_ALLOWED_1000000_asyncwriter=1000050
- MAX_ALLOCS_ALLOWED_1000_addHandlers=45050
Expand Down Expand Up @@ -79,7 +79,7 @@ services:
update-benchmark-baseline:
image: swift-nio:22.04-main
environment:
- SWIFT_VERSION=main
- SWIFT_VERSION=nightly-main

shell:
image: swift-nio:22.04-main
Expand Down
91 changes: 91 additions & 0 deletions docker/docker-compose.2204.next.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
version: "3"

services:

runtime-setup:
image: swift-nio:22.04-next
build:
args:
base_image: "swiftlang/swift:nightly-6.0-jammy"

unit-tests:
image: swift-nio:22.04-next

integration-tests:
image: swift-nio:22.04-next

documentation-check:
image: swift-nio:22.04-next

test:
image: swift-nio:22.04-next
environment:
- SWIFT_VERSION=nightly-next
- MAX_ALLOCS_ALLOWED_10000000_asyncsequenceproducer=21
- MAX_ALLOCS_ALLOWED_1000000_asyncwriter=1000050
- MAX_ALLOCS_ALLOWED_1000_addHandlers=45050
- MAX_ALLOCS_ALLOWED_1000_addHandlers_sync=38050
- MAX_ALLOCS_ALLOWED_1000_addRemoveHandlers_handlercontext=8050
- MAX_ALLOCS_ALLOWED_1000_addRemoveHandlers_handlername=8050
- MAX_ALLOCS_ALLOWED_1000_addRemoveHandlers_handlertype=8050
- MAX_ALLOCS_ALLOWED_1000_autoReadGetAndSet=18050
- MAX_ALLOCS_ALLOWED_1000_autoReadGetAndSet_sync=0
- MAX_ALLOCS_ALLOWED_1000_copying_bytebufferview_to_array=1050
- MAX_ALLOCS_ALLOWED_1000_copying_circularbuffer_to_array=1050
- MAX_ALLOCS_ALLOWED_1000_getHandlers=8050
- MAX_ALLOCS_ALLOWED_1000_getHandlers_sync=36
- MAX_ALLOCS_ALLOWED_1000_reqs_1_conn=26400
- MAX_ALLOCS_ALLOWED_1000_rst_connections=147000
- MAX_ALLOCS_ALLOWED_1000_tcpbootstraps=3050
- MAX_ALLOCS_ALLOWED_1000_tcpconnections=155050
- MAX_ALLOCS_ALLOWED_1000_udp_reqs=6050
- MAX_ALLOCS_ALLOWED_1000_udpbootstraps=2050
- MAX_ALLOCS_ALLOWED_1000_udpconnections=76050
- MAX_ALLOCS_ALLOWED_1_reqs_1000_conn=393000
- MAX_ALLOCS_ALLOWED_bytebuffer_lots_of_rw=2050
- MAX_ALLOCS_ALLOWED_creating_10000_headers=0
- MAX_ALLOCS_ALLOWED_decode_1000_ws_frames=2050
- MAX_ALLOCS_ALLOWED_encode_1000_ws_frames_holding_buffer=3
- MAX_ALLOCS_ALLOWED_encode_1000_ws_frames_holding_buffer_with_mask=2050
- MAX_ALLOCS_ALLOWED_encode_1000_ws_frames_holding_buffer_with_space=3
- MAX_ALLOCS_ALLOWED_encode_1000_ws_frames_holding_buffer_with_space_with_mask=2050
- MAX_ALLOCS_ALLOWED_encode_1000_ws_frames_new_buffer=3050
- MAX_ALLOCS_ALLOWED_encode_1000_ws_frames_new_buffer_with_mask=5050
- MAX_ALLOCS_ALLOWED_encode_1000_ws_frames_new_buffer_with_space=3050
- MAX_ALLOCS_ALLOWED_encode_1000_ws_frames_new_buffer_with_space_with_mask=5050
- MAX_ALLOCS_ALLOWED_execute_hop_10000_tasks=0
- MAX_ALLOCS_ALLOWED_future_erase_result=4050
- MAX_ALLOCS_ALLOWED_future_lots_of_callbacks=53050
- MAX_ALLOCS_ALLOWED_get_100000_headers_canonical_form=700050
- MAX_ALLOCS_ALLOWED_get_100000_headers_canonical_form_trimming_whitespace=700050
- MAX_ALLOCS_ALLOWED_get_100000_headers_canonical_form_trimming_whitespace_from_long_string=700050
- MAX_ALLOCS_ALLOWED_get_100000_headers_canonical_form_trimming_whitespace_from_short_string=700050
- MAX_ALLOCS_ALLOWED_modifying_1000_circular_buffer_elements=0
- MAX_ALLOCS_ALLOWED_modifying_byte_buffer_view=6050
- MAX_ALLOCS_ALLOWED_ping_pong_1000_reqs_1_conn=343
- MAX_ALLOCS_ALLOWED_read_10000_chunks_from_file=110200
- MAX_ALLOCS_ALLOWED_schedule_10000_tasks=50100
- MAX_ALLOCS_ALLOWED_schedule_and_run_10000_tasks=50050
- MAX_ALLOCS_ALLOWED_scheduling_10000_executions=89
- MAX_ALLOCS_ALLOWED_udp_1000_reqs_1_conn=6200
- MAX_ALLOCS_ALLOWED_udp_1_reqs_1000_conn=165050
- WARN_AS_ERROR_ARG=-Xswiftc -warnings-as-errors
- IMPORT_CHECK_ARG=--explicit-target-dependency-import-check error
# - SANITIZER_ARG=--sanitize=thread # TSan broken still

performance-test:
image: swift-nio:22.04-next

update-benchmark-baseline:
image: swift-nio:22.04-next
environment:
- SWIFT_VERSION=nightly-next

shell:
image: swift-nio:22.04-next

echo:
image: swift-nio:22.04-next

http:
image: swift-nio:22.04-next