[WIP] Add performance testing #63

evrardjp · 2021-05-18T07:01:39Z

Without this, we are not comparing our BCI to UBI.
This is a problem, as we are not proving the value of our
efforts, from a technical standpoint.
We should show why our containers are probably better from
a technical perspective (or show where we can improve).

This fixes it by adding multiple builds of of the same
application, in 3 different runtimes: BCI, SLE current default
image, and UBI from RH. To make it significant, I chose
to use software which makes sense for ISVs or SUSE
Hub, so I chose grafana.

The grafana UBI files come from RH dockerfile in https://catalog.redhat.com/software/containers/rhel8/grafana/5edf9c35dd19c7063a62aff6?container-tabs=dockerfile

Closes: #41

evrardjp · 2021-05-18T07:02:36Z

self = DockerRuntime(marks=None, build_command='docker build .', runner_binary='docker', _runtime_functional=True), stdout = ''

    def get_image_id_from_stdout(self, stdout: str) -> str:
        # docker build prints this into the last non-empty line:
        # Successfully built 1e3c746e8069
        # -> grab the last line (see podman) & the last entry
>       last_line = list(
            filter(None, map(lambda l: l.strip(), stdout.split("\n")))
        )[-1]
E       IndexError: list index out of range

That's because stdout is empty:

CommandResult(command=b'docker build .', exit_status=0, stdout=None, stderr=b'#1 [internal] load build definition from Dockerfile\n#1 sha256:8b3833760bbc33cf0e9a2a352d88447990e297935136e77284eb85d183812c72\n#1 transferring dockerfile: 75B done\n#1 DONE 0.0s\n\n#2 [internal] load .dockerignore\n#2 sha256:6aa514c4bbf2349aa16efaca71cf742ec1b916d5b0c1076bfa60ca49597ab213\n#2 transferring context: 2B done\n#2 DONE 0.0s\n\n#3 [internal] load metadata for docker.io/library/ubuntu:bionic\n#3 sha256:42b24fe0142b829277f500329d7c50fc0d654412fc6d296283027ac72a38314d\n#3 DONE 0.7s\n\n#4 [1/2] FROM docker.io/library/ubuntu:bionic@sha256:538529c9d229fb55f50e6746b119e899775205d62c0fc1b7e679b30d02ecb6e8\n#4 sha256:fd6c9ed36b7ae183be04e452dbad1302f769da67a97754b8644470bc461a059c\n#4 DONE 0.0s\n\n#5 [2/2] WORKDIR /TODO\n#5 sha256:27f20393f7b82d6f80b1f8623c952ae9cb3a6c794aafcd6c7925101715235fe8\n#5 CACHED\n\n#6 exporting to image\n#6 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00\n#6 exporting layers done\n#6 writing image sha256:0062df290c70333608a837ae460300c1fc3d7cf60e248ae4cb5bdef7b4559a0b done\n#6 DONE 0.0s\n') and vars: {'exit_status': 0, '_stdout_bytes': b'', '_stderr_bytes': b'#1 [internal] load build definition from Dockerfile\n#1 sha256:8b3833760bbc33cf0e9a2a352d88447990e297935136e77284eb85d183812c72\n#1 transferring dockerfile: 75B done\n#1 DONE 0.0s\n\n#2 [internal] load .dockerignore\n#2 sha256:6aa514c4bbf2349aa16efaca71cf742ec1b916d5b0c1076bfa60ca49597ab213\n#2 transferring context: 2B done\n#2 DONE 0.0s\n\n#3 [internal] load metadata for docker.io/library/ubuntu:bionic\n#3 sha256:42b24fe0142b829277f500329d7c50fc0d654412fc6d296283027ac72a38314d\n#3 DONE 0.7s\n\n#4 [1/2] FROM docker.io/library/ubuntu:bionic@sha256:538529c9d229fb55f50e6746b119e899775205d62c0fc1b7e679b30d02ecb6e8\n#4 sha256:fd6c9ed36b7ae183be04e452dbad1302f769da67a97754b8644470bc461a059c\n#4 DONE 0.0s\n\n#5 [2/2] WORKDIR /TODO\n#5 sha256:27f20393f7b82d6f80b1f8623c952ae9cb3a6c794aafcd6c7925101715235fe8\n#5 CACHED\n\n#6 exporting to image\n#6 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00\n#6 exporting layers done\n#6 writing image sha256:0062df290c70333608a837ae460300c1fc3d7cf60e248ae4cb5bdef7b4559a0b done\n#6 DONE 0.0s\n', '_stdout': None, '_stderr': None, 'command': b'docker build .', '_backend': <testinfra.backend.local.LocalBackend object at 0x7f0ec66e5b80>}

Here is the output of the build:

Looking better:
[+] Building 1.6s (8/8) FINISHED                                                                                                                                       
 => [internal] load build definition from Dockerfile                                                                                                              0.0s
 => => transferring dockerfile: 92B                                                                                                                               0.0s
 => [internal] load .dockerignore                                                                                                                                 0.0s
 => => transferring context: 2B                                                                                                                                   0.0s
 => [internal] load metadata for docker.io/library/ubuntu:bionic                                                                                                  1.3s
 => [auth] library/ubuntu:pull token for registry-1.docker.io                                                                                                     0.0s
 => [1/3] FROM docker.io/library/ubuntu:bionic@sha256:538529c9d229fb55f50e6746b119e899775205d62c0fc1b7e679b30d02ecb6e8                                            0.0s
 => CACHED [2/3] WORKDIR /TODO                                                                                                                                    0.0s
 => [3/3] RUN echo "woot"                                                                                                                                         0.2s
 => exporting to image                                                                                                                                            0.0s
 => => exporting layers                                                                                                                                           0.0s
 => => writing image sha256:f67fe10a7de209534d274c565291423b968190f0e4f39d8eb3cfc8012bed2509                                                                      0.0s

How do the build here differ from the previous one we are doing?

evrardjp · 2021-05-18T07:18:32Z

The docker build --no-cache didn't change a thing

tox.ini

evrardjp · 2021-05-18T10:51:45Z

test_performance.py

+        # (the container would be gone then)
+        host.run(
+            f"{container_runtime.runner_binary} stop "
+            + container_runtime.get_image_id_from_stdout(cmd.stdout)


does that work properly ? See also L56. (Or maybe build and run don't have the same kind of output?)

It appears to do the right thing on my machine: no containers are left on the system with both podman and docker after running the performance tests.

evrardjp · 2021-05-18T11:32:13Z

Rebased, squashed a few commits together.

evrardjp · 2021-05-18T11:33:19Z

dockerfiles/perftest-kafka/UBI8-Standard/Dockerfile

+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+


To be closer to the reality between UBI and BCI, we need to also have a base container built, or need to build both base containers in this test.

evrardjp · 2021-05-21T11:09:26Z

This is still a draft... More work needs to be done to compare the different containers:

Making those container build and run for the differnet distros
Discuss whether or not the pytest junit files are enough. If not, parse podman build results for more details/profiling.
Add not only build time testing, but also other runtime/performance testing

Without this, we are not comparing our BCI to UBI. This is a problem, as we are not proving the value of our efforts, from a technical standpoint. We should show why our containers are probably better from a technical perspective (or show where we can improve). This fixes it by adding multiple builds of of the same application, in 3 different runtimes: BCI, SLE current default image, and UBI from RH. To make it significant, I chose to use software which makes sense for ISVs or SUSE Hub, so I chose Kafka. The Kafka UBI files come from confluentinc/kafka-images/kafka. The Dockerfile has been renamed, and follow up changes will need to be tracked in git, so we can easily compare them.

Without this, we are reusing the default env. This is a problem, as it does extra stuff we don't need, like fetching containers. To represent a user behaviour, no prefetch should be done.

stdlib, external, then internal libs.

This will make sure the latest containers are built. It will remove a few transient issues.

dirkmueller · 2023-03-17T09:19:40Z

Closing this for now. Please don't hesitate to reopen if there is time to rebase and finish it off!

evrardjp changed the title ~~Add performance testing~~ [WIP] Add performance testing May 18, 2021

evrardjp force-pushed the perf-test branch from a5ea373 to 58db6fd Compare May 18, 2021 07:40

dcermak marked this pull request as draft May 18, 2021 10:45

evrardjp commented May 18, 2021

View reviewed changes

evrardjp force-pushed the perf-test branch from 13e4ed4 to d0594e5 Compare May 18, 2021 11:32

evrardjp commented May 18, 2021

View reviewed changes

dcermak force-pushed the perf-test branch 3 times, most recently from 0469dc9 to a75bda2 Compare May 19, 2021 05:45

evrardjp force-pushed the perf-test branch from a75bda2 to 6a7c815 Compare May 20, 2021 11:48

dcermak force-pushed the perf-test branch from 6a7c815 to ec1872f Compare May 20, 2021 12:37

evrardjp and others added 8 commits June 7, 2021 08:04

Add fixture to launch zookeeper from confluent

373a504

Fix the tag in the kafka Dockerfile

ffc7ed1

Add performance tests to tox.ini

c27e346

Without this, we are reusing the default env. This is a problem, as it does extra stuff we don't need, like fetching containers. To represent a user behaviour, no prefetch should be done.

Make the kafka fixture only available for perf

245e78a

Cleanup imports

743287d

stdlib, external, then internal libs.

Replace kafka by grafana for perftest

227fe76

Do not use cache to build container

93d3ea7

This will make sure the latest containers are built. It will remove a few transient issues.

dcermak force-pushed the perf-test branch from ec1872f to 93d3ea7 Compare June 7, 2021 06:06

dirkmueller closed this Mar 17, 2023

dirkmueller deleted the perf-test branch October 20, 2023 23:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Add performance testing #63

[WIP] Add performance testing #63

evrardjp commented May 18, 2021 •

edited

Loading

evrardjp commented May 18, 2021 •

edited

Loading

evrardjp commented May 18, 2021

evrardjp May 18, 2021 •

edited

Loading

dcermak May 18, 2021

evrardjp commented May 18, 2021

evrardjp May 18, 2021

evrardjp commented May 21, 2021

dirkmueller commented Mar 17, 2023

[WIP] Add performance testing #63

[WIP] Add performance testing #63

Conversation

evrardjp commented May 18, 2021 • edited Loading

evrardjp commented May 18, 2021 • edited Loading

evrardjp commented May 18, 2021

evrardjp May 18, 2021 • edited Loading

Choose a reason for hiding this comment

dcermak May 18, 2021

Choose a reason for hiding this comment

evrardjp commented May 18, 2021

evrardjp May 18, 2021

Choose a reason for hiding this comment

evrardjp commented May 21, 2021

dirkmueller commented Mar 17, 2023

evrardjp commented May 18, 2021 •

edited

Loading

evrardjp commented May 18, 2021 •

edited

Loading

evrardjp May 18, 2021 •

edited

Loading