Dockerized .NET app throws AccessViolationException when running on M1 Mac host

[Tyrrrz]

Hello.

This was reported on one of my projects here: Tyrrrz/DiscordChatExporter#1016

Judging by the stack trace, the exception appears to be thrown by one of the compiled regular expressions:

System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at DynamicClass.Regex22_Scan(System.Text.RegularExpressions.RegexRunner, System.ReadOnlySpan`1<Char>)

qemu: uncaught target signal 6 (Aborted) - core dumped

For reference, this is how the regexes are used, although nothing is particularly special about them. They're all created with the Compiled flag: https://github.com/Tyrrrz/DiscordChatExporter/blob/b9cd40f21164e697d180539a3e577df47fd03ed9/DiscordChatExporter.Core/Markdown/Parsing/MarkdownParser.cs

And here's the Dockerfile:

https://github.com/Tyrrrz/DiscordChatExporter/blob/b9cd40f21164e697d180539a3e577df47fd03ed9/DiscordChatExporter.Cli.dockerfile

The image runs perfectly fine on Windows, Linux, and non-ARM-based macOS. Unfortunately, I don't have the means to test it firsthand, but it seems that the users in the comments to that issue can reproduce the error very consistently on M1 Macs.

Can you suggest what could be the issue and how can I go about resolving it?

Thank you

[devbytyler]

Also experiencing this, but it was working for me back in January 2023. Both docker and the image (amd64 only) have had updates since then.

My error is slightly different:

Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

at System.Collections.Generic.HashSet1[[System.__Canon, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].AddIfNotPresent(System.__Canon, Int32 ByRef)
...
qemu: uncaught target signal 6 (Aborted) - core dumped

• Using Mac M1 Max.
• Docker 4.19.0

[Tyrrrz]

From what I've discovered, it seems that the issue is that Docker can't emulate arm64 properly on other architectures, and the best solution right now is to provide a separate arm64 image. Apparently there's a way to produce multi-arch images too, but I haven't looked much into that. Kinda invalidates the whole "containerize once, run everywhere" premise of Docker, but I guess the world is not all sunshine and rainbows when the abstraction starts leaking.

[Tyrrrz]

I'll self-answer this and close the discussion because I found the solution. Below is a short overview of what you need to do to correctly build a Docker image for your .NET app, to seamlessly work on both x64 and arm64.

In essence, Docker uses QEMU to emulate arm64 images on x64 (and vice versa). Dotnet does not support being emulated through QEMU, which is why you may get AccessViolatedExceptions. The only solution is to provide an image that doesn't require emulation -- i.e. one matching the host platform. You can push multiple Docker images for different platforms under the same tag using a process called multi-arch build.

The simplest way to do that is by using Docker Buildx which is a set of extensions for docker build. It is capable of building images for multiple platforms, creating the corresponding manifest files, and then automatically pushing them to the registry – all in a single command invocation.

Here's how the build process looks, taken from my GitHub Actions workflow file:

name: docker

on: push

jobs:
  deploy-latest:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Install Docker Buildx
        uses: docker/setup-buildx-action@v2

      - name: Login to DockerHub
        run: >
          echo ${{ secrets.DOCKER_TOKEN }} |
          docker login --username tyrrrz --password-stdin

      - name: Build & push image
        run: >
          docker buildx build
          --file DiscordChatExporter.Cli.dockerfile
          --platform linux/amd64,linux/arm64
          --push
          --tag tyrrrz/discordchatexporter:latest
          .

The important part here is the docker buildx build command and the arguments passed to it. You will want to change the --file and --tag to whatever you use in your project.

The --push option instructs the command to push the resulting images straight to the registry. You can't do the traditional docker build followed by docker push here since you'll actually have two images and a manifest instead of just one image, and docker push currently does not support that.

The --platform option is where the multi-arch build comes into play. In order for your image to support this, however, you need to make a few changes to the Dockerfile. This is taken from my project, with some unrelated parts removed:

# -- Build
# Specify the platform here so that we pull the SDK image matching the host platform,
# instead of the target platform specified during build by the `--platform` option.
# Use the .NET 8.0 preview because the `--arch` option is only available in that version.
# https://github.com/dotnet/dotnet-docker/issues/4388#issuecomment-1459038566
# TODO: switch images to Alpine once .NET 8.0 is released.
# Currently, the correct preview version is only available on Debian.
# https://github.com/dotnet/dotnet-docker/blob/main/samples/selecting-tags.md
FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:8.0-preview AS build

# Expose the target architecture set by the `docker build --platform` option, so that
# we can build the assembly for the correct platform.
ARG TARGETARCH

WORKDIR /build

COPY favicon.ico .
COPY NuGet.config .
COPY Directory.Build.props .
COPY DiscordChatExporter.Core DiscordChatExporter.Core
COPY DiscordChatExporter.Cli DiscordChatExporter.Cli

# Publish a self-contained assembly so we can use a slimmer runtime image
RUN dotnet publish DiscordChatExporter.Cli \
    --configuration Release \
    --self-contained \
    --use-current-runtime \
    --arch $TARGETARCH \
    --output publish/

# -- Run
# Use `runtime-deps` instead of `runtime` because we have a self-contained assembly
FROM --platform=$TARGETPLATFORM mcr.microsoft.com/dotnet/runtime-deps:7.0 AS run

COPY --from=build /build/publish /opt/dce
ENTRYPOINT ["/opt/dce/DiscordChatExporter.Cli"]

The critical part here is the FROM --platform=$BUILDPLATFORM ... in the build stage, which instructs Docker to pull the specified image (dotnet/sdk:8.0-preview) for the host platform (i.e. the computer where you build the image, or the CI server), instead of the platform set by the --platform option (which is what happens by default). This ensures that the build stage of the image can run without emulation.

For dotnet publish, you need to specify the correct platform to target, which might be different from $BUILDPLATFORM (for example, building on linux/amd64 but targeting linux/arm64). Dotnet RIDs are different from Docker platform IDs, so using the --runtime option can be a bit cumbersome. Instead, you can use the --arch option to switch just the architecture while keeping the platform.

The --arch option is added in .NET 8, so you'll have to pull the preview version of the SDK (at the time of writing) to use it. It also understands Docker platform IDs directly, so no need to do any conversion here and just put $TARGETARCH.

I am also building my assembly as self-contained to get a smaller runtime image size, but this is not required.

Finally, the runtime stage of the image should remain pretty much unchanged. I added --platform=$TARGETPLATFORM there for explicitness and symmetry, but that's the default anyway, so it's not required. Note that you also don't need to use the .NET 8 runtime – the SDK can build for lower runtime versions too, and I'm building for .NET 7 in this case.

Once you put everything in place and push your images to the registry, this is how it should look:

Now, when the user pulls your image, Docker will resolve the correct instance based on the host platform. This allows the underlying .NET app to run natively and without emulation.