"Network is unreachable" in identity container

[webmozart]

Steps To Reproduce

We have two nightly cron jobs that

• restart Bitwarden in order to regenerate the Letsencrypt certificates, if necessary
• upgrade Bitwarden to the latest version and free disk space

29 1 * * * /home/bitwarden/bitwarden.sh restart

47 2 * * * /home/bitwarden/bitwarden.sh updateself && /home/bitwarden/bitwarden.sh update && /usr/bin/docker system prune --all --volumes --force

This has worked reliably for five years. Since one of the latest upgrades, however, Bitwarden doesn't work reliably after these cron jobs almost every day. We had problems with the nginx container not being able to access the Docker network, but now, these have gone and instead it is the identity container that fails to connect.

Expected Result

Users can log into Bitwarden and use the application.

Actual Result

Users cannot log into Bitwarden.

Recreating the identity container with docker-compose up -d --force-recreate identity temporarily fixes the problem.

Screenshots or Videos

No response

Additional Context

We first started experiencing this problem on February 17 2024. Since then, this happens almost daily.

Relevant logs from the "identity" container:

2024-02-29 01:29:30.081 +00:00 [Information] Identity started.
2024-02-29 02:00:52.295 +00:00 [Information] Identity started.
2024-02-29 08:45:06.924 +00:00 [Fatal] Unhandled exception: "Network is unreachable"
System.Net.Sockets.SocketException (101): Network is unreachable
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ConnectAsync(Socket socket)
   at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP)
   at System.Net.Sockets.Socket.BeginConnect(IPAddress address, Int32 port, AsyncCallback requestCallback, Object state)
   at MailKit.Net.SocketUtils.ConnectAsync(String host, Int32 port, IPEndPoint localEndPoint, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task.FinishSlow(Boolean userDelegateExecute)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskToAsyncResult.End(IAsyncResult asyncResult)
   at MailKit.Net.SocketUtils.ConnectAsync(String host, Int32 port, IPEndPoint localEndPoint, CancellationToken cancellationToken)
   at MailKit.Net.SocketUtils.ConnectAsync(String host, Int32 port, IPEndPoint localEndPoint, Int32 timeout, CancellationToken cancellationToken)
   at MailKit.MailService.ConnectNetworkAsync(String host, Int32 port, CancellationToken cancellationToken)
   at MailKit.Net.Smtp.SmtpClient.ConnectAsync(String host, Int32 port, SecureSocketOptions options, CancellationToken cancellationToken)
   at Bit.Core.Services.MailKitSmtpMailDeliveryService.SendEmailAsync(MailMessage message) in /home/runner/work/server/server/src/Core/Services/Implementations/MailKitSmtpMailDeliveryService.cs:line 86
   at Bit.Core.Services.HandlebarsMailService.SendTwoFactorEmailAsync(String email, String token) in /home/runner/work/server/server/src/Core/Services/Implementations/HandlebarsMailService.cs:line 116
   at Bit.Core.Services.UserService.SendTwoFactorEmailAsync(User user) in /home/runner/work/server/server/src/Core/Services/Implementations/UserService.cs:line 389
   at Bit.Identity.IdentityServer.BaseRequestValidator`1.BuildTwoFactorResultAsync(User user, Organization organization, T context) in /home/runner/work/server/server/src/Identity/IdentityServer/BaseRequestValidator.cs:line 288
   at Bit.Identity.IdentityServer.BaseRequestValidator`1.ValidateAsync(T context, ValidatedTokenRequest request, CustomValidatorRequestContext validatorContext) in /home/runner/work/server/server/src/Identity/IdentityServer/BaseRequestValidator.cs:line 138
   at Bit.Identity.IdentityServer.ResourceOwnerPasswordValidator.ValidateAsync(ResourceOwnerPasswordValidationContext context) in /home/runner/work/server/server/src/Identity/IdentityServer/ResourceOwnerPasswordValidator.cs:line 105
   at Duende.IdentityServer.Validation.TokenRequestValidator.ValidateResourceOwnerCredentialRequestAsync(NameValueCollection parameters) in /_/src/IdentityServer/Validation/Default/TokenRequestValidator.cs:line 597
   at Duende.IdentityServer.Validation.TokenRequestValidator.RunValidationAsync(Func`2 validationFunc, NameValueCollection parameters) in /_/src/IdentityServer/Validation/Default/TokenRequestValidator.cs:line 269
   at Duende.IdentityServer.Validation.TokenRequestValidator.ValidateRequestAsync(TokenRequestValidationContext context) in /_/src/IdentityServer/Validation/Default/TokenRequestValidator.cs:line 191
   at Duende.IdentityServer.Endpoints.TokenEndpoint.ProcessTokenRequestAsync(HttpContext context) in /_/src/IdentityServer/Endpoints/TokenEndpoint.cs:line 117
   at Duende.IdentityServer.Endpoints.TokenEndpoint.ProcessAsync(HttpContext context) in /_/src/IdentityServer/Endpoints/TokenEndpoint.cs:line 81
   at Duende.IdentityServer.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IdentityServerOptions options, IEndpointRouter router, IUserSession userSession, IEventService events, IIssuerNameService issuerNameService, ISessionCoordinationService sessionCoordinationService) in /_/src/IdentityServer/Hosting/IdentityServerMiddleware.cs:line 101
2024-02-29 08:45:06.939 +00:00 [Error] Connection id ""0HN1OP959NGEQ"", Request id ""0HN1OP959NGEQ:00000001"": An unhandled exception was thrown by the application.
System.Net.Sockets.SocketException (101): Network is unreachable
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ConnectAsync(Socket socket)
   at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP)
   at System.Net.Sockets.Socket.BeginConnect(IPAddress address, Int32 port, AsyncCallback requestCallback, Object state)
   at MailKit.Net.SocketUtils.ConnectAsync(String host, Int32 port, IPEndPoint localEndPoint, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task.FinishSlow(Boolean userDelegateExecute)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskToAsyncResult.End(IAsyncResult asyncResult)
   at MailKit.Net.SocketUtils.ConnectAsync(String host, Int32 port, IPEndPoint localEndPoint, CancellationToken cancellationToken)
   at MailKit.Net.SocketUtils.ConnectAsync(String host, Int32 port, IPEndPoint localEndPoint, Int32 timeout, CancellationToken cancellationToken)
   at MailKit.MailService.ConnectNetworkAsync(String host, Int32 port, CancellationToken cancellationToken)
   at MailKit.Net.Smtp.SmtpClient.ConnectAsync(String host, Int32 port, SecureSocketOptions options, CancellationToken cancellationToken)
   at Bit.Core.Services.MailKitSmtpMailDeliveryService.SendEmailAsync(MailMessage message) in /home/runner/work/server/server/src/Core/Services/Implementations/MailKitSmtpMailDeliveryService.cs:line 86
   at Bit.Core.Services.HandlebarsMailService.SendTwoFactorEmailAsync(String email, String token) in /home/runner/work/server/server/src/Core/Services/Implementations/HandlebarsMailService.cs:line 116
   at Bit.Core.Services.UserService.SendTwoFactorEmailAsync(User user) in /home/runner/work/server/server/src/Core/Services/Implementations/UserService.cs:line 389
   at Bit.Identity.IdentityServer.BaseRequestValidator`1.BuildTwoFactorResultAsync(User user, Organization organization, T context) in /home/runner/work/server/server/src/Identity/IdentityServer/BaseRequestValidator.cs:line 288
   at Bit.Identity.IdentityServer.BaseRequestValidator`1.ValidateAsync(T context, ValidatedTokenRequest request, CustomValidatorRequestContext validatorContext) in /home/runner/work/server/server/src/Identity/IdentityServer/BaseRequestValidator.cs:line 138
   at Bit.Identity.IdentityServer.ResourceOwnerPasswordValidator.ValidateAsync(ResourceOwnerPasswordValidationContext context) in /home/runner/work/server/server/src/Identity/IdentityServer/ResourceOwnerPasswordValidator.cs:line 105
   at Duende.IdentityServer.Validation.TokenRequestValidator.ValidateResourceOwnerCredentialRequestAsync(NameValueCollection parameters) in /_/src/IdentityServer/Validation/Default/TokenRequestValidator.cs:line 597
   at Duende.IdentityServer.Validation.TokenRequestValidator.RunValidationAsync(Func`2 validationFunc, NameValueCollection parameters) in /_/src/IdentityServer/Validation/Default/TokenRequestValidator.cs:line 269
   at Duende.IdentityServer.Validation.TokenRequestValidator.ValidateRequestAsync(TokenRequestValidationContext context) in /_/src/IdentityServer/Validation/Default/TokenRequestValidator.cs:line 191
   at Duende.IdentityServer.Endpoints.TokenEndpoint.ProcessTokenRequestAsync(HttpContext context) in /_/src/IdentityServer/Endpoints/TokenEndpoint.cs:line 117
   at Duende.IdentityServer.Endpoints.TokenEndpoint.ProcessAsync(HttpContext context) in /_/src/IdentityServer/Endpoints/TokenEndpoint.cs:line 81
   at Duende.IdentityServer.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IdentityServerOptions options, IEndpointRouter router, IUserSession userSession, IEventService events, IIssuerNameService issuerNameService, ISessionCoordinationService sessionCoordinationService) in /_/src/IdentityServer/Hosting/IdentityServerMiddleware.cs:line 101
   at Duende.IdentityServer.Hosting.IdentityServerMiddleware.Invoke(HttpContext context, IdentityServerOptions options, IEndpointRouter router, IUserSession userSession, IEventService events, IIssuerNameService issuerNameService, ISessionCoordinationService sessionCoordinationService) in /_/src/IdentityServer/Hosting/IdentityServerMiddleware.cs:line 117
   at Duende.IdentityServer.Hosting.MutualTlsEndpointMiddleware.Invoke(HttpContext context, IAuthenticationSchemeProvider schemes) in /_/src/IdentityServer/Hosting/MutualTlsEndpointMiddleware.cs:line 95
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Duende.IdentityServer.Hosting.DynamicProviders.DynamicSchemeAuthenticationMiddleware.Invoke(HttpContext context) in /_/src/IdentityServer/Hosting/DynamicProviders/DynamicSchemes/DynamicSchemeAuthenticationMiddleware.cs:line 50
   at Duende.IdentityServer.Hosting.BaseUrlMiddleware.Invoke(HttpContext context) in /_/src/IdentityServer/Hosting/BaseUrlMiddleware.cs:line 27
   at Bit.Core.Utilities.CurrentContextMiddleware.Invoke(HttpContext httpContext, ICurrentContext currentContext, GlobalSettings globalSettings) in /home/runner/work/server/server/src/Core/Utilities/CurrentContextMiddleware.cs:line 19
   at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
   at Bit.SharedWeb.Utilities.ServiceCollectionExtensions.<>c__DisplayClass11_0.<<UseDefaultMiddleware>b__1>d.MoveNext() in /home/runner/work/server/server/src/SharedWeb/Utilities/ServiceCollectionExtensions.cs:line 560
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Builder.Extensions.UsePathBaseMiddleware.InvokeCore(HttpContext context, PathString matchedPath, PathString remainingPath)
   at Bit.Identity.Startup.<>c__DisplayClass10_1.<<Configure>b__2>d.MoveNext() in /home/runner/work/server/server/src/Identity/Startup.cs:line 182
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

docker-compose.yml:

#
# Useful references:
# https://docs.docker.com/compose/compose-file/
# https://docs.docker.com/compose/reference/overview/#use--f-to-specify-name-and-path-of-one-or-more-compose-files
# https://docs.docker.com/compose/reference/envvars/
#
#########################################################################
# WARNING: This file is generated. Do not make changes to this file.    #
# They will be overwritten on update. If you want to make additions to  #
# this file, you can create a `docker-compose.override.yml` file in the #
# same directory and it will be merged into this file at runtime. You   #
# can also manage various settings used in this file from the           #
# ./bwdata/config.yml file for your installation.                       #
#########################################################################

version: '3'

services:
  mssql:
    image: bitwarden/mssql:2024.2.2
    container_name: bitwarden-mssql
    restart: always
    stop_grace_period: 60s
    volumes:
      - ../mssql/data:/var/opt/mssql/data
      - ../logs/mssql:/var/opt/mssql/log
      - ../mssql/backups:/etc/bitwarden/mssql/backups
    env_file:
      - mssql.env
      - ../env/uid.env
      - ../env/mssql.override.env

  web:
    image: bitwarden/web:2024.2.2
    container_name: bitwarden-web
    restart: always
    volumes:
      - ../web:/etc/bitwarden/web
    env_file:
      - global.env
      - ../env/uid.env

  attachments:
    image: bitwarden/attachments:2024.2.2
    container_name: bitwarden-attachments
    restart: always
    volumes:
      - ../core/attachments:/etc/bitwarden/core/attachments
    env_file:
      - global.env
      - ../env/uid.env

  api:
    image: bitwarden/api:2024.2.2
    container_name: bitwarden-api
    restart: always
    volumes:
      - ../core:/etc/bitwarden/core
      - ../ca-certificates:/etc/bitwarden/ca-certificates
      - ../logs/api:/etc/bitwarden/logs
    env_file:
      - global.env
      - ../env/uid.env
      - ../env/global.override.env
    networks:
      - default
      - public

  identity:
    image: bitwarden/identity:2024.2.2
    container_name: bitwarden-identity
    restart: always
    volumes:
      - ../identity:/etc/bitwarden/identity
      - ../core:/etc/bitwarden/core
      - ../ca-certificates:/etc/bitwarden/ca-certificates
      - ../logs/identity:/etc/bitwarden/logs
    env_file:
      - global.env
      - ../env/uid.env
      - ../env/global.override.env
    networks:
      - default
      - public

  sso:
    image: bitwarden/sso:2024.2.2
    container_name: bitwarden-sso
    restart: always
    volumes:
      - ../identity:/etc/bitwarden/identity
      - ../core:/etc/bitwarden/core
      - ../ca-certificates:/etc/bitwarden/ca-certificates
      - ../logs/sso:/etc/bitwarden/logs
    env_file:
      - global.env
      - ../env/uid.env
      - ../env/global.override.env
    networks:
      - default
      - public

  admin:
    image: bitwarden/admin:2024.2.2
    container_name: bitwarden-admin
    restart: always
    depends_on:
      - mssql
    volumes:
      - ../core:/etc/bitwarden/core
      - ../ca-certificates:/etc/bitwarden/ca-certificates
      - ../logs/admin:/etc/bitwarden/logs
    env_file:
      - global.env
      - ../env/uid.env
      - ../env/global.override.env
    networks:
      - default
      - public

  icons:
    image: bitwarden/icons:2024.2.2
    container_name: bitwarden-icons
    restart: always
    volumes:
      - ../ca-certificates:/etc/bitwarden/ca-certificates
      - ../logs/icons:/etc/bitwarden/logs
    env_file:
      - global.env
      - ../env/uid.env
    networks:
      - default
      - public

  notifications:
    image: bitwarden/notifications:2024.2.2
    container_name: bitwarden-notifications
    restart: always
    volumes:
      - ../ca-certificates:/etc/bitwarden/ca-certificates
      - ../logs/notifications:/etc/bitwarden/logs
    env_file:
      - global.env
      - ../env/uid.env
      - ../env/global.override.env
    networks:
      - default
      - public

  events:
    image: bitwarden/events:2024.2.2
    container_name: bitwarden-events
    restart: always
    volumes:
      - ../ca-certificates:/etc/bitwarden/ca-certificates
      - ../logs/events:/etc/bitwarden/logs
    env_file:
      - global.env
      - ../env/uid.env
      - ../env/global.override.env
    networks:
      - default
      - public

  nginx:
    image: bitwarden/nginx:2024.2.2
    container_name: bitwarden-nginx
    restart: always
    depends_on:
      - web
      - admin
      - api
      - identity
    ports:
      - '80:8080'
      - '443:8443'
    volumes:
      - ../nginx:/etc/bitwarden/nginx
      - ../letsencrypt:/etc/letsencrypt
      - ../ssl:/etc/ssl
      - ../logs/nginx:/var/log/nginx
    env_file:
      - ../env/uid.env
    networks:
      - default
      - public


networks:
  default:
    internal: true
  public:
    internal: false

Build Version

2024.2.2

Environment

Self-Hosted

Environment Details

• OS: Ubuntu 20.04.6 LTS
• Environment: AWS EC2 t3a.medium, 2 vCPUs, 4 GB RAM, 25 GB storage

Issue Tracking Info

• I understand that work is tracked outside of Github. A PR will be linked to this issue should one be opened to address it, but Bitwarden doesn't use fields like "assigned", "milestone", or "project" to track progress.

[NovaSilentium]

Hi there,

I attempted to reproduce your issue and was unable to do so.

please can you write us back using our “Contact support” form located on our Help Center (https://bitwarden.com/help/).

We use GitHub issues as a place to track bugs and other development related issues.

You can include a link to this issue in the message content.

Alternatively, you can also search for an answer in our help documentation or get help from other Bitwarden users on our community forums (https://community.bitwarden.com/c/support/).

The issue here will be closed.