Error performing DeleteSendsJob

[tjc0726]

Steps To Reproduce

1. Go to 'web-report'
2. Click on 'data leak'
3. Click on 'check leak'

Expected Result

send me the report

Actual Result

says error while generate the report

Screenshots or Videos

No response

Additional Context

The bitwarden/admin:2024.8.0 docker's logger shows:

fail: Bit.Admin.Auth.Jobs.DatabaseExpiredGrantsJob[2]
      Error performing DeleteSendsJob.
      Microsoft.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server)
         at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
         at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, SqlCommand command, Boolean callerHasConnectionLock, Boolean asyncClose)
         at Microsoft.Data.SqlClient.TdsParser.Connect(ServerInfo serverInfo, SqlInternalConnectionTds connHandler, TimeoutTimer timeout, SqlConnectionString connectionOptions, Boolean withFailover)
         at Microsoft.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, TimeoutTimer timeout, Boolean withFailover)
         at Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout)
         at Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance)
         at Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken, DbConnectionPool pool, Func`3 accessTokenCallback)
         at Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
         at Microsoft.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
         at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
         at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
         at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
         at Microsoft.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen()
      --- End of stack trace from previous location ---
         at Dapper.SqlMapper.QueryAsync[T](IDbConnection cnn, Type effectiveType, CommandDefinition command) in /_/Dapper/SqlMapper.Async.cs:line 433
         at Bit.Infrastructure.Dapper.Tools.Repositories.SendRepository.GetManyByDeletionDateAsync(DateTime deletionDateBefore) in /home/runner/work/server/server/src/Infrastructure.Dapper/Tools/Repositories/SendRepository.cs:line 45
         at Bit.Admin.Tools.Jobs.DeleteSendsJob.ExecuteJobAsync(IJobExecutionContext context) in /home/runner/work/server/server/src/Admin/Tools/Jobs/DeleteSendsJob.cs:line 27
         at Bit.Core.Jobs.BaseJob.Execute(IJobExecutionContext context) in /home/runner/work/server/server/src/Core/Jobs/BaseJob.cs:line 19
      ClientConnectionId:00000000-0000-0000-0000-000000000000
fail: Bit.Admin.HostedServices.DatabaseMigrationHostedService[0]
      Database unavailable for migration. Trying again (attempt #2)...
      Microsoft.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server)
         at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
         at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
         at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
         at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
         at Microsoft.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
         at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry, SqlConnectionOverrides overrides)
         at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)
         at Microsoft.Data.SqlClient.SqlConnection.Open()
         at Bit.Migrator.DbMigrator.PrepareDatabase(CancellationToken cancellationToken) in /home/runner/work/server/server/util/Migrator/DbMigrator.cs:line 100
         at Bit.Migrator.DbMigrator.MigrateMsSqlDatabaseWithRetries(Boolean enableLogging, Boolean repeatable, String folderName, Boolean dryRun, CancellationToken cancellationToken) in /home/runner/work/server/server/util/Migrator/DbMigrator.cs:line 39
         at Bit.Migrator.SqlServerDbMigrator.MigrateDatabase(Boolean enableLogging, CancellationToken cancellationToken) in /home/runner/work/server/server/util/Migrator/SqlServerDbMigrator.cs:line 20
         at Bit.Admin.HostedServices.DatabaseMigrationHostedService.StartAsync(CancellationToken cancellationToken) in /home/runner/work/server/server/src/Admin/HostedServices/DatabaseMigrationHostedService.cs:line 29
      ClientConnectionId:00000000-0000-0000-0000-000000000000
warn: Microsoft.AspNetCore.Hosting.Diagnostics[15]
      Overriding HTTP_PORTS '8080' and HTTPS_PORTS ''. Binding to values defined by URLS instead 'http://+:5000'.
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://[::]:5000
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /app

while the sql seems fine according to log:

2024-11-10 13:40:28.19 Server      UTC adjustment: 0:00
2024-11-10 13:40:28.19 Server      (c) Microsoft Corporation.
2024-11-10 13:40:28.19 Server      All rights reserved.
2024-11-10 13:40:28.19 Server      Server process ID is 408.
2024-11-10 13:40:28.19 Server      Logging SQL Server messages in file '/var/opt/mssql/log/errorlog'.
2024-11-10 13:40:28.20 Server      Registry startup parameters: 
	 -d /var/opt/mssql/data/master.mdf
	 -l /var/opt/mssql/data/mastlog.ldf
	 -e /var/opt/mssql/log/errorlog
2024-11-10 13:40:28.22 Server      SQL Server detected 1 sockets with 4 cores per socket and 4 logical processors per socket, 4 total logical processors; using 4 logical processors based on SQL Server licensing. This is an informational message; no user action is required.
2024-11-10 13:40:28.22 Server      SQL Server is starting at normal priority base (=7). This is an informational message only. No user action is required.
2024-11-10 13:40:28.22 Server      Detected 2720 MB of RAM. This is an informational message; no user action is required.
2024-11-10 13:40:28.22 Server      Using conventional memory in the memory manager.
2024-11-10 13:40:28.23 Server      Detected pause instruction latency: 40 cycles.
2024-11-10 13:40:28.23 Server      SQL Server detected the following NUMA node configuration (NUMA Node number 0, Processor Group number 0, CPU Mask 0x000000000000000f).
2024-11-10 13:40:28.23 Server      Page exclusion bitmap is enabled.
2024-11-10 13:40:28.27 Server      Buffer pool extension is not supported on Linux platform.
2024-11-10 13:40:28.27 Server      Buffer Pool: Allocating 262144 bytes for 255010 hashPages.
2024-11-10 13:40:28.52 Server      Buffer pool extension is already disabled. No action is necessary.
2024-11-10 13:40:29.18 Server      Installing Client TLS certificates to the store.
2024-11-10 13:40:29.19 Server      CPU vectorization level(s) detected:  SSE SSE2 SSE3 SSSE3 SSE41 SSE42 AVX AVX2 POPCNT BMI1 BMI2
2024-11-10 13:40:29.36 Server      Successfully initialized the TLS configuration. Allowed TLS protocol versions are ['1.0 1.1 1.2']. Allowed TLS ciphers are ['ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:!DHE-RSA-AES256-GCM-SHA384:!DHE-RSA-AES128-GCM-SHA256:!DHE-RSA-AES256-SHA:!DHE-RSA-AES128-SHA'].
2024-11-10 13:40:29.50 Server      Query Store settings initialized with enabled = 1, 
2024-11-10 13:40:29.53 Server      The maximum number of dedicated administrator connections for this instance is '1'
2024-11-10 13:40:29.56 Server      Node configuration: node 0: CPU mask: 0x000000000000000f:0 Active CPU mask: 0x000000000000000f:0. This message provides a description of the NUMA configuration for this computer. This is an informational message only. No user action is required.
2024-11-10 13:40:29.58 Server      Using dynamic lock allocation.  Initial allocation of 2500 Lock blocks and 5000 Lock Owner blocks per node.  This is an informational message only.  No user action is required.
2024-11-10 13:40:29.64 Server      In-Memory OLTP initialized on lowend machine.
2024-11-10 13:40:29.72 Server      CLR version v4.0.30319 loaded.
2024-11-10 13:40:29.86 Server      [INFO] Created Extended Events session 'hkenginexesession'
2024-11-10 13:40:29.87 Server      Database Instant File Initialization: enabled. For security and performance considerations see the topic 'Database Instant File Initialization' in SQL Server Books Online. This is an informational message only. No user action is required.
ForceFlush is enabled for this instance. 
2024-11-10 13:40:29.89 Server      Total Log Writer threads: 2. This is an informational message; no user action is required.
2024-11-10 13:40:29.92 Server      clflushopt is selected for pmem flush operation.
2024-11-10 13:40:29.93 Server      Software Usage Metrics is disabled.
2024-11-10 13:40:29.98 spid18s     [1]. Feature Status: PVS: 0. CTR: 0. ConcurrentPFSUpdate: 1. ConcurrentGAMUpdate: 1. ConcurrentSGAMUpdate: 1, CleanupUnderUserTransaction: 0. TranLevelPVS: 0
2024-11-10 13:40:30.00 spid18s     Starting up database 'master'.
ForceFlush feature is enabled for log durability.
2024-11-10 13:40:30.11 Server      Common language runtime (CLR) functionality initialized.
2024-11-10 13:40:30.74 spid18s     Service Master Key could not be decrypted using one of its encryptions. See sys.key_encryptions for details.
2024-11-10 13:40:30.74 spid18s     An error occurred during Service Master Key initialization. SQLErrorCode=33095, State=14, LastOsError=2.
2024-11-10 13:40:30.76 spid18s     SQL Server Audit is starting the audits. This is an informational message. No user action is required.
2024-11-10 13:40:30.77 spid18s     SQL Server Audit has started the audits. This is an informational message. No user action is required.
2024-11-10 13:40:30.80 spid18s     XE session 'system_health' started.
2024-11-10 13:40:30.89 spid18s     SQL Trace ID 1 was started by login "sa".
2024-11-10 13:40:30.91 spid18s     Server name is 'fc31698ce639'. This is an informational message only. No user action is required.
2024-11-10 13:40:30.95 spid35s     Always On: The availability replica manager is starting. This is an informational message only. No user action is required.
2024-11-10 13:40:30.95 spid35s     Always On: The availability replica manager is waiting for the instance of SQL Server to allow client connections. This is an informational message only. No user action is required.
2024-11-10 13:40:30.95 spid18s     [4]. Feature Status: PVS: 0. CTR: 0. ConcurrentPFSUpdate: 1. ConcurrentGAMUpdate: 1. ConcurrentSGAMUpdate: 1, CleanupUnderUserTransaction: 0. TranLevelPVS: 0
2024-11-10 13:40:30.95 spid21s     [32767]. Feature Status: PVS: 0. CTR: 0. ConcurrentPFSUpdate: 1. ConcurrentGAMUpdate: 1. ConcurrentSGAMUpdate: 1, CleanupUnderUserTransaction: 0. TranLevelPVS: 0
2024-11-10 13:40:30.96 spid21s     Starting up database 'mssqlsystemresource'.
2024-11-10 13:40:30.97 spid18s     Starting up database 'msdb'.
2024-11-10 13:40:30.97 spid21s     The resource database build version is 16.00.4115. This is an informational message only. No user action is required.
2024-11-10 13:40:31.06 spid21s     [3]. Feature Status: PVS: 0. CTR: 0. ConcurrentPFSUpdate: 1. ConcurrentGAMUpdate: 1. ConcurrentSGAMUpdate: 1, CleanupUnderUserTransaction: 0. TranLevelPVS: 0
2024-11-10 13:40:31.06 spid21s     Starting up database 'model'.
2024-11-10 13:40:31.31 spid21s     Clearing tempdb database.
2024-11-10 13:40:31.57 Server      External governance manager initialized
2024-11-10 13:40:31.83 spid21s     [2]. Feature Status: PVS: 0. CTR: 0. ConcurrentPFSUpdate: 1. ConcurrentGAMUpdate: 1. ConcurrentSGAMUpdate: 1, CleanupUnderUserTransaction: 0. TranLevelPVS: 0
2024-11-10 13:40:31.83 spid21s     Starting up database 'tempdb'.
2024-11-10 13:40:31.83 spid33s     A self-generated certificate was successfully loaded for encryption.
2024-11-10 13:40:31.84 spid33s     Server is listening on [ 'any' <ipv6> 1433] accept sockets 1.
2024-11-10 13:40:31.84 spid33s     Server is listening on [ 'any' <ipv4> 1433] accept sockets 1.
2024-11-10 13:40:31.84 spid33s     Dedicated administrator connection support was not started because it is disabled on this edition of SQL Server. If you want to use a dedicated administrator connection, restart SQL Server using the trace flag 7806. This is an informational message only. No user action is required.
2024-11-10 13:40:31.85 spid33s     Server is listening on [ ::1 <ipv6> 1431] accept sockets 1.
2024-11-10 13:40:31.85 spid33s     Server is listening on [ 127.0.0.1 <ipv4> 1431] accept sockets 1.
2024-11-10 13:40:31.85 spid33s     SQL Server is now ready for client connections. This is an informational message; no user action is required.
2024-11-10 13:40:31.97 spid35s     The Service Broker endpoint is in disabled or stopped state.
2024-11-10 13:40:31.97 spid35s     The Database Mirroring endpoint is in disabled or stopped state.
2024-11-10 13:40:32.00 spid35s     Service Broker manager has started.
2024-11-10 13:40:32.02 spid18s     Recovery is complete. This is an informational message only. No user action is required.
2024-11-10 13:40:32.83 spid51      [5]. Feature Status: PVS: 0. CTR: 0. ConcurrentPFSUpdate: 1. ConcurrentGAMUpdate: 1. ConcurrentSGAMUpdate: 1, CleanupUnderUserTransaction: 0. TranLevelPVS: 0
2024-11-10 13:40:32.83 spid51      Starting up database 'vault'.
2024-11-10 13:40:32.83 spid51      RemoveStaleDbEntries: Cleanup of stale DB entries called for database ID: [5]
2024-11-10 13:40:32.84 spid51      RemoveStaleDbEntries: Cleanup of stale DB entries skipped because master db is not memory optimized. DbId: 5.
2024-11-10 13:40:33.62 spid51      Parallel redo is started for database 'vault' with worker pool size [2].
2024-11-10 13:40:33.71 spid51      Parallel redo is shutdown for database 'vault' with worker pool size [2].
2024-11-10 13:40:35.77 spid20s     Attribute synchronization initialized
2024-11-10 13:40:35.77 spid20s     Attribute synchronization manager initialized

Build Version

2024.8.0

Environment

Self-Hosted

Environment Details

No response

Issue Tracking Info

• I understand that work is tracked outside of Github. A PR will be linked to this issue should one be opened to address it, but Bitwarden doesn't use fields like "assigned", "milestone", or "project" to track progress.

[sammbw]

Hello @tjc0726 - do you have have a Have I Been Pwned API key in your global.override.env file? This is detailed here: https://bitwarden.com/help/reports/#data-breach-report-individual-vaults-only

If you do have the API key, could you try updating your environment to 2024.10.2 and confirm if this still occurs?

[tjc0726]

HI, I dont have the hibp api ,but as it said in your website this feature should be free to use

[sammbw]

Hello @tjc0726 thanks for replying - this only applies to Bitwarden Cloud accounts. For self-hosted environments, an API key is a requirement per the help article I linked previously.

This issue will be closed.