Hopefully optimize the DB and split the YAML schema out #27
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
SaphireLattice
force-pushed
the
feature/optimize-db
branch
from
9d1533e to
35a936a
Compare
SaphireLattice
force-pushed
the
feature/optimize-db
branch
from
35a936a to
6d4f144
Compare
Simyon264
reviewed
Aug 21, 2024
Comment on lines
+36
to
+77
| INSERT INTO "JobDepartments" | ||
| ( "Job", "Department" ) | ||
| VALUES | ||
| ( 'Captain', 'Command' ), | ||
| ( 'HeadOfPersonnel', 'Command' ), | ||
| ( 'ChiefMedicalOfficer', 'Command' ), | ||
| ( 'ResearchDirector', 'Command' ), | ||
| ( 'HeadOfSecurity', 'Command' ), | ||
| ( 'ChiefEngineer', 'Command' ), | ||
| ( 'Quartermaster', 'Command' ), | ||
| ( 'Borg', 'Science' ), | ||
| ( 'Scientist', 'Science' ), | ||
| ( 'ResearchAssistant', 'Science' ), | ||
| ( 'Warden', 'Security' ), | ||
| ( 'Detective', 'Security' ), | ||
| ( 'SecurityOfficer', 'Security' ), | ||
| ( 'SecurityCadet', 'Security' ), | ||
| ( 'MedicalDoctor', 'Medical' ), | ||
| ( 'Chemist', 'Medical' ), | ||
| ( 'Paramedic', 'Medical' ), | ||
| ( 'Psychologist', 'Medical' ), | ||
| ( 'MedicalIntern', 'Medical' ), | ||
| ( 'StationEngineer', 'Engineering' ), | ||
| ( 'AtmosphericTechnician', 'Engineering' ), | ||
| ( 'TechnicalAssistant', 'Engineering' ), | ||
| ( 'Janitor', 'Service' ), | ||
| ( 'Chef', 'Service' ), | ||
| ( 'Botanist', 'Service' ), | ||
| ( 'Bartender', 'Service' ), | ||
| ( 'Chaplain', 'Service' ), | ||
| ( 'Lawyer', 'Service' ), | ||
| ( 'Musician', 'Service' ), | ||
| ( 'Reporter', 'Service' ), | ||
| ( 'Zookeeper', 'Service' ), | ||
| ( 'Librarian', 'Service' ), | ||
| ( 'ServiceWorker', 'Service' ), | ||
| ( 'Clown', 'Service' ), | ||
| ( 'Mime', 'Service' ), | ||
| ( 'CargoTechnician', 'Cargo' ), | ||
| ( 'SalvageSpecialist', 'Cargo' ), | ||
| ( 'Passenger', 'The tide' ); | ||
| """ |
There was a problem hiding this comment.
A bit nitpick-y: This doesn't include any of the RMC14 jobs. It would be nice to have them also count (for example) to command. Think of CMCMO or something. But I don't really know any of the roles so /shrug/
There was a problem hiding this comment.
Should they be added into existing departments?
Could also make an extra category for Xenos because... apparently those are jobs
There was a problem hiding this comment.
Maybe? Unsure. It would certainly look fancy.
| @@ -50,5 +52,11 @@ public static IHostBuilder CreateHostBuilder(string[] args) => | |||
| { | |||
| webBuilder.UseKestrel(); | |||
| webBuilder.UseStartup<Startup>(); | |||
| }) | |||
| .ConfigureServices(s => { | |||
There was a problem hiding this comment.
Whats the reason for pulling this out of the startup? Not an issue, just curious.
There was a problem hiding this comment.
They depend on DbContext, but need setting up somewhere, and so they ended up outside. And put after dbcontext has migrated, so they don't error out as they rush to do things.
And migration has to be in other part of Startup because otherwise the EF dotnet tool just breaks in weird ways
Simyon264
requested changes
Aug 21, 2024
There was a problem hiding this comment.
Ingest fails
System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')
at System.Collections.Generic.List`1.get_Item(Int32 index)
at ReplayBrowser.Services.ReplayParser.ReplayParserService.<>c__DisplayClass17_1.<ParseReplay>b__6(JobDepartment j) in D:\Projects\ReplayBrowser\ReplayBrowser\Services\ReplayParser\ReplayParserService.cs:line 271
at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Func`2 predicate, Boolean& found)
at System.Linq.Enumerable.SingleOrDefault[TSource](IEnumerable`1 source, Func`2 predicate)
at ReplayBrowser.Services.ReplayParser.ReplayParserService.<>c__DisplayClass17_0.<ParseReplay>b__5(Player pl) in D:\Projects\ReplayBrowser\ReplayBrowser\Services\ReplayParser\ReplayParserService.cs:line 271
at System.Collections.Generic.List`1.ForEach(Action`1 action)
at ReplayBrowser.Services.ReplayParser.ReplayParserService.<>c__DisplayClass17_0.<ParseReplay>b__2(ReplayParticipant p) in D:\Projects\ReplayBrowser\ReplayBrowser\Services\ReplayParser\ReplayParserService.cs:line 270
at System.Collections.Generic.List`1.ForEach(Action`1 action)
at ReplayBrowser.Services.ReplayParser.ReplayParserService.ParseReplay(Stream stream, String replayLink) in D:\Projects\ReplayBrowser\ReplayBrowser\Services\ReplayParser\ReplayParserService.cs:line 270
at ReplayBrowser.Services.ReplayParser.ReplayParserService.<>c__DisplayClass16_1.<<ConsumeQueue>b__1>d.MoveNext() in D:\Projects\ReplayBrowser\ReplayBrowser\Services\ReplayParser\ReplayParserService.cs:line 163
2024-08-21 18:14:55.742 +02:00 [ERR] Failed executing DbCommand (24ms) [Parameters=[@p0='2024-08-02T22:00:00.0000000Z' (Nullable = true) (DbType = DateTime), @p1='03:20:12.0879879' (Nullable = false), @p2='446496', @p3='04:08:03.1517835' (Nullable = false), @p4='1013', @p5='Secret' (Nullable = false), @p6='https://sector-umbra.net/replays/2024/08/2024-08-03-round_534.zip' (Nullable = false), @p7='Saltern', @p8=NULL (DbType = Object), @p9=NULL, @p10=NULL (DbType = Int32), @p11=NULL (Nullable = false), @p12=NULL, @p13='195665681', @p14='1074567837'], CommandType='"Text"', CommandTimeout='600']
INSERT INTO "Replays" ("Date", "Duration", "EndTick", "EndTime", "FileCount", "Gamemode", "Link", "Map", "Maps", "RoundEndText", "RoundId", "ServerId", "ServerName", "Size", "UncompressedSize")
VALUES (@p0, @p1, @p2, @p3, @p4, @p5, @p6, @p7, @p8, @p9, @p10, @p11, @p12, @p13, @p14)
RETURNING "Id", "RoundEndTextSearchVector";
2024-08-21 18:14:55.778 +02:00 [ERR] An exception occurred in the database while saving changes for context type 'ReplayBrowser.Data.ReplayDbContext'.
Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred while saving the entity changes. See the inner exception for details.
---> Npgsql.PostgresException (0x80004005): 23502: null value in column "ServerId" of relation "Replays" violates not-null constraint
DETAIL: Detail redacted as it may contain sensitive data. Specify 'Include Error Detail' in the connection string to include this information.
at Npgsql.Internal.NpgsqlConnector.ReadMessageLong(Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
Exception data:
Severity: ERROR
SqlState: 23502
MessageText: null value in column "ServerId" of relation "Replays" violates not-null constraint
Detail: Detail redacted as it may contain sensitive data. Specify 'Include Error Detail' in the connection string to include this information.
SchemaName: public
TableName: Replays
ColumnName: ServerId
File: execMain.c
Line: 2009
Routine: ExecConstraints
--- End of inner exception stack trace ---
at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred while saving the entity changes. See the inner exception for details.
---> Npgsql.PostgresException (0x80004005): 23502: null value in column "ServerId" of relation "Replays" violates not-null constraint
DETAIL: Detail redacted as it may contain sensitive data. Specify 'Include Error Detail' in the connection string to include this information.
at Npgsql.Internal.NpgsqlConnector.ReadMessageLong(Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
Exception data:
Severity: ERROR
SqlState: 23502
MessageText: null value in column "ServerId" of relation "Replays" violates not-null constraint
Detail: Detail redacted as it may contain sensitive data. Specify 'Include Error Detail' in the connection string to include this information.
SchemaName: public
TableName: Replays
ColumnName: ServerId
File: execMain.c
Line: 2009
Routine: ExecConstraints
--- End of inner exception stack trace ---
at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
|
here goes nothing |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It's a big one, wheeeee :D
Current list of changes:
BEFORE MERGING: BACKUP THE DB