Potential flaky bwc test failure around model management

[jmazanec15]

Description

Looks like some bwc tests are failing. Failure looks to be around ModelMetadata:

»  fatal error in thread [opensearch[knnBwcCluster-restart-1][clusterApplierService#updateTask][T#1]], exiting
»  java.lang.AssertionError: new global checkpoint [-1] is lower than previous one [9]
»  	at org.opensearch.index.seqno.ReplicationTracker.updateGlobalCheckpointOnPrimary(ReplicationTracker.java:1704)
»  	at org.opensearch.index.seqno.ReplicationTracker.activatePrimaryMode(ReplicationTracker.java:1349)
»  	at org.opensearch.index.shard.IndexShard.updateShardState(IndexShard.java:652)
»  	at org.opensearch.indices.cluster.IndicesClusterStateService.updateShard(IndicesClusterStateService.java:711)
»  	at org.opensearch.indices.cluster.IndicesClusterStateService.createOrUpdateShards(IndicesClusterStateService.java:651)
»  	at org.opensearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:294)
»  	at org.opensearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:608)
»  	at org.opensearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:595)
»  	at org.opensearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:563)
»  	at org.opensearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:486)
»  	at org.opensearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:188)
»  	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854)
»  	at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:283)
»  	at 

> Run with --scan to get full insights.
> Get more help at https://help.gradle.org./

BUILD FAILED in 8m 6s
»  	at org.opensearch.index.mapper.DocumentParser.parseArray(DocumentParser.java:584) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:445) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:415) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:137) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:92) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:251) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.index.shard.IndexShard.prepareIndex(IndexShard.java:1112) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1069) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:2281) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.index.shard.IndexShard.runTranslogRecovery(IndexShard.java:2337) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.index.shard.IndexShard.lambda$openEngineAndRecoverFromTranslog$16(IndexShard.java:2385) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.index.translog.InternalTranslogManager.recoverFromTranslogInternal(InternalTranslogManager.java:150) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.index.translog.InternalTranslogManager.recoverFromTranslog(InternalTranslogManager.java:132) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:2400) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:664) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:123) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:344) [opensearch-core-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:121) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:2772) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
»  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
»  	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
»  Caused by: java.lang.IllegalArgumentException: Illegal format for model metadata. Must be of the form "<KNNEngine>,<SpaceType>,<Dimension>,<ModelState>,<Timestamp>,<Description>,<Error>" or "<KNNEngine>,<SpaceType>,<Dimension>,<ModelState>,<Timestamp>,<Description>,<Error>,<NodeAssignment>".
»  	at org.opensearch.knn.indices.ModelMetadata.fromString(ModelMetadata.java:292) ~[?:?]
»  	at org.opensearch.knn.indices.ModelDao$OpenSearchKNNModelDao.getMetadata(ModelDao.java:465) ~[?:?]
»  	at org.opensearch.knn.index.mapper.ModelFieldMapper.parseCreateField(ModelFieldMapper.java:51) ~[?:?]
»  	at org.opensearch.index.mapper.FieldMapper.parse(FieldMapper.java:269) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	... 26 more

Im wondering if we're missing a case related to model metadata.

Related failures:

1. https://github.com/opensearch-project/k-NN/actions/runs/8320121253/job/22764646196?pr=1545
2. https://github.com/opensearch-project/k-NN/actions/runs/8329040045/job/22790428643?pr=1545

[ryanbogan]

I believe this bug is caused because the changes here are not present in the PR code. The change linked has been backported to the 2.x branch and is present in the 2.13 snapshot. It added an additional field to ModelMetadata. Therefore, the fromString method should now contain a string of length 9, while the PR code throws an exception if the string is not 7 or 8. Rebasing the PR with main should fix the failures.

[jmazanec15]

That explains it. Thanks @ryanbogan. Closing