Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

createIndex if 'NOT indexExists' fails (version 4.15) #283

Open
jmayday opened this issue Aug 22, 2022 · 9 comments
Open

createIndex if 'NOT indexExists' fails (version 4.15) #283

jmayday opened this issue Aug 22, 2022 · 9 comments

Comments

@jmayday
Copy link

jmayday commented Aug 22, 2022

Demo project: https://github.com/jmayday/liquibase-demo
Liquibase version: 4.15.0 #4001 built at 2022-08-05 16:17+0000

I'm adding Liquibase integration to project and I would like to prepare a change set creating indexes if they are missing only. Someone already asked about indexExists precondition here #69 but there was no answer.

This is change set without any precondition and it fails with error below (which is all correct, as there is already index with such name):

<changeSet id="changeset_20220822_1250_1" author="jmayday">
    <ext:createIndex collectionName="testCollection">
        <ext:keys>
            { userId: 1, type: 1}
        </ext:keys>
        <ext:options>
            { unique: false, name: "userId_1" }
        </ext:options>
    </ext:createIndex>
</changeSet>

The error message (again - all is fine, index is already existing so we can't 2nd one with same name):

Caused by: com.mongodb.MongoCommandException: Command failed with error 86 (IndexKeySpecsConflict): 'Index must have unique name.The existing index: { v: 2, key: { userId: 1 }, name: "userId_1", ns: "dev1.testCollection" } has the same name as the requested index: { v: 2, key: { userId: 1, type: 1 }, name: "userId_1", ns: "dev1.testCollection" }' on server localhost:27017. The full response is {"ok": 0.0, "errmsg": "Index must have unique name.The existing index: { v: 2, key: { userId: 1 }, name: \"userId_1\", ns: \"dev1.testCollection\" } has the same name as the requested index: { v: 2, key: { userId: 1, type: 1 }, name: \"userId_1\", ns: \"dev1.testCollection\" }", "code": 86, "codeName": "IndexKeySpecsConflict"}
	at com.mongodb.internal.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:198)

But if I extend the changeSet with preConditions, then an exception is being thrown:

<changeSet id="changeset_20220822_1250_1" author="jmayday">
    <preConditions onFail="MARK_RAN">
        <not>
            <indexExists indexName="userId_1"/>
        </not>
    </preConditions>
    <ext:createIndex collectionName="testCollection">
        <ext:keys>
            { userId: 1, type: 1}
        </ext:keys>
        <ext:options>
            { unique: false, name: "userId_1" }
        </ext:options>
    </ext:createIndex>
</changeSet>

The error:

Exception in thread "main" liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for changeset changesets/changeset_20220819_1428.xml::changeset_20220822_1250_1::jmayday:
     Reason: 
          changelog.xml : Index Exists Precondition: userId_1 : class liquibase.ext.mongodb.database.MongoConnection cannot be cast to class liquibase.database.jvm.JdbcConnection (liquibase.ext.mongodb.database.MongoConnection and liquibase.database.jvm.JdbcConnection are in unnamed module of loader 'app')

Dependency tree:

[INFO] com.myproject:liquibase:jar:1.0.0-SNAPSHOT
[INFO] +- org.liquibase.ext:liquibase-mongodb:jar:4.15.0:compile
[INFO] |  +- org.liquibase:liquibase-core:jar:4.15.0:compile
[INFO] |  |  +- javax.xml.bind:jaxb-api:jar:2.3.1:compile
[INFO] |  |  |  \- javax.activation:javax.activation-api:jar:1.2.0:compile
[INFO] |  |  +- org.yaml:snakeyaml:jar:1.27:compile
[INFO] |  |  \- com.opencsv:opencsv:jar:5.6:compile
[INFO] |  |     +- org.apache.commons:commons-lang3:jar:3.12.0:compile
[INFO] |  |     +- org.apache.commons:commons-text:jar:1.9:compile
[INFO] |  |     \- org.apache.commons:commons-collections4:jar:4.4:compile
[INFO] |  +- org.mongodb:mongodb-driver-sync:jar:4.6.1:compile
[INFO] |  +- org.mongodb:mongodb-driver-core:jar:4.6.1:compile
[INFO] |  |  \- org.mongodb:bson-record-codec:jar:4.6.1:runtime
[INFO] |  +- org.mongodb:bson:jar:4.6.1:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.13.3:compile
[INFO] |  |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.13.3:compile
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.13.3:compile
[INFO] |  \- org.projectlombok:lombok:jar:1.18.22:compile
[INFO] +- commons-cli:commons-cli:jar:1.5.0:compile
[INFO] \- ch.qos.logback:logback-classic:jar:1.2.11:compile
[INFO]    +- ch.qos.logback:logback-core:jar:1.2.11:compile
[INFO]    \- org.slf4j:slf4j-api:jar:1.7.30:compile

When debugging I find method liquibase.snapshot.JdbcDatabaseSnapshot#getMetaDataFromCache to do casting which throws exception because getDatabase().getConnection() is a MongoConnection:

public CachingDatabaseMetaData getMetaDataFromCache() throws SQLException {
    if (cachingDatabaseMetaData == null) {
        DatabaseMetaData databaseMetaData = null;
        if (getDatabase().getConnection() != null) {
            databaseMetaData = ((JdbcConnection) getDatabase().getConnection()).getUnderlyingConnection().getMetaData();
        }

        cachingDatabaseMetaData = new CachingDatabaseMetaData(this.getDatabase(), databaseMetaData);
    }
    return cachingDatabaseMetaData;
}

Whole stacktrace below:

Exception in thread "main" liquibase.exception.LiquibaseException: liquibase.exception.MigrationFailedException: Migration failed for changeset changesets/changeset_20220819_1428.xml::changeset_20220822_1250_1::jmayday:
     Reason: 
          changelog.xml : Index Exists Precondition: userId_1 : class liquibase.ext.mongodb.database.MongoConnection cannot be cast to class liquibase.database.jvm.JdbcConnection (liquibase.ext.mongodb.database.MongoConnection and liquibase.database.jvm.JdbcConnection are in unnamed module of loader 'app')

	at liquibase.changelog.ChangeLogIterator.run(ChangeLogIterator.java:126)
	at liquibase.Liquibase.lambda$null$0(Liquibase.java:263)
	at liquibase.Scope.lambda$child$0(Scope.java:180)
	at liquibase.Scope.child(Scope.java:189)
	at liquibase.Scope.child(Scope.java:179)
	at liquibase.Scope.child(Scope.java:158)
	at liquibase.Scope.child(Scope.java:243)
	at liquibase.Liquibase.lambda$update$1(Liquibase.java:262)
	at liquibase.Scope.lambda$child$0(Scope.java:180)
	at liquibase.Scope.child(Scope.java:189)
	at liquibase.Scope.child(Scope.java:179)
	at liquibase.Scope.child(Scope.java:158)
	at liquibase.Liquibase.runInScope(Liquibase.java:2414)
	at liquibase.Liquibase.update(Liquibase.java:209)
	at liquibase.Liquibase.update(Liquibase.java:195)
	at liquibase.Liquibase.update(Liquibase.java:191)
	at liquibase.Liquibase.update(Liquibase.java:183)
	at com.myproject.Liqui.main(Liqui.java:62)
Caused by: liquibase.exception.MigrationFailedException: Migration failed for changeset changesets/changeset_20220819_1428.xml::changeset_20220822_1250_1::jmayday:
     Reason: 
          changelog.xml : Index Exists Precondition: userId_1 : class liquibase.ext.mongodb.database.MongoConnection cannot be cast to class liquibase.database.jvm.JdbcConnection (liquibase.ext.mongodb.database.MongoConnection and liquibase.database.jvm.JdbcConnection are in unnamed module of loader 'app')

	at liquibase.changelog.ChangeSet.execute(ChangeSet.java:632)
	at liquibase.changelog.visitor.UpdateVisitor.visit(UpdateVisitor.java:56)
	at liquibase.changelog.ChangeLogIterator$2.lambda$null$0(ChangeLogIterator.java:113)
	at liquibase.Scope.lambda$child$0(Scope.java:180)
	at liquibase.Scope.child(Scope.java:189)
	at liquibase.Scope.child(Scope.java:179)
	at liquibase.Scope.child(Scope.java:158)
	at liquibase.changelog.ChangeLogIterator$2.lambda$run$1(ChangeLogIterator.java:112)
	at liquibase.Scope.lambda$child$0(Scope.java:180)
	at liquibase.Scope.child(Scope.java:189)
	at liquibase.Scope.child(Scope.java:179)
	at liquibase.Scope.child(Scope.java:158)
	at liquibase.Scope.child(Scope.java:243)
	at liquibase.changelog.ChangeLogIterator$2.run(ChangeLogIterator.java:93)
	at liquibase.Scope.lambda$child$0(Scope.java:180)
	at liquibase.Scope.child(Scope.java:189)
	at liquibase.Scope.child(Scope.java:179)
	at liquibase.Scope.child(Scope.java:158)
	at liquibase.Scope.child(Scope.java:243)
	at liquibase.Scope.child(Scope.java:247)
	at liquibase.changelog.ChangeLogIterator.run(ChangeLogIterator.java:65)
	... 17 more
Caused by: liquibase.exception.PreconditionErrorException: Precondition Error
	at liquibase.precondition.core.IndexExistsPrecondition.check(IndexExistsPrecondition.java:123)
	at liquibase.precondition.core.NotPrecondition.check(NotPrecondition.java:35)
	at liquibase.precondition.core.AndPrecondition.check(AndPrecondition.java:40)
	at liquibase.precondition.core.PreconditionContainer.check(PreconditionContainer.java:213)
	at liquibase.changelog.ChangeSet.execute(ChangeSet.java:589)
	... 37 more
Caused by: liquibase.exception.DatabaseException: java.lang.ClassCastException: class liquibase.ext.mongodb.database.MongoConnection cannot be cast to class liquibase.database.jvm.JdbcConnection (liquibase.ext.mongodb.database.MongoConnection and liquibase.database.jvm.JdbcConnection are in unnamed module of loader 'app')
	at liquibase.snapshot.jvm.IndexSnapshotGenerator.snapshotObject(IndexSnapshotGenerator.java:304)
	at liquibase.snapshot.jvm.JdbcSnapshotGenerator.snapshot(JdbcSnapshotGenerator.java:66)
	at liquibase.snapshot.SnapshotGeneratorChain.snapshot(SnapshotGeneratorChain.java:49)
	at liquibase.snapshot.DatabaseSnapshot.include(DatabaseSnapshot.java:312)
	at liquibase.snapshot.DatabaseSnapshot.init(DatabaseSnapshot.java:105)
	at liquibase.snapshot.DatabaseSnapshot.<init>(DatabaseSnapshot.java:58)
	at liquibase.snapshot.JdbcDatabaseSnapshot.<init>(JdbcDatabaseSnapshot.java:34)
	at liquibase.snapshot.SnapshotGeneratorFactory.createSnapshot(SnapshotGeneratorFactory.java:215)
	at liquibase.snapshot.SnapshotGeneratorFactory.createSnapshot(SnapshotGeneratorFactory.java:244)
	at liquibase.snapshot.SnapshotGeneratorFactory.has(SnapshotGeneratorFactory.java:133)
	at liquibase.precondition.core.IndexExistsPrecondition.check(IndexExistsPrecondition.java:103)
	... 41 more
Caused by: java.lang.ClassCastException: class liquibase.ext.mongodb.database.MongoConnection cannot be cast to class liquibase.database.jvm.JdbcConnection (liquibase.ext.mongodb.database.MongoConnection and liquibase.database.jvm.JdbcConnection are in unnamed module of loader 'app')
	at liquibase.snapshot.JdbcDatabaseSnapshot.getMetaDataFromCache(JdbcDatabaseSnapshot.java:45)
	at liquibase.snapshot.jvm.IndexSnapshotGenerator.snapshotObject(IndexSnapshotGenerator.java:159)
	... 51 more
@jmayday jmayday changed the title Can't make indexExists precondition working Can't make indexExists precondition working (version 4.15) Aug 22, 2022
@jmayday jmayday changed the title Can't make indexExists precondition working (version 4.15) createIndex if 'NOT indexExists' (version 4.15) Aug 22, 2022
@jmayday jmayday changed the title createIndex if 'NOT indexExists' (version 4.15) createIndex if 'NOT indexExists' fails (version 4.15) Aug 23, 2022
@r-michal-ah
Copy link

hello, any update on it?

@kb-mendozaACN
Copy link

Can we have this fixed pls?

@jmayday
Copy link
Author

jmayday commented Sep 21, 2023

@r-michal-ah @kb-mendozaACN I don't think it's worth waiting. I've started using Mongock, maybe it will be good choice for you guys.

@kb-mendozaACN
Copy link

kb-mendozaACN commented Sep 21, 2023

@jmayday unfortunately, we can not introduce more tools. have you tried creating via mongodb:runCommand? I wanted to try it as well, but checking here first, just in case someone already tried and failed too.

@jmayday
Copy link
Author

jmayday commented Sep 21, 2023

Sorry, I don't understand. What you mean exactly?

@jmayday
Copy link
Author

jmayday commented Sep 21, 2023

I think I tried. But I don't remember exactly, as it was year ago :). I think I exhausted options to use liquibase with Mongo, and there was no support at all (as we see in this thread), so I started checking alternatives and ended up using Mongock.

@pcalouche
Copy link

I would love to see this implemented to. The workaround I found was not fail the change set if there was an error. Better than nothing.

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
  xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext"
  xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
        https://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-latest.xsd
        http://www.liquibase.org/xml/ns/dbchangelog-ext https://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd">

  <!--
    Mongo Liquibase does not have a precondition to check if an index exists, so set
    failOnError to be false. See https://github.com/liquibase/liquibase-mongodb/issues/283.
  -->
  <changeSet id="4" author="psc" failOnError="false">
    <ext:dropIndex collectionName="testers">
      <ext:keys>{firstName: 1}</ext:keys>
    </ext:dropIndex>
    <rollback>
      <ext:createIndex collectionName="testers">
        <ext:keys>{firstName: 1}</ext:keys>
        <ext:options>{name: "firstName_asc"}</ext:options>
      </ext:createIndex>
    </rollback>
    <comment>
      Drops the index on the firstName field in the testers collection
    </comment>
  </changeSet>

</databaseChangeLog>

@maxlupin
Copy link

maxlupin commented Jan 31, 2025

I would also love to see this implemented. On my side the workaround I found was to implement a CustomPrecondition documented in liquibase.

Here is an example of changeset for the creation of an index with name theGreatIndexedField_uniq on the collection myWonderfulCollection for the property theGreatIndexedField :

{
  "changeSet": {
    "id": "1",
    "author": "maxime",
    "preConditions": [
      {
        "onFail": "MARK_RAN"
      },
      {
        "customPrecondition": [
          {
            "className": "org.myproject.utils.liquibase.IndexShouldNotExistsPrecondition"
          },
          {
            "param": {
              "name": "collectionName",
              "value": "myWonderfulCollection"
            }
          },
          {
            "param": {
              "name": "indexName",
              "value": "theGreatIndexedField_uniq"
            }
          }
        ]
      }
    ],
    "changes": [
      {
        "createIndex": {
          "collectionName": "myWonderfulCollection",
          "keys": {
            "$rawJson": {
              "theGreatIndexedField": 1,
              "type":1
            }
          },
          "options": {
            "$rawJson": {
              "unique": true,
              "name": "theGreatIndexedField_uniq",
              "partialFilterExpression": { "theGreatIndexedField": { "$exists": true } }
            }
          }
        }
      }
    ]
  }
}

The collection has 15 millions documents and theGreatIndexField is null on 2 millions of them. This is why partialFilterExpression is here.

And here is the IndexShouldNotExistsPrecondition class

package com.myProject.utils.liquibase;

import org.bson.Document;
import liquibase.database.Database;
import liquibase.exception.CustomPreconditionErrorException;
import liquibase.exception.CustomPreconditionFailedException;
import liquibase.ext.mongodb.database.MongoLiquibaseDatabase;
import liquibase.precondition.CustomPrecondition;

import java.util.List;
import java.util.Optional;

public class IndexShouldNotExistsPrecondition implements CustomPrecondition {

    private String indexName;
    private String collectionName;

    public String getIndexName() {
        return indexName;
    }

    public void setIndexName(String indexName) {
        this.indexName = indexName;
    }

    public String getCollectionName() {
        return collectionName;
    }

    public void setCollectionName(String collectionName) {
        this.collectionName = collectionName;
    }

    @Override
    public void check(Database database) throws CustomPreconditionFailedException, CustomPreconditionErrorException {
        try {
            Document document = new Document();
            document.put("listIndexes", this.collectionName);
            Document result = ((MongoLiquibaseDatabase) database).getMongoDatabase().runCommand(document);

            List<Document> indexes = result.get("cursor", Document.class).getList("firstBatch", Document.class);

            Optional<Document> optIndex = indexes.stream()
                    .filter(index ->
                            index.getString("name").equals(this.indexName)
                    )
                    .findFirst();

            if (optIndex.isPresent())
                throw new CustomPreconditionFailedException("index already exists");
        } catch (Exception e){
            throw new CustomPreconditionFailedException(e.getMessage());
        }

    }


}

Hope this helps

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants