'msck repair tabe ...' is failing in Spark-SQL for tables with more than 100 partitions

[YevhenKv]

Hi,

We have a tables with 100> partitions. When we run 'msck repair tabe ...' in Spark-SQL it is failing with error:

21/08/09 12:32:32 ERROR BatchCreatePartitionsHelper: BatchCreatePartitions failed to create 100 out of 100 partitions.

However, if we run 'msck repair tabe ...' with hive everything works properly.

EMR: 6.3.0
Glue Data catalog: configured for hive and Spark

Found limitation in Glue API reference, but again, there is no error in hive (not sure if it's not raised or it works)
https://docs.aws.amazon.com/glue/latest/webapi/API_BatchCreatePartition.html

[swiatek25]

Have the same issue. @smoy Any updates on that one?

[swiatek25]

I think I found the problem:

aws-glue-data-catalog-client-for-apache-hive-metastore/aws-glue-datacatalog-client-common/src/main/java/com/amazonaws/glue/catalog/util/BatchCreatePartitionsHelper.java

Line 77 in 83e728b

logger.error(String.format("BatchCreatePartitions failed to create %d out of %d partitions. \n",

This is default logging in case of all types of errors. Your particular case is caused by AlreadyExistsException (partition already exists). Basically could be ignored.

I guess one improvement could be warning about already exists exception when it is allowed (ifNotExists) instead of putting the error information non-contextually.