Spark SQL读取数据异常问题

[Jrebel-i]

现有架构：使用AWS EMR，Hadoop 3.3.6, Hive 3.1.3, Spark 3.5.0，apache-celeborn-0.4.0
yarn日志.txt

[hadoop@ip-10-150-30-111 ~]$ spark-sql \
--conf spark.shuffle.manager=org.apache.spark.shuffle.celeborn.SparkShuffleManager \
--conf spark.shuffle.service.enabled=false
May 15, 2024 10:51:05 AM org.apache.spark.launcher.Log4jHotPatchOption staticJavaAgentOption
WARNING: spark.log4jHotPatch.enabled is set to true, but /usr/share/log4j-cve-2021-44228-hotpatch/jdk17/Log4jHotPatchFat.jar does not exist at the configured location

Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
24/05/15 10:51:07 WARN HiveConf: HiveConf of name hive.server2.thrift.url does not exist
24/05/15 10:51:08 WARN SparkShuffleManager: Detected spark.sql.adaptive.localShuffleReader.enabled (default is true) is enabled, it's highly recommended to disable it when use Celeborn as Remote Shuffle Service to avoid performance degradation.
24/05/15 10:51:09 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
Spark Web UI available at http://ip-10-150-30-111.ec2.internal:4040
Spark master: yarn, Application Id: application_1715769583829_0003
spark-sql (default)> select count(*) from dwd.dwd_api_booking_records_d where day_num='2024-05-01';
24/05/15 10:51:28 WARN SparkStringUtils: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.sql.debug.maxToStringFields'.
6917572301047211749
Time taken: 9.679 seconds, Fetched 1 row(s)
spark-sql (default)> select count(*) from dwd.dwd_api_booking_records_d where day_num='2024-05-01';
6917571880140416839
Time taken: 0.685 seconds, Fetched 1 row(s)

[FMX]

Sorry for the delayed response. Does this problem exist now?
For AWS spark, it's highly recommended that you set the celeborn.client.spark.push.unsafeRow.fastWrite.enabled to true.