dumpIncrementalCheckpoint may block and leave long living checkpoint

[CalvinNeo]

Bug Report

Please answer these questions before submitting your issue. Thanks!

{"namespace":"cse-htap-apply-snapshot-v71-tps-7515842-1-548","pod":"tc-tiflash-0","log":"2024-08-30T01:29:37.329093891+08:00 stdout F [2024/08/29 17:29:36.765 +00:00] [WARN] [PageDirectory.cpp:2132] [\"Meet a stale snapshot [thread id=15] [tracing id=dumpIncrementalCheckpoint] [seq=91488499] [alive time(s)=5722.792490518]\"] [source=uni_write] [thread_id=60]","container":"serverlog"}

The error is a Check initialize() failed phrase, it will not panic.

See also #9406, which is due to fullgc.

{"namespace":"cse-htap-apply-snapshot-v71-tps-7515842-1-548","pod":"tc-tiflash-0","log":"2024-08-30T01:29:45.868776981+08:00 stdout F [2024/08/29 17:29:45.841 +00:00] [ERROR] [Exception.cpp:96] [\"Code: 49, e.displayText() = DB::Exception: Check initialize() failed, remote_fname = s1043/data/dat_31_0, e.what() = DB::Exception, Stack trace:\\n\\n\\n  0x55e991734e33\\tStackTrace::StackTrace() [tiflash+34512435]\\n                \\tdbms/src/Common/StackTrace.cpp:23\\n  0x55e991732506\\tDB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, int) [tiflash+34501894]\\n                \\tdbms/src/Common/Exception.h:46\\n  0x55e9978fbaff\\tDB::S3::S3RandomAccessFile::S3RandomAccessFile(std::__1::shared_ptr<DB::S3::TiFlashS3Client>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) [tiflash+137038591]\\n                \\tdbms/src/Storages/S3/S3RandomAccessFile.cpp:49\\n  0x55e997fac487\\tstd::__1::__shared_ptr_emplace<DB::S3::S3RandomAccessFile, std::__1::allocator<DB::S3::S3RandomAccessFile>>::__shared_ptr_emplace[abi:ue170006]<std::__1::shared_ptr<DB::S3::TiFlashS3Client>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>(std::__1::allocator<DB::S3::S3RandomAccessFile>, std::__1::shared_ptr<DB::S3::TiFlashS3Client>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&&) [tiflash+144053383]\\n                \\t/usr/lib/llvm-17/bin/../include/c++/v1/__memory/construct_at.h:41\\n  0x55e997faa585\\tDB::PS::V3::S3PageReader::read(std::__1::pair<DB::UniversalPageId, DB::PS::V3::PageEntryV3> const&) [tiflash+144045445]\\n                \\t/usr/lib/llvm-17/bin/../include/c++/v1/__memory/shared_ptr.h:1022\\n  0x55e997ff4fd6\\tDB::PS::V3::CPWriteDataSourceBlobStore::read(std::__1::pair<DB::UniversalPageId, DB::PS::V3::PageEntryV3> const&) [tiflash+144351190]\\n                \\tdbms/src/Storages/Page/V3/CheckpointFile/CPWriteDataSource.cpp:25\\n  0x55e997fdb6b9\\tDB::PS::V3::CPFilesWriter::writeEditsAndApplyCheckpointInfo(DB::PS::V3::PageEntriesEdit<DB::UniversalPageId>&, DB::PS::V3::CPFilesWriter::CompactOptions const&, bool) [tiflash+144246457]\\n                \\tdbms/src/Storages/Page/V3/CheckpointFile/CPFilesWriter.cpp:185\\n  0x55e997fb8b31\\tDB::UniversalPageStorage::dumpIncrementalCheckpoint(DB::UniversalPageStorage::DumpCheckpointOptions const&) [tiflash+144104241]\\n                \\tdbms/src/Storages/Page/V3/Universal/UniversalPageStorage.cpp:551\\n  0x55e997fccd57\\tDB::UniversalPageStorageService::uploadCheckpointImpl(metapb::Store const&, std::__1::shared_ptr<DB::S3::IS3LockClient> const&, std::__1::shared_ptr<DB::DM::Remote::IDataStore> const&, bool) [tiflash+144186711]\\n                \\tdbms/src/Storages/Page/V3/Universal/UniversalPageStorageService.cpp:284\\n  0x55e997fcbe92\\tDB::UniversalPageStorageService::uploadCheckpoint() [tiflash+144182930]\\n                \\tdbms/src/Storages/Page/V3/Universal/UniversalPageStorageService.cpp:159\\n  0x55e997fcd9ce\\tstd::__1::__function::__func<DB::UniversalPageStorageService::create(DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::PSDiskDelegator>, DB::PageStorageConfig const&)::$_0, std::__1::allocator<DB::UniversalPageStorageService::create(DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::PSDiskDelegator>, DB::PageStorageConfig const&)::$_0>, bool ()>::operator()() [tiflash+144189902]\\n                \\tdbms/src/Storages/Page/V3/Universal/UniversalPageStorageService.cpp:74\\n  0x55e9977e715d\\tDB::BackgroundProcessingPool::threadFunction(unsigned long) [tiflash+135905629]\\n                \\t/usr/lib/llvm-17/bin/../include/c++/v1/__functional/function.h:517\\n  0x55e9977e7cd5\\tvoid* std::__1::__thread_proxy[abi:ue170006]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::JointThreadInfoJeallocMap>)::$_1>>(void*) [tiflash+135908565]\\n                \\tdbms/src/Storages/BackgroundProcessingPool.cpp:101\\n  0x7f41b74efac3\\tstart_thread [libc.so.6+608963]\\n                \\t./nptl/./nptl/pthread_create.c:442\\n  0x7f41b7581850\\t__GI___clone3 [libc.so.6+1206352]\\n                \\t./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81\"] [source=\"void DB::BackgroundProcessingPool::threadFunction(size_t)\"] [thread_id=15]","container":"errorlog"}
{"namespace":"cse-htap-apply-snapshot-v71-tps-7515842-1-548","pod":"tc-tiflash-0","log":"2024-08-30T01:29:45.868719011+08:00 stdout F [2024/08/29 17:29:45.802 +00:00] [ERROR] [Exception.cpp:96] [\"Code: 49, e.displayText() = DB::Exception: Check initialize() failed, remote_fname = s1043/data/dat_31_0, e.what() = DB::Exception, Stack trace:\\n\\n\\n  0x55e991734e33\\tStackTrace::StackTrace() [tiflash+34512435]\\n                \\tdbms/src/Common/StackTrace.cpp:23\\n  0x55e991732506\\tDB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, int) [tiflash+34501894]\\n                \\tdbms/src/Common/Exception.h:46\\n  0x55e9978fbaff\\tDB::S3::S3RandomAccessFile::S3RandomAccessFile(std::__1::shared_ptr<DB::S3::TiFlashS3Client>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) [tiflash+137038591]\\n                \\tdbms/src/Storages/S3/S3RandomAccessFile.cpp:49\\n  0x55e997fac487\\tstd::__1::__shared_ptr_emplace<DB::S3::S3RandomAccessFile, std::__1::allocator<DB::S3::S3RandomAccessFile>>::__shared_ptr_emplace[abi:ue170006]<std::__1::shared_ptr<DB::S3::TiFlashS3Client>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>(std::__1::allocator<DB::S3::S3RandomAccessFile>, std::__1::shared_ptr<DB::S3::TiFlashS3Client>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&&) [tiflash+144053383]\\n                \\t/usr/lib/llvm-17/bin/../include/c++/v1/__memory/construct_at.h:41\\n  0x55e997faa585\\tDB::PS::V3::S3PageReader::read(std::__1::pair<DB::UniversalPageId, DB::PS::V3::PageEntryV3> const&) [tiflash+144045445]\\n                \\t/usr/lib/llvm-17/bin/../include/c++/v1/__memory/shared_ptr.h:1022\\n  0x55e997ff4fd6\\tDB::PS::V3::CPWriteDataSourceBlobStore::read(std::__1::pair<DB::UniversalPageId, DB::PS::V3::PageEntryV3> const&) [tiflash+144351190]\\n                \\tdbms/src/Storages/Page/V3/CheckpointFile/CPWriteDataSource.cpp:25\\n  0x55e997fdb6b9\\tDB::PS::V3::CPFilesWriter::writeEditsAndApplyCheckpointInfo(DB::PS::V3::PageEntriesEdit<DB::UniversalPageId>&, DB::PS::V3::CPFilesWriter::CompactOptions const&, bool) [tiflash+144246457]\\n                \\tdbms/src/Storages/Page/V3/CheckpointFile/CPFilesWriter.cpp:185\\n  0x55e997fb8b31\\tDB::UniversalPageStorage::dumpIncrementalCheckpoint(DB::UniversalPageStorage::DumpCheckpointOptions const&) [tiflash+144104241]\\n                \\tdbms/src/Storages/Page/V3/Universal/UniversalPageStorage.cpp:551\\n  0x55e997fccd57\\tDB::UniversalPageStorageService::uploadCheckpointImpl(metapb::Store const&, std::__1::shared_ptr<DB::S3::IS3LockClient> const&, std::__1::shared_ptr<DB::DM::Remote::IDataStore> const&, bool) [tiflash+144186711]\\n                \\tdbms/src/Storages/Page/V3/Universal/UniversalPageStorageService.cpp:284\\n  0x55e997fcbe92\\tDB::UniversalPageStorageService::uploadCheckpoint() [tiflash+144182930]\\n                \\tdbms/src/Storages/Page/V3/Universal/UniversalPageStorageService.cpp:159\\n  0x55e997fcd9ce\\tstd::__1::__function::__func<DB::UniversalPageStorageService::create(DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::PSDiskDelegator>, DB::PageStorageConfig const&)::$_0, std::__1::allocator<DB::UniversalPageStorageService::create(DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::PSDiskDelegator>, DB::PageStorageConfig const&)::$_0>, bool ()>::operator()() [tiflash+144189902]\\n                \\tdbms/src/Storages/Page/V3/Universal/UniversalPageStorageService.cpp:74\\n  0x55e9977e715d\\tDB::BackgroundProcessingPool::threadFunction(unsigned long) [tiflash+135905629]\\n                \\t/usr/lib/llvm-17/bin/../include/c++/v1/__functional/function.h:517\\n  0x55e9977e7cd5\\tvoid* std::__1::__thread_proxy[abi:ue170006]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::JointThreadInfoJeallocMap>)::$_1>>(void*) [tiflash+135908565]\\n                \\tdbms/src/Storages/BackgroundProcessingPool.cpp:101\\n  0x7f41b74efac3\\tstart_thread [libc.so.6+608963]\\n                \\t./nptl/./nptl/pthread_create.c:442\\n  0x7f41b7581850\\t__GI___clone3 [libc.so.6+1206352]\\n                \\t./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81\"] [source=\"CPDataDumpStats DB::PS::V3::CPFilesWriter::writeEditsAndApplyCheckpointInfo(universal::PageEntriesEdit &, const CPFilesWriter::CompactOptions &, bool)\"] [thread_id=15]","container":"errorlog"}
{"namespace":"cse-htap-apply-snapshot-v71-tps-7515842-1-548","pod":"tc-tiflash-0","log":"2024-08-30T01:29:41.868092408+08:00 stdout F [2024/08/29 17:29:41.606 +00:00] [ERROR] [CPFilesWriter.cpp:210] [\"failed to read page, record={type:VAR_ENT, page_id:0x01010200000000000005A6010000000000000023, ori_id:0x.0, version:576457.0, entry:PageEntry{file: 0, offset: 0x0, size: 1491, checksum: 0x0, tag: 0, field_offsets: [], checkpoint_info: {local_data_reclaimed: true, data_location: {data_file_id: lock/s1043/dat_31_0.lock_s1043_31, offset_in_file: 120005, size_in_file: 1491}}}, being_ref_count:1}\"] [thread_id=15]","container":"errorlog"}
{"namespace":"cse-htap-apply-snapshot-v71-tps-7515842-1-548","pod":"tc-tiflash-0","log":"2024-08-30T01:29:41.868063227+08:00 stdout F [2024/08/29 17:29:41.604 +00:00] [ERROR] [S3RandomAccessFile.cpp:197] [\"S3 GetObject failed:  s3error= s3exception_name=NoSuchKey s3msg=The specified key does not exist. request_id=17F0433BD8E74212, cur_retry=3\"] [source=s1043/data/dat_31_0] [thread_id=15]","container":"errorlog"}
{"namespace":"cse-htap-apply-snapshot-v71-tps-7515842-1-548","pod":"tc-tiflash-0","log":"2024-08-30T01:29:41.868060459+08:00 stdout F [2024/08/29 17:29:41.604 +00:00] [ERROR] [S3Common.cpp:145] [\"tag=AWSXmlClient message=HTTP response code: 404\\nResolved remote host IP address: 10.233.120.129\\nRequest ID: 17F0433BD8E74212\\nException name: NoSuchKey\\nError message: The specified key does not exist.\\n12 response headers:\\naccept-ranges : bytes\\ncontent-length : 393\\ncontent-security-policy : block-all-mixed-content\\ncontent-type : application/xml\\ndate : Thu, 29 Aug 2024 17:29:41 GMT\\nserver : MinIO\\nstrict-transport-security : max-age=31536000; includeSubDomains\\nvary : Origin\\nx-amz-id-2 : e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\nx-amz-request-id : 17F0433BD8E74212\\nx-content-type-options : nosniff\\nx-xss-protection : 1; mode=block\"] [source=AWSClient] [thread_id=15]","container":"errorlog"}
{"namespace":"cse-htap-apply-snapshot-v71-tps-7515842-1-548","pod":"tc-tiflash-0","log":"2024-08-30T01:29:41.868057129+08:00 stdout F [2024/08/29 17:29:41.604 +00:00] [WARN] [S3Common.cpp:145] [\"tag=AWSErrorMarshaller message=Encountered AWSError 'NoSuchKey': The specified key does not exist.\"] [source=AWSClient] [thread_id=15]","container":"errorlog"}
{"namespace":"cse-htap-apply-snapshot-v71-tps-7515842-1-548","pod":"tc-tiflash-0","log":"2024-08-30T01:29:41.868051834+08:00 stdout F [2024/08/29 17:29:41.603 +00:00] [ERROR] [S3RandomAccessFile.cpp:197] [\"S3 GetObject failed:  s3error= s3exception_name=NoSuchKey s3msg=The specified key does not exist. request_id=17F0433BD8D4F0B7, cur_retry=2\"] [source=s1043/data/dat_31_0] [thread_id=15]","container":"errorlog"}
{"namespace":"cse-htap-apply-snapshot-v71-tps-7515842-1-548","pod":"tc-tiflash-0","log":"2024-08-30T01:29:41.86804876+08:00 stdout F [2024/08/29 17:29:41.603 +00:00] [ERROR] [S3Common.cpp:145] [\"tag=AWSXmlClient message=HTTP response code: 404\\nResolved remote host IP address: 10.233.120.129\\nRequest ID: 17F0433BD8D4F0B7\\nException name: NoSuchKey\\nError message: The specified key does not exist.\\n12 response headers:\\naccept-ranges : bytes\\ncontent-length : 393\\ncontent-security-policy : block-all-mixed-content\\ncontent-type : application/xml\\ndate : Thu, 29 Aug 2024 17:29:41 GMT\\nserver : MinIO\\nstrict-transport-security : max-age=31536000; includeSubDomains\\nvary : Origin\\nx-amz-id-2 : e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\nx-amz-request-id : 17F0433BD8D4F0B7\\nx-content-type-options : nosniff\\nx-xss-protection : 1; mode=block\"] [source=AWSClient] [thread_id=15]","container":"errorlog"}
{"namespace":"cse-htap-apply-snapshot-v71-tps-7515842-1-548","pod":"tc-tiflash-0","log":"2024-08-30T01:29:41.868045879+08:00 stdout F [2024/08/29 17:29:41.603 +00:00] [WARN] [S3Common.cpp:145] [\"tag=AWSErrorMarshaller message=Encountered AWSError 'NoSuchKey': The specified key does not exist.\"] [source=AWSClient] [thread_id=15]","container":"errorlog"}
{"namespace":"cse-htap-apply-snapshot-v71-tps-7515842-1-548","pod":"tc-tiflash-0","log":"2024-08-30T01:29:41.868042001+08:00 stdout F [2024/08/29 17:29:41.602 +00:00] [ERROR] [S3RandomAccessFile.cpp:197] [\"S3 GetObject failed:  s3error= s3exception_name=NoSuchKey s3msg=The specified key does not exist. request_id=17F0433BD8BEFD2F, cur_retry=1\"] [source=s1043/data/dat_31_0] [thread_id=15]","container":"errorlog"}
{"namespace":"cse-htap-apply-snapshot-v71-tps-7515842-1-548","pod":"tc-tiflash-0","log":"2024-08-30T01:29:41.868034957+08:00 stdout F [2024/08/29 17:29:41.602 +00:00] [ERROR] [S3Common.cpp:145] [\"tag=AWSXmlClient message=HTTP response code: 404\\nResolved remote host IP address: 10.233.120.129\\nRequest ID: 17F0433BD8BEFD2F\\nException name: NoSuchKey\\nError message: The specified key does not exist.\\n12 response headers:\\naccept-ranges : bytes\\ncontent-length : 393\\ncontent-security-policy : block-all-mixed-content\\ncontent-type : application/xml\\ndate : Thu, 29 Aug 2024 17:29:41 GMT\\nserver : MinIO\\nstrict-transport-security : max-age=31536000; includeSubDomains\\nvary : Origin\\nx-amz-id-2 : e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\nx-amz-request-id : 17F0433BD8BEFD2F\\nx-content-type-options : nosniff\\nx-xss-protection : 1; mode=block\"] [source=AWSClient] [thread_id=15]","container":"errorlog"}
{"namespace":"cse-htap-apply-snapshot-v71-tps-7515842-1-548","pod":"tc-tiflash-0","log":"2024-08-30T01:29:41.868001815+08:00 stdout F [2024/08/29 17:29:41.602 +00:00] [WARN] [S3Common.cpp:145] [\"tag=AWSErrorMarshaller message=Encountered AWSError 'NoSuchKey': The specified key does not exist.\"] [source=AWSClient] [thread_id=15]","container":"errorlog"}
{"namespace":"cse-htap-apply-snapshot-v71-tps-7515842-1-548","pod":"tc-tiflash-0","log":"2024-08-29T23:54:14.74550759+08:00 stdout F [2024/08/29 15:54:14.473 +00:00] [INFO] [CPDataFileStat.cpp:191] [\"CheckpointData stats {compaction=[{key=lock/s1042/dat_31_1.lock_s1041_29 age=5117.473 size=9088821 rate=6.83%}, {key=lock/s1042/dat_30_1.lock_s1041_29 age=5149.473 size=1328993 rate=5.19%}, {key=lock/s1042/dat_30_1.lock_s1041_28 age=5149.473 size=1328993 rate=3.89%}, {key=lock/s1041/dat_67_1.lock_s1041_67 age=3646.473 size=47091142 rate=8.93%}, {key=lock/s1041/dat_56_1.lock_s1041_56 age=4055.473 size=82152078 rate=4.77%}, {key=lock/s1041/dat_53_1.lock_s1041_53 age=4164.473 size=269117314 rate=0.61%}, {key=lock/s1041/dat_44_1.lock_s1041_44 age=4492.473 size=92031170 rate=1.25%}, {key=lock/s1041/dat_42_1.lock_s1041_42 age=4567.473 size=139999410 rate=8.90%}, {key=lock/s1041/dat_59_1.lock_s1041_59 age=3944.473 size=47218313 rate=5.59%}, {key=lock/s1041/dat_54_1.lock_s1041_54 age=4129.473 size=39210060 rate=7.80%}, {key=lock/s1041/dat_57_1.lock_s1041_57 age=4017.473 size=84781228 rate=0.48%}, {key=lock/s1041/dat_66_1.lock_s1041_66 age=3683.473 size=226027989 rate=2.90%}, {key=lock/s1042/dat_31_0.lock_s1042_31 age=5116.473 size=103837516 rate=0.00%}, {key=lock/s1043/dat_32_0.lock_s1043_32 age=5123.473 size=129087329 rate=0.00%}, {key=lock/s1213/dat_30_0.lock_s1213_30 age=5141.473 size=55866608 rate=0.00%}, {key=lock/s1041/dat_41_1.lock_s1041_41 age=4604.473 size=77687967 rate=2.69%}, {key=lock/s1043/dat_33_1.lock_s1041_29 age=5089.473 size=6554306 rate=21.74%}, {key=lock/s1041/dat_68_1.lock_s1041_68 age=3609.473 size=103548372 rate=2.69%}, {key=lock/s1042/dat_30_0.lock_s1042_30 age=5149.473 size=16691457 rate=0.00%}, {key=lock/s1043/dat_31_0.lock_s1043_31 age=5155.473 size=221689 rate=0.05%}] unchanged=[{key=lock/s1041/dat_130_1.lock_s1041_130 age=1526.473 size=122487936 rate=18.26%}, {key=lock/s1041/dat_131_1.lock_s1041_131 age=1493.473 size=16421322 rate=41.69%}, {key=lock/s1041/dat_133_1.lock_s1041_133 age=1428.473 size=21551290 rate=26.13%}, {key=lock/s1041/dat_158_1.lock_s1041_158 age=603.473 size=100099161 rate=9.93%}, {key=lock/s1041/dat_107_1.lock_s1041_107 age=2285.473 size=41159714 rate=65.38%}, {key=lock/s1041/dat_141_1.lock_s1041_141 age=1165.473 size=43061219 rate=26.29%}, {key=lock/s1041/dat_159_1.lock_s1041_159 age=571.473 size=48100034 rate=58.19%}, {key=lock/s1041/dat_162_1.lock_s1041_162 age=471.473 size=15800136 rate=36.27%}, {key=lock/s1041/dat_173_1.lock_s1041_173 age=106.473 size=24340879 rate=75.41%}, {key=lock/s1041/dat_161_1.lock_s1041_161 age=504.473 size=48911068 rate=34.89%}, {key=lock/s1041/dat_114_1.lock_s1041_114 age=2053.473 size=22782614 rate=35.64%}, {key=lock/s1041/dat_118_1.lock_s1041_118 age=1922.473 size=39827979 rate=7.35%}, {key=lock/s1041/dat_163_1.lock_s1041_163 age=438.473 size=66157396 rate=37.60%}, {key=lock/s1041/dat_166_1.lock_s1041_166 age=339.473 size=40417143 rate=13.36%}, {key=lock/s1041/dat_165_1.lock_s1041_165 age=372.473 size=62448686 rate=14.20%}, {key=lock/s1041/dat_167_1.lock_s1041_167 age=306.473 size=55011820 rate=18.54%}, {key=lock/s1041/dat_168_1.lock_s1041_168 age=273.473 size=48849701 rate=17.30%}, {key=lock/s1041/dat_112_1.lock_s1041_112 age=2119.473 size=53815144 rate=21.47%}, {key=lock/s1041/dat_169_1.lock_s1041_169 age=240.473 size=33940577 rate=91.04%}, {key=lock/s1041/dat_155_1.lock_s1041_155 age=704.473 size=8228463 rate=40.81%}, {key=lock/s1041/dat_127_1.lock_s1041_127 age=1626.473 size=41304655 rate=27.91%}, {key=lock/s1041/dat_175_1.lock_s1041_175 age=38.473 size=42944830 rate=8.96%}, {key=lock/s1041/dat_108_1.lock_s1041_108 age=2252.473 size=43247431 rate=16.66%}, {key=lock/s1041/dat_175_0.lock_s1041_175 age=38.473 size=65924153 rate=6.85%}, {key=lock/s1041/dat_174_1.lock_s1041_174 age=73.473 size=38678106 rate=0.10%}, {key=lock/s1041/dat_170_1.lock_s1041_170 age=208.473 size=38939007 rate=65.12%}, {key=lock/s1041/dat_71_1.lock_s1041_71 age=3497.473 size=233414892 rate=8.05%}, {key=lock/s1041/dat_142_1.lock_s1041_142 age=1132.473 size=62269172 rate=6.48%}, {key=lock/s1041/dat_149_1.lock_s1041_149 age=901.473 size=108400782 rate=35.01%}, {key=lock/s1041/dat_171_1.lock_s1041_171 age=175.473 size=21428786 rate=29.02%}, {key=lock/s1041/dat_123_1.lock_s1041_123 age=1757.473 size=62691274 rate=3.81%}, {key=lock/s1041/dat_172_1.lock_s1041_172 age=140.473 size=80574700 rate=127.00%}, {key=lock/s1041/dat_150_1.lock_s1041_150 age=869.473 size=53864947 rate=105.17%}, {key=lock/s1041/dat_147_1.lock_s1041_147 age=968.473 size=37312243 rate=12.45%}, {key=lock/s1041/dat_140_1.lock_s1041_140 age=1198.473 size=52382591 rate=8.29%}, {key=lock/s1041/dat_143_1.lock_s1041_143 age=1100.473 size=34481222 rate=5.45%}, {key=lock/s1041/dat_132_1.lock_s1041_132 age=1461.473 size=56304223 rate=8.48%}, {key=lock/s1041/dat_82_1.lock_s1041_82 age=3118.473 size=15179373 rate=14.25%}, {key=lock/s1041/dat_87_1.lock_s1041_87 age=2950.473 size=101877119 rate=2.31%}, {key=lock/s1213/dat_82_0.lock_s1213_82 age=3290.473 size=6663 rate=1.74%}, {key=lock/s1041/dat_116_1.lock_s1041_116 age=1987.473 size=82199120 rate=18.81%}, {key=lock/s1041/dat_160_1.lock_s1041_160 age=538.473 size=68367488 rate=51.89%}, {key=lock/s1041/dat_152_1.lock_s1041_152 age=802.473 size=37300618 rate=57.28%}, {key=lock/s1041/dat_151_1.lock_s1041_151 age=835.473 size=45917317 rate=24.23%}, {key=lock/s1041/dat_129_1.lock_s1041_129 age=1559.473 size=26723938 rate=10.17%}, {key=lock/s1041/dat_174_0.lock_s1041_174 age=73.473 size=52912815 rate=3.41%}, {key=lock/s1041/dat_85_1.lock_s1041_85 age=3018.473 size=69209568 rate=0.00%}, {key=lock/s1041/dat_78_1.lock_s1041_78 age=3251.473 size=10646 rate=1.88%}, {key=lock/s1041/dat_91_1.lock_s1041_91 age=2815.473 size=2649581 rate=0.79%}, {key=lock/s1041/dat_126_1.lock_s1041_126 age=1659.473 size=29391133 rate=18.45%}, {key=lock/s1041/dat_125_1.lock_s1041_125 age=1692.473 size=35083771 rate=16.09%}, {key=lock/s1041/dat_115_1.lock_s1041_115 age=2021.473 size=20977559 rate=13.96%}, {key=lock/s1041/dat_104_1.lock_s1041_104 age=2386.473 size=33363724 rate=3.81%}, {key=lock/s1041/dat_128_1.lock_s1041_128 age=1593.473 size=52703238 rate=3.41%}, {key=lock/s1041/dat_84_1.lock_s1041_84 age=3052.473 size=5781886 rate=93.29%}, {key=lock/s1041/dat_83_1.lock_s1041_83 age=3084.473 size=54513791 rate=10.05%}, {key=lock/s1041/dat_80_1.lock_s1041_80 age=3185.473 size=24047230 rate=6.28%}, {key=lock/s1041/dat_70_1.lock_s1041_70 age=3535.473 size=20533573 rate=2.68%}, {key=lock/s1041/dat_156_1.lock_s1041_156 age=671.473 size=71369744 rate=10.21%}, {key=lock/s1041/dat_153_1.lock_s1041_153 age=769.473 size=46797204 rate=58.24%}, {key=lock/s1041/dat_120_1.lock_s1041_120 age=1857.473 size=19491805 rate=12.16%}, {key=lock/s1041/dat_119_1.lock_s1041_119 age=1890.473 size=41068254 rate=36.28%}, {key=lock/s1041/dat_113_1.lock_s1041_113 age=2087.473 size=41534327 rate=5.58%}, {key=lock/s1041/dat_110_1.lock_s1041_110 age=2185.473 size=50855812 rate=20.54%}, {key=lock/s1041/dat_89_1.lock_s1041_89 age=2882.473 size=56783685 rate=4.14%}, {key=lock/s1041/dat_69_1.lock_s1041_69 age=3572.473 size=101321333 rate=6.41%}, {key=lock/s1041/dat_86_1.lock_s1041_86 age=2985.473 size=34985546 rate=2.51%}, {key=lock/s1041/dat_103_1.lock_s1041_103 age=2419.473 size=60156293 rate=7.61%}, {key=lock/s1041/dat_75_1.lock_s1041_75 age=3349.473 size=83842069 rate=2.98%}, {key=lock/s1041/dat_81_1.lock_s1041_81 age=3151.473 size=7951317 rate=49.38%}, {key=lock/s1041/dat_91_0.lock_s1041_91 age=2815.473 size=61820259 rate=0.00%}]} RemoteGCThreshold{min_age=3600 min_file_threshold=131072 valid_rate=0.50%}\"] [source=UniPSService] [thread_id=15]","container":"serverlog"}
{"namespace":"cse-htap-apply-snapshot-v71-tps-7515842-1-548","pod":"tc-tiflash-0","log":"2024-08-29T23:54:14.745482363+08:00 stdout F [2024/08/29 15:54:14.430 +00:00] [INFO] [PageDirectory.cpp:2281] [\"Dumped snapshot to edits.[sequence=91488499]\"] [source=uni_write_176] [thread_id=15]","container":"serverlog"}
{"namespace":"cse-htap-apply-snapshot-v71-tps-7515842-1-548","pod":"tc-tiflash-0","log":"2024-08-29T23:54:14.745444407+08:00 stdout F [2024/08/29 15:54:14.160 +00:00] [INFO] [PageDirectory.cpp:2281] [\"Dumped snapshot to edits.[sequence=91488499]\"] [source=uni_write] [thread_id=15]","container":"serverlog"}
{"namespace":"cse-htap-apply-snapshot-v71-tps-7515842-1-548","pod":"tc-tiflash-0","log":"2024-08-29T23:53:43.736439928+08:00 stdout F [2024/08/29 15:53:43.284 +00:00] [INFO] [UniversalPageStorageService.cpp:294] [\"Upload checkpoint success, upload_sequence=175 incremental_bytes=93557768 compact_bytes=0\"] [source=UniPSService] [thread_id=15]","container":"serverlog"}

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiFlash version? (Required)

[JaySon-Huang]

Remove the "affects-8.5" because it is mainly caused by remote page ingested by FastAddPeer feature. And that feature is not enabled by default in the 8.5 version

[JaySon-Huang]

@CalvinNeo Does this issue workaround by introducing the cache in CPWriteDataSourceBlobStore?
#9614