TiFlash compute node crashes after executing ALTER RANGE in TiDB

[JaySon-Huang]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Reported from https://asktug.com/t/topic/1037727

ALTER PLACEMENT POLICY evict_sata_dw CONSTRAINTS="[-disk=sata-new, -disk=dw-ssd]" SURVIVAL_PREFERENCES="[host]";
ALTER RANGE global PLACEMENT POLICY evict_sata_dw;

2. What did you expect to see? (Required)

3. What did you see instead (Required)

[2024/12/28 12:52:05.786 +08:00] [ERROR] [BaseDaemon.cpp:370] [########################################] [source=BaseDaemon] [thread_id=41]
[2024/12/28 12:52:05.788 +08:00] [ERROR] [BaseDaemon.cpp:371] ["(from thread 42) Received signal Segmentation fault(11)."] [source=BaseDaemon] [thread_id=41]
[2024/12/28 12:52:05.788 +08:00] [ERROR] [BaseDaemon.cpp:401] ["Address: 0x10"] [source=BaseDaemon] [thread_id=41]
[2024/12/28 12:52:05.788 +08:00] [ERROR] [BaseDaemon.cpp:407] ["Access: read."] [source=BaseDaemon] [thread_id=41]
[2024/12/28 12:52:05.788 +08:00] [ERROR] [BaseDaemon.cpp:416] ["Address not mapped to object."] [source=BaseDaemon] [thread_id=41]
[2024/12/28 12:52:05.788 +08:00] [ERROR] [BaseDaemon.cpp:563] ["\n       0x7772a31\tfaultSignalHandler(int, siginfo_t*, void*) [tiflash+125250097]\n                \tlibs/libdaemon/src/BaseDaemon.cpp:214\n  0x7f65aff03520\t<unknown symbol> [libc.so.6+271648]\n  0x7f65aff58ef4\t__pthread_mutex_lock [libc.so.6+622324]\n  0x7f65b4af0f46\tstd::__1::mutex::lock() [libc++.so.1+421702]\n  0x7f65b4af192a\tstd::__1::__shared_mutex_base::lock_shared() [libc++.so.1+424234]\n       0x1f9eaa6\tDB::TiDBSchemaSyncerManager::getOrCreateSchemaSyncer(unsigned int) [tiflash+33155750]\n                \tdbms/src/TiDB/Schema/TiDBSchemaManager.h:120\n       0x8b1d78b\tDB::AtomicGetStorageSchema(std::__1::shared_ptr<DB::Region> const&, DB::TMTContext&) [tiflash+145872779]\n                \tdbms/src/Storages/KVStore/Decode/PartitionStreams.cpp:473\n       0x8a9e243\tDB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&) [tiflash+145351235]\n                \tdbms/src/Storages/KVStore/MultiRaft/PrehandleSnapshot.cpp:588\n       0x8a9d625\tDB::KVStore::preHandleSnapshotToFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, std::__1::optional<unsigned long>, DB::TMTContext&) [tiflash+145348133]\n                \tdbms/src/Storages/KVStore/MultiRaft/PrehandleSnapshot.cpp:206\n       0x8a80843\tPreHandleSnapshot [tiflash+145229891]\n                \tdbms/src/Storages/KVStore/FFI/ProxyFFI.cpp:672\n  0x7f65b1ba571c\tproxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h791b034f987b027e [libtiflash_proxy.so+26081052]\n  0x7f65b16c9f2e\tengine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::hc02c06315952cb47 [libtiflash_proxy.so+20987694]\n  0x7f65b22b41f6\tyatp::task::future::RawTask$LT$F$GT$::poll::h1156ccd37a9b2f70 [libtiflash_proxy.so+33481206]\n  0x7f65b3e2143b\t_$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::h879bddc8d67b170f [libtiflash_proxy.so+62239803]\n  0x7f65b3e11e8c\tstd::sys_common::backtrace::__rust_begin_short_backtrace::h17fc1674134e9d3e [libtiflash_proxy.so+62176908]\n  0x7f65b3e1290c\tcore::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc086da2e6c76c260 [libtiflash_proxy.so+62179596]\n  0x7f65b350d015\tstd::sys::unix::thread::Thread::new::thread_start::hd2791a9cabec1fda [libtiflash_proxy.so+52719637]\n                \t/rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/sys/unix/thread.rs:108\n  0x7f65aff55ac3\t<unknown symbol> [libc.so.6+608963]"] [source=BaseDaemon] [thread_id=41]
[2024/12/28 12:52:25.517 +08:00] [WARN] [S3Common.cpp:145] ["tag=ClientConfiguration message=User specified profile: [] is not found, will use the SDK resolved one."] [source=AWSClient] [thread_id=1]
[2024/12/28 12:52:28.495 +08:00] [ERROR] [BaseDaemon.cpp:370] [########################################] [source=BaseDaemon] [thread_id=41]
[2024/12/28 12:52:28.495 +08:00] [ERROR] [BaseDaemon.cpp:371] ["(from thread 40) Received signal Segmentation fault(11)."] [source=BaseDaemon] [thread_id=41]
[2024/12/28 12:52:28.495 +08:00] [ERROR] [BaseDaemon.cpp:401] ["Address: 0x10"] [source=BaseDaemon] [thread_id=41]
[2024/12/28 12:52:28.495 +08:00] [ERROR] [BaseDaemon.cpp:407] ["Access: read."] [source=BaseDaemon] [thread_id=41]
[2024/12/28 12:52:28.495 +08:00] [ERROR] [BaseDaemon.cpp:416] ["Address not mapped to object."] [source=BaseDaemon] [thread_id=41]
[2024/12/28 12:52:31.051 +08:00] [FATAL] [Exception.cpp:106] ["Code: 49, e.displayText() = DB::Exception: Illegal region range, should not happen, start_key=748000000000000CFF535F698000000000FF0000020130633162FF30383137FF2D3437FF62392D3436FF3130FF2D613135612DFF32FF62303163633065FFFF3364333000000000FFFB01363232363163FF3638FF2D35323961FF2D3431FF39622D39FF6434622DFF336134FF3633333239FF3565FF326300000000FB00FE end_key=748000000000000CFF7B5F728000000000FF0007310000000000FA, e.what() = DB::Exception, Stack trace:\n\n\n       0x8aed691\tDB::Exception::Exception<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&) [tiflash+145675921]\n                \tdbms/src/Common/Exception.h:53\n       0x8aed2a5\tDB::RegionRangeKeys::RegionRangeKeys(DB::StringObject<true>&&, DB::StringObject<true>&&) [tiflash+145674917]\n                \tdbms/src/Storages/KVStore/MultiRaft/RegionState.cpp:149\n       0x8aec7c1\tDB::RegionState::updateRegionRange() [tiflash+145672129]\n                \tdbms/src/Storages/KVStore/MultiRaft/RegionState.cpp:83\n       0x8ade8b2\tDB::RegionMeta::RegionMeta(metapb::Peer, metapb::Region, raft_serverpb::RaftApplyState) [tiflash+145615026]\n                \tdbms/src/Storages/KVStore/MultiRaft/RegionMeta.cpp:489\n       0x8a4ab97\tDB::KVStore::genRegionPtr(metapb::Region&&, unsigned long, unsigned long, unsigned long) [tiflash+145009559]\n                \tdbms/src/Storages/KVStore/KVStore.cpp:685\n       0x8a807d5\tPreHandleSnapshot [tiflash+145229781]\n                \tdbms/src/Storages/KVStore/FFI/ProxyFFI.cpp:660\n  0x7f971619871c\tproxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h791b034f987b027e [libtiflash_proxy.so+26081052]\n  0x7f9715cbcf2e\tengine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::hc02c06315952cb47 [libtiflash_proxy.so+20987694]\n  0x7f97168a71f6\tyatp::task::future::RawTask$LT$F$GT$::poll::h1156ccd37a9b2f70 [libtiflash_proxy.so+33481206]\n  0x7f971841443b\t_$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::h879bddc8d67b170f [libtiflash_proxy.so+62239803]\n  0x7f9718404e8c\tstd::sys_common::backtrace::__rust_begin_short_backtrace::h17fc1674134e9d3e [libtiflash_proxy.so+62176908]\n  0x7f971840590c\tcore::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc086da2e6c76c260 [libtiflash_proxy.so+62179596]\n  0x7f9717b00015\tstd::sys::unix::thread::Thread::new::thread_start::hd2791a9cabec1fda [libtiflash_proxy.so+52719637]\n                \t/rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/sys/unix/thread.rs:108\n  0x7f9714548ac3\t<unknown symbol> [libc.so.6+608963]\n  0x7f97145da850\t<unknown symbol> [libc.so.6+1206352]"] [source="DB::RawCppPtr DB::PreHandleSnapshot(DB::EngineStoreServerWrap *, DB::BaseBuffView, uint64_t, DB::SSTViewVec, uint64_t, uint64_t)"] [thread_id=44]
[2024/12/28 12:52:31.051 +08:00] [FATAL] [Exception.cpp:106] ["Code: 49, e.displayText() = DB::Exception: Illegal region range, should not happen, start_key=7480000000000137FF1A5F698000000000FF000002010FA70F82FF02211002FF0FC00EFF330E6D0E8BFF0E8BFF0E310E2D1044FF0EFF600E2F10510E6DFFFF0F2E0E290E300E2FFFFF00000000000000FF00F7010E2A0E290EFF2D0E32FF00000000FF00000000F7010E2BFF0E310E2B0E29FF0EFF2C0E2F0E320000FDFF0419B53600000000FF00010E2B0E290E2BFF0E2DFF0E2A0E2B0EFF2B0E30FF0E290E2BFF0E2F0E31FF0E2A0EFF2F0E290E2EFF0000FF000000000000F700FE end_key=7480000000000137FF1A5F698000000000FF000002010FB40F5BFF0F100FEAFF021B0EFF33101F1002FF0F82FF021B0E300E33FF0EFF300E310E310E8BFFFF0E2F0E3202210E2CFFFF0E2B0E320E3102FF21FF0E2D0E2C0E2AFF0E2AFF02210E310EFF2B0E4AFF0E2B0221FF0E310E32FF000000FF0000000000F7010EFF290E290E290E31FFFF0000000000000000FFF7010E2F0E2D0E2DFF0E2AFF0E310E3000FF000000FB0419B512FF0000000000010E2BFF0E290E2B0E2DFF0EFF2A0E2B0E290E32FFFF0E290E2E0E2B0E29FFFF0E2F0E2E0E2B0EFF32FF000000000000FF0000F70000000000FA, e.what() = DB::Exception, Stack trace:\n\n\n       0x8aed691\tDB::Exception::Exception<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&) [tiflash+145675921]\n                \tdbms/src/Common/Exception.h:53\n       0x8aed2a5\tDB::RegionRangeKeys::RegionRangeKeys(DB::StringObject<true>&&, DB::StringObject<true>&&) [tiflash+145674917]\n                \tdbms/src/Storages/KVStore/MultiRaft/RegionState.cpp:149\n       0x8aec7c1\tDB::RegionState::updateRegionRange() [tiflash+145672129]\n                \tdbms/src/Storages/KVStore/MultiRaft/RegionState.cpp:83\n       0x8ade8b2\tDB::RegionMeta::RegionMeta(metapb::Peer, metapb::Region, raft_serverpb::RaftApplyState) [tiflash+145615026]\n                \tdbms/src/Storages/KVStore/MultiRaft/RegionMeta.cpp:489\n       0x8a4ab97\tDB::KVStore::genRegionPtr(metapb::Region&&, unsigned long, unsigned long, unsigned long) [tiflash+145009559]\n                \tdbms/src/Storages/KVStore/KVStore.cpp:685\n       0x8a807d5\tPreHandleSnapshot [tiflash+145229781]\n                \tdbms/src/Storages/KVStore/FFI/ProxyFFI.cpp:660\n  0x7f971619871c\tproxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h791b034f987b027e [libtiflash_proxy.so+26081052]\n  0x7f9715cbcf2e\tengine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::hc02c06315952cb47 [libtiflash_proxy.so+20987694]\n  0x7f97168a71f6\tyatp::task::future::RawTask$LT$F$GT$::poll::h1156ccd37a9b2f70 [libtiflash_proxy.so+33481206]\n  0x7f971841443b\t_$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::h879bddc8d67b170f [libtiflash_proxy.so+62239803]\n  0x7f9718404e8c\tstd::sys_common::backtrace::__rust_begin_short_backtrace::h17fc1674134e9d3e [libtiflash_proxy.so+62176908]\n  0x7f971840590c\tcore::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc086da2e6c76c260 [libtiflash_proxy.so+62179596]\n  0x7f9717b00015\tstd::sys::unix::thread::Thread::new::thread_start::hd2791a9cabec1fda [libtiflash_proxy.so+52719637]\n                \t/rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/sys/unix/thread.rs:108\n  0x7f9714548ac3\t<unknown symbol> [libc.so.6+608963]\n  0x7f97145da850\t<unknown symbol> [libc.so.6+1206352]"] [source="DB::RawCppPtr DB::PreHandleSnapshot(DB::EngineStoreServerWrap *, DB::BaseBuffView, uint64_t, DB::SSTViewVec, uint64_t, uint64_t)"] [thread_id=45]
[2024/12/28 12:52:31.051 +08:00] [ERROR] [BaseDaemon.cpp:563] ["\n       0x7772a31\tfaultSignalHandler(int, siginfo_t*, void*) [tiflash+125250097]\n                \tlibs/libdaemon/src/BaseDaemon.cpp:214\n  0x7f97144f6520\t<unknown symbol> [libc.so.6+271648]\n  0x7f971454bef4\t__pthread_mutex_lock [libc.so.6+622324]\n  0x7f97190e3f46\tstd::__1::mutex::lock() [libc++.so.1+421702]\n  0x7f97190e492a\tstd::__1::__shared_mutex_base::lock_shared() [libc++.so.1+424234]\n       0x1f9eaa6\tDB::TiDBSchemaSyncerManager::getOrCreateSchemaSyncer(unsigned int) [tiflash+33155750]\n                \tdbms/src/TiDB/Schema/TiDBSchemaManager.h:120\n       0x8b1d78b\tDB::AtomicGetStorageSchema(std::__1::shared_ptr<DB::Region> const&, DB::TMTContext&) [tiflash+145872779]\n                \tdbms/src/Storages/KVStore/Decode/PartitionStreams.cpp:473\n       0x8a9e243\tDB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&) [tiflash+145351235]\n                \tdbms/src/Storages/KVStore/MultiRaft/PrehandleSnapshot.cpp:588\n       0x8a9d625\tDB::KVStore::preHandleSnapshotToFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, std::__1::optional<unsigned long>, DB::TMTContext&) [tiflash+145348133]\n                \tdbms/src/Storages/KVStore/MultiRaft/PrehandleSnapshot.cpp:206\n       0x8a80843\tPreHandleSnapshot [tiflash+145229891]\n                \tdbms/src/Storages/KVStore/FFI/ProxyFFI.cpp:672\n  0x7f971619871c\tproxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h791b034f987b027e [libtiflash_proxy.so+26081052]\n  0x7f9715cbcf2e\tengine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::hc02c06315952cb47 [libtiflash_proxy.so+20987694]\n  0x7f97168a71f6\tyatp::task::future::RawTask$LT$F$GT$::poll::h1156ccd37a9b2f70 [libtiflash_proxy.so+33481206]\n  0x7f971841443b\t_$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::h879bddc8d67b170f [libtiflash_proxy.so+62239803]\n  0x7f9718404e8c\tstd::sys_common::backtrace::__rust_begin_short_backtrace::h17fc1674134e9d3e [libtiflash_proxy.so+62176908]\n  0x7f971840590c\tcore::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hc086da2e6c76c260 [libtiflash_proxy.so+62179596]\n  0x7f9717b00015\tstd::sys::unix::thread::Thread::new::thread_start::hd2791a9cabec1fda [libtiflash_proxy.so+52719637]\n                \t/rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/sys/unix/thread.rs:108\n  0x7f9714548ac3\t<unknown symbol> [libc.so.6+608963]"] [source=BaseDaemon] [thread_id=41]

4. What is your TiFlash version? (Required)

v7.5.4
deploy with storage and compute disaggregated arch

[JaySon-Huang]

Reproduce steps:

Deploy a cluster with tikv and disaggregated tiflash

server_configs:
  pd:
    replication.location-labels: ["host", "disk"]
... # here ignore other components like tidb,pd config items
tikv_servers:
  - host: 10.2.12.81
    port: 7540
    status_port: 16540
    config:
      server.labels:
        disk: sata
        host: h81
  - host: 10.2.12.81
    port: 7541
    status_port: 16541
    config:
      server.labels:
        disk: sata
        host: h81
  - host: 10.2.12.81
    port: 7542
    status_port: 16542
    config:
      server.labels:
        disk: sata-new
        host: h81
tiflash_servers:
  - host: 10.2.12.79
    tcp_port: 5040
    http_port: 4540
    flash_service_port: 9540
    flash_proxy_port: 9040
    flash_proxy_status_port: 20040
    metrics_port: 20540
    config:
      flash.disaggregated_mode: tiflash_write
      storage.main.dir :   [ /DATA/disk2/jaysonhuang/clusters/tiflash-5040/data ]
      storage.main.capacity:   [ 107374182400 ]
      storage.s3.bucket: jayson
      storage.s3.root: /j40/
      storage.s3.endpoint: http://10.2.12.79:9000
      storage.s3.access_key_id: minioadmin
      storage.s3.secret_access_key: minioadmin
  - host: 10.2.12.79
    tcp_port: 5050
    http_port: 4550
    flash_service_port: 9550
    flash_proxy_port: 9050
    flash_proxy_status_port: 20050
    metrics_port: 20550
    config:
      flash.disaggregated_mode: tiflash_compute
      storage.main.dir :   [ /DATA/disk2/jaysonhuang/clusters/tiflash-5050/data ]
      storage.main.capacity:   [ 107374182400 ]
      storage.remote.cache.dir: /DATA/disk2/jaysonhuang/clusters/tiflash-5050/cache
      storage.remote.cache.capacity: 107374182400
      storage.s3.bucket: jayson
      storage.s3.root: /j40/
      storage.s3.endpoint: http://10.2.12.79:9000
      storage.s3.access_key_id: minioadmin
      storage.s3.secret_access_key: minioadmin

Load TPC-H dataset into the cluster and create tiflash replica

Create placement policy

CREATE PLACEMENT POLICY evict_sata_dw CONSTRAINTS="[-disk=sata-new, -disk=dw-ssd]" SURVIVAL_PREFERENCES="[host]";
ALTER TABLE test.region PLACEMENT POLICY=evict_sata_dw;
-- or alter policy globally
-- ALTER RANGE global PLACEMENT POLICY evict_sata_dw;

The label of store under disagg arch

>  tiup ctl:v8.5.0 pd -u 10.2.12.81:6540 store | jq '.stores[] | {store_id: .store.id, state: .store.state_name, address: .store.address, labels: .store.labels} | tostring'
"{\"store_id\":1,\"state\":\"Up\",\"address\":\"10.2.12.81:7540\",\"labels\":[{\"key\":\"disk\",\"value\":\"sata\"},{\"key\":\"host\",\"value\":\"h81\"}]}"
"{\"store_id\":4,\"state\":\"Up\",\"address\":\"10.2.12.81:7541\",\"labels\":[{\"key\":\"disk\",\"value\":\"sata\"},{\"key\":\"host\",\"value\":\"h81\"}]}"
"{\"store_id\":5,\"state\":\"Up\",\"address\":\"10.2.12.81:7542\",\"labels\":[{\"key\":\"disk\",\"value\":\"sata-new\"},{\"key\":\"host\",\"value\":\"h81\"}]}"
"{\"store_id\":132,\"state\":\"Up\",\"address\":\"10.2.12.79:9550\",\"labels\":[{\"key\":\"engine\",\"value\":\"tiflash_compute\"}]}"
"{\"store_id\":133,\"state\":\"Up\",\"address\":\"10.2.12.79:9540\",\"labels\":[{\"key\":\"engine_role\",\"value\":\"write\"},{\"key\":\"engine\",\"value\":\"tiflash\"}]}"

Those steps will create a placement-rule as follow, and try to add peer on the tiflash compute node

[2024/12/30 16:22:50.176 +08:00] [INFO] [audit.go:126] ["audit log"] [service-info="{ServiceLabel:SetPlacementRules, Method:HTTP/1.1/POST:/pd/api/v1/config/placement-rule, CallerID:tidb-info-syncer, IP:10.2.12.79, Port:48484, StartTime:2024-12-30 16:22:50 +0800 CST, URLParam:{\"partial\":[\"true\"]}, BodyParam:[{\"group_id\":\"TiDB_DDL_124\",\"group_index\":40,\"group_override\":true,\"rules\":[{\"group_id\":\"TiDB_DDL_124\",\"id\":\"table_rule_124_0\",\"index\":40,\"start_key\":\"7480000000000000ff7c00000000000000f8\",\"end_key\":\"7480000000000000ff7d00000000000000f8\",\"role\":\"voter\",\"is_witness\":false,\"count\":3,\"label_constraints\":[{\"key\":\"disk\",\"op\":\"notIn\",\"values\":[\"sata-new\"]},{\"key\":\"disk\",\"op\":\"notIn\",\"values\":[\"dw-ssd\"]},{\"key\":\"engine\",\"op\":\"notIn\",\"values\":[\"tiflash\"]}],\"location_labels\":[\"host\"]}]}]}"]
[2024/12/30 16:22:50.177 +08:00] [INFO] [rule_manager.go:742] ["full config reset"] [config="[{\"group_id\":\"TiDB_DDL_124\",\"group_index\":40,\"group_override\":true,\"rules\":[{\"group_id\":\"TiDB_DDL_124\",\"id\":\"table_rule_124_0\",\"index\":40,\"start_key\":\"7480000000000000ff7c00000000000000f8\",\"end_key\":\"7480000000000000ff7d00000000000000f8\",\"role\":\"voter\",\"is_witness\":false,\"count\":3,\"label_constraints\":[{\"key\":\"disk\",\"op\":\"notIn\",\"values\":[\"sata-new\"]},{\"key\":\"disk\",\"op\":\"notIn\",\"values\":[\"dw-ssd\"]},{\"key\":\"engine\",\"op\":\"notIn\",\"values\":[\"tiflash\"]}],\"location_labels\":[\"host\"],\"create_timestamp\":1735546970}]}]"]
[2024/12/30 16:22:50.204 +08:00] [INFO] [operator_controller.go:510] ["add operator"] [region-id=302] [operator="\"add-rule-peer {add peer: store [132]} (kind:replica,region, region:302(74, 5), createAt:2024-12-30 16:22:50.20420482 +0800 CST m=+655.704389486, startAt:0001-01-01 00:00:00 +0000 UTC, currentStep:0, size:1, steps:[0:{add learner peer 5258 on store 132}, 1:{promote learner peer 5258 on store 132 to voter}], timeout:[11m0s])\""] [additional-info=]
[2024/12/30 16:22:50.204 +08:00] [INFO] [operator_controller.go:786] ["send schedule command"] [region-id=302] [step="add learner peer 5258 on store 132"] [source=create]
[2024/12/30 16:22:50.204 +08:00] [INFO] [operator_controller.go:510] ["add operator"] [region-id=372] [operator="\"add-rule-peer {add peer: store [132]} (kind:replica,region, region:372(75, 6), createAt:2024-12-30 16:22:50.204365351 +0800 CST m=+655.704550016, startAt:0001-01-01 00:00:00 +0000 UTC, currentStep:0, size:1, steps:[0:{add learner peer 5259 on store 132}, 1:{promote learner peer 5259 on store 132 to voter}], timeout:[11m0s])\""] [additional-info=]

[JaySon-Huang]

Disaster recovery measures

If a user run into a crash as above. Then he/she can

Restore the policy to default

ALTER TABLE test.region PLACEMENT POLICY=default;
-- or
ALTER RANGE global PLACEMENT POLICY=default;

scale-in all the tiflash compute node

tiup cluster scale-in <cluster-name> -N ...... # all the tiflash compute node

Wait for the PD remove all peers from existing tiflash compute node

Re-deploy the tiflash compute node

[JaySon-Huang]

Root cause

When creating a placement policy through tidb, tidb will create a placement-rule with "engine" notIn "tiflash" pingcap/tidb#22065
However, under tiflash disaggregated compute and storage arch, the compute node will register store with label {"engine": "tiflash_compute"}.

The logic in PD of choosing store for rule: pkg/schedule/placement/label_constraint.go @ pd

1. If the store's label.key is equal to "engine" (or "exclusive," or starts with '$'), and the rule's constraint does not contain a rule with the same label.key, then do not schedule the peer to this store.
2. Otherwise, further check if the rule's constraint matches the store's label. If it matches, schedule the peer to this store.

So the PD would pick tiflash compute node as target store to place the Region peer.

Workaround

As a workaround, user can explicitly exclude tiflash and tiflash_compute in theirs rule before this behavior is fixed. For example

CREATE PLACEMENT POLICY evict_sata_dw CONSTRAINTS="[-engine=tiflash, -engine=tiflash_compute, -disk=sata-new, -disk=dw-ssd]" SURVIVAL_PREFERENCES="[host]";
ALTER TABLE test.region PLACEMENT POLICY=evict_sata_dw;