tiflash got crash in Debug compalition mode when async_grpc_server is disable

[guo-shaoge]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. use tiup to start a cluster with one or more tiflash
2. build tiflash in Debug mode
3. change config to disable profiles.default.enable_async_server
4. run a simple query

2. What did you expect to see? (Required)

runs ok

3. What did you see instead (Required)

tiflash crash

[2024/12/12 15:17:46.794 +08:00] [ERROR] [BaseDaemon.cpp:559] ["
  0x5b631eb8201a\tfaultSignalHandler(int, siginfo_t*, void*) [tiflash+213696538]
                \tlibs/libdaemon/src/BaseDaemon.cpp:210
  0x7921814951d0\t<unknown symbol> [libc.so.6+250320]
  0x7921814ee3f4\t<unknown symbol> [libc.so.6+615412]
  0x792181495120\tgsignal [libc.so.6+250144]
  0x79218147c4c3\tabort [libc.so.6+148675]
  0x79218147c3df\t<unknown symbol> [libc.so.6+148447]
  0x79218148d177\t__assert_fail [libc.so.6+217463]
  0x5b6320e29e42\tDB::FlashService::EstablishMPPConnection(grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*) [tiflash+250035778]
                \tdbms/src/Flash/FlashService.cpp:637
  0x5b632349e730\ttikvpb::Tikv::Service::Service()::$_53::operator()(tikvpb::Tikv::Service*, grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*) const [tiflash+290359088]
                \tbuild/clang17_debug/contrib/kvproto/cpp/kvproto/tikvpb.grpc.pb.cc:2237
  0x5b632349e6dc\tdecltype(std::declval<tikvpb::Tikv::Service::Service()::$_53&>()(std::declval<tikvpb::Tikv::Service*>(), std::declval<grpc::ServerContext*>(), std::declval<mpp::EstablishMPPConnectionRequest const*>(), std::declval<grpc::ServerWriter<mpp::MPPDataPacket>*>())) std::__1::__invoke[abi:ue170006]<tikvpb::Tikv::Service::Service()::$_53&, tikvpb::Tikv::Service*, grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*>(tikvpb::Tikv::Service::Service()::$_53&, tikvpb::Tikv::Service*&&, grpc::ServerContext*&&, mpp::EstablishMPPConnectionRequest const*&&, grpc::ServerWriter<mpp::MPPDataPacket>*&&) [tiflash+290359004]
                \t/usr/local/bin/../include/c++/v1/__type_traits/invoke.h:340
  0x5b632349e660\tgrpc::Status std::__1::__invoke_void_return_wrapper<grpc::Status, false>::__call[abi:ue170006]<tikvpb::Tikv::Service::Service()::$_53&, tikvpb::Tikv::Service*, grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*>(tikvpb::Tikv::Service::Service()::$_53&, tikvpb::Tikv::Service*&&, grpc::ServerContext*&&, mpp::EstablishMPPConnectionRequest const*&&, grpc::ServerWriter<mpp::MPPDataPacket>*&&) [tiflash+290358880]
                \t/usr/local/bin/../include/c++/v1/__type_traits/invoke.h:407
  0x5b632349e610\tstd::__1::__function::__alloc_func<tikvpb::Tikv::Service::Service()::$_53, std::__1::allocator<tikvpb::Tikv::Service::Service()::$_53>, grpc::Status (tikvpb::Tikv::Service*, grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*)>::operator()[abi:ue170006](tikvpb::Tikv::Service*&&, grpc::ServerContext*&&, mpp::EstablishMPPConnectionRequest const*&&, grpc::ServerWriter<mpp::MPPDataPacket>*&&) [tiflash+290358800]
                \t/usr/local/bin/../include/c++/v1/__functional/function.h:192
  0x5b632349da24\tstd::__1::__function::__func<tikvpb::Tikv::Service::Service()::$_53, std::__1::allocator<tikvpb::Tikv::Service::Service()::$_53>, grpc::Status (tikvpb::Tikv::Service*, grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*)>::operator()(tikvpb::Tikv::Service*&&, grpc::ServerContext*&&, mpp::EstablishMPPConnectionRequest const*&&, grpc::ServerWriter<mpp::MPPDataPacket>*&&) [tiflash+290355748]
                \t/usr/local/bin/../include/c++/v1/__functional/function.h:363
  0x5b6323541ce5\tstd::__1::__function::__value_func<grpc::Status (tikvpb::Tikv::Service*, grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*)>::operator()[abi:ue170006](tikvpb::Tikv::Service*&&, grpc::ServerContext*&&, mpp::EstablishMPPConnectionRequest const*&&, grpc::ServerWriter<mpp::MPPDataPacket>*&&) const [tiflash+291028197]
                \t/usr/local/bin/../include/c++/v1/__functional/function.h:517
  0x5b6323541c70\tstd::__1::function<grpc::Status (tikvpb::Tikv::Service*, grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*)>::operator()(tikvpb::Tikv::Service*, grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*) const [tiflash+291028080]
                \t/usr/local/bin/../include/c++/v1/__functional/function.h:1168
  0x5b6323541c22\tgrpc::internal::ServerStreamingHandler<tikvpb::Tikv::Service, mpp::EstablishMPPConnectionRequest, mpp::MPPDataPacket>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::'lambda'()::operator()() const [tiflash+291028002]
                \tcontrib/grpc/include/grpcpp/impl/codegen/method_handler.h:207
  0x5b63235402d4\tgrpc::Status grpc::internal::CatchingFunctionHandler<grpc::internal::ServerStreamingHandler<tikvpb::Tikv::Service, mpp::EstablishMPPConnectionRequest, mpp::MPPDataPacket>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::'lambda'()>(grpc::internal::ServerStreamingHandler<tikvpb::Tikv::Service, mpp::EstablishMPPConnectionRequest, mpp::MPPDataPacket>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::'lambda'()&&) [tiflash+291021524]
                \tcontrib/grpc/include/grpcpp/impl/codegen/method_handler.h:44
  0x5b632353fe35\tgrpc::internal::ServerStreamingHandler<tikvpb::Tikv::Service, mpp::EstablishMPPConnectionRequest, mpp::MPPDataPacket>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&) [tiflash+291020341]
                \tcontrib/grpc/include/grpcpp/impl/codegen/method_handler.h:206
  0x5b632296cceb\tgrpc::Server::SyncRequest::ContinueRunAfterInterception() [tiflash+278621419]
                \tcontrib/grpc/src/cpp/server/server_cc.cc:433
  0x5b632296c97f\tgrpc::Server::SyncRequest::Run(std::__1::shared_ptr<grpc::Server::GlobalCallbacks> const&, bool) [tiflash+278620543]
                \tcontrib/grpc/src/cpp/server/server_cc.cc:421
  0x5b632296c4e3\tgrpc::Server::SyncRequestThreadManager::DoWork(void*, bool, bool) [tiflash+278619363]
                \tcontrib/grpc/src/cpp/server/server_cc.cc:791
  0x5b6322998be2\tgrpc::ThreadManager::MainWorkLoop() [tiflash+278801378]
                \tcontrib/grpc/src/cpp/thread_manager/thread_manager.cc:211
  0x5b632299890c\tgrpc::ThreadManager::WorkerThread::Run() [tiflash+278800652]
                \tcontrib/grpc/src/cpp/thread_manager/thread_manager.cc:44
  0x5b6322999469\tgrpc::ThreadManager::WorkerThread::WorkerThread(grpc::ThreadManager*)::$_0::operator()(void*) const [tiflash+278803561]
                \tcontrib/grpc/src/cpp/thread_manager/thread_manager.cc:36
  0x5b6322999449\tgrpc::ThreadManager::WorkerThread::WorkerThread(grpc::ThreadManager*)::$_0::__invoke(void*) [tiflash+278803529]
                \tcontrib/grpc/src/cpp/thread_manager/thread_manager.cc:36
  0x5b6323015e62\tgrpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::operator()(void*) const [tiflash+285605474]
                \tcontrib/grpc/src/core/lib/gprpp/thd_posix.cc:140
  0x5b6323015d49\tgrpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::__invoke(void*) [tiflash+285605193]
                \tcontrib/grpc/src/core/lib/gprpp/thd_posix.cc:110
  0x7921814ec39d\t<unknown symbol> [libc.so.6+607133]"] [source=BaseDaemon] [thread_id=139]

4. What is your TiFlash version? (Required)

e3e6780

[guo-shaoge]

Why crash:

1. EstablishMPPConnection assert sender's resource group name is same with receiver's resource group name. (Which means only in Debug Mode, this assert will take effects)
2. But actually the sender meta's resource group name is empty, because in GRPCReceiverContext and tidb's fragment.go, no nobody set resource group name in sender's task meta.

This crash only happens only when TiFlash is built in Debug mode and when async grpc server is disabled (which is enabled by default).

[guo-shaoge]

we can setup sender's resource group name in tidb, just like connection id/connection alias to fix this crash