集群初始化完成可以正常使用，重启集群之后集群就不正常了

[mywgl]

Bug Type (问题类型)

other exception / error (其他异常报错)

Before submit

• 我已经确认现有的 Issues 与 FAQ 中没有相同 / 重复问题 (I have confirmed and searched that there are no similar problems in the historical issue and documents)

Environment (环境信息)

• Server Version: 1.7.0 (Apache Release Version)
• Backend: hstore 3 nodes, HDD
• OS: 2 CPUs, 256 G RAM, CentOS 7.8
• Data Size: xx vertices, xx edges

Expected & Actual behavior (期望与实际表现)

初始化集群之后是能够正常使用的，重启集群之后，pd日志报错 NPE
RaftRpcClient failed to call rpc to 192.168.1.103:8610 .null
WARN redirect to leader with error
1.查看store的日志没有报错，但是发现跟初始化集群时候的查看的leader角色 改变成其他节点上，怀疑是这个原因导致，请问有啥解决方案吗？
2. 为啥每次重启集群之前的leader 就转到其他节点上了，还有就是转移了角色为啥集群就无法正常工作，那这个集群等于存在单点故障问题，不是高可用集群呢？

Vertex/Edge example (问题点 / 边数据举例)

Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)

[dosubot]

Hi @mywgl! I'm Dosu and I’m helping the hugegraph team.

This looks like a known bug tracked in #2959 [1]. The root cause is a NullPointerException in RaftEngine.getLeaderGrpcAddress() — when a PD node is a follower, it tries to forward requests to the leader via a bolt RPC call, but the CompletableFuture.get() returns null, causing the NPE.

The key issue is that after a cluster restart, if the Raft leader election picks a different PD node than the one that was leader before, follower PD nodes fail to redirect requests to the new leader. This causes store registration and partition distribution to break, making the cluster non-functional.

Your two questions:

1. Workaround: As noted in [Bug] 3-node PD cluster fails when pd0 is not raft leader — getLeaderGrpcAddress() NPE in bridge network mode #2959, restarting the non-leader PD nodes to force the original node to win the election can resolve the issue temporarily. However, this is not a proper fix — the underlying getLeaderGrpcAddress() NPE needs to be addressed in the codebase [1].

2. Why leader change breaks the cluster: You're right that this effectively creates a single-point-of-failure issue. The cluster should handle leader changes gracefully — this is a bug, not by design. The redirectToLeader() mechanism is supposed to forward requests from follower PD nodes to the current leader, but the NPE prevents that from working correctly [1].

You may want to check if there is a fix available in a newer version or a related PR (e.g., #2952 was mentioned as related).

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[mywgl]

@dosubot
有啥api请求获取，或者查询什么日志 能够正确获取原来的leader节点的方法吗？

[mywgl]

@dosubot 是重启pd节点还是store节点？

[dosubot]

@mywgl 有以下几种方法可以获取当前 PD 集群的 leader 节点信息：

方法一：通过 REST API 查询

最直接的方式是调用 PD 的 /v1/members 接口，返回结果中包含 pdLeader 字段，标识当前 leader 节点 [1]：

curl -u admin:password http://<pd_host>:<pd_port>/v1/members

也可以用 /v1/cluster 接口获取更完整的集群信息，同样包含 pdLeader 字段 [2]：

curl -u admin:password http://<pd_host>:<pd_port>/v1/cluster

注意：1.7.0 版本开启了认证，需要带上用户名密码 [3]。如果不确定凭据，可以尝试 -u store:admin 或 -u admin:admin。

方法二：查看 PD 日志

在 PD 节点的日志中搜索以下关键字：

• Raft becomes leader — 表示该节点成为了 leader [4]
• Raft lost leader — 表示该节点失去了 leader 角色 [5]

每个 PD 节点分别执行：

grep "becomes leader" /path/to/hugegraph-pd/logs/hugegraph-pd-stdout.log

关于你的场景：由于当前的 leader 转移后 getLeaderGrpcAddress() 会抛出 NPE（如我之前提到的 #2959 中的 bug），你可以先通过上面的 API 或日志确认当前哪个节点是 leader，然后尝试重启非 leader 的 PD 节点，迫使原来的节点重新赢得选举，作为临时的 workaround。

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[bitflicker64]

Hi @mywgl — this is a known bug that has been fixed in #2961, merged into master last month.

What's happening:

After a cluster restart, Raft elects a new leader non-deterministically, whoever gets enough votes first wins. This is normal and expected behavior, not a bug. The actual bug was in the redirect mechanism. When a client sends a request to a follower PD node, that follower needs to forward it to the current leader. To do that it makes an internal RPC call to discover the leader's gRPC address. This RPC was failing and returning null, causing the NPE you see:

RaftRpcClient failed to call rpc to 192.168.1.103:8610 .null
WARN redirect to leader with error

To answer your questions directly:

1. The leader changing on restart is Raft working correctly — not the problem. The problem was follower PD nodes couldn't redirect requests to the new leader. The fix ensures the cluster works regardless of which node wins election.

2. This was not a single point of failure by design — it was a bug. With the fix applied, any PD node can be leader and the cluster handles it gracefully.

To find your current leader:

grep "becomes leader" /path/to/hugegraph-pd/logs/hugegraph-pd-stdout.log

Whichever node prints output is the current leader. You can also confirm via REST API:

curl -u store:store http://<any-pd-host>:8620/v1/members

Look at the pdLeader field in the response.

What to do:

If you're on Docker, pull the latest image which already includes the fix:

docker pull hugegraph/hugegraph:latest

If you're on bare metal CentOS, the fix is currently in master. I'd recommend checking with the maintainers (@imbajin) on when it will be available in a release.

[imbajin]

@mywgl does the problem solved? (after updating the image)

[mywgl]

@imbajin 谢谢，已经解决了，使用master分支重新编译打包，并且重新安装部署上去之后确实解决这个问题了

[mywgl]

@bitflicker64 使用最新的master分支打包出来的安装包，启动hugegraph-server会有一个lsof 的使用报错：
lsof: unacceptable port specification in: -i :
lsof 4.93.2
latest revision: https://github.com/lsof-org/lsof
latest FAQ: https://github.com/lsof-org/lsof/blob/master/00FAQ
latest (non-formatted) man page: https://github.com/lsof-org/lsof/blob/master/Lsof.8
usage: [-?abhKlnNoOPRtUvVX] [+|-c c] [+|-d s] [+D D] [+|-E] [+|-e s] [+|-f[gG]]
[-F [f]] [-g [s]] [-i [i]] [+|-L [l]] [+m [m]] [+|-M] [-o [o]] [-p s]
[+|-r [t]] [-s [p:s]] [-S [t]] [-T [t]] [-u s] [+|-w] [-x [fl]] [--] [names]
Use the ``-h'' option to get more help information.
Starting HugeGraphServer in daemon mode...

经过排查是"start-hugegraph.sh" 里面的check_port "$REST_SERVER_URL" 导致的错误
check_port是util.sh文件的函数，把函数中的：
local port=$(echo "$1" | awk -F':' '{print $3}')
修改成：
local port=$(echo "$1" | awk -F':' '{print $NF}')
就能解决上面的错误，希望官方修改这个错误