Skip to content
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
5ba6d4a
optimize: Optimize RocksDB batch query performance
lokidundun Mar 29, 2026
f5405a0
Refactor getByIds to queryByIds in RocksDBTable
lokidundun Mar 31, 2026
939ace0
Modify queryByIds to use super method temporarily
lokidundun Mar 31, 2026
502c7df
Refactor queryByIds to getByIds with HashSet
lokidundun Apr 2, 2026
352f66b
Update RocksDBTables.java
lokidundun Apr 2, 2026
0d9052d
fix: fix ci
lokidundun Apr 6, 2026
45298f9
optimize: optimize the batch query
lokidundun Apr 7, 2026
223fb28
optimize: optimize rockDb query
lokidundun Apr 16, 2026
ce4e2cb
fix: fix ci test
lokidundun Apr 16, 2026
6fe08a7
fix: preserve input id multiplicity in RocksDB multi-get path
Copilot Apr 17, 2026
0391069
fix: fallback getByIds when rocksdb session has pending changes
Copilot Apr 17, 2026
ce802b9
revert: restore rocksdb query files to ce4e2cb state
Copilot Apr 17, 2026
c06225c
fix: fix wrong usage
lokidundun Apr 17, 2026
a99491b
Merge branch 'improvequery' of https://github.com/lokidundun/incubato…
lokidundun Apr 17, 2026
10cb0a4
test: add test for new multi-get path
lokidundun Apr 17, 2026
d33123e
add test for session.hasChanges()
lokidundun Apr 20, 2026
c1b3599
optimize: extract the common logic
lokidundun Apr 26, 2026
0432f58
fix: refine queryByIdsWithGet hasChanges guard and align tests
lokidundun May 29, 2026
cb98e63
fix query test exception
lokidundun Jun 1, 2026
73588c4
test: verify non-consecutive duplicate ids are preserved in edge queries
lokidundun Jun 4, 2026
f2d2e29
fix ci
lokidundun Jun 4, 2026
b11ec06
add detail test
lokidundun Jun 4, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -783,8 +783,10 @@ protected Iterator<Vertex> queryVerticesByIds(Object[] vertexIds, boolean adjace
// NOTE: allowed duplicated vertices if query by duplicated ids
List<Id> ids = InsertionOrderUtil.newList();
Map<Id, HugeVertex> vertices = new HashMap<>(vertexIds.length);
Set<Id> fetchedIds = InsertionOrderUtil.newSet();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ 这里新增 fetchedIds 之后,非相邻的重复 id 也会被全局去重;而旧逻辑里 IdQuery.query() 只会折叠相邻重复 id。最终返回结果看起来应该还是保持重复输出,但真实的后端访问路径已经变了。建议补一个回归测试,至少覆盖 超过 query.batch_size跨 batch 的重复 id缺失 id + checkMustExist 这几个组合场景,避免后面再改这里时把语义悄悄带偏。

IdQuery batchQuery = null;
final int batchSize = this.batchSize;

@imbajin imbajin Apr 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ 这里把 batching 放在 GraphTransaction 通用层后,会影响所有 backend,而不只是 issue #2674 里提到的 RPC backend。

以 RocksDB 为例,当前 queryByIds() 仍然是逐 id 展开查询,并没有真正走 multi-get;现在强制按 query.batch_size 拆成多个 IdQuery,很可能只是增加额外的 query/iterator 次数。建议把这类分批策略下沉到具体 backend,或者至少通过 feature/store type 把它限定在 HBase/HStore 这类 RPC backend 上,避免把针对性优化变成全局行为变化。

PS: 后续我们应该让 RocksDB 使用上原生的 multi-get API (这应该是之前的 TODO)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. GraphTransaction 不再按 batch_size 拆分 ID,一次性下发完整 IdQuery;
  2. RocksDBTables.Vertex/Edge 已覆写 queryByIds(),在 !session.hasChanges() 时走 multiGetAsList() 原生批量读取,脏 session 安全回退到逐 id scan;


IdQuery query = new IdQuery(type);
for (Object vertexId : vertexIds) {
HugeVertex vertex;
Id id = HugeVertex.getIdValue(vertexId);
Expand All @@ -799,17 +801,25 @@ protected Iterator<Vertex> queryVerticesByIds(Object[] vertexIds, boolean adjace
// Found from local tx
vertices.put(vertex.id(), vertex);
} else {
// Prepare to query from backend store
query.query(id);
// Query from backend
if (!fetchedIds.contains(id)) {
if (batchQuery == null) {
batchQuery = new IdQuery(type);
}
batchQuery.query(id);
fetchedIds.add(id);

if (batchQuery.idsSize() >= batchSize) {
flushIdBatch(batchQuery, vertices);
batchQuery = null;
}
}
}
ids.add(id);
}

if (!query.empty()) {
// Query from backend store
query.mustSortByInput(false);
Iterator<HugeVertex> it = this.queryVerticesFromBackend(query);
QueryResults.fillMap(it, vertices);
if (batchQuery != null && !batchQuery.empty()) {
flushIdBatch(batchQuery, vertices);
}

return new MapperIterator<>(ids.iterator(), id -> {
Expand All @@ -831,6 +841,13 @@ protected Iterator<Vertex> queryVerticesByIds(Object[] vertexIds, boolean adjace
});
}

private void flushIdBatch(IdQuery query, Map<Id, HugeVertex> vertices) {
Query.checkForceCapacity(query.idsSize());
query.mustSortByInput(false);
Iterator<HugeVertex> it = this.queryVerticesFromBackend(query);
QueryResults.fillMap(it, vertices);
}

public Iterator<Vertex> queryVertices() {
Query q = new Query(HugeType.VERTEX);
return this.queryVertices(q);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -309,7 +309,7 @@ protected static BackendEntryIterator newEntryIterator(BackendColumnIterator col
}

protected static BackendEntryIterator newEntryIteratorOlap(
BackendColumnIterator cols, Query query, boolean isOlap) {
BackendColumnIterator cols, Query query, boolean isOlap) {
return new BinaryEntryIterator<>(cols, query, (entry, col) -> {
if (entry == null || !entry.belongToMe(col)) {
HugeType type = query.resultType();
Expand Down
Loading