Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion QUERY_EXPLANATIONS_AND_SAMPLES.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,16 @@
# Query Explanations & Samples
This document covers the full query catalog currently generated by `UsersQueriesRepository` in `src/queries_repository.rs`, including core queries, optional algorithm queries, and phase-1 additions.
This document covers the full query catalog currently generated by `UsersQueriesRepository` in `src/queries_repository.rs`, including core queries, optional algorithm queries, phase-1 additions, and profile-gated extended/fixture-dependent additions.

## Dataset assumptions
- Primary label: `:User`
- Primary relationship: `:Friend`
- Common properties used by queries: `id`, `age`, and `bench_capacity`

## Coverage profiles
- `baseline`: core + phase-1 query set.
- `extended-core`: baseline + `temporal_spatial_roundtrip` (where vendor support exists).
- `fixture-dependent`: extended-core + text/vector smoke queries that require fixture seeding and additional indexes.

## Complete query catalog (alphabetical by query id)
- `aggregate_age`: average age over all users.
- `aggregate_age_distinct`: count distinct age values.
Expand Down Expand Up @@ -33,6 +38,8 @@ This document covers the full query catalog currently generated by `UsersQueries
- `exact_5_hop_traverse_count`: fixed 5-hop traversal.
- `exact_6_hop_traverse_count`: fixed 6-hop traversal.
- `foreach_loop_mutation`: `FOREACH` mutation-loop coverage.
- `fulltext_query_nodes_smoke`: fixture-dependent fulltext query on nodes.
- `fulltext_query_relationships_smoke`: fixture-dependent fulltext query on relationships.
- `id_range_scan`: internal-id range scan.
- `id_seek`: internal-id point lookup.
- `indexed_in_list_predicate`: IN-list index utilization shape.
Expand All @@ -57,13 +64,15 @@ This document covers the full query catalog currently generated by `UsersQueries
- `single_vertex_read`: point lookup by user id.
- `single_vertex_update`: update `rpc_social_credit` on a user.
- `single_vertex_write`: create one `User` node.
- `temporal_spatial_roundtrip`: extended-core temporal/spatial scalar-function roundtrip query.
- `union_all_ids`: `UNION ALL` composition.
- `union_distinct_ids`: `UNION` (distinct) composition.
- `unwind_rows`: row-level list fan-out with `UNWIND`.
- `value_join`: value-based join on `age`.
- `value_join_cnt`: count variant of the value join.
- `var_len_friends`: variable-length traversal (`*1..2`).
- `var_len_with_edge_where_filter`: variable-length traversal with edge filtering.
- `vector_query_nodes_smoke`: fixture-dependent vector-search query on nodes.
- `vertex_on_label_property`: lookup by label + property.
- `vertex_on_label_property_index`: same shape for index-path benchmarking.
- `vertex_on_property`: lookup by property without label predicate.
Expand Down Expand Up @@ -92,9 +101,22 @@ RETURN labels(a), type(r), properties(a), nodes(p), relationships(p), length(p)
LIMIT 1
```

```cypher
// Neo4j
CALL db.index.vector.queryNodes('bench_user_embedding_idx', 10, [0.1, 0.2, 0.3])
YIELD node, score
RETURN id(node), score
LIMIT 10
```

## Vendor-specific notes
- `shortest_path`, `shortest_path_with_filter`, and `all_shortest_paths_len` have vendor-specific query text.
- `aggregate_count_users` uses a Falkor-optimized `db.meta.stats()` path.
- `temporal_spatial_roundtrip` is currently generated only for FalkorDB and Neo4j.
- Fixture-dependent text/vector smoke queries (`vector_query_nodes_smoke`, `fulltext_query_nodes_smoke`, `fulltext_query_relationships_smoke`) use vendor-specific procedures/index names:
- FalkorDB: `db.idx.vector.*`, `db.idx.fulltext.*`
- Neo4j: `db.index.vector.*`, `db.index.fulltext.*`
- Memgraph: `vector_search.*`, `text_search.*`
- `algo_max_flow_single_pair` in Falkor uses one relationship type in config:
- gets a single value from `db.relationshipTypes()`
- passes `relationshipTypes: [relationshipType]`
10 changes: 10 additions & 0 deletions readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,11 @@ NOTE: It is possible to use the load command with externally run vendor endpoint
- `cargo run --release --bin benchmark -- load --vendor neo4j -s small --endpoint neo4j://neo4j:benchmark123@127.0.0.1:7687`
- `cargo run --release --bin benchmark -- load --vendor memgraph -s small --endpoint bolt://127.0.0.1:7687`

Profile-aware loading (runs additional fixture/index setup when required):
- `cargo run --release --bin benchmark -- load --vendor neo4j -s small --query-profile fixture-dependent`
- `cargo run --release --bin benchmark -- load --vendor memgraph -s small --query-profile fixture-dependent`
- `cargo run --release --bin benchmark -- load --vendor falkor -s small --query-profile fixture-dependent`

##### create a set of queries to be used with the run command

-
Expand All @@ -203,6 +208,11 @@ NOTE: preparing a smaller run of 1,000,000 queries:

`cargo run --release --bin benchmark -- generate-queries -s1000000 --dataset small --name=small-readonly --write-ratio 0.0`

Generate with a broader coverage profile:

- `cargo run --release --bin benchmark -- generate-queries -s1000000 --dataset small --name=small-extended --write-ratio 0.0 --vendor neo4j --query-profile extended-core`
- `cargo run --release --bin benchmark -- generate-queries -s1000000 --dataset small --name=small-fixtures --write-ratio 0.0 --vendor memgraph --query-profile fixture-dependent`

##### run the benchmarks

- `cargo run --release --bin benchmark run --vendor falkor --name small-readonly -p40 --mps 4000`
Expand Down
36 changes: 29 additions & 7 deletions scripts/run_large_benchmark.sh
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,11 @@ SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
# ENABLE_ALGO_MSF (default: 0)
# ENABLE_ALGO_HARMONIC (default: 0)
# FREE_PRIMARY_FALKOR_BEFORE_SECOND_RUN (default: 1)
# IN_SCRIPT_QUERY_PROFILE (default: empty; set in this file to force profile)
# baseline - baseline query set
# extended-core - baseline + extended core queries
# fixture-dependent - extended-core + fixture/index-dependent queries
# QUERY_PROFILE (env fallback when IN_SCRIPT_QUERY_PROFILE is empty; default: baseline)
#
# Results:
# RESULTS_DIR (default: Results-YYMMDD-HH:MM)
Expand Down Expand Up @@ -79,6 +84,23 @@ ENABLE_ALGO_MAX_FLOW=${ENABLE_ALGO_MAX_FLOW:-0}
ENABLE_ALGO_MSF=${ENABLE_ALGO_MSF:-0}
ENABLE_ALGO_HARMONIC=${ENABLE_ALGO_HARMONIC:-0}
FREE_PRIMARY_FALKOR_BEFORE_SECOND_RUN=${FREE_PRIMARY_FALKOR_BEFORE_SECOND_RUN:-1}
# Optional in-script query profile override.
# Set this value directly in the script to force a profile for every run.
# Leave empty ("") to keep env-based behavior (QUERY_PROFILE env var, else baseline).
IN_SCRIPT_QUERY_PROFILE=""
if [[ -n "$IN_SCRIPT_QUERY_PROFILE" ]]; then
QUERY_PROFILE="$IN_SCRIPT_QUERY_PROFILE"
else
QUERY_PROFILE=${QUERY_PROFILE:-baseline}
fi

case "$QUERY_PROFILE" in
baseline|extended-core|fixture-dependent) ;;
*)
echo "Invalid QUERY_PROFILE '$QUERY_PROFILE'. Valid options: baseline, extended-core, fixture-dependent." >&2
exit 1
;;
esac

# Derive per-vendor query file names so each engine can use vendor-optimized queries.
QUERIES_FILE_BASE="${QUERIES_FILE}"
Expand Down Expand Up @@ -376,7 +398,7 @@ if [[ "${RUN_FALKOR}" == "1" ]]; then
PYTHONPATH="../falkordb-bulk-loader" python3 "../falkordb-bulk-loader/falkordb_bulk_loader/bulk_insert.py" falkor \
-u "$REDIS_URL" \
-n "$CSV_DIR/User.csv" \
-r "$CSV_DIR/FRIEND.csv" \
-R Friend "$CSV_DIR/FRIEND.csv" \
-j INTEGER -s -i User:id -i User:age \
-c 128 -b 16 -t 16
fi
Expand Down Expand Up @@ -461,7 +483,7 @@ fi

if [[ "${RUN_NEO4J}" == "1" ]]; then
echo "==> Loading large dataset into Neo4j"
cargo run --release --bin benchmark -- load --vendor neo4j --size large --endpoint "$NEO4J_ENDPOINT" -b "$BATCH_SIZE"
cargo run --release --bin benchmark -- load --vendor neo4j --size large --endpoint "$NEO4J_ENDPOINT" -b "$BATCH_SIZE" --query-profile "$QUERY_PROFILE"
fi

if [[ "${RUN_FALKOR}" == "1" ]]; then
Expand Down Expand Up @@ -536,17 +558,17 @@ if [[ "${RUN_MEMGRAPH}" == "1" ]]; then
fi
fi

echo "==> Generating vendor-specific query files (base=${QUERIES_FILE_BASE}, dataset=large, count=${QUERIES_COUNT}, write_ratio=${WRITE_RATIO})"
echo "==> Generating vendor-specific query files (base=${QUERIES_FILE_BASE}, dataset=large, count=${QUERIES_COUNT}, write_ratio=${WRITE_RATIO}, profile=${QUERY_PROFILE})"
echo "==> Algorithm query toggles (pagerank=${ENABLE_ALGO_PAGERANK_BOOL}, max_flow=${ENABLE_ALGO_MAX_FLOW_BOOL}, msf=${ENABLE_ALGO_MSF_BOOL}, harmonic=${ENABLE_ALGO_HARMONIC_BOOL})"
# Always regenerate so each selected vendor gets the latest query catalog + stable q_id fields.
if [[ "${RUN_FALKOR}" == "1" || "${RUN_FALKOR_2}" == "1" ]]; then
cargo run --release --bin benchmark -- generate-queries --vendor falkor --dataset large --size "$QUERIES_COUNT" --name "$FALKOR_QUERIES_FILE" --write-ratio "$WRITE_RATIO" "${ALGO_QUERY_ARGS[@]}"
cargo run --release --bin benchmark -- generate-queries --vendor falkor --dataset large --size "$QUERIES_COUNT" --name "$FALKOR_QUERIES_FILE" --write-ratio "$WRITE_RATIO" --query-profile "$QUERY_PROFILE" "${ALGO_QUERY_ARGS[@]}"
fi
if [[ "${RUN_NEO4J}" == "1" ]]; then
cargo run --release --bin benchmark -- generate-queries --vendor neo4j --dataset large --size "$QUERIES_COUNT" --name "$NEO4J_QUERIES_FILE" --write-ratio "$WRITE_RATIO" "${ALGO_QUERY_ARGS[@]}"
cargo run --release --bin benchmark -- generate-queries --vendor neo4j --dataset large --size "$QUERIES_COUNT" --name "$NEO4J_QUERIES_FILE" --write-ratio "$WRITE_RATIO" --query-profile "$QUERY_PROFILE" "${ALGO_QUERY_ARGS[@]}"
fi
if [[ "${RUN_MEMGRAPH}" == "1" ]]; then
cargo run --release --bin benchmark -- generate-queries --vendor memgraph --dataset large --size "$QUERIES_COUNT" --name "$MEMGRAPH_QUERIES_FILE" --write-ratio "$WRITE_RATIO" "${ALGO_QUERY_ARGS[@]}"
cargo run --release --bin benchmark -- generate-queries --vendor memgraph --dataset large --size "$QUERIES_COUNT" --name "$MEMGRAPH_QUERIES_FILE" --write-ratio "$WRITE_RATIO" --query-profile "$QUERY_PROFILE" "${ALGO_QUERY_ARGS[@]}"
fi

echo "==> Running ${QUERIES_FILE} workload (parallel=${PARALLEL}, mps=${MPS})"
Expand Down Expand Up @@ -580,7 +602,7 @@ if [[ "${RUN_FALKOR_2}" == "1" ]]; then
PYTHONPATH="../falkordb-bulk-loader" python3 "../falkordb-bulk-loader/falkordb_bulk_loader/bulk_insert.py" falkor \
-u "$REDIS_URL_2" \
-n "$CSV_DIR/User.csv" \
-r "$CSV_DIR/FRIEND.csv" \
-R Friend "$CSV_DIR/FRIEND.csv" \
-j INTEGER -s -i User:id -i User:age \
-c 128 -b 16 -t 16

Expand Down
48 changes: 35 additions & 13 deletions scripts/run_medium_benchmark.sh
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,11 @@ SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
# ENABLE_ALGO_MAX_FLOW (default: 1)
# ENABLE_ALGO_MSF (default: 1)
# ENABLE_ALGO_HARMONIC (default: 1)
# IN_SCRIPT_QUERY_PROFILE (default: empty; set in this file to force profile)
# baseline - baseline query set
# extended-core - baseline + extended core queries
# fixture-dependent - extended-core + fixture/index-dependent queries
# QUERY_PROFILE (env fallback when IN_SCRIPT_QUERY_PROFILE is empty; default: baseline)
#
# Results:
# RESULTS_DIR (default: Results-YYMMDD-HH:MM)
Expand Down Expand Up @@ -63,13 +68,30 @@ BATCH_SIZE=${BATCH_SIZE:-5000}
PARALLEL=${PARALLEL:-10}
MPS=${MPS:-3000}
QUERIES_FILE=${QUERIES_FILE:-"medium-readonly"}
QUERIES_COUNT=${QUERIES_COUNT:-25000}
QUERIES_COUNT=${QUERIES_COUNT:-2500}
WRITE_RATIO=${WRITE_RATIO:-0.05}
FALKOR_QUERY_TIMEOUT_MS=${FALKOR_QUERY_TIMEOUT_MS:-900000}
ENABLE_ALGO_PAGERANK=${ENABLE_ALGO_PAGERANK:-1}
ENABLE_ALGO_MAX_FLOW=${ENABLE_ALGO_MAX_FLOW:-1}
ENABLE_ALGO_MSF=${ENABLE_ALGO_MSF:-1}
ENABLE_ALGO_HARMONIC=${ENABLE_ALGO_HARMONIC:-1}
ENABLE_ALGO_PAGERANK=${ENABLE_ALGO_PAGERANK:-0}
ENABLE_ALGO_MAX_FLOW=${ENABLE_ALGO_MAX_FLOW:-0}
ENABLE_ALGO_MSF=${ENABLE_ALGO_MSF:-0}
ENABLE_ALGO_HARMONIC=${ENABLE_ALGO_HARMONIC:-0}
# Optional in-script query profile override.
# Set this value directly in the script to force a profile for every run.
# Leave empty ("") to keep env-based behavior (QUERY_PROFILE env var, else baseline).
IN_SCRIPT_QUERY_PROFILE="fixture-dependent"
if [[ -n "$IN_SCRIPT_QUERY_PROFILE" ]]; then
QUERY_PROFILE="$IN_SCRIPT_QUERY_PROFILE"
else
QUERY_PROFILE=${QUERY_PROFILE:-baseline}
fi

case "$QUERY_PROFILE" in
baseline|extended-core|fixture-dependent) ;;
*)
echo "Invalid QUERY_PROFILE '$QUERY_PROFILE'. Valid options: baseline, extended-core, fixture-dependent." >&2
exit 1
;;
esac

# Derive per-vendor query file names so each engine can use vendor-optimized queries.
QUERIES_FILE_BASE="${QUERIES_FILE}"
Expand Down Expand Up @@ -246,7 +268,7 @@ if [[ "${RUN_FALKOR}" == "1" ]]; then
PYTHONPATH="../falkordb-bulk-loader" python3 ../falkordb-bulk-loader/falkordb_bulk_loader/bulk_insert.py falkor \
-u "$REDIS_URL" \
-n "$CSV_DIR/User.csv" \
-r "$CSV_DIR/FRIEND.csv" \
-R Friend "$CSV_DIR/FRIEND.csv" \
-j INTEGER -s -i User:id -i User:age \
-c 128 -b 16 -t 16
fi
Expand All @@ -262,20 +284,20 @@ if [[ "${RUN_FALKOR_2}" == "1" ]]; then
PYTHONPATH="../falkordb-bulk-loader" python3 ../falkordb-bulk-loader/falkordb_bulk_loader/bulk_insert.py falkor \
-u "$REDIS_URL_2" \
-n "$CSV_DIR/User.csv" \
-r "$CSV_DIR/FRIEND.csv" \
-R Friend "$CSV_DIR/FRIEND.csv" \
-j INTEGER -s -i User:id -i User:age \
-c 128 -b 16 -t 16
fi

if [[ "${RUN_NEO4J}" == "1" ]]; then
echo "==> Loading medium dataset into Neo4j"
cargo run --release --bin benchmark -- load --vendor neo4j --size medium --endpoint "$NEO4J_ENDPOINT" -b "$BATCH_SIZE"
cargo run --release --bin benchmark -- load --vendor neo4j --size medium --endpoint "$NEO4J_ENDPOINT" -b "$BATCH_SIZE" --query-profile "$QUERY_PROFILE"
fi

if [[ "${RUN_MEMGRAPH}" == "1" ]]; then
echo "==> Loading medium dataset into Memgraph (UNWIND loader)"
# --force clears the external Memgraph instance before loading
cargo run --release --bin benchmark -- load --vendor memgraph --size medium --endpoint "$MEMGRAPH_ENDPOINT" -b "$BATCH_SIZE" --force
cargo run --release --bin benchmark -- load --vendor memgraph --size medium --endpoint "$MEMGRAPH_ENDPOINT" -b "$BATCH_SIZE" --force --query-profile "$QUERY_PROFILE"
fi
if [[ "${RUN_FALKOR}" == "1" || "${RUN_FALKOR_2}" == "1" ]]; then
echo "==> Configuring FalkorDB query timeout (${FALKOR_QUERY_TIMEOUT_MS}ms)"
Expand All @@ -287,17 +309,17 @@ if [[ "${RUN_FALKOR_2}" == "1" ]]; then
set_falkor_query_timeout "FalkorDB (secondary)" "$FALKOR_2_HOST" "$FALKOR_2_PORT"
fi

echo "==> Generating vendor-specific query files (base=${QUERIES_FILE_BASE}, dataset=medium, count=${QUERIES_COUNT}, write_ratio=${WRITE_RATIO})"
echo "==> Generating vendor-specific query files (base=${QUERIES_FILE_BASE}, dataset=medium, count=${QUERIES_COUNT}, write_ratio=${WRITE_RATIO}, profile=${QUERY_PROFILE})"
echo "==> Algorithm query toggles (pagerank=${ENABLE_ALGO_PAGERANK_BOOL}, max_flow=${ENABLE_ALGO_MAX_FLOW_BOOL}, msf=${ENABLE_ALGO_MSF_BOOL}, harmonic=${ENABLE_ALGO_HARMONIC_BOOL})"
# Always regenerate so each vendor gets the latest query catalog + stable q_id fields.
if [[ "${RUN_FALKOR}" == "1" || "${RUN_FALKOR_2}" == "1" ]]; then
cargo run --release --bin benchmark -- generate-queries --vendor falkor --dataset medium --size "$QUERIES_COUNT" --name "$FALKOR_QUERIES_FILE" --write-ratio "$WRITE_RATIO" "${ALGO_QUERY_ARGS[@]}"
cargo run --release --bin benchmark -- generate-queries --vendor falkor --dataset medium --size "$QUERIES_COUNT" --name "$FALKOR_QUERIES_FILE" --write-ratio "$WRITE_RATIO" --query-profile "$QUERY_PROFILE" "${ALGO_QUERY_ARGS[@]}"
fi
if [[ "${RUN_NEO4J}" == "1" ]]; then
cargo run --release --bin benchmark -- generate-queries --vendor neo4j --dataset medium --size "$QUERIES_COUNT" --name "$NEO4J_QUERIES_FILE" --write-ratio "$WRITE_RATIO" "${ALGO_QUERY_ARGS[@]}"
cargo run --release --bin benchmark -- generate-queries --vendor neo4j --dataset medium --size "$QUERIES_COUNT" --name "$NEO4J_QUERIES_FILE" --write-ratio "$WRITE_RATIO" --query-profile "$QUERY_PROFILE" "${ALGO_QUERY_ARGS[@]}"
fi
if [[ "${RUN_MEMGRAPH}" == "1" ]]; then
cargo run --release --bin benchmark -- generate-queries --vendor memgraph --dataset medium --size "$QUERIES_COUNT" --name "$MEMGRAPH_QUERIES_FILE" --write-ratio "$WRITE_RATIO" "${ALGO_QUERY_ARGS[@]}"
cargo run --release --bin benchmark -- generate-queries --vendor memgraph --dataset medium --size "$QUERIES_COUNT" --name "$MEMGRAPH_QUERIES_FILE" --write-ratio "$WRITE_RATIO" --query-profile "$QUERY_PROFILE" "${ALGO_QUERY_ARGS[@]}"
fi

echo "==> Running ${QUERIES_FILE} workload (parallel=${PARALLEL}, mps=${MPS})"
Expand Down
Loading
Loading