unification of events + clusters #1080

skull8888888 · 2025-12-09T18:35:50Z

Note

Introduce semantic events pipeline with clustering and source-aware events, unify workers, and add frontend for managing code/semantic events and clusters.

Backend:
- Semantic Events: Add SemanticEventHandler, queues (semantic_event_*), trigger spans cache (semantic_event_trigger_spans), and DB modules for semantic event definitions/triggers; write events to ClickHouse with source.
- Clustering: Add event clustering flow (event_cluster_configs, event_clustering_*), mustache templating, locking, and service call; new CH view events_v0 with clusters.
- Workers: Introduce generic worker framework and migrate spans, browser events, evaluators, payloads, notifications, Quickwit indexer, semantic events, and clustering to it.
- Events: Record span events via record_span_events; add EventSource and propagate to CH/queries; remove old trace summary flow.
- Health/Features: Simplify probes; add SemanticEvents and update Clustering flags.
Frontend:
- Events UI: Split into Semantic and Code tabs/pages; new CRUD/API for semantic event definitions; manage trigger spans and prompts; display events with source filter and attributes first.
- Clustering UI: Add clusters table per event, APIs to get/manage cluster config; start/disable clustering dialogs with mustache editor.
- Table Infra: Infinite table supports optional "Load More" button; header breadcrumbs support links.
DB/SQL:
- Add tables: event_cluster_configs, event_clusters, semantic_event_definitions, semantic_event_trigger_spans; enable vector extension.
- ClickHouse: add events.source and events_v0 view; update queries to filter by source.

^{Written by Cursor Bugbot for commit 712f06e. This will update automatically on new commits. Configure here.}

* tmp * tmp * spans * all * fixes * error * more fixes * fixes

ellipsis-dev · 2025-12-09T18:36:35Z

⚠️ This PR is too big for Ellipsis, but support for larger PRs is coming soon. If you want us to prioritize this feature, let us know at [email protected]

Generated with ❤️ by ellipsis.dev

app-server/src/traces/clustering.rs

app-server/src/db/event_cluster_configs.rs

app-server/src/db/semantic_event_trigger_spans.rs

app-server/src/features/mod.rs

app-server/src/db/events.rs

cursor · 2025-12-09T22:34:09Z

frontend/lib/actions/events/index.ts

 const GetLastEventSchema = z.object({
  projectId: z.string(),
  name: z.string(),
+  eventSource: z.enum(["CODE", "SEMANTIC"]),


Bug: Case mismatch in eventSource query parameter

The GetLastEventSchema defines eventSource as z.enum(["CODE", "SEMANTIC"]) (uppercase), but the backend Rust code stores the source field as lowercase "code" or "semantic" in ClickHouse (via EventSource::to_string()). The frontend pages pass 'CODE' and 'SEMANTIC' to getLastEvent, which queries ClickHouse with these uppercase values. Since ClickHouse string comparisons are case-sensitive, the query source = {source: String} will never match any records, causing getLastEvent to always return no results.

Additional Locations (2)

frontend/app/project/[projectId]/events/code/[id]/page.tsx#L28-L31

frontend/app/project/[projectId]/events/semantic/[id]/page.tsx#L28-L31

* feat: add template prefill * feat: update type, always pass prompt, structured output

app-server/src/db/semantic_event_trigger_spans.rs

frontend/app/api/projects/[projectId]/events/[name]/cluster-config/route.ts

app-server/src/db/event_cluster_configs.rs

cursor · 2025-12-10T16:43:38Z

app-server/src/utils/mod.rs

+            .map_err(|e| {
+                log::warn!("Failed to call service ({}): {:?}", service_url, e);
+                backoff::Error::transient(anyhow::Error::from(e))
+            })?;


Bug: Missing HTTP request timeout in service utility function

The new call_service_with_retry function does not set a timeout on individual HTTP requests. The old clustering code explicitly included .timeout(Duration::from_secs(60)) to prevent requests from hanging indefinitely. The backoff's max_elapsed_time only controls retry timing, not individual request duration. If an external service becomes unresponsive, worker threads calling this function (clustering and semantic events handlers) will block forever, causing the worker pool to gradually degrade.

Additional Locations (1)

app-server/src/traces/clustering.rs#L180-L187

app-server/src/utils/mod.rs

app-server/src/traces/semantic_events.rs

cursor · 2025-12-11T13:39:35Z

app-server/src/db/events.rs

        // 8 bytes for timestamp,
-        return 16 + 16 + 16 + 8 + 8 + self.name.len() + estimate_json_size(&self.attributes);
+        return 16 + 16 + 16 + 8 + self.name.len() + estimate_json_size(&self.attributes);
    }


Bug: Event size estimate missing new source field

The estimate_size_bytes function calculates event size for usage tracking but doesn't include the newly added source field. The old created_at field (8 bytes) was removed from the calculation, but the new source field (which stores "CODE" or "SEMANTIC") wasn't added. This causes size estimates to be slightly smaller than actual, affecting workspace usage limit calculations.

cursor · 2025-12-11T14:23:44Z

app-server/src/traces/semantic_events.rs

+        "project_id": message.project_id.to_string(),
+        "trace_id": message.trace_id.to_string(),
+        "event_definition": serde_json::to_value(event_definition).unwrap(),
+    });


Bug: Missing trigger_span_id in semantic event service request

The request body sent to the semantic event service is missing trigger_span_id. The SemanticEventMessage struct contains this field and it's passed through the queue correctly, but when constructing the HTTP request body, only project_id, trace_id, and event_definition are included. The old code in summary.rs included "trigger_span_id": message.trigger_span_id.to_string() in the request. The external service likely needs this field to identify which span triggered the semantic event analysis.

app-server/src/traces/semantic_events.rs

skull8888888 and others added 6 commits December 3, 2025 13:39

tmp

66b1850

Merge branch 'dev' into patterns

c00cb73

Workers (#1069)

54293cf

* tmp * tmp * spans * all * fixes * error * more fixes * fixes

Feat/patterns UI (#1071)

20b9a47

tmp

e404454

tmp

87f00ee

cursor bot reviewed Dec 9, 2025

View reviewed changes

app-server/src/traces/clustering.rs Show resolved Hide resolved

app-server/src/db/event_cluster_configs.rs Show resolved Hide resolved

app-server/src/db/semantic_event_trigger_spans.rs Outdated Show resolved Hide resolved

app-server/src/features/mod.rs Outdated Show resolved Hide resolved

fixes

98bcc6c

cursor bot reviewed Dec 9, 2025

View reviewed changes

app-server/src/db/events.rs Show resolved Hide resolved

skull8888888 and others added 2 commits December 9, 2025 22:21

fixes

ad4dc1b

feat: separate semantic and code events (#1078)

62a2957

cursor bot reviewed Dec 9, 2025

View reviewed changes

feat: add template prefill (#1081)

1b8749c

* feat: add template prefill * feat: update type, always pass prompt, structured output

cursor bot reviewed Dec 10, 2025

View reviewed changes

app-server/src/db/semantic_event_trigger_spans.rs Outdated Show resolved Hide resolved

frontend/app/api/projects/[projectId]/events/[name]/cluster-config/route.ts Outdated Show resolved Hide resolved

tmp

3675325

cursor bot reviewed Dec 10, 2025

View reviewed changes

olzhik11 and others added 3 commits December 10, 2025 17:56

feat: add constraint

0814b4a

tmp

46a4eee

fixes

07360f6

cursor bot reviewed Dec 11, 2025

View reviewed changes

fixes

a328ed7

cursor bot reviewed Dec 11, 2025

View reviewed changes

updates

712f06e

cursor bot reviewed Dec 11, 2025

View reviewed changes

app-server/src/traces/semantic_events.rs Show resolved Hide resolved

dinmukhamedm mentioned this pull request Dec 11, 2025

remove the aggregate traces feature flag #1085

Merged

skull8888888 merged commit 24766df into dev Dec 11, 2025
5 checks passed

skull8888888 deleted the patterns branch December 11, 2025 15:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

unification of events + clusters #1080

unification of events + clusters #1080

Uh oh!

skull8888888 commented Dec 9, 2025 •

edited by cursor bot

Loading

Uh oh!

ellipsis-dev bot commented Dec 9, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor bot Dec 9, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor bot Dec 10, 2025

Uh oh!

Uh oh!

Uh oh!

cursor bot Dec 11, 2025

Uh oh!

cursor bot Dec 11, 2025

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

unification of events + clusters #1080

unification of events + clusters #1080

Uh oh!

Conversation

skull8888888 commented Dec 9, 2025 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ellipsis-dev bot commented Dec 9, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor bot Dec 9, 2025

Choose a reason for hiding this comment

Bug: Case mismatch in eventSource query parameter

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor bot Dec 10, 2025

Choose a reason for hiding this comment

Bug: Missing HTTP request timeout in service utility function

Uh oh!

Uh oh!

Uh oh!

cursor bot Dec 11, 2025

Choose a reason for hiding this comment

Bug: Event size estimate missing new source field

Uh oh!

cursor bot Dec 11, 2025

Choose a reason for hiding this comment

Bug: Missing trigger_span_id in semantic event service request

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

skull8888888 commented Dec 9, 2025 •

edited by cursor bot

Loading