Fix remaining OSINT signal text truncation by schergr · Pull Request #68 · calesthio/Crucix

schergr · 2026-03-21T17:01:13Z

Summary

Remove 120-char truncation in delta engine when building OSINT signals
Remove 80-char truncation in memory snapshots for urgent Telegram posts
Remove 120-char truncation in ideas/LLM context for OSINT posts
Improve signal formatting in Telegram alerts (bulleted list instead of inline)

The prior fix (753c676) removed truncation at source ingestion and alert formatting, but signals were still arriving at the alerter pre-truncated from upstream. The sendMessage chunker already handles Telegram's 4096-char API limit.

Test plan

Trigger a sweep with urgent OSINT posts and verify full text appears in Telegram alert
Confirm alert messages are properly chunked if they exceed 4096 chars
Verify delta engine correctly deduplicates signals with full-length text

🤖 Generated with Claude Code

Posts were being cut to 300 chars (source ingestion) and 150 chars (alert evaluation), losing valuable OSINT context. The sendMessage chunker already handles the 4096-char Telegram API limit. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The prior fix (753c676) only removed truncation at source ingestion and alert formatting. Signals were still being cut to 120 chars in the delta engine, 80 chars in memory snapshots, and 120 chars in the ideas LLM context — so OSINT posts arrived at the alerter already truncated. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

This PR removes remaining upstream truncation of urgent Telegram/OSINT signal text so full post content can flow through delta computation, memory snapshots, LLM context, and Telegram alert rendering (with improved “Signals” formatting).

Changes:

Removed substring/slice truncation in Telegram source ingestion, delta engine signal construction, and memory snapshot compaction.
Updated LLM “ideas” sweep compaction to include full urgent OSINT post text.
Improved Telegram alert formatting for signals (more items + bulleted list output).

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
lib/llm/ideas.mjs	Stops truncating urgent OSINT post text included in LLM ideas context.
lib/delta/memory.mjs	Stores full urgent post text in compacted memory snapshots.
lib/delta/engine.mjs	Emits full urgent post text in newly-detected OSINT signals.
lib/alerts/telegram.mjs	Expands/reshapes OSINT signal text shown in alerts and formats signals as bullets.
apis/sources/telegram.mjs	Stops truncating Telegram message text extracted via Bot API and web preview parsing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

lib/llm/ideas.mjs

lib/alerts/telegram.mjs

calesthio · 2026-03-22T20:03:52Z

Thanks for opening this. The direction makes sense, but there are two issues I think should be fixed before this is merged:

Telegram alert formatting now sends full raw OSINT post text through parse_mode: Markdown without escaping. In the rule-based OSINT surge path, evaluation.signals can now contain full Telegram post bodies, and _formatTieredAlert() renders them as bullet lines. Real post text commonly contains _, brackets, parentheses, and similar Markdown-significant characters. That means alerts can render incorrectly or be rejected by the Bot API altogether. Please either escape Markdown-sensitive characters before formatting or send this section without Markdown parsing.
The ideas LLM context no longer has a length bound for urgent OSINT posts. Keeping full text in storage/delta/memory is reasonable, but compactSweepForLLM() is supposed to stay compact and now it can be dominated by a handful of long Telegram posts. That creates regression risk for latency, cost, and provider-side input-limit failures. Please keep full text upstream, but add an overall size/token cap when building the ideas prompt.

Once those two are addressed, this looks much closer to mergeable.

Addresses PR review: escape Markdown-sensitive characters in _formatTieredAlert signal bullets to prevent Telegram Bot API rejections, and add a 1500-char budget for URGENT_OSINT in compactSweepForLLM to bound prompt size while keeping full text upstream. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Replace single &calesthio#39; handler with generic numeric/hex entity decoder so &calesthio#39; and other unpadded entities are properly converted - Dedup urgent OSINT posts against all hot memory runs (last 3 sweeps) instead of only the previous sweep, preventing posts that drop out of one sweep from reappearing as "new" in the next Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

schergr · 2026-03-24T18:11:50Z

anything else you need?

calesthio · 2026-03-25T01:49:27Z

Added a follow-up commit on top of this branch to close the remaining review issues:

switched broad OSINT dedup in lib/delta/engine.mjs to prefer stable post identity (postId, or channel/chat + date + text) instead of only the lossy semantic hash
preserved channel/post identity in lib/delta/memory.mjs so cross-run dedup has enough information to suppress exact reposts without hiding genuinely new updates
aligned signal escaping in lib/alerts/telegram.mjs with the bot's existing legacy Markdown parse mode instead of MarkdownV2-style escaping

Rechecked the branch after the patch: sweep still completes, dashboard inject still runs, the new-post dedup false negative is fixed, and this stays scoped to Telegram/delta/ideas paths without touching jarvis core UI code.

calesthio

Reviewed the updated branch including the follow-up fix commit. The truncation removal adds real value, and with the dedup identity + Markdown escaping fixes in place I don’t see a remaining blocker.

schergr · 2026-03-25T17:15:35Z

This PR is ready to merge — can you merge it when you get a chance? We're currently running the unpatched code on master.

commit 53f6d81 Merge: 8c1ea37 5c08355 Author: Calesthio <celesthioailabs@gmail.com> Date: Wed Mar 25 10:21:02 2026 -0700 Merge pull request calesthio#68 from schergr/fix/osint-signal-truncation Fix remaining OSINT signal text truncation commit 5c08355 Author: calesthio <celesthioailabs@gmail.com> Date: Tue Mar 24 18:48:55 2026 -0700 Fix Telegram dedup identity and legacy Markdown escaping commit b7322f1 Author: Greg Scher <greg@212southadvisors.com> Date: Mon Mar 23 13:01:32 2026 -0400 Fix HTML entity decoding and broaden OSINT dedup window - Replace single &calesthio#39; handler with generic numeric/hex entity decoder so &calesthio#39; and other unpadded entities are properly converted - Dedup urgent OSINT posts against all hot memory runs (last 3 sweeps) instead of only the previous sweep, preventing posts that drop out of one sweep from reappearing as "new" in the next Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> commit 31c305c Author: Greg Scher <greg@212southadvisors.com> Date: Mon Mar 23 12:57:37 2026 -0400 Escape Markdown in alert signals and cap OSINT text in ideas prompt Addresses PR review: escape Markdown-sensitive characters in _formatTieredAlert signal bullets to prevent Telegram Bot API rejections, and add a 1500-char budget for URGENT_OSINT in compactSweepForLLM to bound prompt size while keeping full text upstream. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> commit 2d166c2 Author: Greg Scher <greg@212southadvisors.com> Date: Sat Mar 21 12:59:30 2026 -0400 Remove remaining text truncation across delta engine, memory, and ideas The prior fix (753c676) only removed truncation at source ingestion and alert formatting. Signals were still being cut to 120 chars in the delta engine, 80 chars in memory snapshots, and 120 chars in the ideas LLM context — so OSINT posts arrived at the alerter already truncated. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> commit 753c676 Author: Greg Scher <greg@212southadvisors.com> Date: Fri Mar 20 16:49:58 2026 -0400 Remove text truncation limits from Telegram posts Posts were being cut to 300 chars (source ingestion) and 150 chars (alert evaluation), losing valuable OSINT context. The sendMessage chunker already handles the 4096-char Telegram API limit. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

commit 8d99875 Author: Matt <mdrong2@gmail.com> Date: Wed Mar 25 14:19:07 2026 -0500 updated commodities commit a9d77d9 Author: Matt-Drong <mathias.drong@creospan.com> Date: Wed Mar 25 13:44:05 2026 -0500 Squashed commit of the following: commit 53f6d81 Merge: 8c1ea37 5c08355 Author: Calesthio <celesthioailabs@gmail.com> Date: Wed Mar 25 10:21:02 2026 -0700 Merge pull request calesthio#68 from schergr/fix/osint-signal-truncation Fix remaining OSINT signal text truncation commit 5c08355 Author: calesthio <celesthioailabs@gmail.com> Date: Tue Mar 24 18:48:55 2026 -0700 Fix Telegram dedup identity and legacy Markdown escaping commit b7322f1 Author: Greg Scher <greg@212southadvisors.com> Date: Mon Mar 23 13:01:32 2026 -0400 Fix HTML entity decoding and broaden OSINT dedup window - Replace single &calesthio#39; handler with generic numeric/hex entity decoder so &calesthio#39; and other unpadded entities are properly converted - Dedup urgent OSINT posts against all hot memory runs (last 3 sweeps) instead of only the previous sweep, preventing posts that drop out of one sweep from reappearing as "new" in the next Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> commit 31c305c Author: Greg Scher <greg@212southadvisors.com> Date: Mon Mar 23 12:57:37 2026 -0400 Escape Markdown in alert signals and cap OSINT text in ideas prompt Addresses PR review: escape Markdown-sensitive characters in _formatTieredAlert signal bullets to prevent Telegram Bot API rejections, and add a 1500-char budget for URGENT_OSINT in compactSweepForLLM to bound prompt size while keeping full text upstream. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> commit 2d166c2 Author: Greg Scher <greg@212southadvisors.com> Date: Sat Mar 21 12:59:30 2026 -0400 Remove remaining text truncation across delta engine, memory, and ideas The prior fix (753c676) only removed truncation at source ingestion and alert formatting. Signals were still being cut to 120 chars in the delta engine, 80 chars in memory snapshots, and 120 chars in the ideas LLM context — so OSINT posts arrived at the alerter already truncated. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> commit 753c676 Author: Greg Scher <greg@212southadvisors.com> Date: Fri Mar 20 16:49:58 2026 -0400 Remove text truncation limits from Telegram posts Posts were being cut to 300 chars (source ingestion) and 150 chars (alert evaluation), losing valuable OSINT context. The sendMessage chunker already handles the 4096-char Telegram API limit. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> commit b54ddce Author: Matt <mdrong2@gmail.com> Date: Wed Mar 25 02:22:48 2026 -0500 sure why not

commit 212fefd Merge: 3f3d050 1b80539 Author: Calesthio <celesthioailabs@gmail.com> Date: Sat Mar 28 23:14:48 2026 -0700 Merge pull request calesthio#85 from calesthio/codex/gold-silver-macro feat: add gold and silver to macro + markets context commit 1b80539 Author: calesthio <celesthioailabs@gmail.com> Date: Sat Mar 28 22:53:23 2026 -0700 feat(markets): surface gold and silver across dashboard and briefs commit 3f3d050 Merge: 53f6d81 1fb5093 Author: Calesthio <celesthioailabs@gmail.com> Date: Sat Mar 28 22:41:17 2026 -0700 Merge pull request calesthio#62 from 3byss/master feat: LLM provider for Grok commit 1fb5093 Author: calesthio <celesthioailabs@gmail.com> Date: Sat Mar 28 22:36:33 2026 -0700 fix(grok): align token caps and docs commit 18233bc Merge: 457dba2 e9b04b2 Author: 3byss <ziyadaneffati@gmail.com> Date: Fri Mar 27 22:23:07 2026 -0600 Merge remote-tracking branch 'refs/remotes/origin/master' commit 457dba2 Author: 3byss <ziyadaneffati@gmail.com> Date: Fri Mar 27 22:22:21 2026 -0600 Fixed any issues with the PR commit e9b04b2 Merge: caf4430 53f6d81 Author: 3byss <ziyadaneffati@gmail.com> Date: Fri Mar 27 22:19:00 2026 -0600 Merge branch 'calesthio:master' into master commit 9028d42 Merge: caf4430 53f6d81 Author: 3byss <ziyadaneffati@gmail.com> Date: Fri Mar 27 22:13:37 2026 -0600 Merge remote-tracking branch 'upstream/master' commit 53f6d81 Merge: 8c1ea37 5c08355 Author: Calesthio <celesthioailabs@gmail.com> Date: Wed Mar 25 10:21:02 2026 -0700 Merge pull request calesthio#68 from schergr/fix/osint-signal-truncation Fix remaining OSINT signal text truncation commit 5c08355 Author: calesthio <celesthioailabs@gmail.com> Date: Tue Mar 24 18:48:55 2026 -0700 Fix Telegram dedup identity and legacy Markdown escaping commit b7322f1 Author: Greg Scher <greg@212southadvisors.com> Date: Mon Mar 23 13:01:32 2026 -0400 Fix HTML entity decoding and broaden OSINT dedup window - Replace single &calesthio#39; handler with generic numeric/hex entity decoder so &calesthio#39; and other unpadded entities are properly converted - Dedup urgent OSINT posts against all hot memory runs (last 3 sweeps) instead of only the previous sweep, preventing posts that drop out of one sweep from reappearing as "new" in the next Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> commit 31c305c Author: Greg Scher <greg@212southadvisors.com> Date: Mon Mar 23 12:57:37 2026 -0400 Escape Markdown in alert signals and cap OSINT text in ideas prompt Addresses PR review: escape Markdown-sensitive characters in _formatTieredAlert signal bullets to prevent Telegram Bot API rejections, and add a 1500-char budget for URGENT_OSINT in compactSweepForLLM to bound prompt size while keeping full text upstream. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> commit caf4430 Merge: cb27575 8c1ea37 Author: 3byss <ziyadaneffati@gmail.com> Date: Sat Mar 21 16:17:13 2026 -0600 Merge branch 'master' into master commit 2d166c2 Author: Greg Scher <greg@212southadvisors.com> Date: Sat Mar 21 12:59:30 2026 -0400 Remove remaining text truncation across delta engine, memory, and ideas The prior fix (753c676) only removed truncation at source ingestion and alert formatting. Signals were still being cut to 120 chars in the delta engine, 80 chars in memory snapshots, and 120 chars in the ideas LLM context — so OSINT posts arrived at the alerter already truncated. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> commit cb27575 Author: 3byss <ziyadaneffati@gmail.com> Date: Fri Mar 20 16:26:30 2026 -0600 Fixed types and created tests commit 753c676 Author: Greg Scher <greg@212southadvisors.com> Date: Fri Mar 20 16:49:58 2026 -0400 Remove text truncation limits from Telegram posts Posts were being cut to 300 chars (source ingestion) and 150 chars (alert evaluation), losing valuable OSINT context. The sendMessage chunker already handles the 4096-char Telegram API limit. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> commit 17b8d9e Author: 3byss <ziyadaneffati@gmail.com> Date: Fri Mar 20 00:46:44 2026 -0600 Finished the Grok LLM implementation

Greg Scher and others added 2 commits March 20, 2026 16:49

schergr requested a review from calesthio as a code owner March 21, 2026 17:01

Copilot AI review requested due to automatic review settings March 21, 2026 17:01

Copilot started reviewing on behalf of schergr March 21, 2026 17:01 View session

Copilot AI reviewed Mar 21, 2026

View reviewed changes

lib/llm/ideas.mjs Outdated Show resolved Hide resolved

lib/alerts/telegram.mjs Show resolved Hide resolved

lib/alerts/telegram.mjs Show resolved Hide resolved

Greg Scher and others added 2 commits March 23, 2026 12:57

Fix Telegram dedup identity and legacy Markdown escaping

5c08355

calesthio approved these changes Mar 25, 2026

View reviewed changes

calesthio merged commit 53f6d81 into calesthio:master Mar 25, 2026
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix remaining OSINT signal text truncation#68

Fix remaining OSINT signal text truncation#68
calesthio merged 5 commits intocalesthio:masterfrom
schergr:fix/osint-signal-truncation

schergr commented Mar 21, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

calesthio commented Mar 22, 2026

Uh oh!

schergr commented Mar 24, 2026

Uh oh!

calesthio commented Mar 25, 2026 •

edited

Loading

Uh oh!

calesthio left a comment

Uh oh!

schergr commented Mar 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

schergr commented Mar 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

calesthio commented Mar 22, 2026

Uh oh!

schergr commented Mar 24, 2026

Uh oh!

calesthio commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

calesthio left a comment

Choose a reason for hiding this comment

Uh oh!

schergr commented Mar 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

schergr commented Mar 21, 2026 •

edited

Loading

calesthio commented Mar 25, 2026 •

edited

Loading