Skip to content

fix(chat): flag unresolved [cite: N] markers instead of hiding them (#2563) - #2566

Open
tomasbenavidez wants to merge 1 commit into
teng-lin:mainfrom
tomasbenavidez:fix/2563-unresolved-citation-signal
Open

tomasbenavidez wants to merge 1 commit into
teng-lin:mainfrom
tomasbenavidez:fix/2563-unresolved-citation-signal

Conversation

@tomasbenavidez

@tomasbenavidez tomasbenavidez commented Nov 20, 2026 •

Copy link
Copy Markdown

Summary

Implements the client-side signal suggested in #2563: when a chat turn streams without its citation table, callers can now see that the answer's citations could not be linked to sources, instead of getting a clean-looking answer with references: [].

While reproducing it, I found a second, related problem in text mode: ask printed the answer as Rich markup, so [cite: 3] was parsed as a markup tag and silently dropped ("Claim [cite: 3]." printed as "Claim .").

Related Issue

Refs #2563. This only adds the signal on the client side. The root cause (the server omitting the citation table on some follow-up turns) is still being investigated in the issue, so this PR should not close it.

Changes

  • AskResult.unresolved_citation_count: a property that counts raw [cite: N] / [cite: N, M] markers in answer. [N] markers and prose such as [cite: N] with a literal N are not counted.
  • ask_result_view adds unresolved_citations (an int, always present) to the MCP chat_ask / chat_start, REST, and ask --json envelopes. It reads the property with getattr so _app stays inside its import boundary (test_app_has_no_transport_dependency_imports).
  • CLI text ask: prints the answer with markup=False and, when markers are present, warns and suggests --new.
  • --save-as-note: a new NOTE_UNRESOLVED_CITATIONS chat event warns that the plain-text note keeps the raw markers.
  • Regenerated backend_static_coupling.json with scripts/regen_baselines.py. Only line counts changed.
  • CHANGELOG (Unreleased → Added) and the AskResult block in docs/python-api.md.

Test Plan

  • I tested these changes locally (Windows 11, Python 3.13, uv sync --frozen with browser/dev/markdown/mcp/server extras)
  • Tests pass (uv run pytest) for the touched areas: tests/unit/test_unresolved_citations.py (new), tests/unit/app/test_app_chat.py, tests/unit/app/test_app_serialize.py, tests/unit/cli/test_chat.py, tests/unit/mcp/test_chat.py, tests/server, and tests/_guardrails (app boundary, public surface, manifest baselines, cassettes)
  • Linting and formatting pass (uv run pre-commit run --all-files)
  • Type checking passes (uv run mypy src/notebooklm): no new errors. The 6 reported on Windows (os.O_NOFOLLOW/O_DIRECTORY/O_NONBLOCK in mcp/_hostupload.py, two unused ignores in _auth/refresh.py) are already on main.
  • N/A: no Android runtime code changed
  • N/A: no architectural change

The full tests/unit run on Windows has failures that also show up on a clean main checkout: symlink tests without symlink privilege, the bash-based CI workflow tests, timing-sensitive fault-server tests, and the test_browser_oauth_token.py survivor-policy drift. None of them are in chat, views, or CLI chat code. I expect Linux CI to confirm.

New tests use the shared mock_fetch_tokens fixture rather than a new patch.object(auth_module, ...) site, so the shrink-only auth patch baselines do not grow.

Notes

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Chat answers now report unresolved citation markers, including their count in text and JSON output.
    • Saving an answer with unresolved citations displays a warning; the answer text and raw markers are retained.
  • Documentation
    • Documented unresolved citation counts and how they appear in result output.

…eng-lin#2563)

Follow-up turns can stream without their citation table: the answer keeps
raw `[cite: N]` markers, `references` is empty, and nothing tells the caller
the claims cannot be mapped to sources.

- AskResult.unresolved_citation_count counts the raw markers.
- ask_result_view adds `unresolved_citations` to the MCP / REST /
  `ask --json` envelopes.
- Text `ask` prints the answer with markup=False (Rich was swallowing the
  markers as markup tags) and warns, suggesting `--new`.
- `--save-as-note` emits NOTE_UNRESOLVED_CITATIONS so the plain-text note's
  raw markers are not a silent downgrade.
- Regenerate backend_static_coupling baseline (line counts only).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The change counts raw numeric citation markers in answers and exposes the count in result payloads. Text-mode CLI output preserves the markers and displays a warning when the count is nonzero. The plain-text note-saving flow also reports unresolved markers.

Changes: Unresolved citation reporting

Layer / File(s) Summary
Count unresolved citation markers
src/notebooklm/_types/chat.py, tests/unit/test_unresolved_citations.py, tests/fixtures/baselines/backend_static_coupling.json
AskResult now counts raw numeric citation markers in its answer. Unit tests cover marker formats and counts.
Expose counts in results and CLI output
src/notebooklm/_app/views.py, src/notebooklm/cli/chat_cmd.py, tests/unit/app/test_app_serialize.py, tests/unit/cli/test_chat.py, docs/python-api.md, CHANGELOG.md
Result serialization adds unresolved_citations, defaulting to zero. Text-mode output preserves raw markers and warns when the count is nonzero; JSON output includes the count. Tests and documentation describe these outputs.
Report unresolved citations when saving notes
src/notebooklm/_app/chat.py, src/notebooklm/cli/chat_cmd.py, tests/unit/app/test_app_chat.py, tests/unit/cli/test_chat.py
The plain-text note fallback emits a counted unresolved-citation event before note creation. The CLI renders the event as a warning. Tests cover the event and warning.

Estimated code review effort: 2 (Simple) | ~12 minutes

Change: Bug fix

Possibly related issues

  • Issue 2563 describes raw citation markers with an empty references list in follow-up turns. This change adds caller-facing counts and warnings, but the issue notes that the server-side cause remains under investigation and is not resolved here.

Suggested reviewers: teng-lin

Merge Risk: Minimal · up to b37e3

This change adds an always-present unresolved citation count to ask results and warns in the CLI when citation markers have no linked references. The behavior is additive and low risk, and no merge-blocking issue was found.

Pre-merge checks: Passed 4

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title names the main change: unresolved [cite: N] markers are now flagged to callers instead of being hidden. The changeset matches this. It adds AskResult.unresolved_citation_count, a NOTE_UNRESO…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main change: unresolved [cite: N] markers are now flagged to callers instead of being hidden. The changeset matches this. It adds AskResult.unresolved_citation_count, a NOTE_UNRESOLVED_CITATIONS warning, CLI warnings in text-mode ask, and an unresolved_citations field in JSON, MCP, and REST envelopes. The title is specific and concise. It uses the conventional commit form and includes the issue reference #2563, which the objectives say this PR does not close. That reference is a minor point and does not make the title misleading.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: fba15a0d-0e98-4c57-9be4-28161a8cbf03
Commits

Reviewing files that changed from the base of the PR and between c526cba and b37e363.

Files selected for processing: 11

  • CHANGELOG.md
  • docs/python-api.md
  • src/notebooklm/_app/chat.py
  • src/notebooklm/_app/views.py
  • src/notebooklm/_types/chat.py
  • src/notebooklm/cli/chat_cmd.py
  • tests/fixtures/baselines/backend_static_coupling.json
  • tests/unit/app/test_app_chat.py
  • tests/unit/app/test_app_serialize.py
  • tests/unit/cli/test_chat.py
  • tests/unit/test_unresolved_citations.py

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

@lovebuilt

Copy link
Copy Markdown

I checked this PR's marker pattern against the ask --json outputs saved from my #2563 captures: 35 turns (7 first turns, 28 follow-ups) on two copies of the 551-source notebook, notebooklm-py 0.8.6. I ran only the pattern from src/notebooklm/_types/chat.py (commit b37e363b) over the saved answer fields. I have not installed the branch.

turns references count from the PR's pattern
failing follow-ups 4 0 in all 4 37, 34, 30, 36
healthy turns 31 6 or more in all 31 0 in all 31
  • The four counts equal the ones in my ask: follow-up turns intermittently return raw [cite: N] markers and an empty references list (server sends no citation table) #2563 comment.
  • A looser, case-insensitive search for cite found 137 occurrences in the four failing answers, every one inside a pattern match, so no marker shape escapes the pattern in any of the 35 answers.
  • The pattern matched nothing in any healthy answer, and none of the 31 contains the text cite at all. The notebook has 17 report sources with literal [cite: text (2,523 occurrences). 22 of the 31 healthy turns had those reports selected (17 on all 551 sources, 5 on the 17 reports alone); the other 9 ran without them.

Not tested by me: anything that needs the branch installed. That is the unresolved_citations field itself in the ask --json, MCP and REST envelopes, the text-mode output (the markup=False change and the warning line), and the --save-as-note warning.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants