bug(discover): extensionless scripts with valid shebangs are silently skipped

[ibaldr89]

Version

v0.9.0 and current main at 7d6cdb23ef5ca2fd51f5d5b7509b33e112ef15f3

Platform

Linux (x64)

Install channel

Built from source

Binary variant

standard

What happened, and what did you expect?

An extensionless executable script is silently excluded from discovery even when its first line unambiguously declares a supported language:

#!/usr/bin/env python3

def extensionless_probe():
    return 42

cbm_language_for_filename("tool") returns CBM_LANG_COUNT, and detect_file_language() returns immediately without inspecting the shebang. The file therefore produces no File or Function nodes, and graph/code searches are indistinguishable from a true negative.

Expected behavior: preserve existing special-filename and extension detection, but when they return unknown, inspect a bounded first line and infer a supported scripting language from a valid shebang.

Reproduction

1. Create a dummy repository:

mkdir -p /tmp/cbm-shebang-repro/bin
cat >/tmp/cbm-shebang-repro/bin/tool <<'PY'
#!/usr/bin/env python3

def extensionless_probe():
    return 42
PY
chmod +x /tmp/cbm-shebang-repro/bin/tool

2. Index it:

codebase-memory-mcp cli index_repository '{"repo_path":"/tmp/cbm-shebang-repro","mode":"full"}'

3. Search for extensionless_probe with search_graph or search_code.

Actual: zero results; the file is absent from discovery. Expected: Python File/Function nodes and a matching search result.

The executable bit is not essential to parsing; the shebang is the language signal and should also work on platforms without POSIX mode bits.

Proposed fix

• Keep filename/extension mappings authoritative.
• Only when they return unknown, read at most the first line.
• Recognize direct interpreter paths plus /usr/bin/env forms, including env -S.
• Match interpreter basenames exactly (with version suffixes such as python3.12 where applicable).
• Reject malformed/NUL-containing lines and read failures as unknown.
• Add reproduce-first discovery tests for direct Python, env python3, env -S python3 -u, precedence, and negative inputs.

Related work checked

• Option to index new filetypes #962 covers configurable include paths and a not-indexed signal, not language detection for a file reached by the walker.
• fix: add .m extension to EXT_TABLE for content-based disambiguation #306 fixed a missing extension mapping before content disambiguation, but still requires an extension.
• extra_extensions (feat(discover): user-defined extension→language mappings (C port) #73/docs(config): document extra_extensions values and skip behavior #487) maps extensions only and cannot represent an extensionless basename.
• Existing shebang PRs Use /usr/bin/env bash for widely compatible shebang #674/fix(cli): overflow-safe ZIP bounds check; env-bash shebangs #812 change project script portability; they do not add shebang-based source discovery.

Confirmations

• I searched existing open and closed issues/PRs and found no equivalent fix.
• The reproduction uses a shareable dummy snippet, not proprietary code.

[DeusData]

Confirmed on current main (97ce23f), and thank you for the unusually careful write-up — particularly the "Related work checked" section, which saved me the duplicate hunt entirely.

The mechanism is exactly where you place it. detect_file_language() (src/discover/discover.c:651-655) opens with:

CBMLanguage lang = cbm_language_for_filename(entry_name);
if (lang == CBM_LANG_COUNT) {
    return CBM_LANG_COUNT;
}

Every content-based path in that function — the .m disambiguation, .cls (ObjectScript UDL vs Apex), .inc, and the ObjectScript Export XML probe — sits after that early return and each one keys off strrchr(entry_name, '.'). So content inspection in discovery is structurally reachable only for files that already have a recognised extension. An extensionless name can never reach it.

Two details that support the shape you propose:

• The filename table is genuinely filename-based, not extension-only — Dockerfile and Makefile are mapped as whole names (src/discover/language.c:645-647). So "keep filename/extension mappings authoritative, probe the shebang only when they return unknown" layers onto the existing precedence rather than fighting it.
• Shebang parsing already exists in the codebase, but only in src/pipeline/pass_infrascan.c:815-830, where it records the interpreter of a known shell script for infrastructure scanning. It is downstream of discovery and never feeds language detection, so there is no existing helper being bypassed here — this really is a missing capability rather than a wiring mistake.

Your point that the executable bit shouldn't be required is right and worth keeping in the design: mode bits don't survive many Windows checkouts, and the shebang is the actual language signal.

On #1203: it's queued for review, and I'm deliberately not reviewing it in this triage pass. One thing you can act on now — GitHub currently reports it as CONFLICTING against main, which has moved a fair amount since you opened it. A rebase would help; I'd rather review the diff you intend than a stale one.

Thanks for reproducing against both v0.9.0 and a pinned main SHA, and for keeping the repro to a shareable snippet.