Description
Extensions configured in programming_languages.json with no language field receive score=0.0 with scoring_method='skipped-unsupported'. The scorer only line-counts files whose extension is in the NON_CODE_EXTENSIONS constant — it never checks whether the JSON config itself implies line-count eligibility. Adding a weight to the JSON for a null-language extension has no effect unless NON_CODE_EXTENSIONS is also manually updated.
Affected extensions (configured in JSON, absent from NON_CODE_EXTENSIONS):
env, gitattributes, gitignore, gql, graphql, ipynb, move, nim, pde, puml, tex.
Steps to Reproduce
- Create a FileChange for
schema.graphql (status='added', changes=1).
- Pass it to calculate_token_score_from_file_changes() with
FileContentPair(old_content=None, new_content='type Query { hello: String }\n').
- Inspect
file_results[0].
Expected Behavior
scoring_method='line-count', score=1.0 — because graphql is configured in programming_languages.json with weight=1.0 and no language.
Actual Behavior
scoring_method='skipped-unsupported', score=0.0.
Same result for .gitignore (configured weight=0.5) and all other null-language extensions listed above.
Environment
- OS: Linux
- Python version: 3.12
- Commit/Version:
47d2651 (upstream/test)
Additional Context
The routing logic in gittensor/validator/utils/tree_sitter_scoring.py (around line 277):
elif ext in NON_CODE_EXTENSIONS: # line-count path
...
elif not weights.supports_tree_sitter(ext): # skipped-unsupported
...supports_tree_sitter() correctly returns False for null-language configured extensions, but that False only drops them into skipped-unsupported rather than line-count. The fix is to also check whether the extension has a configured language=None entry before concluding it is unsupported.
Related: #986 fixed extensionless configured files that have a tree-sitter language (Dockerfile, Makefile). This issue is the null-language complement.
Description
Extensions configured in programming_languages.json with no
languagefield receivescore=0.0withscoring_method='skipped-unsupported'. The scorer only line-counts files whose extension is in theNON_CODE_EXTENSIONSconstant — it never checks whether the JSON config itself implies line-count eligibility. Adding a weight to the JSON for a null-language extension has no effect unlessNON_CODE_EXTENSIONSis also manually updated.Affected extensions (configured in JSON, absent from
NON_CODE_EXTENSIONS):env,gitattributes,gitignore,gql,graphql,ipynb,move,nim,pde,puml,tex.Steps to Reproduce
schema.graphql(status='added',changes=1).FileContentPair(old_content=None, new_content='type Query { hello: String }\n').file_results[0].Expected Behavior
scoring_method='line-count',score=1.0— becausegraphqlis configured in programming_languages.json withweight=1.0and nolanguage.Actual Behavior
scoring_method='skipped-unsupported',score=0.0.Same result for
.gitignore(configuredweight=0.5) and all other null-language extensions listed above.Environment
47d2651(upstream/test)Additional Context
The routing logic in gittensor/validator/utils/tree_sitter_scoring.py (around line 277):
supports_tree_sitter()correctly returnsFalsefor null-language configured extensions, but thatFalseonly drops them intoskipped-unsupportedrather than line-count. The fix is to also check whether the extension has a configuredlanguage=Noneentry before concluding it is unsupported.Related: #986 fixed extensionless configured files that have a tree-sitter language (Dockerfile, Makefile). This issue is the null-language complement.