Code-prose-composition tagger. #247
Open
+212
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
no0p
force-pushed
the
code-prose-composition-b
branch
2 times, most recently
from
bed5fe8 to
9eae084
Compare
Add a tagger that adds attributes for code-prose-other composition of files based on line classifications. Coverage for code-prose-composition tagger. Improve error messages for spawn method checks. Set multiprocessing start method in test setup Set multiprocessing in test case with error handling. Add before suite hook to set mp start method. The default multiprocessing start method is "fork" which is not compatible with with runtime assertions that it is set to spawn. When running unit tests, it's possible to call an external library that sets the start method to "fork". Here we enforce the start method to be "spawn" for all tests before executing. linting. Remove error log messages.
no0p
force-pushed
the
code-prose-composition-b
branch
from
9eae084 to
db8b744
Compare
Additionally update commentary and word choice.
|
Generated attribute line example: I think you might be able to keep this shorter by cutting the key in line_label, maybe elsewhere |
Whattabatt
approved these changes
Feb 28, 2025
Whattabatt
reviewed
Mar 1, 2025
| from ..core.registry import TaggerRegistry | ||
|
|
||
|
|
||
| @TaggerRegistry.add("code-prose-composition") |
There was a problem hiding this comment.
Other taggers use underscores instead of dashes
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Tagger for Code Prose Composition
Add a tagger that adds attributes for code-prose-other composition of files based on line classifications.
Produces tags for
Recommended filter for mixed prose/code content based on these tags is:
The code entropy adjusts for bias towards code for short string including "code-y" characters like (, ), [, ], : etc due to a lack of nice negative examples. Until time for an improved classifier is available, including a filter for high confidence code predictions via mean entropy works sufficiently well.
One More Thing
Updated a pre-suite hook to set the multiprocessing start method to
spawnto prevent a side effect where test case dependencies may set it to the defaultfork, violating runtime assertions.