-
Notifications
You must be signed in to change notification settings - Fork 7
feat: Redesign architecture and embrace Lightning native patterns #173
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Changes from 37 commits
Commits
Show all changes
47 commits
Select commit
Hold shift + click to select a range
7719988
Update github workflows
ibro45 9c0db7f
Rename LighterConfig to ConfigSchema
ibro45 1682354
Improve github workflows. Add mkdocs extensions
ibro45 317021d
Improve type annotations throughout the codebase
ibro45 41dc125
Refactor System mode detection to use trainer state
ibro45 7725982
Simplify Runner CLI to leverage Sparkwheel auto-detection
ibro45 7e99a8b
Update dependencies and GitHub workflow configuration
ibro45 58c55c3
Update System design documentation
ibro45 8e29a75
Update tests to match refactored CLI and System changes
ibro45 0d7bf12
Refactor core architecture: replace System with Model and Data modules
ibro45 6425f46
Refactor callback system: consolidate writers into flat structure
ibro45 a9f7330
Simplify Runner and remove schema module
ibro45 f1825e3
Update CIFAR10 example for new architecture
ibro45 be149e9
Update test suite for new architecture
ibro45 c8704cc
Rewrite documentation for new architecture
ibro45 17d5691
Update project configuration and CI/CD
ibro45 f1a1520
docs: enhance configuration guide and logging documentation
ibro45 37da1d0
refactor: require MetricCollection wrapper for multiple metrics
ibro45 926ff14
test: add comprehensive DDP testing with mock-based approach
ibro45 d854e52
Add codecov yml
ibro45 e54c071
chore: improve project configuration
ibro45 6bc6351
refactor(runner): streamline config management and CLI architecture
ibro45 173eb84
feat(data): add max_retries parameter to collate_replace_corrupted
ibro45 2bdbcb4
refactor: remove unused utilities and update comments
ibro45 3f8a002
docs: improve examples and clarify multi-GPU behavior
ibro45 5c71150
test: update test suite for runner refactoring
ibro45 072b449
Refactor config saving in Runner
ibro45 4c8e955
Fix tests after removing _save_hyperparameters
ibro45 78733a4
Remove unneeded files
ibro45 1443fe5
docs: improve configuration examples and clarify _disabled_ behavior
ibro45 987c320
chore: update .gitignore patterns
ibro45 1309ae2
fix: register module in sys.modules before execution
ibro45 4d1c5ff
feat(dynamic_imports): add multiprocessing spawn support with hybrid …
ibro45 7be99fb
refactor(tests): improve test descriptions and remove unnecessary com…
ibro45 0dd9d38
Update sparkwheel
ibro45 2897a57
feat(callbacks): enhance CsvWriter and FileWriter with usage examples…
ibro45 f228ef0
fix(freezer): update model parameter access to support LighterModule
ibro45 f7f46a6
feat(model): add runtime checks for criterion in training and validat…
ibro45 1a3c567
fix(freezer): enhance logging for unfrozen layers with exception suffix
ibro45 bc8ac5b
fix(tests): update logging assertions to capture stderr output from L…
ibro45 a8fd4aa
fix(tests): add ANSI stripping function for CI compatibility in freez…
ibro45 68a1ca5
fix(dynamic_imports): improve error handling during dynamic module im…
ibro45 98cdd5e
fix(csv_writer): implement file closure and state reset methods to pr…
ibro45 189179c
fix(csv_writer): add validation for configured keys in outputs to pre…
ibro45 bb9a1a2
fix(csv_writer): adjust setup method call order for correct stage han…
ibro45 2df7442
fix(runner): update stage execution comments for clarity and enhance …
ibro45 a9c95fc
chore(paper): remove paper folder
ibro45 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
| # Codecov Configuration | ||
| # Documentation: https://docs.codecov.com/docs/codecovyml-reference | ||
|
|
||
| coverage: | ||
| precision: 2 # Number of decimal places (0-5) | ||
| round: down # How to round coverage (down/up/nearest) | ||
| range: 70..100 # Color coding range (red at 70%, green at 100%) | ||
|
|
||
| status: | ||
| # Project coverage: overall repository coverage | ||
| project: | ||
| default: | ||
| target: 90% # Minimum coverage threshold | ||
| threshold: null # No additional threshold; fail check if below target | ||
| base: auto # Compare against base branch | ||
| informational: false # Fail the check if below target | ||
|
|
||
| # Patch coverage: coverage on changed lines only | ||
| patch: | ||
| default: | ||
| target: 80% # New code should have at least 80% coverage | ||
| threshold: 0% # No wiggle room for patch coverage | ||
| base: auto | ||
| informational: false # Fail if new code doesn't meet target | ||
|
|
||
| # Pull request comment configuration | ||
| comment: | ||
| layout: "diff, flags, files, footer" # What to show in PR comments | ||
| behavior: default # Comment on all PRs | ||
| require_changes: false # Comment even if coverage unchanged | ||
| require_base: false # Comment even without base report | ||
| require_head: true # Only comment if head report exists | ||
|
|
||
| # Paths to ignore in coverage reports | ||
| ignore: | ||
| - "tests/*" # Test files | ||
| - "tests/**/*" # All test subdirectories | ||
| - "docs/**/*" # Documentation | ||
| - "site/*" # Built documentation site | ||
| - "htmlcov/*" # Coverage HTML reports | ||
| - ".venv/*" # Virtual environment | ||
| - ".tox/*" # Tox environments | ||
| - "**/__pycache__/*" # Python cache | ||
| - "**/conftest.py" # Pytest configuration | ||
| - "update_tests.py" # Utility scripts | ||
|
|
||
| # GitHub Checks configuration | ||
| github_checks: | ||
| annotations: true # Show coverage annotations on changed files |
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| name: 'Setup Environment' | ||
| description: 'Setup Python with uv and install dependencies' | ||
| inputs: | ||
| python-version: | ||
| description: 'Python version' | ||
| required: false | ||
| default: '3.12' | ||
| install-deps: | ||
| description: 'Install dependencies' | ||
| required: false | ||
| default: 'true' | ||
| install-groups: | ||
| description: 'Dependency groups to install' | ||
| required: false | ||
| default: '--all-extras --all-groups' | ||
| runs: | ||
| using: 'composite' | ||
| steps: | ||
| - name: Install uv | ||
| uses: astral-sh/setup-uv@v6 | ||
| with: | ||
| python-version: ${{ inputs.python-version }} | ||
| enable-cache: true | ||
|
|
||
| - name: Install dependencies | ||
| if: inputs.install-deps == 'true' | ||
| run: uv sync ${{ inputs.install-groups }} | ||
| shell: bash |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| #!/usr/bin/env python3 | ||
| """Generate GitHub Actions test summary from JUnit XML.""" | ||
|
|
||
| import sys | ||
| import xml.etree.ElementTree as ET | ||
| from pathlib import Path | ||
|
|
||
|
|
||
| def main() -> int: | ||
| """Parse JUnit XML and write summary to GitHub Actions step summary.""" | ||
| xml_path = Path("test-results.xml") | ||
|
|
||
| if not xml_path.exists(): | ||
| print("⚠️ No test results found", file=sys.stderr) | ||
| return 1 | ||
|
|
||
| try: | ||
| tree = ET.parse(xml_path) | ||
| root = tree.getroot() | ||
|
|
||
| tests = int(root.get("tests", 0)) | ||
| failures = int(root.get("failures", 0)) | ||
| errors = int(root.get("errors", 0)) | ||
| skipped = int(root.get("skipped", 0)) | ||
| passed = tests - failures - errors - skipped | ||
|
|
||
| # Determine status emoji | ||
| if failures + errors > 0: | ||
| status = "❌" | ||
| elif skipped == tests: | ||
| status = "⏭️" | ||
| else: | ||
| status = "✅" | ||
|
|
||
| # Print summary lines | ||
| print(f"{status} **Test Results Summary**") | ||
| print(f"- ✅ Passed: {passed}") | ||
| print(f"- ❌ Failed: {failures}") | ||
| print(f"- ⚠️ Errors: {errors}") | ||
| print(f"- ⏭️ Skipped: {skipped}") | ||
| print(f"- **Total: {tests}**") | ||
|
|
||
| # Exit with error if tests failed | ||
| return 1 if (failures + errors > 0) else 0 | ||
|
|
||
| except ET.ParseError as e: | ||
| print(f"❌ Failed to parse XML: {e}", file=sys.stderr) | ||
| return 1 | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| sys.exit(main()) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,35 +1,47 @@ | ||
| name: "Check PR title" | ||
|
|
||
| on: | ||
| pull_request: | ||
| types: [edited, opened, synchronize, reopened] | ||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.event.pull_request.number }} | ||
| cancel-in-progress: true | ||
|
|
||
| permissions: | ||
| contents: read | ||
| pull-requests: read | ||
| statuses: write | ||
|
|
||
| jobs: | ||
| pr-title-check: | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 5 | ||
| if: ${{ github.event.pull_request.user.login != 'allcontributors[bot]' }} | ||
| steps: | ||
| # Echo the user's login | ||
| - name: Echo user login | ||
| run: echo ${{ github.event.pull_request.user.login }} | ||
|
|
||
| - uses: naveenk1223/action-pr-title@master | ||
| - uses: amannn/[email protected] | ||
| env: | ||
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| with: | ||
| # Require imperative mood (e.g. "Add feature" not "Adds feature") | ||
| # ^ Start of string | ||
| # [A-Z] First character must be an uppercase ASCII letter | ||
| # [a-zA-Z]* Followed by zero or more ASCII letters | ||
| # (?<![^s]s) Negative lookbehind: disallow a single 's' at the end of the first word | ||
| # ( .+)+ At least one space and one or more characters (requires more words) | ||
| # [a-z]* Followed by zero or more lowercase letters | ||
| # (?<![^s]s) Negative lookbehind: disallow single 's' at end of first word | ||
| # ( .+)+ At least one space and one or more characters | ||
| # [^.] Final character must not be a period | ||
| # $ End of string | ||
| regex: "^[A-Z][a-zA-Z]*(?<![^s]s)( .+)+[^.]$" | ||
| # Valid titles: | ||
| # - "Do something" | ||
| # - "Address something" | ||
| # Invalid title: | ||
| # - "do something" | ||
| # - "Do something." | ||
| # - "Does something" | ||
| # - "Do" | ||
| # - "Addresses something" | ||
| min_length: 10 | ||
| max_length: 72 | ||
| subjectPattern: '^[A-Z][a-z]*(?<![^s]s)( .+)+[^.]$' | ||
| subjectPatternError: | | ||
| The PR title must use imperative mood and proper formatting: | ||
| - Start with uppercase letter (e.g., "Add" not "add") | ||
| - Use base form of verb (e.g., "Add" not "Adds" or "Adding") | ||
| - Have multiple words (minimum 10 characters) | ||
| - Not end with a period | ||
| Valid: "Add new feature", "Fix bug in parser" | ||
| Invalid: "add feature", "Adds feature", "Add.", "Fix" | ||
| # Disable type prefixes (we don't use conventional commits format) | ||
| requireScope: false | ||
| ignoreLabels: | | ||
| ignore-title-check | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| name: CI Full | ||
|
|
||
| on: | ||
| push: | ||
| branches: | ||
| - main | ||
| workflow_dispatch: | ||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.ref }} | ||
| cancel-in-progress: false | ||
|
|
||
| permissions: | ||
| contents: read | ||
| checks: write | ||
|
|
||
| env: | ||
| FORCE_COLOR: 1 | ||
|
|
||
| jobs: | ||
| tests: | ||
| name: Python ${{ matrix.python }} | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 30 | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| python: ["3.10", "3.11", "3.12"] | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| with: | ||
| fetch-depth: 0 | ||
|
|
||
| - uses: ./.github/actions/setup | ||
| with: | ||
| python-version: ${{ matrix.python }} | ||
|
|
||
| - name: Run all tests with coverage | ||
| run: | | ||
| uv run coverage run -m pytest tests --durations=10 \ | ||
| --junit-xml=test-results.xml \ | ||
| --html=test-report.html --self-contained-html | ||
|
|
||
| - name: Generate test summary | ||
| if: always() | ||
| run: | | ||
| echo "## Test Results - Python ${{ matrix.python }}" >> $GITHUB_STEP_SUMMARY | ||
| python3 .github/scripts/test_summary.py >> $GITHUB_STEP_SUMMARY | ||
|
|
||
| - name: Upload test results | ||
| if: always() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: test-results-py${{ matrix.python }} | ||
| retention-days: 7 | ||
| path: | | ||
| test-results.xml | ||
| test-report.html | ||
| .coverage | ||
|
|
||
| - name: Generate coverage report | ||
| if: matrix.python == '3.12' | ||
| run: uv run coverage xml | ||
|
|
||
| - name: Upload coverage to Codecov | ||
| if: matrix.python == '3.12' | ||
| uses: codecov/[email protected] | ||
| with: | ||
| files: ./coverage.xml | ||
| fail_ci_if_error: true | ||
| env: | ||
| CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }} |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.