(spike) investigate package build and release of providers

[gallettilance]

Package Publishing Infrastructure - PR Summary

🎯 What Changed

This PR transforms the repository from a single monorepo package to individual PyPI packages for each provider, enabling independent publishing and versioning.

Before:

lightspeed_stack_providers/          # Single package
├── providers/
│   ├── inline/agents/...
│   ├── inline/safety/...
│   └── remote/agents/...

After:

providers/                          # Individual packages
├── agents/
│   ├── lightspeed-inline-agent/   # → lightspeed-inline-agent PyPI package
│   └── lightspeed-agent/          # → lightspeed-agent PyPI package
├── safety/
│   ├── lightspeed-question-validity/  # → lightspeed-question-validity PyPI package
│   └── lightspeed-redaction/      # → lightspeed-redaction PyPI package
└── tool-runtime/
    └── lightspeed-tool-runtime/   # → lightspeed-tool-runtime PyPI package

🚀 Why This Matters

For Users:

• Independent Installation: Install only the providers you need

  pip install lightspeed-inline-agent
  pip install lightspeed-question-validity

• Version Flexibility: Each provider can be updated independently
• Reduced Dependencies: Smaller, focused packages

For Contributors:

• Clear Organization: Each provider has its own directory and package
• Template Generation: make generate-template creates new providers quickly
• Automated Publishing: GitHub Actions handles build, test, and publish
• Lockstep Versioning: All packages share the same version for coordinated releases

For Maintainers:

• Automated CI/CD: Push a tag, get all packages published
• Individual Testing: Test specific packages without building everything
• Version Management: Single source of truth in pyproject.toml

🔧 Key Infrastructure

Build System:

• scripts/build_reorganized.py: Main build script for individual packages
• Makefile: Build, test, and publish targets
• GitHub Actions: Automated publishing on version tags

Package Structure:

provider-name/
├── src/package_name/              # Source code
├── config/provider-name.yaml      # Llama-stack configuration
├── pyproject.toml                # Package metadata
└── README.md                     # Documentation

Version Management:

• Lockstep Versioning: All packages inherit version from main pyproject.toml
• Single Source of Truth: Update version once, all packages updated
• Automated Inheritance: Build script automatically syncs versions

📦 Available Packages

Package	Type	Description
lightspeed-inline-agent	Inline Agent	Agent orchestration with tool filtering
lightspeed-agent	Remote Agent	Remote agent service integration
lightspeed-question-validity	Inline Safety	Content validation and filtering
lightspeed-redaction	Inline Safety	Content redaction and safety
lightspeed-tool-runtime	Remote Tool Runtime	MCP-based tool execution

🎯 Benefits

1. Modularity: Install only what you need
2. Maintainability: Each provider is self-contained
3. Scalability: Easy to add new providers
4. Automation: Fully automated build and publish pipeline
5. Version Control: Coordinated releases with lockstep versioning

🔄 Migration Impact

• No Breaking Changes: Existing functionality preserved
• Backward Compatible: All providers work the same way
• Improved Developer Experience: Better organization and tooling
• Future-Proof: Ready for independent provider evolution

This change enables the repository to scale better, provide more flexibility to users, and maintain cleaner separation of concerns while preserving all existing functionality.