Add sessions to deployment

[safoinme]

Add Session Support to Pipeline Deployments

Overview

Adds stateful session management to ZenML pipeline deployments, enabling multi-turn interactions for chatbots, agents, and stateful applications.

Key Features

• Multiple Backends: In-memory (single-worker) and SQLite-based (multi-worker)
• Automatic Lifecycle: Sessions auto-created, updated, and expired based on TTL
• Thread & Process Safe: Concurrent access support
• Configurable Limits: Size limits and validation

Architecture

• Session Backends: Abstract SessionBackend interface with InMemorySessionBackend and LocalSessionBackend implementations
• Session Manager: High-level orchestration layer handling resolution, persistence, and validation
• Runtime Integration: Session state exposed via get_step_context().session_state in pipeline steps
• Service Integration: PipelineDeploymentService automatically manages session lifecycle during invocations

Usage Example

from zenml.steps import get_step_context

@step
def weather_analyzer(city: str) -> str:
    """Analyze weather with session history tracking."""
    session_state = get_step_context().session_state
    
    # Track query history
    history = session_state.setdefault("history", [])
    history.append(city)
    
    return f"Weather for {city} (Query #{len(history)})"
    
@step
def compare_2_cities(analysis: str) -> str:
    """Compare current city to previous city from session history."""
    session_state = get_step_context().session_state
    history = session_state.get("history", [])
    
    if len(history) < 2:
        return "Not enough history to compare cities yet."
    
    current = history[-1]
    previous = history[-2]
    delta_temp = current["temperature"] - previous["temperature"]
    
    return (
        f"Comparing {current['city']} to {previous['city']}:\n"
        f"• Temperature change: {delta_temp:+.1f}°C"
    )

Pre-requisites

Please ensure you have done the following:

• I have read the CONTRIBUTING.md document.
• I have added tests to cover my changes.
• I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.
• IMPORTANT: I made sure that my changes are reflected properly in the following resources:
  • ZenML Docs
  • Dashboard: Needs to be communicated to the frontend team.
  • Templates: Might need adjustments (that are not reflected in the template tests) in case of non-breaking changes and deprecations.
  • Projects: Depending on the version dependencies, different projects might get affected.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Other (add details above)

[github-actions]

⚠️ This PR has been inactive for 2 weeks and has been marked as stale.
Timeline:

• Week 2 (now): First reminder - PR marked as stale
• Week 4: PR will be automatically closed if no activity
  Please update this PR or leave a comment to keep it active. Any activity will reset the timer and remove the stale label.