DSLIGHTING: Data Science Framework

🎯 Intelligent Agent Workflows • 📊 Interactive Data Visualization
🤖 Automated Code Generation • 📈 End-to-End Task Evaluation

⭐ Star us • 💬 Discussions

2026-08 🎉

Our EMNLP 2026 Findings paper, DSFlow: Evolutionary Workflow Optimization for Generalizable LLM-Based Data Science Automation, introduces evolutionary workflow optimization for building data-science agents that generalize across tasks and LLM backbones. Download PDF

2026-06 🎉

Our paper, DS-Lighting: Making Agent Harnesses Explicit for Data-Science Automation, has been accepted to the KDD 2026 Workshop on AI Data Scientist (AIDataSci). PDF

News 2026.03 · Jump to Benchmarks

DSLighting now officially supports benchmark evaluation for DACode (EMNLP 2024), DABench (ICML 2024), MoSciBench (ICLR 2026), MLE-Bench, and ScienceAgentBench (ICLR 2025).

Run benchmark evaluations with just a few lines of code through DSBenchmark.

New in v2.7.9 (Current)

Feature	Description
Benchmark Mode	Officially supports DABench, DACode, MLEBench, MoSciBench, and ScienceAgentBench / ScienceBench benchmark families for agent evaluation
DAG Mode	Enhanced Directed Acyclic Graph (DAG) runtime for workflow orchestration
~~Web UI~~	Development is currently paused, and the Web UI is no longer supported in this repository

What DSLighting Is

DSLighting is an LLM-driven data science execution framework. DSLighting is an LLM-driven autonomous data science execution engine that turns task descriptions and datasets into iterative code generation, execution, evaluation, and refinement workflows. It supports:

task-oriented agent execution (run_agent, Agent)
benchmark evaluation (DSBenchmark)
architecture-level customization (services/operators/workflows)

The current public API is split into two layers:

dslighting.api: simplified, stable user API
dslighting.arch.*: advanced architecture API for custom agent development

Two Usage Modes

Simplified API (recommended for quick start)
For rapid prototyping and standard data science tasks.
Architecture (recommended for deep customization)
For advanced users building custom operators, workflows, and factories.

Installation

git clone https://github.com/usail-hkust/dslighting.git
cd dslighting

python3.10 -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate

pip install -r requirements.txt  # Core runtime dependencies
pip install -e .
# Optional: full development/research dependency set
# pip install -r requirements_local.txt

If you hit ModuleNotFoundError: aiofiles, run:

pip install aiofiles

Environment Setup

Minimal .env:

API_KEY=your_key
API_BASE=https://api.openai.com/v1
LLM_MODEL=gpt-4o

Optional model-specific overrides (LLM_MODEL_CONFIGS):

{
  "openai/deepseek-ai/DeepSeek-V3.1-Terminus": {
    "api_key": ["key1", "key2"],
    "api_base": "https://api.siliconflow.cn/v1",
    "temperature": 1.0
  }
}

`config.yaml` (Optional)

config.yaml is optional for normal run_agent / Agent usage.

You can run most tasks without it.
It is mainly used by benchmark/runtime configuration and custom model pricing metadata.
Task registries still use per-task data_dir/<task_id>/config.yaml (separate from root config).

Minimal example (config.yaml):

run:
  enable_trajectory_logging: false
  trajectory_filename: trajectory.jsonl

llm_pricing:
  custom_models: {}

Sandbox Backends (Simplified API)

run_agent and Agent support:

local
e2b
ds_sandbox

Use explicit dotenv loading in your script:

from dotenv import load_dotenv
load_dotenv()

`.env` example

SANDBOX_BACKEND=local
SANDBOX_BACKEND_TYPE=docker
SANDBOX_TIMEOUT=21600
E2B_API_KEY=
SANDBOX_WORKSPACE_BASE=/tmp/ds_sandbox_workspaces
SANDBOX_PAUSED_BASE=/tmp/ds_sandbox_paused

Local sandbox

result = run_agent(
    task_id="bike-sharing-demand",
    sandbox_backend="local",
)

E2B sandbox

result = run_agent(
    task_id="bike-sharing-demand",
    sandbox_backend="e2b",
    sandbox_api_key=None,  # read from E2B_API_KEY by default
)

Notes:

install SDK: pip install e2b
set E2B_API_KEY in .env (or pass sandbox_api_key)

DS-Sandbox

result = run_agent(
    task_id="bike-sharing-demand",
    sandbox_backend="ds_sandbox",
    sandbox_backend_type="local",  # or "docker"
)

Notes:

install package: pip install ds-sandbox
defaults use writable paths under /tmp to avoid /opt permission issues

Quick Start (Simplified API)

One-liner execution

from dotenv import load_dotenv
load_dotenv()

from dslighting.api import run_agent

result = run_agent(
    task_id="bike-sharing-demand",
    workflow="aide",   # optional
    model="gpt-4o",    # optional
)

print(result.success, result.score, result.cost)
print(result.duration, result.output, result.error)

Use `Agent` directly

from dotenv import load_dotenv
load_dotenv()

from dslighting.api import Agent

agent = Agent(
    workflow="aide",
    model="gpt-4o",
    max_iterations=5,
)

result = agent.run(task_id="bike-sharing-demand")
print(result)

Architecture Mode (Custom Agent Development)

Use this when you need custom operators/workflows/factories.

What each part does:

Operator: one async capability unit (for example, summarize, plan, execute).
Workflow.solve(...): core async logic of your agent.
WorkflowFactory.create_agent(...): wires services + operators into a workflow instance.
workflow.run(...): sync wrapper around solve(...) for non-async users.

from pathlib import Path
from typing import Any

from dslighting.arch.interfaces import WorkflowFactoryInterface
from dslighting.arch.operators import Operator
from dslighting.arch.services import LLMService, SandboxService, WorkspaceService
from dslighting.arch.state import JournalState
from dslighting.arch.workflows import BaseWorkflow, BaseWorkflowFactory
from dslighting.config import LLMConfig


class SummarizeOperator(Operator):
    async def __call__(self, text: str) -> dict[str, Any]:
        return {"summary": text[:200]}


class MyWorkflow(BaseWorkflow):
    def __init__(self, operators, services, agent_config=None):
        super().__init__(
            operators=operators,
            services=services,
            agent_config=agent_config or {},
        )

    async def solve(
        self,
        description: str,
        io_instructions: str,
        data_dir: Path,
        output_path: Path,
    ) -> dict[str, Any]:
        return await self.operators["summarize"](text=description)


class MyWorkflowFactory(BaseWorkflowFactory, WorkflowFactoryInterface):
    def create_agent(self, **kwargs):
        workspace = WorkspaceService(run_name="custom_arch_run")
        services = {
            "llm": LLMService(config=LLMConfig(model=self.model)),
            "sandbox": SandboxService(workspace=workspace),
            "workspace": workspace,
            "state": JournalState(),
        }
        operators = {
            "summarize": SummarizeOperator(),
        }
        return MyWorkflow(operators=operators, services=services, agent_config=kwargs)


# Recommended for normal scripts: sync call via workflow.run(...)
def main():
    workflow = MyWorkflowFactory(model="gpt-4o").create_agent(max_iterations=3)
    result = workflow.run(data="data/competitions/bike-sharing-demand")
    print(result)

if __name__ == "__main__":
    main()

For most users (no custom workflow), use Agent.run(...):

from dslighting.api import Agent

agent = Agent(workflow="aide", model="gpt-4o")
result = agent.run(task_id="bike-sharing-demand")
print(result)

Usage rule:

Normal scripts: use workflow.run(...) or agent.run(...).
Already inside async def: use await workflow.solve(...) (do not call workflow.run(...) there).

RAG Usage (DSAgent / AutoMind)

RAG is enabled through workflow namespace params and backed by VDBService.

File Layout

case_dir should contain Python case files (*.py), for example:

experience_replay/
  case_001.py
  case_002.py

Simplified API Examples

from dslighting.api import run_agent

result = run_agent(
    task_id="bike-sharing-demand",
    workflow="dsagent",
    dsagent={"enable_rag": True, "case_dir": "./experience_replay"},
)

from dslighting.api import run_agent

result = run_agent(
    task_id="bike-sharing-demand",
    workflow="automind",
    automind={"enable_rag": True, "case_dir": "./experience_replay"},
)

Notes

RAG params must be namespaced (dsagent={...} / automind={...}).
enable_rag is False by default.
Flat keys like enable_rag=... or case_dir=... are rejected.
If case_dir does not exist or has no *.py files, retrieval returns empty results.

Result Object (Single Source of Truth)

Agent.run() and run_agent() return AgentResult:

@dataclass
class AgentResult:
    success: bool
    output: Any
    cost: float = 0.0
    duration: float = 0.0
    score: float | None = None
    artifacts_path: Path | None = None
    workspace_path: Path | None = None
    error: str | None = None
    metadata: dict[str, Any] = field(default_factory=dict)

Benchmarks

Use DSBenchmark for multi-task benchmark runs.

from dotenv import load_dotenv
load_dotenv()

from dslighting.api import DSBenchmark
from dslighting.core import ConfigBuilder

config = ConfigBuilder().build_config(
    workflow="aide",
    model="gpt-4o",
)

benchmark = DSBenchmark("dabench", data_dir="/path/to/dabench")
result = benchmark.run(config=config)

print(result.results_path)
print(result.metadata_path)

DSBenchmark.run(config=...) expects a fully resolved DSLightingConfig. If you rely on .env values or LLM_MODEL_CONFIGS, build the config with ConfigBuilder first. Passing a bare DSLightingConfig() no longer triggers benchmark-side LLM env fallback.

Supported benchmark families include DABench, DACode, MLEBench variants, MoSciBench, and ScienceBench.

DSFlow baseline

The standalone DSFlow implementation is integrated as a two-stage meta-optimization workflow. It screens candidate workflows with a coarse plan/code score, fine-evaluates the top-k candidates on the benchmark, and runs the selected workflow in test mode.

from dslighting.api import DSBenchmark
from dslighting.core import ConfigBuilder

config = ConfigBuilder().build_config(
    workflow="dsflow",
    model="gpt-4o",
    dsflow={
        "max_rounds": 4,
        "top_k_selection": 2,
        "task_sample_size": 3,
    },
)

benchmark = DSBenchmark("mlebench", data_dir="/path/to/mlebench-data")
result = benchmark.run(config=config)

OpenAI-compatible endpoints that require additional HTTP headers can be configured per run without mutating global OpenAI client state:

config = ConfigBuilder().build_config(
    workflow="dsflow",
    model="your-model",
    api_key="your-api-key",
    api_base="https://your-endpoint.example/v1/",
    provider="openai",
    default_headers={"x-foo": "true"},
)

The selected workflow is saved as best_workflow.py in the run workspace.

To skip meta-optimization and evaluate a previously saved workflow, set dsflow={"best_workflow_path": "/path/to/best_workflow.py"}. Standalone DSFlow workflows using the former dsat import path are migrated at load time.

DABench

Download the prepared dataset release from Google Drive.
Upstream benchmark and citation link: InfiAgent / DABench (ICML 2024).

Point data_dir to the extracted DABench root directory that contains the dabench-* task folders, or set:

export DSLIGHTING_DABENCH_DATA=/path/to/dabench

Then run it with DSBenchmark:

from dotenv import load_dotenv
load_dotenv()

from dslighting.api import DSBenchmark
from dslighting.core import ConfigBuilder

config = ConfigBuilder().build_config(
    workflow="aide",
    model="gpt-4o",
)

benchmark = DSBenchmark("dabench", data_dir="/path/to/dabench")
result = benchmark.run(config=config)

print(result.results_path)
print(result.metadata_path)

Details, including a reusable launcher script, are available in examples/benchmark/dabench/README.md.

DACode

Download the prepared dataset release from Google Drive.
Upstream benchmark and citation link: DA-Code (EMNLP 2024).

Point data_dir to the prepared DACode root directory that contains the dacode-* task folders, or set:

export DSLIGHTING_DACODE_DATA=/path/to/dacode

Then run it with DSBenchmark:

from dotenv import load_dotenv
load_dotenv()

from dslighting.api import DSBenchmark
from dslighting.core import ConfigBuilder

config = ConfigBuilder().build_config(
    workflow="aide",
    model="gpt-4o",
)

benchmark = DSBenchmark("dacode", data_dir="/path/to/dacode")
result = benchmark.run(config=config)

print(result.results_path)
print(result.metadata_path)

Details, including a reusable launcher script, are available in examples/benchmark/dacode/README.md.

MLE-Bench

Upstream benchmark and dataset instructions: openai/mle-bench.

If you want to run MLE-Bench with DSLighting, follow the upstream repository instructions to download and prepare the dataset first. Then point data_dir to your local MLE-Bench data root, or set:

export DSLIGHTING_MLEBENCH_DATA=/path/to/mlebench

Then run it with DSBenchmark:

from dotenv import load_dotenv
load_dotenv()

from dslighting.api import DSBenchmark
from dslighting.core import ConfigBuilder

config = ConfigBuilder().build_config(
    workflow="aide",
    model="gpt-4o",
)

benchmark = DSBenchmark("mlebench", data_dir="/path/to/mlebench")
result = benchmark.run(config=config)

print(result.results_path)
print(result.metadata_path)

MoSciBench

Download the prepared dataset release from Google Drive.
Upstream benchmark and citation link: MoSciBench (ICLR 2026).

The public release is deduplicated for distribution size. Some public inputs are shared across tasks in the same family, so they are stored once under metadata/ instead of being duplicated in every task folder.

For DSLighting benchmark runs, use the competitions/ directory as data_dir. If you want each task folder to be fully self-contained for batch runs, first copy the shared family data from metadata/ back into each competitions/*/prepared/public/.

Quick expansion script:

DATA_ROOT=/path/to/moscibench

for family_dir in "$DATA_ROOT"/metadata/mosci-*; do
  family=$(basename "$family_dir")
  src="$family_dir/prepared/public/"
  [ -d "$src" ] || continue

  for public_dir in "$DATA_ROOT"/competitions/"${family}"-*/prepared/public; do
    [ -d "$public_dir" ] || continue
    rsync -a "$src" "$public_dir/"
  done
done

After expansion, point data_dir to competitions/, or set:

export DSLIGHTING_MOSCIBENCH_DATA=/path/to/moscibench/competitions

Then run it with DSBenchmark:

from dotenv import load_dotenv
load_dotenv()

from dslighting.api import DSBenchmark
from dslighting.core import ConfigBuilder

config = ConfigBuilder().build_config(
    workflow="aide",
    model="gpt-4o",
)

benchmark = DSBenchmark("moscibench", data_dir="/path/to/moscibench/competitions")
result = benchmark.run(config=config)

print(result.results_path)
print(result.metadata_path)

Details, including a reusable launcher script, are available in examples/benchmark/moscibench/README.md.

ScienceAgentBench / ScienceBench

Download the prepared dataset release from Google Drive.
Upstream benchmark and citation link: ScienceAgentBench (ICLR 2025).

ScienceAgentBench task directories are already self-contained. Point data_dir to the extracted root directory that contains sciencebench-* task folders, or set:

export DSLIGHTING_SCIENCEBENCH_DATA=/path/to/scienceagentbench

Then run it with DSBenchmark:

from dotenv import load_dotenv
load_dotenv()

from dslighting.api import DSBenchmark
from dslighting.core import ConfigBuilder

config = ConfigBuilder().build_config(
    workflow="aide",
    model="gpt-4o",
)

benchmark = DSBenchmark("sciencebench", data_dir="/path/to/scienceagentbench")
result = benchmark.run(config=config)

print(result.results_path)
print(result.metadata_path)

Details, including a reusable launcher script and extra dependency notes, are available in examples/benchmark/sciencebench/README.md.

If DSBenchmark raises an import error (for example missing pandas), install the missing package first:

pip install pandas

🏗️ Core Architecture

┌─────────────────────────────────────────┐
│ 1) Agent Orchestration Layer            │
│    Workflow lifecycle and scheduling     │
│    dslighting/workflows, runner.py       │
└──────────────┬──────────────────────────┘
               │
┌──────────────▼──────────────────────────┐
│ 2) Cognitive / Operator Layer           │
│    Plan/Generate/Execute/Review          │
│    dslighting/ops, prompts, state        │
└──────────────┬──────────────────────────┘
               │
┌──────────────▼──────────────────────────┐
│ 3) Execution / Service Layer            │
│    LLMService, Sandbox, Workspace, DAG   │
│    dslighting/services, runtime          │
└──────────────┬──────────────────────────┘
               │
┌──────────────▼──────────────────────────┐
│ 4) Domain Core Layer                    │
│    Config, tasks, interfaces, results    │
│    dslighting/core, benchmark, datasets  │
└──────────────┬──────────────────────────┘
               │
┌──────────────▼──────────────────────────┐
│ 5) Infra / Foundation Layer             │
│    error, monitoring, checkpoint, utils  │
│    dslighting/error, monitoring, utils   │
└─────────────────────────────────────────┘

Migration Notes (Latest Standard)

Use these imports going forward:

simplified layer: from dslighting.api import Agent, run_agent, DSBenchmark
config construction: from dslighting.core import ConfigBuilder
config object types: from dslighting.config import DSLightingConfig, WorkflowConfig
architecture layer: from dslighting.arch...

Avoid introducing new imports from removed/deprecated compatibility paths.

⭐ Star History

License

AGPL-3.0. See LICENSE.

Citation

If you use DSLighting in research, please cite:

@misc{liu2026dslighting,
  title = {DS-Lighting: Making Agent Harnesses Explicit for Data-Science Automation},
  author = {Fan Liu and Hao Liu},
  year = {2026},
  note = {Accepted to the KDD 2026 Workshop on AI for Data Science (AIDataSci)},
  url = {https://openreview.net/forum?id=K7ohsDwj1m}
}

Contributing

Issues and PRs are welcome: https://github.com/usail-hkust/dslighting

Support

GitHub Issues: https://github.com/usail-hkust/dslighting/issues
GitHub Discussions: https://github.com/usail-hkust/dslighting/discussions

Name		Name	Last commit message	Last commit date
assets		assets
data		data
docs		docs
dslighting		dslighting
examples/benchmark		examples/benchmark
experience_replay		experience_replay
experiments/benchmark		experiments/benchmark
tests		tests
.env.example		.env.example
.gitignore		.gitignore
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
MANIFEST.in		MANIFEST.in
README.md		README.md
config.yaml		config.yaml
config.yaml.example		config.yaml.example
dslighting_cli.py		dslighting_cli.py
pyproject.toml		pyproject.toml
pytest.ini		pytest.ini
requirements.txt		requirements.txt
requirements_local.txt		requirements_local.txt
setup.py		setup.py

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

DSLIGHTING: Data Science Framework

2026-08 🎉

2026-06 🎉

New in v2.7.9 (Current)

What DSLighting Is

Two Usage Modes

Installation

Environment Setup

config.yaml (Optional)

Sandbox Backends (Simplified API)

.env example

Local sandbox

E2B sandbox

DS-Sandbox

Quick Start (Simplified API)

One-liner execution

Use Agent directly

Architecture Mode (Custom Agent Development)

RAG Usage (DSAgent / AutoMind)

File Layout

Simplified API Examples

Notes

Result Object (Single Source of Truth)

Benchmarks

DSFlow baseline

DABench

DACode

MLE-Bench

MoSciBench

ScienceAgentBench / ScienceBench

🏗️ Core Architecture

Migration Notes (Latest Standard)

⭐ Star History

License

Citation

Contributing

Support

About

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

`config.yaml` (Optional)

`.env` example

Use `Agent` directly