Skip to content

Commit a8ff821

Browse files
committed
chore: replace ghrc.io (malicious) with ghcr.io (correct)
1 parent 479d595 commit a8ff821

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

41 files changed

+139
-128
lines changed

docs/architecture/03_module_layer.md

Lines changed: 68 additions & 63 deletions
Original file line numberDiff line numberDiff line change
@@ -11,12 +11,14 @@ Modules define both specifications and compute graphs without performing computa
1111
The module layer offers significant advantages over directly using CLI commands:
1212

1313
### Practical Benefits
14+
1415
- **Standardization** - Consistent interfaces across different algorithm types
1516
- **Composability** - Modules can be connected into larger workflows
1617
- **Containerization** - Built-in container specification for reproducibility
1718
- **Documentation** - Structured approach to capturing metadata and citations
1819

1920
### Architectural Significance
21+
2022
- **Backend Independence** - Run the same module on different execution systems
2123
- **Inspection** - Examine inputs, outputs, and operations before execution
2224
- **Automated UI Generation** - Specifications support interface generation
@@ -27,16 +29,17 @@ The module layer offers significant advantages over directly using CLI commands:
2729
The module layer's defining characteristic is its dual focus on specifications and compute graphs:
2830

2931
1. **Specification (Spec)** - Defines what a module does:
30-
- Input ports with types, descriptions, and validation rules
31-
- Output ports with types and descriptions
32-
- Documentation and metadata
33-
- Parameter constraints and defaults
32+
33+
- Input ports with types, descriptions, and validation rules
34+
- Output ports with types and descriptions
35+
- Documentation and metadata
36+
- Parameter constraints and defaults
3437

3538
2. **Compute Graph** - Defines how operations should be structured:
36-
- Container configurations
37-
- Command construction
38-
- Input/output relationships
39-
- Execution sequence
39+
- Container configurations
40+
- Command construction
41+
- Input/output relationships
42+
- Execution sequence
4043

4144
Crucially, modules define computation but don't perform it. This separation enables inspection and modification before execution, and allows the same module to run on different platforms without changing its definition.
4245

@@ -47,35 +50,35 @@ Just as the CLI layer is organized into command groups that map to algorithm set
4750
Each module set typically follows a consistent pattern with three types of modules:
4851

4952
1. **Load Data Modules** - Generate data loading configurations
50-
- Define which images to process
51-
- Create CSV files for CellProfiler to locate images
52-
- Organize data by batch, plate, well, and site
53+
- Define which images to process
54+
- Create CSV files for CellProfiler to locate images
55+
- Organize data by batch, plate, well, and site
5356
2. **Pipeline Generation Modules** - Create processing pipeline definitions
54-
- Generate CellProfiler pipeline files
55-
- Configure pipeline parameters based on experiment settings
56-
- Define processing operations
57+
- Generate CellProfiler pipeline files
58+
- Configure pipeline parameters based on experiment settings
59+
- Define processing operations
5760
3. **Execution Modules** - Execute the pipeline on prepared data
58-
- Run pipelines with appropriate parallelism
59-
- Manage resource allocation
60-
- Organize outputs according to experimental structure
61+
- Run pipelines with appropriate parallelism
62+
- Manage resource allocation
63+
- Organize outputs according to experimental structure
6164

6265
This pattern mirrors the organization of algorithms and CLI commands, but adds the standardized abstraction and container orchestration capabilities of the module layer.
6366

6467
Common module sets include:
6568

6669
1. **CP Modules** - For Cell Painting workflows:
67-
- `cp_illum_calc` - Illumination calculation modules
68-
- `cp_illum_apply` - Illumination correction modules
69-
- `cp_segcheck` - Segmentation check modules
70+
- `cp_illum_calc` - Illumination calculation modules
71+
- `cp_illum_apply` - Illumination correction modules
72+
- `cp_segcheck` - Segmentation check modules
7073
2. **SBS Modules** - For Sequencing By Synthesis workflows:
71-
- `sbs_illum_calc` - SBS illumination calculation modules
72-
- `sbs_illum_apply` - SBS illumination correction modules
73-
- `sbs_align` - SBS alignment modules
74-
- `sbs_preprocess` - SBS preprocessing modules
74+
- `sbs_illum_calc` - SBS illumination calculation modules
75+
- `sbs_illum_apply` - SBS illumination correction modules
76+
- `sbs_align` - SBS alignment modules
77+
- `sbs_preprocess` - SBS preprocessing modules
7578
3. **Common Modules** - For general operations:
76-
- `gen_index` - Index generation modules
77-
- `gen_inv` - Inventory management modules
78-
- `analysis` - Analysis modules
79+
- `gen_index` - Index generation modules
80+
- `gen_inv` - Inventory management modules
81+
- `analysis` - Analysis modules
7982

8083
## Module Implementation
8184

@@ -241,7 +244,7 @@ class CPSegcheckGenCPPipeModule(StarrynightModule):
241244
"cppipe_path": [spec.outputs["cppipe_path"].value]
242245
},
243246
config=ContainerConfig(
244-
image="ghrc.io/leoank/starrynight:dev",
247+
image="ghcr.io/leoank/starrynight:dev",
245248
cmd=cmd,
246249
env={},
247250
),
@@ -255,9 +258,9 @@ This example illustrates several important aspects of module implementation:
255258

256259
1. **Module Identity** - The `uid()` method provides a unique identifier
257260
2. **Module Specification** - The `_spec()` method defines inputs, outputs, and metadata and also set default values.
258-
4. **Compute Graph Creation** - The module uses the `_create_pipe` function to generate a Pipecraft pipeline
259-
5. **CLI Command Construction** - The module constructs a CLI command that will be executed in a container
260-
6. **Container Specification** - The module defines the container image and execution environment
261+
3. **Compute Graph Creation** - The module uses the `_create_pipe` function to generate a Pipecraft pipeline
262+
4. **CLI Command Construction** - The module constructs a CLI command that will be executed in a container
263+
5. **Container Specification** - The module defines the container image and execution environment
261264

262265
### Module Configuration
263266

@@ -305,12 +308,14 @@ The Bilayers project enables algorithm developers to write a single configuratio
305308
StarryNight leverages the Bilayers specification system to standardize its module interfaces. The integration works through several mechanisms:
306309

307310
1. **Schema Download and Synchronization**: StarryNight maintains a local copy of the Bilayers validation schema, which is automatically downloaded from the Bilayers repository:
311+
308312
```python
309313
# From starrynight/modules/common.py
310314
VALIDATE_SCHEMA_URL = "https://raw.githubusercontent.com/bilayer-containers/bilayers/master/tests/test_config/validate_schema.yaml"
311315
```
312316

313317
2. **Pydantic Model Generation**: The Bilayers LinkML schema is converted into Pydantic models that StarryNight uses for runtime validation:
318+
314319
```python
315320
def update_module_schema() -> None:
316321
"""Download and update the module schema from bilayers."""
@@ -326,26 +331,26 @@ StarryNight leverages the Bilayers specification system to standardize its modul
326331
The module layer uses Bilayers to create standardized definitions of:
327332

328333
- **Input Specifications**:
329-
- Type definitions (image, file, directory, array, measurement)
330-
- Validation rules and constraints
331-
- Default values and optional flags
332-
- Descriptions for documentation
333-
- CLI tag mappings for command generation
334+
- Type definitions (image, file, directory, array, measurement)
335+
- Validation rules and constraints
336+
- Default values and optional flags
337+
- Descriptions for documentation
338+
- CLI tag mappings for command generation
334339
- **Output Specifications**:
335340
- Output types and formats
336341
- File naming patterns
337342
- Directory structures
338343
- Relationships to inputs
339344
- **Parameter Definitions**:
340-
- UI element types (checkbox, integer, float, dropdown, textbox)
341-
- Value constraints and defaults
342-
- Grouping for beginner/advanced modes
343-
- Help text and documentation
345+
- UI element types (checkbox, integer, float, dropdown, textbox)
346+
- Value constraints and defaults
347+
- Grouping for beginner/advanced modes
348+
- Help text and documentation
344349
- **Algorithm Metadata**:
345-
- Citations and references
346-
- Docker image specifications
347-
- License information
348-
- Algorithm descriptions
350+
- Citations and references
351+
- Docker image specifications
352+
- License information
353+
- Algorithm descriptions
349354

350355
### Example: How a Module Uses Bilayers
351356

@@ -388,9 +393,9 @@ def _spec(self) -> SpecContainer:
388393

389394
1. **Standardization**: All modules follow the same specification format, making them predictable and easy to understand.
390395
2. **Interoperability**: Because StarryNight uses the Bilayers specification, there's potential for:
391-
- Importing Bilayers-compatible tools from other projects
392-
- Exporting StarryNight modules for use in other Bilayers-compatible systems
393-
- Leveraging the broader Bilayers ecosystem of tools and interfaces
396+
- Importing Bilayers-compatible tools from other projects
397+
- Exporting StarryNight modules for use in other Bilayers-compatible systems
398+
- Leveraging the broader Bilayers ecosystem of tools and interfaces
394399
3. **Automatic UI Generation**: While StarryNight doesn't currently generate Gradio or Jupyter interfaces from these specs, the Bilayers-compliant specifications make this possible in the future.
395400
4. **Validation**: The LinkML-based schema provides robust validation of module specifications, catching configuration errors early.
396401
5. **Documentation**: The structured format ensures that all modules have consistent documentation for their inputs, outputs, and parameters.
@@ -511,26 +516,26 @@ In pipeline composition, modules are created and configured individually, but th
511516
Creating a new module set involves implementing classes for each stage of processing:
512517

513518
1. **Plan the Module Set**:
514-
- Identify the algorithm set to wrap
515-
- Determine inputs, outputs, and parameters
516-
- Design the module structure (typically load data, pipeline generation, and execution)
519+
- Identify the algorithm set to wrap
520+
- Determine inputs, outputs, and parameters
521+
- Design the module structure (typically load data, pipeline generation, and execution)
517522
2. **Create Module Classes**:
518-
- Implement subclasses of `StarryNightModule` for each stage
519-
- Define unique identifiers and specifications
520-
- Implement `from_config` methods
521-
- Create pipeline generation methods
523+
- Implement subclasses of `StarryNightModule` for each stage
524+
- Define unique identifiers and specifications
525+
- Implement `from_config` methods
526+
- Create pipeline generation methods
522527
3. **Define Specifications**:
523-
- Use Bilayers to define inputs, outputs, and parameters
524-
- Document parameters with clear descriptions
525-
- Define validation rules
528+
- Use Bilayers to define inputs, outputs, and parameters
529+
- Document parameters with clear descriptions
530+
- Define validation rules
526531
4. **Implement Pipeline Creation**:
527-
- Use Pipecraft to define compute graphs
528-
- Specify container configurations
529-
- Map inputs and outputs properly
532+
- Use Pipecraft to define compute graphs
533+
- Specify container configurations
534+
- Map inputs and outputs properly
530535
5. **Test the Modules**:
531-
- Test individual modules
532-
- Test automatic configuration
533-
- Test pipeline integration
536+
- Test individual modules
537+
- Test automatic configuration
538+
- Test pipeline integration
534539

535540
## Relationship to Adjacent Layers
536541

docs/architecture/05_execution_layer.md

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -255,6 +255,7 @@ StarryNight uses containerization for reproducible algorithm execution. This is
255255
### Container Definition
256256

257257
The `Container` class in `pipecraft/node.py` defines execution environments with:
258+
258259
- `image`: Docker/Singularity image reference
259260
- `cmd`: Command to run within the container
260261
- `env`: Environment variables
@@ -273,7 +274,7 @@ Container(
273274
"cp_illum_calc_dir": [...]
274275
},
275276
config=ContainerConfig(
276-
image="ghrc.io/leoank/starrynight:dev",
277+
image="ghcr.io/leoank/starrynight:dev",
277278
cmd=["starrynight", "cp", "-p", spec.inputs[0].path, ...],
278279
env={},
279280
),
@@ -283,6 +284,7 @@ Container(
283284
### Backend Integration
284285

285286
The `SnakeMakeBackend` in `pipecraft/backend/snakemake.py` translates container specifications to Snakemake rules:
287+
286288
- Container images become Snakemake container directives
287289
- Input/output paths define rule dependencies
288290
- Commands define the shell execution
@@ -303,6 +305,7 @@ rule ${container.name.replace(" ", "_").lower()}:
303305
### Execution Flow
304306

305307
The execution process follows these steps:
308+
306309
1. Modules define containers with appropriate configurations
307310
2. The pipeline connects containers in sequential or parallel arrangements
308311
3. The backend compiles the pipeline to Snakemake rules

docs/architecture/08_practical_integration.md

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
This document provides a concrete example of how StarryNight's architectural layers work together in practice by examining `exec_pcp_generic_pipe.py`, an example pipeline implementation file that demonstrates the PCP Generic workflow. While the [architecture overview](00_architecture_overview.md) and individual layer documents ([Algorithm](01_algorithm_layer.md), [CLI](02_cli_layer.md), [Module](03_module_layer.md), [Pipeline](04_pipeline_layer.md), [Execution](05_execution_layer.md), [Configuration](06_configuration_layer.md)) explain each architectural layer conceptually, this walkthrough shows how these components integrate in a real workflow.
44

55
!!!note "Pedagogical Approach"
6-
This document deliberately uses the step-by-step implementation in `exec_pcp_generic_pipe.py` to clearly demonstrate individual components and their interactions. This approach:
6+
This document deliberately uses the step-by-step implementation in `exec_pcp_generic_pipe.py` to clearly demonstrate individual components and their interactions. This approach:
77

88
- Allows researchers to inspect intermediate results between pipeline stages
99
- Matches biological research workflows where verification at each stage is crucial
@@ -151,7 +151,7 @@ pcp_experiment = PCPGeneric.from_index(index_path, pcp_exp_init.model_dump())
151151
## Anatomy of a Pipeline Step
152152

153153
!!!note "CellProfiler Integration Pattern"
154-
The three-phase pattern described below (Generate Load Data → Generate Pipeline File → Execute Pipeline) is specific to how StarryNight integrates with CellProfiler. This pattern isn't a requirement of the StarryNight architecture, but rather a practical approach for this particular integration. Other tools may use different patterns while still adhering to the module abstraction.
154+
The three-phase pattern described below (Generate Load Data → Generate Pipeline File → Execute Pipeline) is specific to how StarryNight integrates with CellProfiler. This pattern isn't a requirement of the StarryNight architecture, but rather a practical approach for this particular integration. Other tools may use different patterns while still adhering to the module abstraction.
155155

156156
With the experiment configured, we can now examine one complete pipeline step (CP calculate illumination). Each step follows a consistent three-phase pattern:
157157

@@ -228,12 +228,14 @@ This module finds both the LoadData file and the pipeline file created in the pr
228228
Looking at this example, we can see how all the architecture layers work together across the two main phases:
229229

230230
### Pipeline Composition Phase
231+
231232
1. **Configuration Layer**: `DataConfig` and experiment configuration drive behavior across all layers
232233
2. **Module Layer**: Defines standardized components (like `CPCalcIllumInvokeCPModule`) with specifications and compute graphs
233234
3. **Pipeline Layer**: In this example, we're executing modules one by one, but they can be composed into a complete pipeline as seen in `create_pcp_generic_pipeline`
234235
4. **Execution Layer (design time)**: `SnakeMakeBackend` translates module compute graphs into Snakemake rules
235236

236237
### Runtime Execution Phase
238+
237239
5. **Execution Layer (runtime)**: Schedules container execution based on Snakemake rules
238240
6. **Container Runtime**: Executes commands in isolated environments
239241
7. **CLI Layer**: Provides command-line tools that parse arguments and call algorithms
@@ -257,7 +259,7 @@ MODULE_REGISTRY: dict[str, StarrynightModule] = {
257259
```
258260

259261
!!!note "Integration with Broader System"
260-
The registry is not used in this example, but it serves as a critical integration point with other StarryNight components:
262+
The registry is not used in this example, but it serves as a critical integration point with other StarryNight components:
261263

262264
- Enables **Conductor** to discover and invoke available modules dynamically
263265
- Allows **Canvas** to present available modules in its user interface
@@ -288,7 +290,7 @@ Container(
288290
"cp_illum_calc_dir": [...]
289291
},
290292
config=ContainerConfig(
291-
image="ghrc.io/leoank/starrynight:dev",
293+
image="ghcr.io/leoank/starrynight:dev",
292294
cmd=["starrynight", "cp", "-p", spec.inputs["cppipe_path"].value, ...],
293295
env={},
294296
),
@@ -346,13 +348,14 @@ This approach enables complex parallel execution patterns, where CP and SBS proc
346348
When implementing your own modules, follow these patterns:
347349

348350
!!!note "Module vs. Algorithm Extension"
349-
This section focuses on extending StarryNight with new **modules** rather than new algorithms. Modules provide standardized interfaces to existing algorithms, whether those algorithms are part of StarryNight's core or from external tools. To add your own algorithms to StarryNight, see the ["Adding a New Algorithm"](#adding-a-new-algorithm) section below.
351+
This section focuses on extending StarryNight with new **modules** rather than new algorithms. Modules provide standardized interfaces to existing algorithms, whether those algorithms are part of StarryNight's core or from external tools. To add your own algorithms to StarryNight, see the ["Adding a New Algorithm"](#adding-a-new-algorithm) section below.
350352

351353
1. **Module Structure**: Consider your module's specific requirements:
352-
- For CellProfiler integrations, use the three-phase pattern shown earlier
353-
- For other tools, design appropriate module structures based on tool requirements
354-
- Ensure your modules have clear inputs, outputs, and containerized execution specifications
354+
- For CellProfiler integrations, use the three-phase pattern shown earlier
355+
- For other tools, design appropriate module structures based on tool requirements
356+
- Ensure your modules have clear inputs, outputs, and containerized execution specifications
355357
2. **Registry Integration**: Define a unique ID and register your module in the registry:
358+
356359
```python
357360
@staticmethod
358361
def uid() -> str:

0 commit comments

Comments
 (0)