wip

Author: GeigerJ2

docs/news/posts/2025-09-05-orm.md

@@ -1,7 +1,7 @@
 ---
 blogpost: true
 category: Blog
-tags: usability
+tags: architecture
 author: Julian Geiger
 date: 2025-09-05
 ---
@@ -11,11 +11,10 @@ date: 2025-09-05
 AiiDA provides an Object-Relational Mapping (ORM) system that abstracts database operations while supporting multiple database backends.
 In this post, we'll explore how AiiDA leverages SQLAlchemy to create a flexible, multi-backend ORM to separate concerns between Python objects and database persistence.
 
-## Architecture Overview: The Three-Layer Design
+## Architecture overview: The multi-layer design
 
-AiiDA's ORM follows a three-layer architecture that provides clean separation between the user interface, business logic, and data persistence:
+AiiDA's ORM follows a four-layer architecture that provides clean separation between the user interface, business logic, and data persistence:
 
-<!-- TODO: check again, we have 4 top-level sections, but only 3 here -->
 ```
 ┌─────────────────────┐
 │   User Interface    │  ← Node (Python ORM class)
@@ -24,8 +23,11 @@ AiiDA's ORM follows a three-layer architecture that provides clean separation be
 │  Backend Interface  │  ← BackendNode (Abstract base class)
 │  (implementation)   │
 ├─────────────────────┤
-│ Database Backends   │  ← SqlaNode (SQLAlchemy implementation)
-│ (psql_dos/sqlite)   │
+│  Database Backends  │  ← SqlaNode (SQLAlchemy implementation)
+│  (psql_dos/sqlite)  │
+├─────────────────────┤
+│   Database Layer    │  ← DBNode (SQLAlchemy models)
+│      (models)       │
 └─────────────────────┘
 ```
 
@@ -41,7 +43,7 @@ The three layers serve distinct purposes:
 
 So let's start from the bottom, shall we?
 
-## The database: SQLAlchemy models
+## The database: `DbNode`
 
 AiiDA uses SQLAlchemy's declarative approach to define database tables.
 This means the database schema is defined using Python classes rather than raw SQL.
@@ -221,7 +223,7 @@ This design provides:
 
 __Storing data in the DB__
 
-Actual node data storage in the SQL db is achieved through the `store` method of the `SqlaNode`:
+Actual data storage in the SQL db is achieved through the `store` method of the `SqlaNode`:
 
 ```python
 def store(self, links=None, clean=True):
@@ -250,6 +252,7 @@ def store(self, links=None, clean=True):
 
 This approach ensures data consistency while supporting nested transactions for complex operations.
 
+!!! info ""
 
 ## The user interface: `Node`
 
@@ -324,7 +327,7 @@ It also uses @cached_property to ensure these namespace objects are created only
 
 ## Honorable mentions
 
-### Link Management: Modeling Relationships
+### Link management: modeling relationships
 
 AiiDA models relationships between nodes through a separate `DbLink` table:
 
@@ -350,7 +353,32 @@ The `label` field describes the specific role of the link (e.g., "structure", "p
 This metadata enables complex provenance queries like "find all structures that were relaxed using PBE exchange-correlation functional."
 The `ondelete='CASCADE'` on the output relationship ensures that when a node is deleted, all its incoming links are also removed, maintaining referential integrity.
 
-### Pydantic Model Integration: Modern Serialization
+### The QueryBuilder
+
+The `QueryBuilder` is AiiDA's main API provided to retrieve data from the database.
+It provides a uniform, backend-agnostic interface:
+
+```python
+# Simple node query
+qb = QueryBuilder()
+qb.append(Node, filters={'label': {'like': 'calculation%'}})
+results = qb.all()
+
+# Complex relationship query
+qb = QueryBuilder()
+qb.append(StructureData, tag='structure')
+qb.append(CalcJobNode, with_incoming='structure', tag='calc')
+qb.append(FolderData, with_incoming='calc', project=['*'])
+```
+
+The QueryBuilder automatically:
+
+* __Converts ORM Classes__: Maps Node classes to their corresponding database entities
+* __Handles Relationships__: Translates with_incoming/with_outgoing to proper SQL joins
+* __Type Filtering_: Automatically adds filters for node types and subtypes
+* __Backend Translation__: Converts the same query syntax to PostgreSQL or SQLite SQL
+
+### Pydantic models: Modern Serialization
 
 Recently, AiiDA incorporated Pydantic models for modern data validation and serialization:
 
@@ -401,15 +429,37 @@ This enables:
 - **API Integration**: Direct compatibility with FastAPI and other modern frameworks
 - **Schema Documentation**: Auto-generated OpenAPI schemas for web interfaces
 
-### NodeCollections
+### NodeCollections: The interface pattern
 
-### Querybuilder
+The `NodeCollection` class provides a clean interface for managing collections of nodes:
+
+```python
+class NodeCollection(EntityCollection[NodeType], Generic[NodeType]):
+    """The collection of nodes."""
+    
+    def delete(self, pk: int) -> None:
+        """Delete a Node from the collection with the given id"""
+        node = self.get(id=pk)
+        
+        if node.base.links.get_incoming().all():
+            raise exceptions.InvalidOperation(f'cannot delete Node<{node.pk}> because it has incoming links')
+            
+        if node.base.links.get_outgoing().all():
+            raise exceptions.InvalidOperation(f'cannot delete Node<{node.pk}> because it has outgoing links')
+            
+        self._backend.nodes.delete(pk)
+```
+This collection pattern provides several benefits:
+
+* __Type safety__: Generic typing ensures you get the correct node type back
+* __Validation__: Prevents deletion of nodes with existing links to maintain provenance integrity
+* __Backend abstraction__: Hides database-specific operations behind a clean interface
 
 ## Key Takeaways
 
-AiiDA's ORM architecture demonstrates several important design principles:
+AiiDA's ORM architecture makes use of several important design principles:
 
-1. **Multi-Layer Abstraction**: The three-layer design (User Interface → Backend Interface → Database Implementation) provides clean separation while allowing backend-specific optimizations.
+1. **Multi-Layer Abstraction**: The multi-layer design (User Interface → Backend Interface → Database Implementation) provides clean separation while allowing backend-specific optimizations.
 
 2. **Automatic Backend Adaptation**: The PostgreSQL-to-SQLite conversion system shows how to maintain feature parity across different database engines without code duplication.
 
@@ -425,9 +475,7 @@ AiiDA's ORM architecture demonstrates several important design principles:
 
 8. **Collection Patterns**: The collection system provides a consistent, intuitive interface for data access that scales from simple lookups to complex queries.
 
-This architecture allows AiiDA to provide a powerful, flexible ORM that adapts to different database backends while maintaining a consistent user experience. The careful balance between abstraction and performance, combined with modern Python practices, makes it an excellent example of how to build scalable scientific software that can evolve with changing requirements and technologies.
-
-The system's ability to automatically convert PostgreSQL models to SQLite equivalents, while maintaining query compatibility through sophisticated backend-specific implementations, demonstrates how thoughtful architecture can provide both flexibility and performance without compromising on either.
+This architecture allows AiiDA to provide a powerful, flexible ORM that adapts to different database backends while maintaining a consistent user experience.
 
 <!-- TODO: -->
 <!-- Add statement that the whole infrastructure also had to be implemented for the other, specialized data types of AiiDA -->