You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
AiiDA provides an Object-Relational Mapping (ORM) system that abstracts database operations while supporting multiple database backends.
12
-
In this post, we'll explore how AiiDA leverages SQLAlchemy to create a flexible, multi-backend ORM to separate concerns between Python objects and database persistence.
11
+
In this post, we'll take a deep dive into the implementation of AiiDA's Object-Relational Mapping (ORM) system.
12
+
We'll explore how AiiDA leverages [SQLAlchemy](https://github.com/sqlalchemy/sqlalchemy) to create a flexible (ORM) backend that abstracts database operations and separates concerns between the user-facing Python objects and the underlying database persistence.
13
13
14
14
## Architecture overview: The multi-layer design
15
15
16
16
AiiDA's ORM follows a four-layer architecture that provides clean separation between the user interface, business logic, and data persistence:
17
17
18
18
```
19
19
┌─────────────────────┐
20
-
│ User Interface │ ← Node (Python ORM class)
20
+
│ User interface │ ← Node (Python ORM class)
21
21
│ (orm/nodes) │
22
22
├─────────────────────┤
23
-
│ Backend Interface │ ← BackendNode (Abstract base class)
23
+
│ Backend interface │ ← BackendNode (Abstract base class)
This design allows AiiDA to support multiple database backends (currently PostgreSQL and SQLite) while providing a unified Python interface for users.
35
-
Importantly, users never have to interact directly with database-specific code.
36
-
Instead, they work with the high-level `Node` class (and its child classes), which automatically delegates operations to the appropriate backend implementation.
34
+
Each layer serves a distinct purposes:
37
35
38
-
The three layers serve distinct purposes:
36
+
*__User interface layer__: Provides a clean, Pythonic API that hides database complexity
37
+
*__Backend interface layer__: Defines contracts that all database implementations must follow
38
+
*__Database implementation layer__: Handles the specifics of different database systems
39
+
*__Database model layer__: Defines the actual table schemas and relationships using SQLAlchemy's declarative approach
39
40
40
-
*__User Interface Layer__: Provides a clean, Pythonic API that hides database complexity
41
-
*__Backend Interface Layer__: Defines contracts that all database implementations must follow
42
-
*__Database Implementation Layer__: Handles the specifics of different database systems
41
+
Importantly, you, the user never have to interact directly with database-specific code.
42
+
Instead, you can work with the high-level `Node` class (and its derived classes), while AiiDA automatically delegates database operations to the appropriate backend implementation.
43
+
This design further allows AiiDA to support multiple database backends (currently PostgreSQL and SQLite), while providing a unified Python interface for users.
43
44
44
-
So let's start from the bottom, shall we?
45
+
Let's start from the bottom and work our way up, shall we?
45
46
46
47
## The database: `DbNode`
47
48
48
49
AiiDA uses SQLAlchemy's declarative approach to define database tables.
49
50
This means the database schema is defined using Python classes rather than raw SQL.
from sqlalchemy.dialects.postgresql importJSONB, UUID
@@ -88,30 +91,34 @@ class DbNode(Base):
88
91
user = relationship('DbUser', backref='dbnodes')
89
92
```
90
93
91
-
<!-- NOTE: -->
92
-
<!-- Where base is derived from SQLAlchemy's `declarative_base`. -->
94
+
The `Base` class from which `DbNode` inherits is derived from SQLAlchemy's `declarative_base()`, which provides the metaclass and base functionality that allows Python classes to be automatically mapped to database tables.
93
95
94
-
__The key features of AiiDA's SQLAlchemy models:__
This declarative approach means that table schemas, constraints, relationships, and indexes are all defined in Python code rather than separate SQL files, making the database structure self-documenting and version-controllable alongside the application logic.
95
104
96
-
1.**JSON(B) for Flexibility**: The use of PostgreSQL's `JSONB` type (`JSON` for SQLite) for `attributes`, `extras`, and `repository_metadata` provides schema flexibility while maintaining query performance.
97
-
This is crucial for scientific computing where you can't predict what data users will want to store.
98
-
Traditional relational databases would require creating new columns for every new property, but JSON columns let you store arbitrary structured data.
105
+
__Key features of AiiDA's SQLAlchemy models:__
99
106
100
-
2.**UUID-based Identity**: Each node has both an integer primary key (`id` in the table, `pk` in the Python API) for database efficiency and a UUID for global uniqueness and portability.
107
+
1.**Use of JSON(B) for flexibility**: The use of PostgreSQL's `JSONB` (and SQLite's `JSON`) type for `attributes`, `extras`, and `repository_metadata` provides schema flexibility while maintaining query performance.
108
+
This is crucial for scientific computing where one can't predict what data users will want to store.
109
+
Traditional relational databases would require creating new columns for every new property, but JSON columns let you store arbitrary structured data (as long as it's JSON-serializable).
110
+
2.**UUID-based Identity**: Each node has both an integer primary key (`id` in the table, `pk` in the Python API[^1]) for database efficiency and a UUID for global uniqueness and portability.
101
111
The integer `id` is user-friendly, fast for database joins and indexing, while the UUID allows nodes to be moved between different AiiDA installations without conflicts.
102
-
103
112
3.**Automatic Timestamps**: Creation and modification times are automatically managed through SQLAlchemy's `default` and `onupdate` parameters.
104
-
105
-
<!-- TODO: Need to understand this better myself -->
106
113
4.**Strategic Indexing**: Important columns like `node_type`, `process_type`, and timestamps are indexed for query performance.
107
114
Without indexes, queries like "find all calculation nodes from last month" would be extremely slow.
108
115
The indexes make these common queries fast even with millions of nodes.
109
116
110
-
<!-- TODO: Check again how SqlaNode and Node interact -->
111
-
112
117
## The contract: `BackendNode`
113
118
114
-
Above the database layer sits the abstract `BackendNode` class, which defines the interface contract that all database backend implementations must follow:
@@ -142,12 +149,12 @@ It defines essential properties like `uuid`, `node_type`, `process_type`, and me
142
149
143
150
The abstract class serves as a __contract__ - any backend implementation must provide all these methods and properties.
144
151
This guarantees that switching between PostgreSQL and SQLite backends won't break user code, since both implementations satisfy the same interface.
145
-
You can think of it like a universal power adapter: regardless of whether you're plugging into a European, US, or UK outlet, your device gets the same voltage and current.
146
-
The ``BackendNode`` abstract base class provides that standardization for database operations.
147
152
148
153
## The bridge: `SqlaNode`
149
154
150
-
The `SqlaNode` class bridges the abstract `BackendNode` interface with the concrete SQLAlchemy models:
that handles session management automatically. This design provides several key capabilities:
213
+
214
+
*__Model wrapping__: Encapsulates SQLAlchemy model instances with automatic session tracking
215
+
*__Type safety__: Generic typing ensures proper relationships between related objects (users, computers)
216
+
*__Backend abstraction__: Provides a consistent interface regardless of database backend
217
+
*__Direct access__: The `bare_model` property allows bypassing AiiDA's automatic management when needed
254
218
255
-
!!! info ""
219
+
The key insight is that `SqlaNode` provides two levels of access:
220
+
221
+
*`node.model`: The wrapped model with automatic session tracking
222
+
*`node.bare_model`: Direct access to the raw SQLAlchemy mode
256
223
257
224
## The user interface: `Node`
258
225
259
-
At the top level, users interact with the `Node` class, which provides a Pythonic interface:
226
+
At the top level, users interact with the `Node` class, which uses compososition to contain an `SqlaNode` instance, and provides a Pythonic interface to users:
@@ -275,22 +242,38 @@ class Node(Entity['BackendNode', NodeCollection], metaclass=AbstractNodeMeta):
275
242
super().__init__(backend_entity)
276
243
```
277
244
278
-
Here, AiiDA uses _composition_ instead of direct _inheritance_ - the `Node` class _contains_ a `BackendNode` instance rather than being one.
279
-
The creation magic happens in `backend.nodes.create()` which returns an SqlaNode instance as shown above (automatically selecting the correct backend).
245
+
The creation magic happens in `backend.nodes.create()` which returns an `SqlaNode` instance[^2] as shown above (automatically selecting the correct backend).
246
+
However, note that from the user's perspective, just a `Node` is created-the backend selection is transparent.
247
+
248
+
Here's how all of this looks like in action in a `verdi shell`:
280
249
281
-
<!-- TODO: add note that for psql sqlanode is created, for sqlite an sqlitenode -->
282
-
However, note that from the user's perspective, just a `Node` is created.
283
-
The backend selection is transparent.
250
+
```python
251
+
In [1]: n = Int(1).store()
252
+
253
+
In [2]: n.backend_entity
254
+
Out[2]: <aiida.storage.psql_dos.orm.nodes.SqlaNode at 0x7d8ac6d53880>
255
+
256
+
In [3]: n.backend_entity.model
257
+
Out[3]: <aiida.storage.psql_dos.orm.utils.ModelWrapper at 0x7d8ac6c6fcd0>
AiiDA uses a namespace pattern on the `Node`class to organize functionality:
288
266
289
267
```python
290
-
@cached_property
291
-
defbase(self) -> NodeBase:
292
-
"""Return the node base namespace."""
293
-
return NodeBase(self)
268
+
class Node(Entity['BackendNode', NodeCollection], metaclass=AbstractNodeMeta):
269
+
"""Base class for all nodes in AiiDA."""
270
+
271
+
...
272
+
273
+
@cached_property
274
+
def base(self) -> NodeBase:
275
+
"""Return the node base namespace."""
276
+
return NodeBase(self)
294
277
295
278
class NodeBase:
296
279
"""A namespace for node related functionality."""
@@ -311,19 +294,14 @@ class NodeBase:
311
294
return NodeLinks(self._node)
312
295
```
313
296
314
-
This namespace pattern was introduced because having all properties and methods directly on the `Node` class became overwhelming and to prevent name conflicts.
315
-
With dozens of methods for different functionalities (attributes, repository, links, caching, comments), the API became cluttered.
316
-
317
-
The namespace approach groups related functionality together:
This namespace pattern was introduced because having all properties and methods directly on the `Node`class became overwhelming, making the API cluttered, and prone to name conflicts.
321
298
322
-
This makes the API more discoverable and prevents method name conflicts.
323
-
It also uses @cached_property to ensure these namespace objects are created only once per node instance.
299
+
The namespace approach further groups related functionality together:
<!-- NOTE: how is collection pattern different from /complements QB? -->
304
+
and the use of `@cached_property` ensures that these namespace objects are created only once per node instance.[^3]
327
305
328
306
## Honorable mentions
329
307
@@ -355,9 +333,10 @@ The `ondelete='CASCADE'` on the output relationship ensures that when a node is
355
333
356
334
### The QueryBuilder
357
335
358
-
The `QueryBuilder` is AiiDA's main API provided to retrieve data from the database.
336
+
The `QueryBuilder`is AiiDA's main Python API to retrieve data from the database.
359
337
It provides a uniform, backend-agnostic interface:
360
338
339
+
<!--TODO: change example -->
361
340
```python
362
341
# Simple node query
363
342
qb = QueryBuilder()
@@ -477,8 +456,20 @@ AiiDA's ORM architecture makes use of several important design principles:
477
456
478
457
This architecture allows AiiDA to provide a powerful, flexible ORM that adapts to different database backends while maintaining a consistent user experience.
479
458
459
+
460
+
__Footnotes__
461
+
[^1]:
462
+
To avoid confusion with Python's `id()` function.
463
+
[^2]:
464
+
For the SQLite backend, an `SqliteNode`is actually created, which, however, inherits from`SqlaNode`.
465
+
[^3]:
466
+
Although we do understand that the `node.base` approach can be non-intuitive for first time users.
467
+
468
+
480
469
<!--TODO: -->
481
470
<!-- Add statement that the whole infrastructure also had to be implemented for the other, specialized data types of AiiDA -->
482
471
<!-- Add some admonititions?-->
483
472
<!-- add gh permalinks -->
484
473
474
+
<!--NOTE: how is collection pattern different from/complements QB?-->
0 commit comments