How to reuse profiles set up on different machines

[basnijholt]

TL;DR: I would like to be able to use the existing profiles (from ~/.aiida files) on new machines with a new empty PostgreSQL server running.

I am working on a computational cluster with a shared NFS but where the machine I am working on (with RabbitMQ and PostgreSQL) might change from time to time.

On machine 1: I run

• verdi quicksetup --profile my_profile
• verdi --profile my_profile computer setup -n --config computer_config.yml
• verdi --profile my_profile computer configure core.ssh my_computer -n

Then on machine 2, I also have RabbitMQ and PostgreSQL running and have access to the ~/.aiida config files.

Now running:

verdi -p my_profile daemon start-circus

raises:

django.db.utils.OperationalError: FATAL:  database "my_profile_basnijholt_e1ab231934e0944fc863db88ac471ddd" does not exist

Full output

(py39) cw-basnijholt  ➜  verdi -p my_profile daemon start-circus
Traceback (most recent call last):
  File "/nfshome/basnijholt/mambaforge/envs/py39/lib/python3.9/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
    self.connect()
  File "/nfshome/basnijholt/mambaforge/envs/py39/lib/python3.9/site-packages/django/db/backends/base/base.py", line 195, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/nfshome/basnijholt/mambaforge/envs/py39/lib/python3.9/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/nfshome/basnijholt/mambaforge/envs/py39/lib/python3.9/site-packages/psycopg2/__init__.py", line 127, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL:  database "my_profile_basnijholt_e1ab231934e0944fc863db88ac471ddd" does not exist


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nfshome/basnijholt/mambaforge/envs/py39/bin/verdi", line 12, in <module>
    sys.exit(verdi())
  File "/nfshome/basnijholt/mambaforge/envs/py39/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/nfshome/basnijholt/mambaforge/envs/py39/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/nfshome/basnijholt/mambaforge/envs/py39/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/nfshome/basnijholt/mambaforge/envs/py39/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/nfshome/basnijholt/mambaforge/envs/py39/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/nfshome/basnijholt/mambaforge/envs/py39/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/nfshome/basnijholt/repos/public/aiida-core/aiida/cmdline/utils/decorators.py", line 70, in wrapper
    load_backend_if_not_loaded()
  File "/nfshome/basnijholt/repos/public/aiida-core/aiida/cmdline/utils/decorators.py", line 48, in load_backend_if_not_loaded
    manager.get_backend()  # This will load the backend of the loaded profile, if not already loaded
  File "/nfshome/basnijholt/repos/public/aiida-core/aiida/manage/manager.py", line 209, in get_backend
    self._load_backend()
  File "/nfshome/basnijholt/repos/public/aiida-core/aiida/manage/manager.py", line 132, in _load_backend
    backend_manager.load_backend_environment(profile, validate_schema=schema_check)
  File "/nfshome/basnijholt/repos/public/aiida-core/aiida/backends/manager.py", line 120, in load_backend_environment
    self.validate_schema(profile)
  File "/nfshome/basnijholt/repos/public/aiida-core/aiida/backends/manager.py", line 249, in validate_schema
    self.validate_schema_generation(profile)
  File "/nfshome/basnijholt/repos/public/aiida-core/aiida/backends/manager.py", line 308, in validate_schema_generation
    schema_generation_database = self.get_schema_generation_database()
  File "/nfshome/basnijholt/repos/public/aiida-core/aiida/backends/djsite/manager.py", line 93, in get_schema_generation_database
    result = backend.execute_raw(r"""SELECT val FROM db_dbsetting WHERE key = 'schema_generation';""")
  File "/nfshome/basnijholt/repos/public/aiida-core/aiida/orm/implementation/django/backend.py", line 201, in execute_raw
    with self.cursor() as cursor:
  File "/nfshome/basnijholt/mambaforge/envs/py39/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/nfshome/basnijholt/repos/public/aiida-core/aiida/orm/implementation/django/backend.py", line 191, in cursor
    yield self._get_connection().cursor()
  File "/nfshome/basnijholt/mambaforge/envs/py39/lib/python3.9/site-packages/django/db/backends/base/base.py", line 256, in cursor
    return self._cursor()
  File "/nfshome/basnijholt/mambaforge/envs/py39/lib/python3.9/site-packages/django/db/backends/base/base.py", line 233, in _cursor
    self.ensure_connection()
  File "/nfshome/basnijholt/mambaforge/envs/py39/lib/python3.9/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
    self.connect()
  File "/nfshome/basnijholt/mambaforge/envs/py39/lib/python3.9/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/nfshome/basnijholt/mambaforge/envs/py39/lib/python3.9/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
    self.connect()
  File "/nfshome/basnijholt/mambaforge/envs/py39/lib/python3.9/site-packages/django/db/backends/base/base.py", line 195, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/nfshome/basnijholt/mambaforge/envs/py39/lib/python3.9/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/nfshome/basnijholt/mambaforge/envs/py39/lib/python3.9/site-packages/psycopg2/__init__.py", line 127, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: FATAL:  database "my_profile_basnijholt_e1ab231934e0944fc863db88ac471ddd" does not exist

Sofar I figured out that I can get the profile to work with a fresh database:

rm -fr ~/.aiida/repository/my_profile/container

and

from aiida.manage.configuration import get_profile
from aiida.manage.external.postgres import Postgres

profile = get_profile()

dbinfo_su = {
    "host": profile.database_hostname,
    "port": profile.database_port,
    "user": "postgres",
    "password": None,
}
postgres = Postgres(interactive=False, quiet=False, dbinfo=dbinfo_su)
db_username, db_name = postgres.create_dbuser_db_safe(
    dbname=profile.database_name,
    dbuser=profile.database_username,
    dbpass=profile.database_password,
)
postgres.determine_setup()

from aiida.manage.manager import get_manager

manager = get_manager()
backend = manager._load_backend(schema_check=False, repository_check=False)

backend.migrate()

However, I do not think this is a good solution. Does anyone have any suggestions?

[zhubonan]

This is because the second machine is accessing a different PostgreSQL database I think - so the profile's database (my_profile_basnijholt_e1ab231934e0944fc863db88ac471ddd) is not found. It is better to check with verdi status prior to launching the daemon.

In theory, you should be able to launch RabbitMQ and PosgreSQL on the second machine using the same underlying data on the NFS mount, as long as they are off on the first machine (e.g. both machine cannot serve the same services at the same time).