v0.8.1 messed with accounts

[omergoktas]

After upgrading to v0.8.1 from v0.8.0, my entire user accounts disappeared. I have been using "mysql" backend for data and "internal" for user accounts. Any idea how to fix that?

My configuration:

authentication.fallback-admin.secret = "redacted"
authentication.fallback-admin.user = "omergoktas"
certificate.master.cert = "%{file:/opt/stalwart-mail/etc/certs/fullchain.pem}%"
certificate.master.default = true
certificate.master.private-key = "%{file:/opt/stalwart-mail/etc/certs/privkey.pem}%"
cluster.node-id = 1
directory.internal.cache.entries = 500
directory.internal.store = "mysql"
directory.internal.type = "internal"
lookup.default.hostname = "omergoktas.com"
server.http.permissive-cors = false
server.http.url = "protocol + '://' + key_get('default', 'hostname') + ':' + local_port"
server.http.use-x-forwarded = true
server.listener.http.bind = "127.0.0.1:8080"
server.listener.http.protocol = "http"
server.listener.http.proxy.override = false
server.listener.http.socket.override = false
server.listener.http.tls.implicit = false
server.listener.http.tls.override = false
server.listener.imaptls.bind = "0.0.0.0:993"
server.listener.imaptls.protocol = "imap"
server.listener.imaptls.proxy.override = false
server.listener.imaptls.socket.override = false
server.listener.imaptls.tls.implicit = true
server.listener.imaptls.tls.override = false
server.listener.sieve.bind = "0.0.0.0:4190"
server.listener.sieve.protocol = "managesieve"
server.listener.sieve.proxy.override = false
server.listener.sieve.socket.override = false
server.listener.sieve.tls.implicit = true
server.listener.sieve.tls.override = false
server.listener.smtp.bind = "0.0.0.0:25"
server.listener.smtp.protocol = "smtp"
server.listener.smtp.proxy.override = false
server.listener.smtp.socket.override = false
server.listener.smtp.tls.implicit = false
server.listener.smtp.tls.override = false
server.listener.submission.bind = "0.0.0.0:587"
server.listener.submission.protocol = "smtp"
server.listener.submission.proxy.override = false
server.listener.submission.socket.override = false
server.listener.submission.tls.implicit = false
server.listener.submission.tls.override = false
server.listener.submissions.bind = "0.0.0.0:465"
server.listener.submissions.protocol = "smtp"
server.listener.submissions.proxy.override = false
server.listener.submissions.socket.override = false
server.listener.submissions.tls.implicit = true
server.listener.submissions.tls.override = false
server.max-connections = 100
server.socket.backlog = 1024
server.socket.nodelay = true
server.socket.reuse-addr = false
server.socket.reuse-port = false
server.tls.ignore-client-order = true
server.tls.timeout = "30s"
signature.rsa-omergoktas.com.algorithm = "rsa-sha256"
signature.rsa-omergoktas.com.canonicalization = "relaxed/relaxed"
signature.rsa-omergoktas.com.domain = "omergoktas.com"
signature.rsa-omergoktas.com.headers.0000 = "From"
signature.rsa-omergoktas.com.headers.0001 = "To"
signature.rsa-omergoktas.com.headers.0002 = "Date"
signature.rsa-omergoktas.com.headers.0003 = "Subject"
signature.rsa-omergoktas.com.headers.0004 = "Message-ID"
signature.rsa-omergoktas.com.private-key = "%{file:/opt/stalwart-mail/etc/certs/dkimpriv.key}%"
signature.rsa-omergoktas.com.report = true
signature.rsa-omergoktas.com.selector = "default"
storage.blob = "mysql"
storage.data = "mysql"
storage.directory = "internal"
storage.fts = "mysql"
storage.lookup = "mysql"
store.mysql.compression = "lz4"
store.mysql.database = "stalwart"
store.mysql.host = "127.0.0.1"
store.mysql.max-allowed-packet = 104857600
store.mysql.password = "redacted"
store.mysql.pool.max-connections = 10
store.mysql.pool.min-connections = 3
store.mysql.port = 3306
store.mysql.purge.frequency = "0 3 *"
store.mysql.query.domains = "SELECT 1 FROM emails WHERE address LIKE CONCAT('%@', ?) LIMIT 1"
store.mysql.query.emails = "SELECT address FROM emails WHERE name = ? AND type != 'list' ORDER BY type DESC, address ASC"
store.mysql.query.expand = "SELECT p.address FROM emails AS p JOIN emails AS l ON p.name = l.name WHERE p.type = 'primary' AND l.address = ? AND l.type = 'list' ORDER BY p.address LIMIT 50"
store.mysql.query.members = "SELECT member_of FROM group_members WHERE name = ?"
store.mysql.query.name = "SELECT name, type, secret, description, quota FROM accounts WHERE name = ? AND active = true"
store.mysql.query.recipients = "SELECT name FROM emails WHERE address = ? ORDER BY name ASC"
store.mysql.query.verify = "SELECT address FROM emails WHERE address LIKE CONCAT('%', ?, '%') AND type = 'primary' ORDER BY address LIMIT 5"
store.mysql.timeout = "15s"
store.mysql.type = "mysql"
store.mysql.user = "stalwart"
tracer.log.enable = true
tracer.log.level = "trace"
tracer.log.type = "journal"

Reverted back to v0.8.0 from my backups and everything works fine again.

[mdecimus]

Version 0.8.1 did not make any changes to the database layout or how information is accessed from the directory. Are you certain that the data you had was not from a 0.7 setup that was not migrated? Because what you describe is what would happen when an older database layout is used with a new version, which is not the case when upgrading from 0.8.0 to 0.8.1.

[omergoktas]

I reverted all the changes back with a backup. I manually added the option storage.encryption.enable = false to my toml file and then tried upgrading from v0.8.0 to v0.8.1 again with the hopes that it would work this time. But still didn't work. Encrytion at rest feature is not enabled by itself this time, but there's gotta be another issue.

[omergoktas]

Something definitely messes with the database structure, here is the database structure before and after the version upgrade:

Before vs After (many new tables with 16K size added to the database):

[mdecimus]

Based on the database layout you provided, this is what I suspected. You are currently using 0.7, not 0.8.0. To upgrade to 0.8.x you need to follow the instructions in the UPGRADING file in the repository.

[omergoktas]

Ok, I'll try the export/import method.

[omergoktas]

Following the steps described here seems to have worked: Upgrading from v0.7.3 to v0.8.0

Even though I am pretty sure I set up the server with v0.8.0, I am still open to the possibility that I might have mistakenly pulled the previous executable (v0.7.3) while setting up my server. If I am not wrong, you may receive more complaints about database issues between versions.

In any case, the solution you provided worked. Thank you, @mdecimus.