Firestore emulator in datastore mode does not export on exit

[Nathanmalnoury]

[REQUIRED] Environment info

firebase-tools:
I'm using the gcloud CLI :

$ /usr/lib64/google-cloud-sdk/platform/cloud-firestore-emulator/cloud_firestore_emulator --version
cloud-firestore-emulator 1.19.9

Platform:

Fedora Linux 40 (Workstation Edition)

[REQUIRED] Test case

https://github.com/Nathanmalnoury/repro-firestore-emulator-datastore-mode-persistence

[REQUIRED] Steps to reproduce

Included in test repo

[REQUIRED] Expected behavior

I would expect to see saved data in my configured "--export-on-exit" folder, to then be able to import them on the next start via --import-data.

[REQUIRED] Actual behavior

It stays empty when interrupting the emulator using Ctrl + C.

/Projects/test/building-an-app-3 (main ✔) gcloud emulators firestore start --database-mode datastore-mode   --export-on-exit ./saved_data  --host-port 0.0.0.0:8081 --project [PROJECT-ID] --verbosity debug 

DEBUG: Running [gcloud.emulators.firestore.start] with arguments: [--database-mode: "datastore-mode", --export-on-exit: "./saved_data", --host-port: "<googlecloudsdk.calliope.arg_parsers.HostPort object at 0x7f1313866310>", --project: "[PROJECT ID]", --verbosity: "debug"]
DEBUG: Found Cloud SDK root: /usr/lib64/google-cloud-sdk
Executing: /usr/lib64/google-cloud-sdk/platform/cloud-firestore-emulator/cloud_firestore_emulator start --host=0.0.0.0 --port=8081 --database-mode=datastore-mode --export-on-exit=./saved_data
[firestore] Oct 29, 2024 11:28:12 AM com.google.cloud.datastore.emulator.firestore.websocket.WebSocketServer start
[firestore] INFO: Started WebSocket server on ws://0.0.0.0:37901
[firestore] API endpoint: http://0.0.0.0:8081
[firestore] If you are using a library that supports the FIRESTORE_EMULATOR_HOST environment variable, run:
[firestore] 
[firestore]    export FIRESTORE_EMULATOR_HOST=0.0.0.0:8081
[firestore] 
[firestore] If you are running a Firestore in Datastore Mode project, run:
[firestore] 
[firestore]    export DATASTORE_EMULATOR_HOST=0.0.0.0:8081
[firestore] 
[firestore] Note: Support for Datastore Mode is in preview. If you encounter any bugs please file at https://github.com/firebase/firebase-tools/issues.
[firestore] Dev App Server is now running.
[firestore] 
[firestore] Oct 29, 2024 11:28:51 AM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[firestore] INFO: Detected HTTP/2 connection.
[firestore] Oct 29, 2024 11:28:51 AM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[firestore] INFO: Detected HTTP/2 connection.
[firestore] Oct 29, 2024 11:28:52 AM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[firestore] INFO: Detected HTTP/2 connection.
[firestore] Oct 29, 2024 11:28:52 AM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[firestore] INFO: Detected HTTP/2 connection.
[firestore] Oct 29, 2024 11:28:52 AM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[firestore] INFO: Detected HTTP/2 connection.
[firestore] Oct 29, 2024 11:28:53 AM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[firestore] INFO: Detected HTTP/2 connection.
[firestore] Oct 29, 2024 11:28:53 AM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[firestore] INFO: Detected HTTP/2 connection.
^C

Command killed by keyboard interrupt

[google-oss-bot]

This issue does not seem to follow the issue template. Make sure you provide all the required information.

[Nathanmalnoury]

To add some context to this issue, my company is based on ndb which is a google official client library for Google Cloud Datastore.

We've been using the legacy datastore emulator for local testing before deploying to pre-production and production projects. Data persistence between runs is crucial for our testing process, and I wasn't able to achieve with the firestore emulator.
The core issue is that the 2.3.0 release of ndb (1st March 2024) rendered the ndb client incompatible with the legacy emulator (described in this issue).
We find ourselves in an uncomfortable position where we can't update our ndb client because that would mean degrading our local setup to a point that would make it practically unusable.

I've spent some time reading the docs and trying to get the emulator to persist data, but I also may have missed something..

Thank you in advance for your time and help!

[aalej]

Hey @Nathanmalnoury, sorry to hear you encountered this issue and thanks for the thorough report. I appreciate you taking the time to put together an mcve! I was able to reproduce the behavior you mentioned, I'll raise this to our engineering team so that they can take a look.

As a temporary workaround, you could try using Import entities into the emulator and Export entities in the emulator to import/export data into/in the emulators.

[AlexisH]

Same issue, the data is not persisted, I don't see anything in the output dir. Was also previously using legacy datastore emulator, but I persisted the data in a different directory.

Ubuntu 22.04.5 LTS
cloud-datastore-emulator 2.3.1
cloud-firestore-emulator 1.19.9

gcloud emulators firestore start --database-mode=datastore-mode --export-on-exit=/home/me/_firestore --host-port=127.0.0.1:8081

[joehan]

Hey all, quick question to help us debug this issue - does the exported data show up if you run firebase emulators:export outputDir in a separate terminal while the emulators are running?

[AlexisH]

I just installed firebase CLI for testing purpose.
firebase emulators:export does not find the running emulator if I start it with gcloud emulators firestore start ...
If I start the emulator via firebase emulators:start --only firestore --project foo then the export command finds it.
If I start the emulator via firebase emulators:start --only firestore --project foo --export-on-exit outputdir then it correctly dumps to outputdir on exit.
But using this emulator I can't get my python app to work I get Database mode not set to Datastore Mode. If using the gCloud CLI, please specify the --database-mode flag. How can I set this flag using the firebase CLI ?