Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create migration for pickle fields to save data in JSON #159

Open
cmutel opened this issue Nov 28, 2023 · 0 comments
Open

Create migration for pickle fields to save data in JSON #159

cmutel opened this issue Nov 28, 2023 · 0 comments

Comments

@cmutel
Copy link
Member

cmutel commented Nov 28, 2023

Storing data in pickle fields is a security issue. We should use JSON with a datetime-aware encoder instead. To do this, we need to work with the constraints of SQLite (where we can't just add a column easily). We also need a script step added in updates, and a test fixture of a project directory (created with bw2io.backup_project_directory where we can apply the migration.

Data first needs to be loaded from the pickled columns and tested to make sure it can be serialized to JSON (i.e. no functions or executable code, just basic data types). If this condition is violated an error should be raised.

I think all the pickle fields are in https://github.com/brightway-lca/brightway2-data/blob/main/bw2data/backends/schema.py, but this needs to be checked.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant