feat(tags): Export and Import Functionality for Superset Dashboards and Charts

[asher-lab]

SUMMARY

This PR is related to Discussion: #30629 by Spens

Motivation

We have adopted the tagging functionality and it is working quite well for us for dashboards and and charts. Our use case and process requires us to build dashboards on a development server, export the dashboards, commit the YAMLs to source control, and finally import the ZIPs to our production server. We would like the tags we assign on the development server to be captured in the export so they are replicated to our production server.

Proposed Change

UI elements are not affected by this change. All changes are related to updated logic/processing of the existing export and import functionality.

Only tags of type "custom" are exported. Other tags that exist but are not exported are "type" and "owner".

Export Logic
When an object (Dashboard/Chart) is exported, a new parameter is added to the exported YAML file. The parameter is "tags" and the value is an array of tag names. Here is an example:

slice_name: Vaccine Candidates per Approach & Stage
description: null
tags:
  - CovidTag
  - Items that start with the letter V
  - Superset Example Data
certified_by: null

...
Because tags have additional information (for example, a description), a new file is needed to hold this information. The file is called "tags.yaml" and lives in the top level of the zip file. For example:

dashboard_export_20241017T025146.zip
  charts (folder)
  dashboards (folder)
  databases (folder)
  datasets (folder)
  metadata.yaml (file)
  tags.yaml (file)

The format of the tags.yaml file is an array of tags

tags:
- tag_name: CovidTag
  description: Everything related to Covid
- tag_name: Items that start with the letter V
  description: Just for fun, tag Vanilla, Vikings, and anything that starts with V
- tag_name: Superset Example Data
  description: Native example data included with Superset

Import Logic
When an object is imported, the code checks to see if any tags included in the import file.
If yes, the tag is created if it does not already exist then the tag is assigned to the imported object. If the tag already exists, it is just assigned with no changes to the tag.

Details

1. A tag can only be created if the user doing the import has permission to create tags. Otherwise the tagging is ignored.
2. A created tag picks up the current time as the "created_on" and "changed_on" properties for the tag
3. A created tag picks up the current user as the "created_by_fk" and "changed_by_fk"
4. A created tag picks up the description from the tags.yaml file described above
5. If a tag already exists, the description is not updated (for consistency with other export/import behavior - for example, if you import a dashboard and the chart included in the zip already exists, Superset does not update that existing chart)
   Existing Superset behavior: When importing objects that already exist, the user is prompted whether it is ok to overwrite. If no, the import process aborts. If yes, the old object is replaced with the new object.

To be consistent with this behavior, when importing an object and specifying "overwrite", old tags that are no longer specified with the object are removed, new tags that didn't originally exist are created, old tags that still remain in the new set of tags are retained (descriptions are not updated for example)

1. It is important to note that in this proposal, imported tags are not added to existing tags
   This is intentional because for the use case of trying to mirror the settings and properties from one system to another, you must have the ability to remove tags from the target system. Otherwise, for example, if you rename a tag, the new name would get added and the old name would stick around which is not the desired behavior.

Thank you to anyone taking the time to read and comment!

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[michael-s-molina]

Thank you for the PR @asher-lab. Could you add unit tests for the new feature with the feature flag on and off?

[asher-lab]

Good day, @michael-s-molina. We've added logic to handle cases when the Tagging System is disabled in Superset. When disabled, the tags.yaml file and any associated tags within charts.yaml and dashboard.yaml will not be exported, and the same applies to the import operation.

Could you provide guidance on creating a unit test for this new feature? I have limited experience with writing unit tests, so any help would be appreciated.

Thank you!

[michael-s-molina]

Could you provide guidance on creating a unit test for this new feature? I have limited experience with writing unit tests, so any help would be appreciated.

Hi @asher-lab. You can check this file for examples.

[asher-lab]

Hi @michael-s-molina . Added unit tests for the new feature with the feature flag TAGGING_SYSTEM on and off. Thank you.

[codecov]

Codecov Report

Attention: Patch coverage is 64.45783% with 59 lines in your changes missing coverage. Please review.

Project coverage is 83.76%. Comparing base (76d897e) to head (094770e).
Report is 968 commits behind head on master.

Files with missing lines	Patch %	Lines
superset/commands/tag/export.py	38.46%	32 Missing ⚠️
...perset/commands/dashboard/importers/v1/__init__.py	50.00%	7 Missing ⚠️
superset/commands/importers/v1/utils.py	82.92%	7 Missing ⚠️
superset/commands/chart/export.py	75.00%	4 Missing ⚠️
superset/commands/chart/importers/v1/__init__.py	66.66%	4 Missing ⚠️
superset/commands/dashboard/export.py	72.72%	3 Missing ⚠️
superset/commands/dataset/importers/v1/__init__.py	75.00%	1 Missing ⚠️
superset/commands/importers/v1/examples.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #30833       +/-   ##
===========================================
+ Coverage   60.48%   83.76%   +23.27%     
===========================================
  Files        1931      537     -1394     
  Lines       76236    39106    -37130     
  Branches     8568        0     -8568     
===========================================
- Hits        46114    32758    -13356     
+ Misses      28017     6348    -21669     
+ Partials     2105        0     -2105     

Flag	Coverage Δ
hive	48.75% <33.13%> (-0.42%)	⬇️
javascript	?
mysql	76.58% <43.37%> (?)
postgres	76.68% <43.37%> (?)
presto	53.21% <33.13%> (-0.60%)	⬇️
python	83.76% <64.45%> (+20.27%)	⬆️
sqlite	76.14% <43.37%> (?)
unit	60.78% <54.21%> (+3.16%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[asher-lab]

I seen a bunch of tests: 10 are failing and 23 are successful. Could you provide what tests that I need to look at?

[Vitor-Avila]

Thank you so much @asher-lab! This is super cool and very useful 😍 🙌 I left a few comments, and I plan to give it another pass Tomorrow.

[asher-lab]

Hi @Vitor-Avila , we have fixed the integration test issue, updated it to use "tags" instead of "tag" and update filter using TagType. Thank you!

[asher-lab]

Hi @Vitor-Avila and @michael-s-molina, if these are merged in, when is the expected date it be released?

[rusackas]

Running CI 🤞 This would definitely be part of 5.0, which needs a new estimated date. It will have missed the boat for 4.1.0, but could make a 4.2.0, though it's unclear if/when that would happen. It would not make a 4.1.1 release since that would only include fixes. Feel free to join us on slack in the release strategy or release announcement channels, if you want to get involved with the process.

[asher-lab]

Currently fixing the tests that are failing. Will be making a commit in order to fix those. Thank you. I realized that we need to work with pre-commit checks and integration tests...

[asher-lab]

Good morning team, @rusackas @Vitor-Avila and @michael-s-molina. Could I get another run for this one please. I did run it on my own fork and all tests have passed. 🤞

[Vitor-Avila]

hey @asher-lab,
Sorry for the delay on my end. Approved the workflows and hoping to check it today

[betodealmeida]

Taking a look at this, but in the meantime a few comments:

1. I like the approach of having tags.yml in the ZIP, I think it aligns super well with the intent of the import/export flow.
2. Using inspect to figure out the caller makes me nervous, let's see if we can find a better solution.
3. Every time we change import/export we need to make sure that (1) old exports still work, and (2) new exports work in older version — that is, everything should be backwards and forwards compatible, ideally.

[Vitor-Avila]

Taking a look at this, but in the meantime a few comments:

1. I like the approach of having tags.yml in the ZIP, I think it aligns super well with the intent of the import/export flow.
2. Using inspect to figure out the caller makes me nervous, let's see if we can find a better solution.
3. Every time we change import/export we need to make sure that (1) old exports still work, and (2) new exports work in older version — that is, everything should be backwards and forwards compatible, ideally.

Thanks @betodealmeida! I can help testing 3 if needed. I agree with 1 and 2 - I'm wondering if there's a better way to handle this flow.

Another consideration is if we should adopt the similar uuid approach for tags, and also allow exporting tags via the UI (as an individual object type), and then the uuid would dictate if the tag gets updated or created. I don't think it's worth in the current scope, but if we end up expanding its features (for example, the ability to assign permissions to tags, etc) might be beneficial. This can also be re-evaluated if we ever get there

[asher-lab]

Hello @betodealmeida , we have refrained from the use of inspect. Kindly let me know your comments on this. Looking forward for the next steps. cc @Vitor-Avila requesting to run the check again please 😊 Thank you.