Why DOLT_DIFF_SUMMARY shows record without data change and schema change?

[mateusz-lachowski-codilime]

Dolt version: v1.47.1

I've encounter following issue using select * from dolt_diff_summary(...) because in some cases it's returning records with both data_change and schema_change set to false. For example:

SELECT * FROM DOLT_DIFF_SUMMARY('5ps4828ppa912injf4vj89h9meaarepu', '76q0rsmorlg35o64uf87d37n2hbiae43')

|from_table_name              |to_table_name                |diff_type|data_change|schema_change|
|-----------------------------|-----------------------------|---------|-----------|-------------|
|dcim_location                |dcim_location                |modified |1          |0            |
|extras_configcontext_clusters|extras_configcontext_clusters|modified |0          |0            |

But other methods return "proper" data:

SELECT * FROM DOLT_DIFF_STAT('5ps4828ppa912injf4vj89h9meaarepu', '76q0rsmorlg35o64uf87d37n2hbiae43')

|table_name         |rows_unmodified|rows_added|rows_deleted|rows_modified|cells_added|cells_deleted|cells_modified|old_row_count|new_row_count|old_cell_count|new_cell_count|
|-------------------|---------------|----------|------------|-------------|-----------|-------------|--------------|-------------|-------------|--------------|--------------|
|dcim_location      |0              |0         |0           |1            |0          |0            |3             |1            |1            |22            |22            |

And dolt diff return nothing for specified table, either on main branch activated or new branch.

SELECT * FROM DOLT_DIFF('5ps4828ppa912injf4vj89h9meaarepu', '76q0rsmorlg35o64uf87d37n2hbiae43', 'extras_configcontext_clusters')

select * from dolt_diff_extras_configcontext_clusters

I've found out that this is happening especially when comparing commit hashes from two different branches. In above case, 5ps commit is the last commit on main branch and 76q is the first and only commit on newly created branch out of main. When I'm comparing two commits from the same branch results are correct.

Expected results: all of the results from calling dolt_diff_summary should be with either data_change or schema_change flag set to true or not shown at all.

[timsehn]

Interesting. This means the actual hash of the table is different but Dolt can't display the change. There are some "hidden" elements in table schema called tags.

Is your database able to be shared?

[nicktobey]

Hey Mateusz!

I agree this is confusing, and we should figure out what's happening.

I have a theory what might be causing this. Does your extras_configcontext_clusters table use an auto-incrementing primary key? Tables in Dolt store the highest auto-incrementing primary key they've generated, in order to guarantee that it doesn't generate the same primary key twice even if the original row gets deleted. This stored value affects the table's hash despite not being part of either the schema or the data. If this value and only this value were to change (for instance, by inserting a new row and then immediately deleting it), it would cause the table to report as modified despite there being no schema change and no data change.

[nicktobey]

We released v.1.50.1 today, which should have a fix for this issue. Please let us know if the issue persists after updating.

[mateusz-lachowski-codilime]

Hi :) Thanks for quick response on that. It seems that v.1.50.1 resolve issue for newly created branches. Update was done via rebuilding the Docker image with newer version of Dolt, but volume data was preserved.

I followed the path of creating a new branch, adding some change and then running dolt_diff_summary and now summary is correct.

But when I'm querying the old "commits", for example the same from the first post of this issue, the results are still the same. Maybe it's ok and fix is only for "future" history?

[nicktobey]

That's odd. The change only affects how the data is read, not how it's stored, so it shouldn't matter whether the branch was created before or after the fix.

However, the code path for accessing table history is slightly different from the code path for accessing the live data on a branch, so it's possible that could be affecting things. Can you give me an example of a query that is now working, alongside an example query that's still broken for you?

[bpf120]

@mateusz-lachowski-codilime , thanks for using Dolt! We'd love to learn more about your use case. Feel free to email me or swing by our Discord if you want to share.