[SPARK-48720][SQL] Align the command `ALTER TABLE ... UNSET TBLPROPERTIES ...` in v1 and v2 #47097

panbingkun · 2024-06-26T07:54:04Z

What changes were proposed in this pull request?

The pr aims to:

align the command ALTER TABLE ... UNSET TBLPROPERTIES ... in v1 and v2.
(this means that in the v1, regardless of whether IF EXISTS is specified or not, when unset a non-existent property, it is ignored and no longer fails.)
update the description of ALTER TABLE ... UNSET TBLPROPERTIES ... in the doc docs/sql-ref-syntax-ddl-alter-table.md.
unify v1 and v2 ALTER TABLE ... UNSET TBLPROPERTIES ... tests.
Add the following scenario for ALTER TABLE ... SET TBLPROPERTIES ... testing
A.table to alter does not exist
B.alter table set reserved properties

Why are the changes needed?

align the command ALTER TABLE ... UNSET TBLPROPERTIES ... in v1 and v2, avoid confusing end-users.
to improve test coverage.
align with other similar tests, eg: AlterTableSetTblProperties*

Does this PR introduce any user-facing change?

Yes, in the v1, regardless of whether IF EXISTS is specified or not, when unset a non-existent property, it is ignored and no longer fails

How was this patch tested?

Update some UT & Pass GA.

Was this patch authored or co-authored using generative AI tooling?

No.

… TBLPROPERTIES ...` in v2

panbingkun · 2024-06-27T11:48:01Z

In Unify v1 and v2 ALTER TABLE .. UNSET TBLPROPERTIES tests migration, I found syntax:

ALTER TABLE ... UNSET TBLPROPERTIES IF EXISTS ('key1');

v1:

If the property `key1` of the table do not exist, it will throw exception `UNSET_NONEXISTENT_PROPERTIES`

v2 :

If the property `key1` of the table do not exist,  it will ignore the property `key1` (non-existent) and finally succeed (slience)

I originally planned to align its logic, which was to throw the same exception UNSET_NONEXISTENT_PROPERTIES as v1. However, based on the previous discussion in [SPARK-48668][SQL] Support ALTER NAMESPACE ... UNSET PROPERTIES in v2 #47038, it seems that we can keep the current logic and only reflect this difference in UT, while also note it in the document:

spark/docs/sql-ref-syntax-ddl-alter-table.md

Line 244 in baf461b

`ALTER TABLE UNSET` is used to drop the table property.

@cloud-fan What do you think?

cloud-fan · 2024-07-01T08:26:23Z

Yea UNSET non-existing property should be noop. BTW we should throw table not found error if table does not exist.

panbingkun · 2024-07-02T00:42:46Z

Okay, let me update it.

panbingkun · 2024-07-02T07:23:14Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala

@@ -1075,25 +1075,6 @@ class DDLParserSuite extends AnalysisTest {
        ifExists = true))
  }

-  // ALTER TABLE table_name UNSET TBLPROPERTIES [IF EXISTS] ('comment', 'key');


The tests have been migrated to AlterTableUnsetTblPropertiesParserSuite

panbingkun · 2024-07-02T07:27:21Z

sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLParserSuite.scala

@@ -107,18 +107,6 @@ class DDLParserSuite extends AnalysisTest with SharedSparkSession {
        stop = 98))
  }

-  test("alter table unset properties - property values must NOT be set") {


The test has been migrated to AlterTableUnsetTblPropertiesParserSuite

panbingkun · 2024-07-02T07:36:30Z

sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala

-    sql("ALTER TABLE tab1 UNSET TBLPROPERTIES IF EXISTS ('c', 'xyz')")
-    assert(getProps == Map("x" -> "y"))
-  }
-


The above test was migrated to AlterTableUnsetTblPropertiesSuite* and split into table to alter does not exist, alter table unset properties and alter table unset non-existent properties, as alter table unset non-existent properties behaves differently in v1 and v2.

panbingkun · 2024-07-02T07:40:03Z

...test/scala/org/apache/spark/sql/execution/command/v1/AlterTableUnsetTblPropertiesSuite.scala

+      // property to unset does not exist
+      checkError(
+        exception = intercept[AnalysisException] {
+          sql(s"ALTER TABLE $t UNSET TBLPROPERTIES ('k3', 'k4')")


This scenario uses different behaviors in v1 and v2.

panbingkun · 2024-07-02T07:41:12Z

@cloud-fan It's ready now, thanks.

cloud-fan · 2024-07-02T14:30:45Z

docs/sql-ref-syntax-ddl-alter-table.md


-- Unset Table Properties
+**Note:** If the specified property key does not exist, when you use the v1 command and do not specify `IF EXISTS`,
+it will throw the error-condition `UNSET_NONEXISTENT_PROPERTIES` and finally `failed`,


shall we change the v1 behavior as well? It's not a breaking change to change from failure to no failure.

and we should avoid mentioning v1/v2 in the doc. It's an internal concept and should not be exposed to end users

shall we change the v1 behavior as well? It's not a breaking change to change from failure to no failure.

Sure, let me update it.

and we should avoid mentioning v1/v2 in the doc. It's an internal concept and should not be exposed to end users

Okay.

shall we change the v1 behavior as well? It's not a breaking change to change from failure to no failure.

Done.

cloud-fan · 2024-07-03T07:37:26Z

docs/sql-ref-syntax-ddl-alter-table.md

+##### Syntax
+
+```sql
+-- Unset Properties
 ALTER TABLE table_identifier UNSET TBLPROPERTIES [ IF EXISTS ] ( key1, key2, ... )


let's remove [ IF EXISTS ] as a way to deprecate. It's no longer effective.

LuciferYang

+1, LGTM

…TIES ...` in v1 and v2 ### What changes were proposed in this pull request? The pr aims to: - align the command `ALTER TABLE ... UNSET TBLPROPERTIES ...` in v1 and v2. (this means that in the v1, regardless of whether `IF EXISTS` is specified or not, when unset a `non-existent` property, it is `ignored` and no longer `fails`.) - update the description of `ALTER TABLE ... UNSET TBLPROPERTIES ...` in the doc `docs/sql-ref-syntax-ddl-alter-table.md`. - unify v1 and v2 `ALTER TABLE ... UNSET TBLPROPERTIES ...` tests. - Add the following `scenario` for `ALTER TABLE ... SET TBLPROPERTIES ...` testing A.`table to alter does not exist` B.`alter table set reserved properties` ### Why are the changes needed? - align the command `ALTER TABLE ... UNSET TBLPROPERTIES ...` in v1 and v2, avoid confusing end-users. - to improve test coverage. - align with other similar tests, eg: `AlterTableSetTblProperties*` ### Does this PR introduce _any_ user-facing change? Yes, in the `v1`, regardless of whether `IF EXISTS` is specified or not, when unset a `non-existent` property, it is `ignored` and no longer `fails` ### How was this patch tested? Update some UT & Pass GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#47097 from panbingkun/alter_unset_table. Authored-by: panbingkun <[email protected]> Signed-off-by: yangjie01 <[email protected]>

[SPARK-48720][SQL] Fix IF EXISTS ignored for `ALTER TABLE ... UNSET…

d5d1f9b

… TBLPROPERTIES ...` in v2

github-actions bot added the SQL label Jun 26, 2024

panbingkun added 2 commits June 26, 2024 18:48

Merge branch 'master' into alter_unset_table

88e4a30

Merge branch 'master' into alter_unset_table

31cbde1

panbingkun added 3 commits July 2, 2024 09:38

Merge branch 'master' into alter_unset_table

1d3b582

update

0de8ddf

update

5733fbb

github-actions bot added the DOCS label Jul 2, 2024

panbingkun changed the title ~~[SPARK-48720][SQL] Fix IF EXISTS ignored for ALTER TABLE ... UNSET … TBLPROPERTIES ... in v2~~ [SPARK-48720][SQL] Explain the diff between command ALTER TABLE ... UNSET TBLPROPERTIES ... in v1 and v2 in doc docs/sql-ref-syntax-ddl-alter-table.md. Jul 2, 2024

panbingkun changed the title ~~[SPARK-48720][SQL] Explain the diff between command ALTER TABLE ... UNSET TBLPROPERTIES ... in v1 and v2 in doc~~ [SPARK-48720][SQL] Clarify the command ALTER TABLE ... UNSET TBLPROPERTIES ... diff between v1 and v2 in doc Jul 2, 2024

panbingkun commented Jul 2, 2024

View reviewed changes

panbingkun marked this pull request as ready for review July 2, 2024 07:41

cloud-fan reviewed Jul 2, 2024

View reviewed changes

panbingkun added 2 commits July 3, 2024 08:44

Merge branch 'master' into alter_unset_table

4d5df8b

remove ifExists logical

9a8dcf3

panbingkun changed the title ~~[SPARK-48720][SQL] Clarify the command ALTER TABLE ... UNSET TBLPROPERTIES ... diff between v1 and v2 in doc~~ [SPARK-48720][SQL] Align the command ALTER TABLE ... UNSET TBLPROPERTIES ... in v1 and v2 Jul 3, 2024

panbingkun changed the title ~~[SPARK-48720][SQL] Align the command ALTER TABLE ... UNSET TBLPROPERTIES ... in v1 and v2~~ [SPARK-48720][SQL] Align the command ALTER TABLE ... UNSET TBLPROPERTIES ... in v1 and v2 Jul 3, 2024

fix

fe674fe

cloud-fan reviewed Jul 3, 2024

View reviewed changes

cloud-fan approved these changes Jul 3, 2024

View reviewed changes

panbingkun added 2 commits July 3, 2024 16:00

update docs/sql-ref-syntax-ddl-alter-table.md

1c0c260

Merge branch 'master' into alter_unset_table

a964a0f

LuciferYang approved these changes Jul 8, 2024

View reviewed changes

LuciferYang closed this in 73126b1 Jul 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-48720][SQL] Align the command `ALTER TABLE ... UNSET TBLPROPERTIES ...` in v1 and v2 #47097

[SPARK-48720][SQL] Align the command `ALTER TABLE ... UNSET TBLPROPERTIES ...` in v1 and v2 #47097

panbingkun commented Jun 26, 2024 •

edited

Loading

panbingkun commented Jun 27, 2024 •

edited

Loading

cloud-fan commented Jul 1, 2024 •

edited

Loading

panbingkun commented Jul 2, 2024

panbingkun Jul 2, 2024

panbingkun Jul 2, 2024

panbingkun Jul 2, 2024

panbingkun Jul 2, 2024

panbingkun commented Jul 2, 2024

cloud-fan Jul 2, 2024

cloud-fan Jul 2, 2024

panbingkun Jul 2, 2024 •

edited

Loading

panbingkun Jul 2, 2024

panbingkun Jul 3, 2024

cloud-fan Jul 3, 2024

panbingkun Jul 3, 2024

LuciferYang left a comment

[SPARK-48720][SQL] Align the command ALTER TABLE ... UNSET TBLPROPERTIES ... in v1 and v2 #47097

[SPARK-48720][SQL] Align the command ALTER TABLE ... UNSET TBLPROPERTIES ... in v1 and v2 #47097

Conversation

panbingkun commented Jun 26, 2024 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

panbingkun commented Jun 27, 2024 • edited Loading

cloud-fan commented Jul 1, 2024 • edited Loading

panbingkun commented Jul 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

panbingkun commented Jul 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

panbingkun Jul 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LuciferYang left a comment

Choose a reason for hiding this comment

[SPARK-48720][SQL] Align the command `ALTER TABLE ... UNSET TBLPROPERTIES ...` in v1 and v2 #47097

[SPARK-48720][SQL] Align the command `ALTER TABLE ... UNSET TBLPROPERTIES ...` in v1 and v2 #47097

panbingkun commented Jun 26, 2024 •

edited

Loading

panbingkun commented Jun 27, 2024 •

edited

Loading

cloud-fan commented Jul 1, 2024 •

edited

Loading

panbingkun Jul 2, 2024 •

edited

Loading