[MohammadHijjawi97]

To be able to merge a pull request, there are a few checks:
[x] the black style should be enforced (make lint)
[ ] the docs should be regenerated (make docs)
[ ] the pull request should pass the tests (make test)
[ ] the examples should be generated (make examples)

Problem

word_summary_vc splits values on whitespace and strips punctuation from each token. Tokens made only of punctuation (-, --, ..., &) become empty strings and are counted as a word. The empty entry then shows up in the Words table of Categorical and Text columns, often at the top:

word_summary_vc(pd.Series(["hello - world", "foo -- bar!", "a ... b"]).value_counts())["word_counts"]
# '' -> 3, hello -> 1, world -> 1, ...

Solution

Drop empty tokens after stripping, before counting.

Tests

Added test_word_summary_vc_ignores_punctuation_only_tokens in tests/unit/test_pandas/test_describe_categorical_pandas.py; it fails before the change and passes after. Ran tests/unit/test_pandas and tests/unit/test_describe.py.

[fabclmnt]

Thanks for the contribution!
Yes indeed, punctuation should not be included and the fix works for “-” and “...”. The root cause is that string. Nevertheless, punctuation is ASCII-only, so Unicode punctuation (em/en dashes, “…”, « », “ ”) still ends up as standalone “words” — “Star Wars — Episode IV” reproduces the original bug.

Could you strip with a Unicode-aware pattern instead, e.g. .str.replace(r"^[\W_]+|[\W_]+$", "", regex=True) and then drop empty tokens? That keeps inner punctuation (“don’t”, “e-mail”) as today.

Please also add test cases for “—”, “–”, “…” and a column where every value is punctuation-only (expect no word_counts), and update the TODO: remove punctuation

[MohammadHijjawi97]

Thanks for the review. I switched the stripping to .str.replace(r"^[\W_]+|[\W_]+$", "", regex=True) and kept dropping empty tokens, so inner punctuation like "don't" and "e-mail" is unchanged, and I replaced the TODO with a comment.

I added tests for —, –, …, « », “ ” and for punctuation-only columns (no word_counts). They fail on the old code and pass now, and tests/unit passes apart from a parquet test that needs pyarrow, which passed once I installed it.

One thing to flag: \W also matches combining marks, so words ending in one are truncated (Hindi नमस्ते becomes नमस्त, and Thai is affected too), and emoji-only tokens are no longer counted. Let me know if you'd like that handled differently.