Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement pandas v2.1.1 #886

Open
SarahJohnsonONS opened this issue Sep 12, 2023 · 0 comments
Open

Implement pandas v2.1.1 #886

SarahJohnsonONS opened this issue Sep 12, 2023 · 0 comments

Comments

@SarahJohnsonONS
Copy link
Contributor

SarahJohnsonONS commented Sep 12, 2023

Upgrading to pandas v2.1.0 raises the following Pyright errors:

/home/runner/work/csvcubed/csvcubed/src/csvcubed/inspect/sparql_handler/data_cube_repository.py
  /home/runner/work/csvcubed/csvcubed/src/csvcubed/inspect/sparql_handler/data_cube_repository.py:534:21 - error: Argument of type "ArrayLike | Unknown | Any" cannot be assigned to parameter "maybe_columnar_data" of type "PandasDataTypes" in function "pandas_input_to_columnar_optional_str"
    Type "ArrayLike | Unknown | Any" cannot be assigned to type "PandasDataTypes"
      Type "ExtensionArray" cannot be assigned to type "PandasDataTypes"
        "ExtensionArray" is incompatible with "DataFrame"
        "ExtensionArray" is incompatible with "Series"
        Type cannot be assigned to type "None" (reportGeneralTypeIssues)
/home/runner/work/csvcubed/csvcubed/src/csvcubed/utils/pandas.py
  /home/runner/work/csvcubed/csvcubed/src/csvcubed/utils/pandas.py:41:19 - error: Argument of type "Set[str]" cannot be assigned to parameter "na_values" of type "Sequence[str] | Mapping[str, Sequence[str]] | None" in function "read_csv"
    Type "Set[str]" cannot be assigned to type "Sequence[str] | Mapping[str, Sequence[str]] | None"
      "Set[str]" is incompatible with "Sequence[str]"
      "Set[str]" is incompatible with "Mapping[str, Sequence[str]]"
      Type cannot be assigned to type "None" (reportGeneralTypeIssues)
/home/runner/work/csvcubed/csvcubed/src/csvcubed/utils/qb/validation/observations.py
  /home/runner/work/csvcubed/csvcubed/src/csvcubed/utils/qb/validation/observations.py:134:78 - error: Cannot access member "isna" for type "ndarray[Any, Unknown]"
    Member "isna" is unknown (reportGeneralTypeIssues)
  /home/runner/work/csvcubed/csvcubed/src/csvcubed/utils/qb/validation/observations.py:134:78 - error: Cannot access member "isna" for type "NDArray[Unknown]"
    Member "isna" is unknown (reportGeneralTypeIssues)
  /home/runner/work/csvcubed/csvcubed/src/csvcubed/utils/qb/validation/observations.py:134:78 - error: Cannot access member "isna" for type "NDArray[Any]"
    Member "isna" is unknown (reportGeneralTypeIssues)
  /home/runner/work/csvcubed/csvcubed/src/csvcubed/utils/qb/validation/observations.py:141:62 - error: Cannot access member "index" for type "ndarray[Any, Unknown]"
    Member "index" is unknown (reportGeneralTypeIssues)
  /home/runner/work/csvcubed/csvcubed/src/csvcubed/utils/qb/validation/observations.py:141:62 - error: Cannot access member "index" for type "NDArray[Unknown]"
    Member "index" is unknown (reportGeneralTypeIssues)

See Other API changes for potential ideas to fix the Pyright errors.

Once fixed, revert to pandas-version: ['pandas@latest'] in the test_in_environments jobs in pull-request.yaml, release.yaml and main-push.yaml

Update 25/9/23

  • pandas v2.1.1 has already been released, so I've updated to this version.
  • Added np.ndarray and pd.api.extensions.ExtensionArray to PandasDataTypes in inputs.py which resolved the Type "ArrayLike | Unknown | Any" cannot be assigned to type "PandasDataTypes" / Type "ExtensionArray" cannot be assigned to type "PandasDataTypes" issue.
  • Converted SPECIFIED_NA_VALUES to a List in utils/pandas.py, and changed the type of na_values in def read_csv() to Sequence.

Still outstanding:

  • In utils/qb/validations/observations.py, still no solution for Cannot access member "isna" for type "ndarray[Any, Unknown]" and Cannot access member "index" for type "ndarray[Any, Unknown]" in _validate_missing_observation_values (L115)
@SarahJohnsonONS SarahJohnsonONS self-assigned this Sep 18, 2023
@NickPapONS NickPapONS changed the title Implement pandas v2.1.0 Implement pandas v2.1.1 Oct 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants