samples(sdmx): Add sample scripts for OECD and Eurostat #1600
Conversation
This commit adds a new SDMX tools module with utilities to fetch metadata and data from SDMX APIs. It also updates the requirements to include the sdmx1 package.
This commit refactors the SDMX utility functions to address review comments. It replaces print statements with logging, improves error handling by allowing exceptions to propagate, and adds type hints for better code clarity.
This commit addresses review comments by: - Renaming `utils.py` to `dataflow.py`. - Adding the request URL to the success logs. - Providing more context in error logs. - Generalizing a log message to be format-agnostic.
This commit adds a new `samples` directory to the `tools/sdmx` module. It includes scripts that demonstrate how to use the dataflow utilities to fetch data and metadata from both OECD and Eurostat, along with a README explaining how to run them.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @SandeepTuniki, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request significantly enhances the usability of the existing dataflow utilities within the "tools/sdmx" module by introducing a collection of practical, runnable Python scripts. These examples serve to demonstrate how to effectively interact with and retrieve economic data and metadata from various sources, specifically OECD and Eurostat APIs, providing clear guidance for users.
Highlights
- New Samples Directory: A dedicated "samples" directory has been added within "tools/sdmx" to house runnable example scripts.
- OECD Data Examples: Includes three new scripts demonstrating how to fetch OECD GDP metadata, a specific slice of GDP data, and a comprehensive example for fetching both full metadata and the entire GDP dataset.
- Eurostat Data Example: A new script has been added to showcase fetching a slice of GDP data from Eurostat, highlighting the multi-source capability of the dataflow utilities.
- Comprehensive Documentation: A "README.md" file is included in the "samples" directory, detailing each script's purpose and providing clear instructions on how to execute them.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request adds several helpful sample scripts for the SDMX utilities. The examples are clear and demonstrate the intended usage well. My review includes a few suggestions to improve the scripts' robustness and maintainability by standardizing how they are run, where they save output files, and how they handle errors. These changes will make the samples cleaner and more aligned with Python best practices.
| You can execute each script from the root of the repository, for example: | ||
|
|
||
| ```bash | ||
| python3 tools/sdmx/samples/fetch_oecd_gdp_metadata.py |
There was a problem hiding this comment.
To make the sample scripts more robust and avoid manipulating sys.path, it's better to execute them as Python modules from the project root using the -m flag. This is a standard practice that improves portability.
| python3 tools/sdmx/samples/fetch_oecd_gdp_metadata.py | |
| python3 -m tools.sdmx.samples.fetch_oecd_gdp_metadata |
| python3 tools/sdmx/samples/fetch_oecd_gdp_metadata.py | ||
| ``` | ||
|
|
||
| The scripts will download the requested data/metadata and save it as `.xml` or `.csv` files in the project's root directory. |
There was a problem hiding this comment.
Saving output files directly in the project's root directory can cause clutter. It would be cleaner to save them into a dedicated output directory within the tools/sdmx/samples/ folder. This keeps the project's root directory tidy.
| The scripts will download the requested data/metadata and save it as `.xml` or `.csv` files in the project's root directory. | |
| The scripts will download the requested data/metadata and save it as `.xml` or `.csv` files in the `tools/sdmx/samples/output/` directory. |
|
|
||
| import logging | ||
| import sys | ||
| import os |
There was a problem hiding this comment.
To handle specific network errors from the requests library, it's good practice to import HTTPError. This allows for more specific exception handling later in the script. This should be added to all sample scripts.
| import os | |
| import os | |
| from requests.exceptions import HTTPError |
| # Add the project root to the Python path | ||
| sys.path.append( | ||
| os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..', '..'))) |
There was a problem hiding this comment.
Modifying sys.path directly can be fragile and is generally discouraged. Since the README.md is being updated to run scripts as modules with the -m flag, this sys.path manipulation is no longer necessary and should be removed from all the sample scripts.
| # --- 1. Define Parameters for the Eurostat GDP Dataset --- | ||
| agency_id = "ESTAT" | ||
| dataflow_id = "TEC00001" | ||
| output_path = "eurostat_gdp_data.csv" |
There was a problem hiding this comment.
Saving output files to the project root can clutter the directory. It's better to create a dedicated output directory within the samples folder and save all generated files there. This approach should be applied consistently across all the new sample scripts, adjusting the final filename accordingly.
| output_path = "eurostat_gdp_data.csv" | |
| output_dir = os.path.join("tools", "sdmx", "samples", "output") | |
| os.makedirs(output_dir, exist_ok=True) | |
| output_path = os.path.join(output_dir, "eurostat_gdp_data.csv") |
| except Exception as e: | ||
| logging.error(f"Failed to download data. Error: {e}") |
There was a problem hiding this comment.
Catching a broad Exception can hide the specific cause of a failure. It's better to catch more specific exceptions first, like HTTPError for network-related problems, and then use a general Exception as a fallback. This makes error handling more robust and debugging easier.
This change should be applied to all sample scripts. Note that for fetch_oecd_full_gdp_dataset.py, you should keep the return statement inside the exception block for metadata fetching to ensure the script exits upon failure.
| except Exception as e: | |
| logging.error(f"Failed to download data. Error: {e}") | |
| except HTTPError as e: | |
| logging.error(f"Failed to download data due to a network error: {e}") | |
| except Exception as e: | |
| logging.error(f"An unexpected error occurred during data download: {e}") |
|
This PR is obsolete now. It is superseded by #1601 |
(Note: This PR is an alternate option for #1601. Only one of these is necessary)
This pull request introduces a
samplesdirectory within thetools/sdmxmodule to provide clear, runnable examples of how to use the dataflow utilities.Key Additions:
fetch_oecd_gdp_metadata.py: Demonstrates downloading metadata.fetch_oecd_gdp_data.py: Shows how to fetch a specific slice of data.fetch_oecd_full_gdp_dataset.py: A comprehensive example for fetching both metadata and a full dataset.fetch_eurostat_gdp_data.py: Demonstrates fetching data from Eurostat, showcasing the multi-source capability of the utilities.README.mdwithin thesamplesdirectory explaining the purpose of each script and how to execute them.