Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add build clarification to package guide #101

Merged
merged 8 commits into from
Nov 1, 2023
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/python-package-development-process.png
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This diagram is great! It captures the key ideas we need to convey here

I'm confused about why we show documentation going into the build distribution though? None of the tree diagrams below show documentation in the sdist or wheel

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh interesting! my lens was a bit different. i was that as just a bucket of packaging "things" and then you build. not "these are the things that go into the dist" BUT your comment makes sense that most people would see that as documentation going into the build. so we can revice that graphic to just say package code, metadata (and optionally?? tests) given it's all about publishing!! and maybe i have a docs arrow that goes to a website bubble??

i know the test thing is really controversial and it turns out this controversy is not just in our ecosystem!!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about this? i don't want to make this graphic too complex but essentially there are some different things that could go into those SDists vs wheels. i suspect that will get confusing for newer maintainers however.

Screen Shot 2023-09-25 at 3 50 51 PM

lwasser marked this conversation as resolved.
Show resolved Hide resolved
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
43 changes: 32 additions & 11 deletions package-structure-code/intro.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,41 +14,62 @@ best fitted for your workflow.
:gutter: 3

:::{grid-item-card}
:link: python-package-build-tools
:link: python-package-structure
:link-type: doc

Finding the right packaging tool(s)
1. Package file structure
^^^

Learn more about the suite of packaging tools out there.
And learn which tool might be best for you.
src layout, flat layout and where should tests folders live? No matter what your level of packaging knowledge is, this page will help you decide upon a package structure that follows modern python best practices.
:::

:::{grid-item-card}
:link: python-package-structure
:link-type: doc

Package file structure
2. Learn about building your package
^^^
src layout, flat layout and where should tests folders live? No matter what your level of packaging knowledge is, this page will help you decide upon a package structure that follows modern python best practices.
Building a Python package refers to the process of placing your package code
lwasser marked this conversation as resolved.
Show resolved Hide resolved
metadata (and tests) into a specific format that PyPI can read and that
your users can install. Learn more about building a Python package here.
:::

:::{grid-item-card}
:link: python-package-build-tools
:link-type: doc

✨ Publish to PyPI and Conda ✨
✨ 3. What Python package tool should you use? ✨
^^^

Learn more about the suite of packaging tools out there.
And learn which tool might be best for you.
:::

:::{grid-item-card}
:link: python-package-build-tools
:link-type: doc

✨ 4. Publish your package to PyPI and Conda ✨
^^^
If you have a pure python package, it's a straight forward
process to publish to both PyPI and then a Conda channel such as
conda-forge. Learn more here.
:::

:::{grid-item-card}
:link: python-package-versions
:link-type: doc

✨ 5. Setup package versioning ✨
^^^
Semver (numeric versioning) and Calver (versioning using the date) are 2
common ways to version a package. Which one should you pick? Learn more here.
:::

:::{grid-item-card}
:link: code-style-linting-format
:link-type: doc

✨ Code style & linters ✨
6. Code style & linters ✨
^^^
Black, blue, flake8, Ruff - which tools can help you ensure your
package follows best practices for code format? Learn more about the options and why this is important here.
Expand Down Expand Up @@ -134,8 +155,8 @@ if you are looking for pyOpenSci's Python package review requirements!
Intro <self>

Python package structure <python-package-structure>
pyproject.toml Package Metadata <pyproject-toml-python-package-metadata>
What are SDist & Wheel Files? <python-package-distribution-files-sdist-wheel>
pyproject.toml Package Metadata <pyproject-toml-python-package-metada>
Build Your Package <python-package-distribution-files-sdist-wheel>
Package Build Tools <python-package-build-tools>
Complex Builds <complex-python-package-builds>
```
Expand Down
4 changes: 3 additions & 1 deletion package-structure-code/python-package-build-tools.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
# Python Package Build Tools
# Python Packaging Tools

<!-- TODO: add a small discussion on what pinning is?-->

## Tools for building your package

There are a several different build tools that you can use to [create your Python package's _sdist_ and _wheel_ distributions](python-package-distribution-files-sdist-wheel). Below, we discuss the features,
benefits and limitations of the most commonly used Python packaging tools.
We focus on pure-python packages in this guide. However, we also
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,72 @@
# The Python Package Source and Wheel Distributions
# Learn about Building a Python Package

:::{figure-md} build-workflow
<img src="/images/python-package-development-process.png" alt="Alt tag to be added when image is final" width="700px">

You need to build your Python package in order to publish it to PyPI or Conda. For PyPI, the build process essentially organizes your code and metadata into a distribution format that PyPI can read and provide to potential users.
:::
lwasser marked this conversation as resolved.
Show resolved Hide resolved

## What is building a Python package?

To [publish your Python package](build-workflow) and make it easy for anyone to install, you first need to build it.

But, what does it mean to build a Python package?

[As shown in the figure above](build-workflow), when you build your Python package, you convert the source files into something called a distribution package. A distribution package contains your source code and metadata about the package, in the format required by the Python Package Index, so that it can be installed by tools like pip.

:::{note}
The term package used to mean many different things in Python and other languages. On this page, we adapt the convention of the [Python Packaging Authority](https://www.pypa.io/en/latest/) and refer to the product of the
build step as a **distribution package**.
:::

This process of organizing and formatting your
code, documentation, tests and metadata into a format that both pip
and PyPI can use, is called a build step.

### Project metadata and PyPI

For instance, when you publish to PyPI, you will notice that each package has metadata listed. Let’s have a look at [xclim](https://pypi.org/project/xclim/), one of our [pyOpenSci packages](https://www.pyopensci.org/python-packages.html). Notice that on the PyPI landing page you see some metadata about the package including python, maintainer information and more. PyPI is able to populate this metadata because it was defined using correct syntax and classifiers by Xclim's maintainers, [pyproject.toml file](https://github.com/Ouranosinc/xclim/blob/master/pyproject.toml). This metadata when the xclim package is built, is translated into a distribution file that allows PyPI to read the metadata and print it out on their website.

lwasser marked this conversation as resolved.
Show resolved Hide resolved
```{figure} ../images/python-build-package/pypi-metadata-classifiers.png
:scale: 50 %
:align: center
:alt: Image showing the left side bar of PiPy for the package xclim. The section at the top says Classifier. Below there is a list of items including Development status, intended audience, License, natural language, operating system, programming language and topic. Below each of those sections are various classifier options." width="300px">

When you add the classifier section to your pyproject.toml
and your package is built, the build tool organizes the metadata into a format that PyPI can understand and
represent on your pypi landing page. These classifiers also allow users to sort through packages by version of python they support, categories and more.
```

:::{figure-md} fig-target
<img src="../images/python-build-package/pypi-metadata-keywords-license.png" alt="t." width="700px">

:::

:::{figure-md} fig-target
<img src="../images/python-build-package/pypi-metadata-maintainers.png" alt="t." width="700px">

:::

## How to create the distribution format that PyPI and Pip expects?

You could in theory create your own scripts to organize your code the way PyPI wants it to be. However, just like there are packages that handle known structures such as Pandas for data frames and Numpy for arrays, there are packages and tools that help you create package build distribution files.

```{note}

There are a suite of packaging tools that can either help you with
the entire packaging process or just one step of the process. For instance
setuptools is a commonly used build back end that can be used to create your
SDist and wheel. Whereas tools like Hatch, PDM, Poetry and flit help with other
parts of the packaging process.

While this can cause some confusion and
complexity in the packaging ecosystem - for the most part, each tool provides
the same distribution output (with minor differences that most users may not
care about). Learn more about those tools on this page.
```

Below, you will learn about the two distribution files that PyPI expects you to publish: SDist and Wheel. You will learn about
their structure and what files belong in each.

There are two core distribution files
that you need to create to publish your Python package to
Expand Down
Loading