Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiple conflicting extras installed when using explicit index assignments #9289

Open
anaoum opened this issue Nov 20, 2024 · 4 comments
Open
Assignees
Labels
bug Something isn't working

Comments

@anaoum
Copy link

anaoum commented Nov 20, 2024

When conflicting extra groups share a dependant, both extra groups seem to be installed.

Consider the following example adapted from the docs:

[project]
name = "project"
version = "0.1.0"
requires-python = ">=3.11,<3.12"
dependencies = []

[project.optional-dependencies]
cpu = [
  "torch>=2.5.1",
  "torchvision>=0.20.1",
]
cu124 = [
  "torch>=2.5.1",
  "torchvision>=0.20.1",
]
metrics = [
  "torchmetrics>=1.2.0",
]

[tool.uv]
conflicts = [
  [
    { extra = "cpu" },
    { extra = "cu124" },
  ],
]

[tool.uv.sources]
torch = [
  { index = "pytorch-cpu", extra = "cpu", marker = "platform_system != 'Darwin'" },
  { index = "pytorch-cu124", extra = "cu124" },
]
torchvision = [
  { index = "pytorch-cpu", extra = "cpu", marker = "platform_system != 'Darwin'" },
  { index = "pytorch-cu124", extra = "cu124" },
]

[[tool.uv.index]]
name = "pytorch-cpu"
url = "https://download.pytorch.org/whl/cpu"
explicit = true

[[tool.uv.index]]
name = "pytorch-cu124"
url = "https://download.pytorch.org/whl/cu124"
explicit = true

The key difference is the additional extra metrics that includes the torchmetrics package that depends on torch.

When I lock and sync both cpu and metrics extras, I get multiple versions of torch installed:

$ uv sync --extra cpu --extra metrics
Resolved 34 packages in 1ms
Uninstalled 3 packages in 199ms
Installed 3 packages in 1.10s
 ~ torch==2.5.1
 ~ torch==2.5.1+cpu
 ~ torch==2.5.1+cu124
@charliermarsh
Copy link
Member

charliermarsh commented Nov 20, 2024

Interesting, thanks. I don't think the extra indexes are relevant, since you can reproduce with:

[project]
name = "project"
version = "0.1.0"
requires-python = ">=3.11,<3.12"
dependencies = []

[project.optional-dependencies]
foo = [
  "idna==3.10",
]
bar = [
  "idna==3.9",
]
baz = [
  "anyio",
]

[tool.uv]
conflicts = [
  [
    { extra = "foo" },
    { extra = "bar" },
  ],
]

From there, uv tree gives you:

project v0.1.0
├── idna v3.9 (extra: bar)
├── anyio v4.6.2.post1 (extra: baz)
│   ├── idna v3.9
│   ├── idna v3.10
│   └── sniffio v1.3.1
└── idna v3.10 (extra: foo)

@charliermarsh charliermarsh added the bug Something isn't working label Nov 20, 2024
@anaoum
Copy link
Author

anaoum commented Nov 20, 2024

Thanks. It also happens if the two conflicting extra groups share the common dependant:

[project]
name = "project"
version = "0.1.0"
requires-python = ">=3.11,<3.12"
dependencies = []

[project.optional-dependencies]
foo = [
  "idna==3.10",
  "anyio",
]
bar = [
  "idna==3.9",
  "anyio",
]

[tool.uv]
conflicts = [
  [
    { extra = "foo" },
    { extra = "bar" },
  ],
]

@BurntSushi
Copy link
Member

I think i see a path to fixing this. I think it has two components:

  • In ResolverOutput::from_state (and possibly also marker_reachability), generate extra = "..." markers from each Resolution's conflicting groups, if they have them. Then apply those markers via intersection to each of the edges in that Resolution. So in Charlie's example above, that means the idna==3.9 dependency would have a extra == "bar" marker and the idna==3.10 dependency would have a extra == "foo" or extra == "baz" marker. Or something similar.
  • Update InstallTarget::to_resolution in uv-resolver/lock to pass in extras when doing marker evaluation.

One hiccup here is that this doesn't take groups into account, so we might not be able to reuse markers for this. We'll instead need an additional field to encode this extra/group inclusion logic.

@BurntSushi
Copy link
Member

BurntSushi commented Nov 21, 2024

Working it out by hand, I think the conditional logic for each idna dependency in each of the three forks present in this example is:

 idna==3.9: extra != "foo" and (extra == "bar" or extra == "baz")
idna==3.10: extra != "bar" and (extra == "foo" or extra == "baz")
idna==3.10: (extra != "foo" and extra != "bar") and extra == "baz"

Which combines to:

 idna==3.9: extra != "foo" and (extra == "bar" or extra == "baz")
idna==3.10: (extra != "bar" and (extra == "foo" or extra == "baz"))
            or ((extra != "foo" and extra != "bar") and extra == "baz")

And simplifies to:

 idna==3.9: extra != "foo" and (extra == "bar" or extra == "baz")
idna==3.10: (extra != "bar" and extra == "baz") or (extra != "bar" and extra == "foo")

Note that because of how extras work, the above notation is quite misleading. It's more clearly written like this:

 idna==3.9: "foo" not in extras and ("bar" in extras or "baz" in extras)
idna==3.10: ("bar" not in extras and "baz" in extras) or ("bar" not in extras and "foo" in extras)

BurntSushi added a commit that referenced this issue Nov 22, 2024
When we generate conflict markers for each resolution after the
resolver runs, it turns out that generating them just from exclusion
rules is not sufficient.

For example, if `foo` and `bar` are declared as conflicting extras, then
we end up with the following forks:

    A: extra != 'foo'
    B: extra != 'bar'
    C: extra != 'foo' and extra != 'bar'

Now let's take an example where these forks don't share the same version
for all packages. Consider a case where `idna==3.9` is in forks A and C,
but where `idna==3.10` is in fork B. If we combine the markers in forks
A and C through disjunction, we get the following:

     idna==3.9: extra != 'foo' or (extra != 'foo' and extra != 'bar')
    idna==3.10: extra != 'bar'

Which simplifies to:

     idna==3.9: extra != 'foo'
    idna==3.10: extra != 'bar'

But these are clearly not disjoint. Both dependencies could be selected,
for example, when neither `foo` nor `bar` are active. We can remedy this
by keeping around the inclusion rules for each fork:

    A: extra != 'foo' and extra == 'bar'
    B: extra != 'bar' and extra == 'foo'
    C: extra != 'foo' and extra != 'bar'

And so for `idna`, we have:

     idna==3.9: (extra != 'foo' and extra == 'bar') or (extra != 'foo' and extra != 'bar')
    idna==3.10: extra != 'bar' and extra == 'foo'

Which simplifies to:

     idna==3.9: extra != 'foo'
    idna==3.10: extra != 'bar' and extra == 'foo'

And these *are* properly disjoint. There is no way for them both to be
active. This also correctly accounts for fork C where neither `foo` nor
`bar` are active, and yet, `idna==3.9` is still enabled but `idna==3.10`
is not. (In the [motivating example], this comes from `baz` being enabled.)
That is, this captures the idea that for `idna==3.10` to be installed,
there must actually be a specific extra that is enabled. That's what
makes it disjoint from `idna==3.9`.

We aren't quite done yet, because this does add *too many* conflict
markers to dependency edges that don't need it. In the next commit,
we'll add in our world knowledge to simplify these conflict markers.

[motivating example]: #9289
BurntSushi added a commit that referenced this issue Nov 23, 2024
When we generate conflict markers for each resolution after the
resolver runs, it turns out that generating them just from exclusion
rules is not sufficient.

For example, if `foo` and `bar` are declared as conflicting extras, then
we end up with the following forks:

    A: extra != 'foo'
    B: extra != 'bar'
    C: extra != 'foo' and extra != 'bar'

Now let's take an example where these forks don't share the same version
for all packages. Consider a case where `idna==3.9` is in forks A and C,
but where `idna==3.10` is in fork B. If we combine the markers in forks
A and C through disjunction, we get the following:

     idna==3.9: extra != 'foo' or (extra != 'foo' and extra != 'bar')
    idna==3.10: extra != 'bar'

Which simplifies to:

     idna==3.9: extra != 'foo'
    idna==3.10: extra != 'bar'

But these are clearly not disjoint. Both dependencies could be selected,
for example, when neither `foo` nor `bar` are active. We can remedy this
by keeping around the inclusion rules for each fork:

    A: extra != 'foo' and extra == 'bar'
    B: extra != 'bar' and extra == 'foo'
    C: extra != 'foo' and extra != 'bar'

And so for `idna`, we have:

     idna==3.9: (extra != 'foo' and extra == 'bar') or (extra != 'foo' and extra != 'bar')
    idna==3.10: extra != 'bar' and extra == 'foo'

Which simplifies to:

     idna==3.9: extra != 'foo'
    idna==3.10: extra != 'bar' and extra == 'foo'

And these *are* properly disjoint. There is no way for them both to be
active. This also correctly accounts for fork C where neither `foo` nor
`bar` are active, and yet, `idna==3.9` is still enabled but `idna==3.10`
is not. (In the [motivating example], this comes from `baz` being enabled.)
That is, this captures the idea that for `idna==3.10` to be installed,
there must actually be a specific extra that is enabled. That's what
makes it disjoint from `idna==3.9`.

We aren't quite done yet, because this does add *too many* conflict
markers to dependency edges that don't need it. In the next commit,
we'll add in our world knowledge to simplify these conflict markers.

[motivating example]: #9289
BurntSushi added a commit that referenced this issue Nov 23, 2024
When we generate conflict markers for each resolution after the
resolver runs, it turns out that generating them just from exclusion
rules is not sufficient.

For example, if `foo` and `bar` are declared as conflicting extras, then
we end up with the following forks:

    A: extra != 'foo'
    B: extra != 'bar'
    C: extra != 'foo' and extra != 'bar'

Now let's take an example where these forks don't share the same version
for all packages. Consider a case where `idna==3.9` is in forks A and C,
but where `idna==3.10` is in fork B. If we combine the markers in forks
A and C through disjunction, we get the following:

     idna==3.9: extra != 'foo' or (extra != 'foo' and extra != 'bar')
    idna==3.10: extra != 'bar'

Which simplifies to:

     idna==3.9: extra != 'foo'
    idna==3.10: extra != 'bar'

But these are clearly not disjoint. Both dependencies could be selected,
for example, when neither `foo` nor `bar` are active. We can remedy this
by keeping around the inclusion rules for each fork:

    A: extra != 'foo' and extra == 'bar'
    B: extra != 'bar' and extra == 'foo'
    C: extra != 'foo' and extra != 'bar'

And so for `idna`, we have:

     idna==3.9: (extra != 'foo' and extra == 'bar') or (extra != 'foo' and extra != 'bar')
    idna==3.10: extra != 'bar' and extra == 'foo'

Which simplifies to:

     idna==3.9: extra != 'foo'
    idna==3.10: extra != 'bar' and extra == 'foo'

And these *are* properly disjoint. There is no way for them both to be
active. This also correctly accounts for fork C where neither `foo` nor
`bar` are active, and yet, `idna==3.9` is still enabled but `idna==3.10`
is not. (In the [motivating example], this comes from `baz` being enabled.)
That is, this captures the idea that for `idna==3.10` to be installed,
there must actually be a specific extra that is enabled. That's what
makes it disjoint from `idna==3.9`.

We aren't quite done yet, because this does add *too many* conflict
markers to dependency edges that don't need it. In the next commit,
we'll add in our world knowledge to simplify these conflict markers.

[motivating example]: #9289
BurntSushi added a commit that referenced this issue Nov 23, 2024
When we generate conflict markers for each resolution after the
resolver runs, it turns out that generating them just from exclusion
rules is not sufficient.

For example, if `foo` and `bar` are declared as conflicting extras, then
we end up with the following forks:

    A: extra != 'foo'
    B: extra != 'bar'
    C: extra != 'foo' and extra != 'bar'

Now let's take an example where these forks don't share the same version
for all packages. Consider a case where `idna==3.9` is in forks A and C,
but where `idna==3.10` is in fork B. If we combine the markers in forks
A and C through disjunction, we get the following:

     idna==3.9: extra != 'foo' or (extra != 'foo' and extra != 'bar')
    idna==3.10: extra != 'bar'

Which simplifies to:

     idna==3.9: extra != 'foo'
    idna==3.10: extra != 'bar'

But these are clearly not disjoint. Both dependencies could be selected,
for example, when neither `foo` nor `bar` are active. We can remedy this
by keeping around the inclusion rules for each fork:

    A: extra != 'foo' and extra == 'bar'
    B: extra != 'bar' and extra == 'foo'
    C: extra != 'foo' and extra != 'bar'

And so for `idna`, we have:

     idna==3.9: (extra != 'foo' and extra == 'bar') or (extra != 'foo' and extra != 'bar')
    idna==3.10: extra != 'bar' and extra == 'foo'

Which simplifies to:

     idna==3.9: extra != 'foo'
    idna==3.10: extra != 'bar' and extra == 'foo'

And these *are* properly disjoint. There is no way for them both to be
active. This also correctly accounts for fork C where neither `foo` nor
`bar` are active, and yet, `idna==3.9` is still enabled but `idna==3.10`
is not. (In the [motivating example], this comes from `baz` being enabled.)
That is, this captures the idea that for `idna==3.10` to be installed,
there must actually be a specific extra that is enabled. That's what
makes it disjoint from `idna==3.9`.

We aren't quite done yet, because this does add *too many* conflict
markers to dependency edges that don't need it. In the next commit,
we'll add in our world knowledge to simplify these conflict markers.

[motivating example]: #9289
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants