Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(issue-search): support IN for semver release search #76313

Open
wants to merge 9 commits into
base: master
Choose a base branch
from

Conversation

JoshFerge
Copy link
Member

@JoshFerge JoshFerge commented Aug 15, 2024

  • adds support for the IN Operator for semver releases.
  • need to support the NOT IN operator as well, as well as any negation conditions
  • tests added

Fixes #76286
Fixes SENTRY-3HAA

@JoshFerge JoshFerge requested review from a team as code owners August 15, 2024 23:41
@github-actions github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Aug 15, 2024
Copy link

codecov bot commented Aug 16, 2024

❌ 13 Tests Failed:

Tests completed Failed Passed Skipped
23296 13 23283 213
View the top 3 failed tests by shortest run time
tests.sentry.search.events.test_filter.SemverBuildFilterConverterTest::test
Stack Traces | 2.4s run time
#x1B[1m#x1B[31m.venv/lib/python3.12.../models/fields/__init__.py#x1B[0m:2123: in get_prep_value
    return int(value)
#x1B[1m#x1B[31mE   TypeError: int() argument must be a string, a bytes-like object or a real number, not 'list'#x1B[0m

#x1B[33mThe above exception was the direct cause of the following exception:#x1B[0m
#x1B[1m#x1B[.../search/events/test_filter.py#x1B[0m:462: in test
    self.run_test("=", "123", "IN", [release.version, release_2.version])
#x1B[1m#x1B[.../search/events/test_filter.py#x1B[0m:223: in run_test
    converted = self.converter(filter, self.key, params)
#x1B[1m#x1B[.../search/events/test_filter.py#x1B[0m:445: in converter
    return _semver_build_filter_converter(*args, **kwargs)
#x1B[1m#x1B[.../search/events/filter.py#x1B[0m:499: in _semver_build_filter_converter
    Release.objects.filter_by_semver_build(
#x1B[1m#x1B[.../sentry/models/release.py#x1B[0m:112: in filter_by_semver_build
    return self.get_queryset().filter_by_semver_build(
#x1B[1m#x1B[.../models/releases/util.py#x1B[0m:98: in filter_by_semver_build
    qs = getattr(qs, query_func)(build_number_filters)
#x1B[1m#x1B[31m.venv/lib/python3.12.../db/models/query.py#x1B[0m:1476: in filter
    return self._filter_or_exclude(False, args, kwargs)
#x1B[1m#x1B[31m.venv/lib/python3.12.../db/models/query.py#x1B[0m:1494: in _filter_or_exclude
    clone._filter_or_exclude_inplace(negate, args, kwargs)
#x1B[1m#x1B[31m.venv/lib/python3.12.../db/models/query.py#x1B[0m:1501: in _filter_or_exclude_inplace
    self._query.add_q(Q(*args, **kwargs))
#x1B[1m#x1B[31m.venv/lib/python3.12.../models/sql/query.py#x1B[0m:1609: in add_q
    clause, _ = self._add_q(q_object, self.used_aliases)
#x1B[1m#x1B[31m.venv/lib/python3.12.../models/sql/query.py#x1B[0m:1641: in _add_q
    child_clause, needed_inner = self.build_filter(
#x1B[1m#x1B[31m.venv/lib/python3.12.../models/sql/query.py#x1B[0m:1468: in build_filter
    return self._add_q(
#x1B[1m#x1B[31m.venv/lib/python3.12.../models/sql/query.py#x1B[0m:1641: in _add_q
    child_clause, needed_inner = self.build_filter(
#x1B[1m#x1B[31m.venv/lib/python3.12.../models/sql/query.py#x1B[0m:1555: in build_filter
    condition = self.build_lookup(lookups, col, value)
#x1B[1m#x1B[31m.venv/lib/python3.12.../models/sql/query.py#x1B[0m:1385: in build_lookup
    lookup = lookup_class(lhs, rhs)
#x1B[1m#x1B[31m.venv/lib/python3.12.../db/models/lookups.py#x1B[0m:30: in __init__
    self.rhs = self.get_prep_lookup()
#x1B[1m#x1B[31m.venv/lib/python3.12.../db/models/lookups.py#x1B[0m:369: in get_prep_lookup
    return super().get_prep_lookup()
#x1B[1m#x1B[31m.venv/lib/python3.12.../db/models/lookups.py#x1B[0m:88: in get_prep_lookup
    return self.lhs.output_field.get_prep_value(self.rhs)
#x1B[1m#x1B[31m.venv/lib/python3.12.../models/fields/__init__.py#x1B[0m:2125: in get_prep_value
    raise e.__class__(
#x1B[1m#x1B[31mE   TypeError: Field 'build_number' expected a number but got [123].#x1B[0m
tests.sentry.models.test_release.ReleaseFilterBySemverBuildTest::test_text
Stack Traces | 2.49s run time
#x1B[1m#x1B[.../sentry/models/test_release.py#x1B[0m:1052: in test_text
    self.run_test("exact", "", [release_1, release_2, release_3])
#x1B[1m#x1B[.../sentry/models/test_release.py#x1B[0m:1019: in run_test
    assert set(
#x1B[1m#x1B[31mE   AssertionError: assert set() == {<Release at ....2.4+123abc'>}#x1B[0m
#x1B[1m#x1B[31mE     #x1B[0m
#x1B[1m#x1B[31mE     Extra items in the right set:#x1B[0m
#x1B[1m#x1B[31mE     <Release at 0x7fbb6367d7c0: id=171, organization_id=4555205052661760, version='[email protected]+123'>#x1B[0m
#x1B[1m#x1B[31mE     <Release at 0x7fbb63651b20: id=172, organization_id=4555205052661760, version='[email protected]+1234'>#x1B[0m
#x1B[1m#x1B[31mE     <Release at 0x7fbb636025a0: id=173, organization_id=4555205052661760, version='[email protected]+123abc'>#x1B[0m
#x1B[1m#x1B[31mE     #x1B[0m
#x1B[1m#x1B[31mE     Full diff:#x1B[0m
#x1B[1m#x1B[31mE     + set()#x1B[0m
#x1B[1m#x1B[31mE     - {#x1B[0m
#x1B[1m#x1B[31mE     -     <Release at 0x7fbb636025a0: id=173, organization_id=4555205052661760, version='[email protected]+123abc'>,#x1B[0m
#x1B[1m#x1B[31mE     -     <Release at 0x7fbb63651b20: id=172, organization_id=4555205052661760, version='[email protected]+1234'>,#x1B[0m
#x1B[1m#x1B[31mE     -     <Release at 0x7fbb6367d7c0: id=171, organization_id=4555205052661760, version='[email protected]+123'>,#x1B[0m
#x1B[1m#x1B[31mE     - }#x1B[0m
tests.sentry.models.test_release.ReleaseFilterBySemverBuildTest::test_no_build
Stack Traces | 3.22s run time
#x1B[1m#x1B[31m.venv/lib/python3.12.../models/fields/__init__.py#x1B[0m:2123: in get_prep_value
    return int(value)
#x1B[1m#x1B[31mE   TypeError: int() argument must be a string, a bytes-like object or a real number, not 'list'#x1B[0m

#x1B[33mThe above exception was the direct cause of the following exception:#x1B[0m
#x1B[1m#x1B[.../sentry/models/test_release.py#x1B[0m:1028: in test_no_build
    self.run_test("gt", "100", [])
#x1B[1m#x1B[.../sentry/models/test_release.py#x1B[0m:1019: in run_test
    assert set(
#x1B[1m#x1B[.../sentry/models/release.py#x1B[0m:112: in filter_by_semver_build
    return self.get_queryset().filter_by_semver_build(
#x1B[1m#x1B[.../models/releases/util.py#x1B[0m:98: in filter_by_semver_build
    qs = getattr(qs, query_func)(build_number_filters)
#x1B[1m#x1B[31m.venv/lib/python3.12.../db/models/query.py#x1B[0m:1476: in filter
    return self._filter_or_exclude(False, args, kwargs)
#x1B[1m#x1B[31m.venv/lib/python3.12.../db/models/query.py#x1B[0m:1494: in _filter_or_exclude
    clone._filter_or_exclude_inplace(negate, args, kwargs)
#x1B[1m#x1B[31m.venv/lib/python3.12.../db/models/query.py#x1B[0m:1501: in _filter_or_exclude_inplace
    self._query.add_q(Q(*args, **kwargs))
#x1B[1m#x1B[31m.venv/lib/python3.12.../models/sql/query.py#x1B[0m:1609: in add_q
    clause, _ = self._add_q(q_object, self.used_aliases)
#x1B[1m#x1B[31m.venv/lib/python3.12.../models/sql/query.py#x1B[0m:1641: in _add_q
    child_clause, needed_inner = self.build_filter(
#x1B[1m#x1B[31m.venv/lib/python3.12.../models/sql/query.py#x1B[0m:1468: in build_filter
    return self._add_q(
#x1B[1m#x1B[31m.venv/lib/python3.12.../models/sql/query.py#x1B[0m:1641: in _add_q
    child_clause, needed_inner = self.build_filter(
#x1B[1m#x1B[31m.venv/lib/python3.12.../models/sql/query.py#x1B[0m:1555: in build_filter
    condition = self.build_lookup(lookups, col, value)
#x1B[1m#x1B[31m.venv/lib/python3.12.../models/sql/query.py#x1B[0m:1385: in build_lookup
    lookup = lookup_class(lhs, rhs)
#x1B[1m#x1B[31m.venv/lib/python3.12.../db/models/lookups.py#x1B[0m:30: in __init__
    self.rhs = self.get_prep_lookup()
#x1B[1m#x1B[31m.venv/lib/python3.12.../db/models/lookups.py#x1B[0m:88: in get_prep_lookup
    return self.lhs.output_field.get_prep_value(self.rhs)
#x1B[1m#x1B[31m.venv/lib/python3.12.../models/fields/__init__.py#x1B[0m:2125: in get_prep_value
    raise e.__class__(
#x1B[1m#x1B[31mE   TypeError: Field 'build_number' expected a number but got [100].#x1B[0m

To view more test analytics, go to the Test Analytics Dashboard
📢 Thoughts on this report? Let us know!

src/sentry/search/events/filter.py Show resolved Hide resolved
else:
semver_filters = []
for v in version:
_, versions = get_versions(v, operator)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not really sure get_versions is necessary for IN. Since in will just be doing exact matches, all the sorting junk that we do isn't really relevant.

@getsantry
Copy link
Contributor

getsantry bot commented Sep 7, 2024

This pull request has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you add the label WIP, I will leave it alone unless WIP is removed ... forever!


"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

@getsantry getsantry bot added Stale and removed Stale labels Sep 7, 2024
@JoshFerge JoshFerge force-pushed the jferg/support-in-releases branch from 82f152f to a6e3bef Compare September 11, 2024 15:57
@JoshFerge JoshFerge requested a review from a team as a code owner September 11, 2024 15:57
JoshFerge added a commit that referenced this pull request Sep 11, 2024
support `in` for for release package for issues search. 

related (but this one is much easier):
#76313
@getsantry
Copy link
Contributor

getsantry bot commented Oct 3, 2024

This pull request has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you add the label WIP, I will leave it alone unless WIP is removed ... forever!


"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

@JoshFerge
Copy link
Member Author

will be working on this, came up in triage duty again

Comment on lines +215 to +230
# do our best.
operator = constants.OPERATOR_NEGATION_MAP[operator]
# Note that the `order_by` here is important for index usage. Postgres seems
# to seq scan with this query if the `order_by` isn't included, so we
# include it even though we don't really care about order for this query
qs_flipped = (
Release.objects.filter_by_semver(organization_id, parse_semver(version, operator))
.order_by(*map(_flip_field_sort, order_by))
.values_list("version", flat=True)[: constants.MAX_SEARCH_RELEASES]
)

exclude_versions = list(qs_flipped)
if exclude_versions and len(exclude_versions) < len(versions):
# Do a negative search instead
final_operator = Op.NOT_IN
versions = exclude_versions
exclude_versions = list(qs_flipped)
if exclude_versions and len(exclude_versions) < len(versions):
# Do a negative search instead
final_operator = Op.NOT_IN
versions = exclude_versions
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these lines are a codepath which i haven't changed, just the indentation makes it a diff, and codecov reports no coverage on these which causes the codecov to be low

@JoshFerge JoshFerge requested a review from wedamija October 31, 2024 16:30
@@ -80,16 +80,49 @@ def filter_by_semver_build(
"release_id", flat=True
)
)
if isinstance(build, str):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if it would make sense to just do

if isinstance(build, str):
    build = [build]

Then go down the IN path

Comment on lines +236 to +243
Release.objects.filter_by_semver(
organization_id,
parse_semver(v, operator),
project_ids=builder.params.project_ids,
)
.values_list("version", flat=True)
.order_by(*order_by)[: constants.MAX_SEARCH_RELEASES]
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we're doing exact match here I guess we don't care about extra filtering logic in the single version branch?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, i don't think its super necessary. i'll add a comment and monitor as we release, but i think because we're only doing matches, the extra filtering logic isn't as important.

@getsantry
Copy link
Contributor

getsantry bot commented Nov 28, 2024

This pull request has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you add the label WIP, I will leave it alone unless WIP is removed ... forever!


"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

@getsantry getsantry bot added the Stale label Nov 28, 2024
@thisisthekap
Copy link

Don't close this PR while remaining unmerged! We are eagerly awaiting this improvement! 😊

@getsantry
Copy link
Contributor

getsantry bot commented Jan 3, 2025

This pull request has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you add the label WIP, I will leave it alone unless WIP is removed ... forever!


"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

@getsantry getsantry bot added the Stale label Jan 3, 2025
@thisisthekap
Copy link

Another comment to keep this open. We are eagerly waiting for this feature! :)

@getsantry getsantry bot removed the Stale label Jan 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Scope: Backend Automatically applied to PRs that change backend components
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Selecting multiple releases in the Issues search is failing to load
3 participants