Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add API spec for Search Response Processors #440

Closed
wants to merge 6 commits into from
Closed

Add API spec for Search Response Processors #440

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
fea5c10
Add smithy-generated existing spec
dbwiddis Jul 19, 2024
a347df3
Add specs for sort and split processors
dbwiddis Jul 19, 2024
d034e66
Add a test
dbwiddis Jul 19, 2024
78507c2
Fix lint errors
dbwiddis Jul 19, 2024
7da742e
More lint errors, change log
dbwiddis Jul 19, 2024
59f5728
Merge branch 'main' into search-response-procs
dbwiddis Jul 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .cspell
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ OSCPU
Oversample
Rebalance
Reindex
Rerank
Reranker
Rethrottle
Rolespan
Rollup
Expand Down Expand Up @@ -138,6 +140,7 @@ readingform
rebalance
recoverysource
reindex
rerank
relo
reloadcerts
remotestore
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Added `is_hidden` to `/{index}/_alias/{name}` and `/{index}/_aliases/{name}` ([#429](https://github.com/opensearch-project/opensearch-api-specification/pull/429))
- Added `ignore_unmapped` to `GeoDistanceQuery` ([#427](https://github.com/opensearch-project/opensearch-api-specification/pull/427))
- Added missing variants of `indices.put_alias` ([#434](https://github.com/opensearch-project/opensearch-api-specification/pull/434))
- Added missing search response processors and new `sort` and `split` processors ([#440](https://github.com/opensearch-project/opensearch-api-specification/pull/440))

### Changed

Expand Down
215 changes: 214 additions & 1 deletion spec/schemas/search_pipeline._common.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ components:
response_processors:
type: array
items:
$ref: '#/components/schemas/RequestProcessor'
$ref: '#/components/schemas/ResponseProcessor'
phase_results_processors:
type: array
items:
Expand Down Expand Up @@ -187,6 +187,219 @@ components:
type: string
required:
- sample_factor
ResponseProcessor:
oneOf:
- type: object
title: personalize_search_ranking
properties:
personalize_search_ranking:
$ref: '#/components/schemas/PersonalizeSearchRankingResponseProcessor'
required:
- personalize_search_ranking
- type: object
title: retrieval_augmented_generation
properties:
retrieval_augmented_generation:
$ref: '#/components/schemas/RetrievalAugmentedGenerationResponseProcessor'
required:
- retrieval_augmented_generation
- type: object
title: rename_field
properties:
rename_field:
$ref: '#/components/schemas/RenameFieldResponseProcessor'
required:
- rename_field
- type: object
title: rerank
properties:
rerank:
$ref: '#/components/schemas/RerankResponseProcessor'
required:
- rerank
- type: object
title: collapse
properties:
collapse:
$ref: '#/components/schemas/CollapseResponseProcessor'
required:
- collapse
- type: object
title: truncate_hits
properties:
truncate_hits:
$ref: '#/components/schemas/TruncateHitsResponseProcessor'
required:
- truncate_hits
- type: object
title: sort
properties:
sort:
$ref: '#/components/schemas/SortResponseProcessor'
required:
- sort
- type: object
title: split
properties:
split:
$ref: '#/components/schemas/SplitResponseProcessor'
required:
- split
PersonalizeSearchRankingResponseProcessor:
type: object
properties:
tag:
type: string
description:
type: string
ignore_failure:
type: boolean
campaign_arn:
type: string
recipe:
type: string
weight:
type: number
format: float
item_id_field:
type: string
iam_role_arn:
type: string
required:
- campaign_arn
- recipe
- weight
RetrievalAugmentedGenerationResponseProcessor:
type: object
properties:
tag:
type: string
description:
type: string
model_id:
type: string
context_field_list:
type: array
items:
type: string
system_prompt:
type: string
user_instructions:
type: string
required:
- context_field_list
- model_id
RenameFieldResponseProcessor:
type: object
properties:
tag:
type: string
description:
type: string
ignore_failure:
type: boolean
field:
type: string
target_field:
type: string
required:
- field
- target_field
RerankContext:
type: object
properties:
document_fields:
type: array
items:
type: string
required:
- document_fields
MLOpenSearchReranker:
type: object
properties:
model_id:
type: string
required:
- model_id
RerankResponseProcessor:
type: object
properties:
tag:
type: string
description:
type: string
ignore_failure:
type: boolean
ml_opensearch:
$ref: '#/components/schemas/MLOpenSearchReranker'
context:
$ref: '#/components/schemas/RerankContext'
CollapseResponseProcessor:
type: object
properties:
tag:
type: string
description:
type: string
ignore_failure:
type: boolean
field:
type: string
context_prefix:
type: string
required:
- field
TruncateHitsResponseProcessor:
type: object
properties:
tag:
type: string
description:
type: string
ignore_failure:
type: boolean
target_size:
type: integer
format: int32
context_prefix:
type: string
SortResponseProcessor:
type: object
properties:
tag:
type: string
description:
type: string
ignore_failure:
type: boolean
field:
type: string
order:
type: string
target_field:
type: string
required:
- field
SplitResponseProcessor:
type: object
properties:
tag:
type: string
description:
type: string
ignore_failure:
type: boolean
field:
type: string
separator:
type: string
preserve_trailing:
type: boolean
target_field:
type: string
required:
- field
- separator
PhaseResultsProcessor:
oneOf:
- type: object
Expand Down
37 changes: 37 additions & 0 deletions tests/_core/search/pipeline.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
$schema: ../../../json_schemas/test_story.schema.yaml

description: |
Test the creation of a search pipeline with sort and split response processors.
epilogues:
- path: /_search/pipeline/sorting_pipeline
method: DELETE
status: [200, 404]
version: '>= 2.16'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CI failed because we lock the build to a specific version on dockerhub here - update it to the latest one that also has the new pipeline added.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CI failed because we lock the build to a specific version on dockerhub here - update it to the latest one that also has the new pipeline added.

The PR adding this feature has not been merged yet. What is the usual pattern for writing a test for unmerged code, since submitting this API PR is a prerequisite to merging the PR? 🐔 🥚

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you have two choices.

  1. Wait till the PR is merged and there's a build with it.
  2. Split this PR in 2, one for existing features and one for the new one, and (1) for the new one.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had already assumed option 1 was the expectation.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @dbwiddis, looking forward to getting all this closed out!

chapters:
- synopsis: Create search pipeline for sorting and splitting.
path: /_search/pipeline/{id}
method: PUT
parameters:
id: sorting_pipeline
request_body:
payload:
response_processors:
- sort:
field: message
order: asc
target_field: sorted_message
- split:
field: message
separator: ', '
target_field: split_message
response:
status: 200
payload:
acknowledged: true
- synopsis: Query created pipeline.
path: /_search/pipeline/{id}
method: GET
parameters:
id: sorting_pipeline
response:
status: 200
Loading