Skip to content

Support nextPageToken and the the new Jira /search/jql endpoint (#6015) #6063

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Support nextPageToken and the the new Jira /search/jql endpoint (#6015) #6063

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions ...jira-source/src/main/java/org/opensearch/dataprepper/plugins/source/jira/JiraService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,15 +97,15 @@ private void searchForNewTicketsAndAddToQueue(JiraSourceConfig configuration, In
Queue<ItemInfo> itemInfoQueue) {
log.trace("Looking for Add/Modified tickets with a Search API call");
StringBuilder jql = createIssueFilterCriteria(configuration, timestamp);
int total;
int startAt = 0;
int total = 0;
String nextPageToken = "";
do {
SearchResults searchIssues = jiraRestClient.getAllIssues(jql, startAt);
SearchResults searchIssues = jiraRestClient.getAllIssues(jql, nextPageToken);
List<IssueBean> issueList = new ArrayList<>(searchIssues.getIssues());
total = searchIssues.getTotal();
startAt += searchIssues.getIssues().size();
total = total + issueList.size();
nextPageToken = searchIssues.getNextPageToken();
addItemsToQueue(issueList, itemInfoQueue);
} while (startAt < total);
} while (nextPageToken != null && !nextPageToken.isEmpty() && !nextPageToken.isBlank());
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could have been simpler to rely on isLast?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought about that, but there could be a case where isLast is false but maybe a bug from Atlassian (yes, bugs even impact large companies) where we don't receive a nextPageToken. Ultimately, regardless of the value of isLast, we need nextPageToken to be a non-empty value to get the next page. Defensive programming..

searchResultsFoundCounter.increment(total);
log.info("Number of tickets found in search api call: {}", total);
}
Expand Down
12 changes: 6 additions & 6 deletions ...ce/src/main/java/org/opensearch/dataprepper/plugins/source/jira/models/SearchResults.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,16 +24,16 @@ public class SearchResults {
@JsonProperty("expand")
private String expand = null;

@JsonProperty("startAt")
private Integer startAt = null;

@JsonProperty("maxResults")
private Integer maxResults = null;

@JsonProperty("total")
private Integer total = null;

@JsonProperty("issues")
private List<IssueBean> issues = null;

@JsonProperty("nextPageToken")
private String nextPageToken = null;

@JsonProperty("isLast")
private Boolean isLast = false;

}
34 changes: 25 additions & 9 deletions ...rce/src/main/java/org/opensearch/dataprepper/plugins/source/jira/rest/JiraRestClient.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,16 +26,18 @@
import static org.opensearch.dataprepper.logging.DataPrepperMarkers.NOISY;
import static org.opensearch.dataprepper.plugins.source.jira.utils.JqlConstants.EXPAND_FIELD;
import static org.opensearch.dataprepper.plugins.source.jira.utils.JqlConstants.EXPAND_VALUE;
import static org.opensearch.dataprepper.plugins.source.jira.utils.JqlConstants.FIELDS_FIELD;
import static org.opensearch.dataprepper.plugins.source.jira.utils.JqlConstants.FIELDS_VALUE;
import static org.opensearch.dataprepper.plugins.source.jira.utils.JqlConstants.JQL_FIELD;

@Slf4j
@Named
public class JiraRestClient extends AtlassianRestClient {

public static final String REST_API_SEARCH = "rest/api/3/search";
public static final String REST_API_SEARCH = "rest/api/3/search/jql";
public static final String REST_API_FETCH_ISSUE = "rest/api/3/issue";
public static final String FIFTY = "50";
public static final String START_AT = "startAt";
public static final String NEXT_PAGE_TOKEN = "nextPageToken";
public static final String MAX_RESULT = "maxResults";
private static final String TICKET_FETCH_LATENCY_TIMER = "ticketFetchLatency";
private static final String SEARCH_CALL_LATENCY_TIMER = "searchCallLatency";
Expand Down Expand Up @@ -72,16 +74,30 @@ public JiraRestClient(RestTemplate restTemplate, AtlassianAuthConfig authConfig,
* @param startAt the start at
* @return InputStream input stream
*/
public SearchResults getAllIssues(StringBuilder jql, int startAt) {
public SearchResults getAllIssues(StringBuilder jql, String nextPageToken) {

String url = authConfig.getUrl() + REST_API_SEARCH;

URI uri = UriComponentsBuilder.fromHttpUrl(url)
.queryParam(MAX_RESULT, FIFTY)
.queryParam(START_AT, startAt)
.queryParam(JQL_FIELD, jql)
.queryParam(EXPAND_FIELD, EXPAND_VALUE)
.buildAndExpand().toUri();
URI uri;

if(nextPageToken!= null && !nextPageToken.isBlank() && !nextPageToken.isEmpty()){
uri = UriComponentsBuilder.fromHttpUrl(url)
.queryParam(MAX_RESULT, FIFTY)
.queryParam(NEXT_PAGE_TOKEN, nextPageToken)
.queryParam(JQL_FIELD, jql)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Subsequent calls that are just fetching the next page, doesn't require jql query param.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid

.queryParam(EXPAND_FIELD, EXPAND_VALUE)
.queryParam(FIELDS_FIELD, FIELDS_VALUE)
.buildAndExpand().toUri();
}
else{
uri = UriComponentsBuilder.fromHttpUrl(url)
.queryParam(MAX_RESULT, FIFTY)
.queryParam(JQL_FIELD, jql)
.queryParam(EXPAND_FIELD, EXPAND_VALUE)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like, with the new FIELDS_FIELD query param added, we don't need previous EXPAND_FIELD. It can be removed in both the blocks

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Jira API documentation still reference both 'expand' and 'fields' so I assumed there is a use-case for both. Without assuming functionality, I felt it better to leave both. https://developer.atlassian.com/cloud/jira/platform/rest/v3/api-group-issue-search/#api-rest-api-3-search-jql-post

Copy link
Collaborator

@san81 san81 Sep 6, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this use case, we only need one of them. So we can just go with the newly introduced field describing the specific fields we want to fetch while searching through the tickets.

.queryParam(FIELDS_FIELD, FIELDS_VALUE)
.buildAndExpand().toUri();
}

return searchCallLatencyTimer.record(
() -> {
try {
Expand Down
2 changes: 2 additions & 0 deletions ...urce/src/main/java/org/opensearch/dataprepper/plugins/source/jira/utils/JqlConstants.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,4 +27,6 @@ public class JqlConstants {
public static final String JQL_FIELD = "jql";
public static final String EXPAND_FIELD = "expand";
public static final String EXPAND_VALUE = "all";
Comment on lines 28 to 29
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two fields can be removed as well?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per above comment, JIRA docs have both expand and fields. Maybe expand will be useful in the future?

public static final String FIELDS_FIELD = "fields";
public static final String FIELDS_VALUE = "*all";
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The query can be optimized to only fetch the specific fields needed for crawling while navigating over search results. While fetching the page, it can fetch all fields.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to make sure that I understand. Are you saying that an initial query is made to retrieve a list of issues, in which case we only need the id/key. Then those are added to a queue, and an iterator fetches the issues one by one?
I suspected that this may be the case, but was not sure.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that is right. Initial search query returns the he list of issues with id, project, created datetime and last modified datetime attributes and then an other threads kicks in to fetch those items individually.

}
14 changes: 5 additions & 9 deletions ...source/src/test/java/org/opensearch/dataprepper/plugins/source/jira/JiraIteratorTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyInt;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.Mockito.doReturn;
import static org.mockito.Mockito.spy;
import static org.mockito.Mockito.when;
Expand Down Expand Up @@ -72,8 +72,7 @@ void testInitialization() {
assertNotNull(jiraIterator);
jiraIterator.initialize(Instant.ofEpochSecond(0));
when(mockSearchResults.getIssues()).thenReturn(new ArrayList<>());
when(mockSearchResults.getTotal()).thenReturn(0);
doReturn(mockSearchResults).when(jiraRestClient).getAllIssues(any(StringBuilder.class), anyInt());
doReturn(mockSearchResults).when(jiraRestClient).getAllIssues(any(StringBuilder.class), anyString());
assertFalse(jiraIterator.hasNext());
}

Expand Down Expand Up @@ -103,8 +102,7 @@ void testItemInfoQueueNotEmpty() {
IssueBean issue1 = createIssueBean(false);
mockIssues.add(issue1);
when(mockSearchResults.getIssues()).thenReturn(mockIssues);
when(mockSearchResults.getTotal()).thenReturn(0);
doReturn(mockSearchResults).when(jiraRestClient).getAllIssues(any(StringBuilder.class), anyInt());
doReturn(mockSearchResults).when(jiraRestClient).getAllIssues(any(StringBuilder.class), anyString());

jiraIterator.initialize(Instant.ofEpochSecond(0));
jiraIterator.setCrawlerQWaitTimeMillis(1);
Expand Down Expand Up @@ -132,8 +130,7 @@ void testFuturesCompleted() throws InterruptedException {
IssueBean issue3 = createIssueBean(false);
mockIssues.add(issue3);
when(mockSearchResults.getIssues()).thenReturn(mockIssues);
when(mockSearchResults.getTotal()).thenReturn(0);
doReturn(mockSearchResults).when(jiraRestClient).getAllIssues(any(StringBuilder.class), anyInt());
doReturn(mockSearchResults).when(jiraRestClient).getAllIssues(any(StringBuilder.class), anyString());

jiraIterator.initialize(Instant.ofEpochSecond(0));
jiraIterator.setCrawlerQWaitTimeMillis(1);
Expand All @@ -149,8 +146,7 @@ void testItemInfoQueueEmpty() {
jiraIterator = createObjectUnderTest();
List<IssueBean> mockIssues = new ArrayList<>();
when(mockSearchResults.getIssues()).thenReturn(mockIssues);
when(mockSearchResults.getTotal()).thenReturn(0);
doReturn(mockSearchResults).when(jiraRestClient).getAllIssues(any(StringBuilder.class), anyInt());
doReturn(mockSearchResults).when(jiraRestClient).getAllIssues(any(StringBuilder.class), anyString());

jiraIterator.initialize(Instant.ofEpochSecond(0));
jiraIterator.setCrawlerQWaitTimeMillis(1);
Expand Down
12 changes: 5 additions & 7 deletions ...-source/src/test/java/org/opensearch/dataprepper/plugins/source/jira/JiraServiceTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,6 @@
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyInt;
import static org.mockito.ArgumentMatchers.anyString;
import static org.mockito.Mockito.doReturn;
import static org.mockito.Mockito.doThrow;
Expand Down Expand Up @@ -204,9 +203,8 @@ public void testGetJiraEntities() throws JsonProcessingException {

SearchResults mockSearchResults = mock(SearchResults.class);
when(mockSearchResults.getIssues()).thenReturn(mockIssues);
when(mockSearchResults.getTotal()).thenReturn(mockIssues.size());

doReturn(mockSearchResults).when(jiraRestClient).getAllIssues(any(StringBuilder.class), anyInt());
doReturn(mockSearchResults).when(jiraRestClient).getAllIssues(any(StringBuilder.class), anyString());

Instant timestamp = Instant.ofEpochSecond(0);
Queue<ItemInfo> itemInfoQueue = new ConcurrentLinkedQueue<>();
Expand All @@ -223,16 +221,16 @@ public void buildIssueItemInfoMultipleFutureThreads() throws JsonProcessingExcep
JiraSourceConfig jiraSourceConfig = createJiraConfiguration(BASIC, issueType, issueStatus, projectKey);
JiraService jiraService = spy(new JiraService(jiraSourceConfig, jiraRestClient, pluginMetrics));
List<IssueBean> mockIssues = new ArrayList<>();
for (int i = 0; i < 50; i++) {
Integer mockIssuesCount = 100;
for (int i = 0; i < mockIssuesCount; i++) {
IssueBean issue1 = createIssueBean(false, false);
mockIssues.add(issue1);
}

SearchResults mockSearchResults = mock(SearchResults.class);
when(mockSearchResults.getIssues()).thenReturn(mockIssues);
when(mockSearchResults.getTotal()).thenReturn(100);

doReturn(mockSearchResults).when(jiraRestClient).getAllIssues(any(StringBuilder.class), anyInt());
doReturn(mockSearchResults).when(jiraRestClient).getAllIssues(any(StringBuilder.class), anyString());

Instant timestamp = Instant.ofEpochSecond(0);
Queue<ItemInfo> itemInfoQueue = new ConcurrentLinkedQueue<>();
Expand Down Expand Up @@ -271,7 +269,7 @@ public void testGetJiraEntitiesException() throws JsonProcessingException {
JiraSourceConfig jiraSourceConfig = createJiraConfiguration(BASIC, issueType, issueStatus, projectKey);
JiraService jiraService = spy(new JiraService(jiraSourceConfig, jiraRestClient, pluginMetrics));

doThrow(RuntimeException.class).when(jiraRestClient).getAllIssues(any(StringBuilder.class), anyInt());
doThrow(RuntimeException.class).when(jiraRestClient).getAllIssues(any(StringBuilder.class), anyString());

Instant timestamp = Instant.ofEpochSecond(0);
Queue<ItemInfo> itemInfoQueue = new ConcurrentLinkedQueue<>();
Expand Down
17 changes: 9 additions & 8 deletions ...rc/test/java/org/opensearch/dataprepper/plugins/source/jira/models/SearchResultsTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,18 +44,18 @@ public void testConstructor() {
assertNotNull(searchResults);

assertNull(searchResults.getExpand());
assertNull(searchResults.getStartAt());
assertNull(searchResults.getMaxResults());
assertNull(searchResults.getTotal());
assertEquals(searchResults.getIsLast(), false);
assertNull(searchResults.getNextPageToken());
assertNull(searchResults.getIssues());
}

@Test
public void testGetters() throws JsonProcessingException {
String expand = "expandTest";
Integer startAt = 1;
String nextPageToken = "tokenTest";
Integer maxResults = 100;
Integer total = 10;
Boolean isLast = true;
List<IssueBean> testIssues = new ArrayList<>();
IssueBean issue1 = new IssueBean();
IssueBean issue2 = new IssueBean();
Expand All @@ -67,19 +67,20 @@ public void testGetters() throws JsonProcessingException {

Map<String, Object> searchResultsMap = new HashMap<>();
searchResultsMap.put("expand", expand);
searchResultsMap.put("startAt", startAt);
searchResultsMap.put("maxResults", maxResults);
searchResultsMap.put("total", total);
searchResultsMap.put("nextPageToken", nextPageToken);
searchResultsMap.put("isLast", isLast);
searchResultsMap.put("issues", testIssues);


String jsonString = objectMapper.writeValueAsString(searchResultsMap);

searchResults = objectMapper.readValue(jsonString, SearchResults.class);

assertEquals(searchResults.getExpand(), expand);
assertEquals(searchResults.getStartAt(), startAt);
assertEquals(searchResults.getMaxResults(), maxResults);
assertEquals(searchResults.getTotal(), total);
assertEquals(searchResults.getNextPageToken(), nextPageToken);
assertEquals(searchResults.getIsLast(), isLast);

List<IssueBean> returnedIssues = searchResults.getIssues();
assertNotNull(returnedIssues);
Expand Down
4 changes: 2 additions & 2 deletions ...src/test/java/org/opensearch/dataprepper/plugins/source/jira/rest/JiraRestClientTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ public void testGetAllIssuesOauth2() {
SearchResults mockSearchResults = mock(SearchResults.class);
doReturn("http://mock-service.jira.com/").when(authConfig).getUrl();
doReturn(new ResponseEntity<>(mockSearchResults, HttpStatus.OK)).when(restTemplate).getForEntity(any(URI.class), any(Class.class));
SearchResults results = jiraRestClient.getAllIssues(jql, 0);
SearchResults results = jiraRestClient.getAllIssues(jql, null);
assertNotNull(results);
}

Expand All @@ -130,7 +130,7 @@ public void testGetAllIssuesBasic() {
SearchResults mockSearchResults = mock(SearchResults.class);
when(authConfig.getUrl()).thenReturn("https://example.com/");
doReturn(new ResponseEntity<>(mockSearchResults, HttpStatus.OK)).when(restTemplate).getForEntity(any(URI.class), any(Class.class));
SearchResults results = jiraRestClient.getAllIssues(jql, 0);
SearchResults results = jiraRestClient.getAllIssues(jql, null);
assertNotNull(results);
}

Expand Down
Loading