Lazily export JSON results #1438

UNEXENU · 2024-08-10T10:28:34Z

JSON results (in the official SPARQL JSON format as well as QLever's custom JSON format that is used by qlever-ui) are now also created on the fly and sent out in chunks without materializing the (possibly huge) complete JSON string in RAM. With this change, all query results are exported in this lazy manner, as the support for all other result types (TSV, CSV, Turtle, XML, Binary) has already been implemented previously.

codecov · 2024-08-10T11:02:08Z

Codecov Report

Attention: Patch coverage is 99.36709% with 1 line in your changes missing coverage. Please review.

Project coverage is 89.41%. Comparing base (f986808) to head (349db27).
Report is 8 commits behind head on master.

Files	Patch %	Lines
src/engine/ExportQueryExecutionTrees.cpp	99.36%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1438      +/-   ##
==========================================
- Coverage   89.44%   89.41%   -0.03%     
==========================================
  Files         346      346              
  Lines       25262    25347      +85     
  Branches     3371     3381      +10     
==========================================
+ Hits        22595    22665      +70     
+ Misses       1507     1495      -12     
- Partials     1160     1187      +27

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

This reverts commit 8c70415.

joka921

In general this already looks very nice.
There's some small things to do.
You can also write a similar exporter for the
qleverJSON , then we can get rid of a lot of if(json) then bla statements in the server.

joka921 · 2024-08-13T08:53:00Z

src/engine/ExportQueryExecutionTrees.cpp

+  co_yield absl::StrCat("{\"head\":{\"vars\":[\"", absl::StrJoin(vars, "\",\""),
+                        "\"]},\"results\":{\"bindings\":[");


To be a little bit safer here,
create an nlohmann::json with the variables, and then use its dump() to fill the head.
(Maybe we have odd variable names etc, we want the exporter to be robust).

joka921 · 2024-08-13T08:54:22Z

src/engine/ExportQueryExecutionTrees.cpp

+      const auto& optionalValue = idToStringAndType(
+          qet.getQec()->getIndex(), currentId, result->localVocab());


This seems like an unused duplicate.

joka921 · 2024-08-13T08:56:26Z

src/engine/ExportQueryExecutionTrees.cpp

+      nlohmann::ordered_json b;
+      if (!xsdType) {
+        // No xsdType, this means that `stringValue` is a plain string literal
+        // or entity.
+        b = stringToBinding(stringValue);
+      } else {
+        b["value"] = stringValue;
+        b["type"] = "literal";
+        b["datatype"] = xsdType;
+      }


This part can also be a separate function
(stringAndTypeToJsonBinding) or something like that.

joka921 · 2024-08-13T08:57:52Z

src/util/CMakeLists.txt

@@ -1,5 +1,5 @@
 add_subdirectory(ConfigManager)
 add_subdirectory(MemorySize)
 add_subdirectory(http)
-add_library(util GeoSparqlHelpers.cpp antlr/ANTLRErrorHandling.cpp ParseException.cpp Conversions.cpp Date.cpp DateYearDuration.cpp Duration.cpp antlr/GenerateAntlrExceptionMetadata.cpp CancellationHandle.cpp StringUtils.cpp)
+add_library(util GeoSparqlHelpers.cpp antlr/ANTLRErrorHandling.cpp ParseException.cpp Conversions.cpp Date.cpp DateYearDuration.cpp Duration.cpp antlr/GenerateAntlrExceptionMetadata.cpp CancellationHandle.cpp StringUtils.cpp LazyJsonParser.cpp)


You can relatively easy delete all theLazyJsonParser files and changes from this PR (git rm` etc and delete the single lines here).
That is in this case probably the easiest way to split this up.

test/ExportQueryExecutionTreesTest.cpp

… for json formats

joka921

This is already really nice.
I think for simplicity you can throw out the old (non-lazy) exporters,
then we have less code duplications.

joka921 · 2024-08-21T07:19:11Z

src/engine/Server.cpp

@@ -720,23 +726,12 @@ boost::asio::awaitable<void> Server::processQuery(
      case octetStream:
      case sparqlXml:
      case turtle:
+      case sparqlJson:
+      case qleverJson:


You can get rid of several lines here:

static constexpr std::array supportedTypes{MediaType::csv, MediaType::tsv, ...} AD_CORRECTNESS_CHECK(ad_utility::contains(supportedTypes, mediaType.value()); co_await sendStreamableResponse(....)

(Always clean up code as you go).

joka921 · 2024-08-21T07:21:06Z

test/ExportQueryExecutionTreesTest.cpp

+TEST(ExportQueryExecutionTrees, LiteralCornercases) {
+  auto runTestCaseWithLiteral = [](std::string literal) {
+    const std::string innerValue = literal.substr(1, literal.rfind("\"") - 1);
+    const std::string suffixValue = literal.substr(literal.rfind("\"") + 1);
+    std::string kg = absl::StrCat("<s> <p> ", literal);


Is this code by you, or is this shown because you haven't merged in the current master?

joka921 · 2024-08-21T07:33:10Z

src/engine/ExportQueryExecutionTrees.cpp

+// _____________________________________________________________________________
+cppcoro::generator<std::string>
+ExportQueryExecutionTrees::computeResultAsQLeverJSONStream(


For efficiency reasons
(The result has to be packed in reasonably large batches)
the inner generators (like for example this one here)
should not be generator<string> but stream_generator (see the other functions).
So the outermost generator of each type
(in particular this one here) should be a stream_generator,
And then best you do your switch for the QLeverJSON also in the single
computeResultAsStream function, which then transforms the stream_generator
back into a generator<string> which then contains large chunks.
If you want to know more details, this is best explained in a meeting.

src/engine/ExportQueryExecutionTrees.cpp

joka921 · 2024-08-21T07:39:42Z

src/engine/ExportQueryExecutionTrees.cpp

+template <>
+ad_utility::streams::stream_generator ExportQueryExecutionTrees::
+    selectQueryResultToStream<ad_utility::MediaType::sparqlJson>(
+        const QueryExecutionTree& qet,
+        const parsedQuery::SelectClause& selectClause,
+        LimitOffsetClause limitAndOffset,
+        CancellationHandle cancellationHandle) {


After putting some thought into this,
Maybe the cleaner way is to give
all those functions the [[maybe_unused]] request timer,
s.t. the qleverJson export can just be a normal template specialization here.

There's one more thing needed for the qleverJson spezialization methods, they both require ParsedQuery::_originalString which can't be accessed by e.g. ParsedQuery::SelectClause which is available in the selectQueryResultToStream template method.
So this cleanup would come at the price of passing the ParsedQuery query to each selectquery-/constructqueryToStream method and manually retrieving the required specific object there.

This is fine now as it is.

joka921

Currently all the tests fail,
But I have identified the problem and proposed a fix.
The remaining things are just very small renamings etc. s.t. we get this clean.
This is almost done, great job!

joka921 · 2024-08-27T10:32:41Z

src/engine/ExportQueryExecutionTrees.cpp

+      } else if (quotePos < entitystr.size() - 2) {
+        AD_CONTRACT_CHECK(entitystr[quotePos + 1] == '^');


As it seems this doesn' work because of our strange fulltext index , that's why the E2E tests fail.
Restore the old behavior, and then add a TODO<joka921> This can be aAD_CONTRACT_CHECK` once the fulltext index vocabulary is stored in a consistent format.

joka921 · 2024-08-27T10:35:51Z

src/engine/Server.cpp

+    static constexpr std::array supportedTypes{
+        MediaType::csv,       MediaType::tsv,    MediaType::octetStream,
+        MediaType::sparqlXml, MediaType::turtle, MediaType::sparqlJson,
+        MediaType::qleverJson};
+    AD_CORRECTNESS_CHECK(
+        ad_utility::contains(supportedTypes, mediaType.value()));


Is this check even needed?
Or can it be further inside (inside sendStreamableResponse or even better inside computeResultAsStream, s.t. we further eliminate code from the Server.cpp .

test/ExportQueryExecutionTreesTest.cpp

joka921 · 2024-08-27T10:41:02Z

src/engine/ExportQueryExecutionTrees.h

      CancellationHandle cancellationHandle);

  // ___________________________________________________________________________
-  static nlohmann::json constructQueryResultBindingsToQLeverJSON(
+  static cppcoro::generator<std::string>
+  constructQueryResultBindingsToQLeverJSONStream(


No need to call it ...Stream as you have deleted the old version. So the original name suffices.

joka921 · 2024-08-27T10:41:40Z

src/engine/ExportQueryExecutionTrees.h

@@ -133,14 +120,15 @@ class ExportQueryExecutionTrees {
   *        then the query result will be obtained via `qet->getResult()`.
   * @return a 2D-Json array corresponding to the IdTable given the arguments
   */
-  static nlohmann::json idTableToQLeverJSONArray(
+  static cppcoro::generator<std::string> idTableToQLeverJSONBindingsStream(


idTableToQLeverJSONBindings suffices (the Stream is not needed anymore, see below).

joka921 · 2024-08-27T10:43:35Z

src/engine/ExportQueryExecutionTrees.h

+  // TODO<unexenu>
+  static ad_utility::streams::stream_generator computeResultAsQLeverJSONStream(


No need for theStream in the name, as you have deleted the other one.

And a good comment is simply "Return the result of the query in the QLeverJSON format."
(There still is a TODO there.

joka921 · 2024-08-27T10:44:25Z

src/engine/ExportQueryExecutionTrees.h

@@ -37,7 +37,8 @@ class ExportQueryExecutionTrees {
  // lazily computes the serialized result in large chunks of bytes.
  static cppcoro::generator<std::string> computeResultAsStream(


As this is the only function left, we can simply call it computeResult (no AsStream needed, as everything is lazy.

joka921 · 2024-08-27T10:49:10Z

src/engine/ExportQueryExecutionTrees.cpp

+template <>
+ad_utility::streams::stream_generator ExportQueryExecutionTrees::
+    selectQueryResultToStream<ad_utility::MediaType::sparqlJson>(
+        const QueryExecutionTree& qet,
+        const parsedQuery::SelectClause& selectClause,
+        LimitOffsetClause limitAndOffset,
+        CancellationHandle cancellationHandle) {


This is fine now as it is.

sonarcloud · 2024-08-27T16:01:07Z

Quality Gate passed

Issues
12 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

joka921

Thank you very much.
This is a milestone for reducing QLever's RAM consumption and increasing its ability to compute and also export huge query results!

* This looks right to me. #1438 accidentally disabled the lazy computation of query results in JSON format. This commit not only reinstates this lazy computation, but also prefers lazy computation for all other export formats.

This fixes a regression introduced in #1438 that I didn't notice wasn't fixed as part of #1470. The runtime information was wrong for lazy operations (wrong number of rows and wrong times). This is now fixed + a unit tests is added that would have prevented this regression in the first place.

So far, when an error occurred after the sending of the result began, the HTTP status code was OK but the result was not. This resulted in confusing behavior in the QLever UI. In particular, if no result was produced at all, the QLever UI would wait for the result until the timeout and then show some generic error message. Since #1438, this affected any query with an error message, with the effect that no more specific error messages were shown in the QLever UI. This is now fixed now as follows: 1. When the error occurs *before* the sending of the result begins (the typical case), a non-200 HTTP code is sent just like before (this was broken by #1438). 2. When the error occurs *after* the sending of the result has begun, a 200 HTTP code is sent (HTTP provides no way to change this code after sending has begun), but when the error occurs a message is sent that begins with a highly unusual character sequence (currently `!!!!>>#`). This message could then be detected by the QLever UI or other tools. It is important to understand that this is not a problem specific to QLever, but a problem of the simplistic HTTP 1.1 protocol.

UNEXENU added 6 commits July 23, 2024 10:07

Base implementation of the LazyJsonParser.

b50aff3

separated different parsing sections, simplified path stack

bd0f993

fixed issues, improved test coverage

d51d600

more fixes

fc32b7b

added sparqlJson export stream

6bb5cff

Merge

e140da6

UNEXENU added 2 commits August 11, 2024 16:34

rebase + small changes

8c70415

Revert "rebase + small changes"

46ba89e

This reverts commit 8c70415.

joka921 requested changes Aug 13, 2024

View reviewed changes

UNEXENU added 5 commits August 13, 2024 11:38

removed accidentally copied LazyJsonParser-branch

ee10e3b

added qleverJson stream exporter, changed server to use stream-export…

0f730f2

… for json formats

fix code duplication

ef043da

fixed timer init-bug in tests

0f42462

simplified export of QLeverJson bindings

6c42102

joka921 requested changes Aug 21, 2024

View reviewed changes

UNEXENU added 2 commits August 23, 2024 11:03

merge and remove non-streaming json functions

d519785

fix coverage and duplication

b6f9263

joka921 requested changes Aug 27, 2024

View reviewed changes

final cleanup

349db27

joka921 approved these changes Aug 27, 2024

View reviewed changes

joka921 changed the title ~~Stream export for JSON~~ Lazily export JSON results Aug 27, 2024

joka921 merged commit ac257c7 into ad-freiburg:master Aug 27, 2024
19 of 20 checks passed

RobinTF mentioned this pull request Aug 27, 2024

Properly convert to milliseconds #1464

Closed

joka921 mentioned this pull request Aug 28, 2024

Reinstore the lazy computation for all export formats #1470

Merged

RobinTF mentioned this pull request Sep 2, 2024

Fix result duration computation for lazy results #1480

Merged

hannahbast mentioned this pull request Sep 5, 2024

Fix handling of exceptions and errors in lazy output #1472

Merged

Qup42 mentioned this pull request Sep 30, 2024

application/sparql-results+json query result is wrong forSELECT with OFFSET #1531

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lazily export JSON results #1438

Lazily export JSON results #1438

UNEXENU commented Aug 10, 2024 •

edited by joka921

Loading

codecov bot commented Aug 10, 2024 •

edited

Loading

joka921 left a comment

joka921 Aug 13, 2024

joka921 Aug 13, 2024

joka921 Aug 13, 2024

joka921 Aug 13, 2024

joka921 left a comment

joka921 Aug 21, 2024

joka921 Aug 21, 2024

joka921 Aug 21, 2024

joka921 Aug 21, 2024

UNEXENU Aug 22, 2024

joka921 Aug 27, 2024

joka921 left a comment

joka921 Aug 27, 2024

joka921 Aug 27, 2024

joka921 Aug 27, 2024

joka921 Aug 27, 2024

joka921 Aug 27, 2024

joka921 Aug 27, 2024

joka921 Aug 27, 2024

sonarcloud bot commented Aug 27, 2024

joka921 left a comment

		co_yield absl::StrCat("{\"head\":{\"vars\":[\"", absl::StrJoin(vars, "\",\""),
		"\"]},\"results\":{\"bindings\":[");

		const auto& optionalValue = idToStringAndType(
		qet.getQec()->getIndex(), currentId, result->localVocab());

		} else if (quotePos < entitystr.size() - 2) {
		AD_CONTRACT_CHECK(entitystr[quotePos + 1] == '^');

		// TODO<unexenu>
		static ad_utility::streams::stream_generator computeResultAsQLeverJSONStream(

		@@ -37,7 +37,8 @@ class ExportQueryExecutionTrees {
		// lazily computes the serialized result in large chunks of bytes.
		static cppcoro::generator<std::string> computeResultAsStream(

Lazily export JSON results #1438

Lazily export JSON results #1438

Conversation

UNEXENU commented Aug 10, 2024 • edited by joka921 Loading

codecov bot commented Aug 10, 2024 • edited Loading

Codecov Report

joka921 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joka921 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joka921 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarcloud bot commented Aug 27, 2024

Quality Gate passed

joka921 left a comment

Choose a reason for hiding this comment

UNEXENU commented Aug 10, 2024 •

edited by joka921

Loading

codecov bot commented Aug 10, 2024 •

edited

Loading