[SUPPORT] Unable to Access PageExecutor Due to Internal Modifier

[YousifS7]

Hello Trino Team,

I hope this message finds you well.

Upon reviewing the codebase, I observed the PageExecutor class, which appears to function similarly to the RecordExecutor class by processing data in pages rather than individual records. This design seems advantageous for querying large datasets in a streaming manner without overwhelming system memory.

However, I noticed that the PageExecutor class is marked with the internal modifier, rendering it inaccessible outside its defining assembly. Could you please clarify whether this access level is intentional? If so, I would appreciate an explanation of the rationale behind restricting access to the PageExecutor class and any recommended approaches for implementing functionality that leverages this class.

Thank you for your time and assistance.

Best regards,

[mosabua]

Did you actually observe any problems using the RecordExecutor? If so.. what were they?

[YousifS7]

@mosabua I successfully implemented a solution using RecordExecutor, and it performed as expected. However, upon reviewing the code, I came across PageExecutor and found it potentially more suitable for our use case. Since granular control over individual records is not required, and our primary goal is to consume data in a streaming manner, I am interested in exploring a solution that leverages PageExecutor to process data in pages.

[mosabua]

I dont think RecordExecutor prevents you from ingesting in a streaming manner at all. What is your actual concern? Did you observe any issues?

Also note that we want to implement support for the Trino spooling protocol eventually.. that will provide even more performance and potential to load data in parallel.

[YousifS7]

@mosabua I did not observer any issues, I'm just curious why is PageExecutor locked with an internal modifier. Is that a mistake or is it intentional? If so, I would appreciate an explanation of the rationale behind restricting access to the PageExecutor class and any recommended approaches for implementing functionality that leverages this class.

[mosabua]

@georgewfisher will have to chime in... if we make it public we will have to update the docs so that users know when to use which class..

[YousifS7]

I wrote an implementation of that class, in case you find it useful for your docs:

ClientSession session = new TrinoConnectionProperties(){
    Server = new Uri(_Host),
    Catalog = catalog
}.GetSession();

PageExecutor pageExecutor = await PageExecutor.Execute(
logger: null,
queryStatusNotifications: null,
session: session,
statement: sqlQuery,
queryParameters: null,
bufferSize: 1024 * 1024 * 20,
isQuery: true,
cancellationToken).ConfigureAwait(false);

bool IsHeaderWritten = false;

var stringBuilder = new StringBuilder();
foreach (var page in pageExecutor)
{
    if (!IsHeaderWritten)
    {
        var header = string.Join(",", page.columns.Select(o => o.name));
        await WriteStreamAsync(stream, header);
        await WriteStreamAsync(stream, Environment.NewLine);
        IsHeaderWritten = true;
    }

    stringBuilder.Clear();

    foreach (var row in page.data)
    {
        var processedData = row.Select(item => EscapeCsvValue(item));
        stringBuilder.AppendJoin(",", processedData);
        stringBuilder.Append(Environment.NewLine);
    }

    await WriteStreamAsync(stream, stringBuilder.ToString());
}

[YousifS7]

Hi @georgewfisher, I’m following up on this issue. Could you kindly provide any insights on the question above?

Thank you