Add support for binary HTTP responses (using base64 encoding)

[italo1983]

Intro added by @mnapoli to clarify the issue following the discussion

API Gateway does not support well returning binary responses (for example PDF files). To support that, we need to return the response as base64 encoded. This is not something that should be done in your application: this is something that we need to implement in Bref.

---

Please apply this mod to LambdaResponse.php file to enable isBase64Encoded flag automatically:

public function toApiGatewayFormat(): array { // The headers must be a JSON object. If the PHP array is empty it is // serialized to [](we want{}`) so we force it to an empty object.
$headers = empty($this->headers) ? new \stdClass : $this->headers;

    // This is the format required by the AWS_PROXY lambda integration
    // See https://stackoverflow.com/questions/43708017/aws-lambda-api-gateway-error-malformed-lambda-proxy-response
    $base64 = $this->validBase64($this->body);
    return [
        'isBase64Encoded' => $base64,
        'statusCode' => $this->statusCode,
        'headers' => $headers,
        'body' => $this->body,
    ];
}

private function validBase64($string)
{
    $decoded = base64_decode($string, true);

    // Check if there is no invalid character in string
    if (!preg_match('/^[a-zA-Z0-9\/\r\n+]*={0,2}$/', $string)) return false;

    // Decode the string in strict mode and send the response
    if (!base64_decode($string, true)) return false;

    // Encode and compare it to original one
    if (base64_encode($decoded) != $string) return false;

    return true;
}`

[mnapoli]

Hi, this is very hard to read and understand. Maybe you could open a pull request instead?

And please could you explain why this is better, and whether there are downsides to encoding everything with base64?

[bubba-h57]

The official docs provide the Output Format of a Lambda Function Proxy Integration as:

{
    "isBase64Encoded": true|false,
    "statusCode": httpStatusCode,
    "headers": { "headerName": "headerValue", ... },
    "multiValueHeaders": { "headerName": ["headerValue", "headerValue2", ...], ... },
    "body": "..."
}

so the API Gateway expects to be told whether it is encoded or not, whether it is encoded or not.

The reasons for using Base64 encoding is typically because the response it binary. From the AWS Documentation:

The output body is marshalled to the frontend as the method response payload. If body is a binary blob, you can encode it as a Base64-encoded string by setting isBase64Encoded to true and configuring / as a Binary Media Type. Otherwise, you can set it to false or leave it unspecified.

Ref: https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html#api-gateway-simple-proxy-for-lambda-output-format
Ref: https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-payload-encodings-configure-with-console.html

[mnapoli]

Since we use PHP-FPM it's not really simple to choose whether to use base64 or not from PHP. I guess the question is: should we always return with base64? If so, are there downsides? I'm guessing that the response size might be bigger?

[bubba-h57]

We should not always return base64. I recommend bref should inspect the headers from PHP-FPM to determine if it should be base64 encoded.

For example, If the PHP-FPM header states that is a Binary Media Type we should encode it prior to sending it on to API Gateway. It could also be a zip file, PDF, or some other type, not just images or video. I can think of a number of use cases for dynamically creating zips, pdf's or images and returning them.

[italo1983]

Exactly, in my case i've used the lambda for creating zip files and pdf but also to output barcode and qrcode in differents formats.. Maybe there's a better way to check if the content is base64 but my code is working very well on a production environment.

PS. When using base64 encoded output we need to configure API Gateway to accept proper media type or using a regular expression (es. */*)

[bubba-h57]

Here is how I handle a similar issue in our laravel-bref-bridge package bootstrap.

https://github.com/stechstudio/laravel-bref-bridge/blob/9b4762f09cd6a8384fb6273506d04ee903d0bbec/src/Services/Bootstrap.php#L266-L278

[mnapoli]

If the PHP-FPM header states that is a Binary Media Type we should encode it prior to sending it on to API Gateway

@bubba-h57 makes a lot of sense! I'm not sure if it will be easy to build a reliable detection of binary responses. I mean there are a lot of content types: http://www.iana.org/assignments/media-types/media-types.xhtml

Another option would be to support a custom Bref-Binary HTTP header that PHP applications could set to ask for the response to be considered as a binary. But I don't really like this option as it's clearly not obvious.

[mnapoli]

Here is an idea: we need to encode in base64 if the response cannot be JSON encoded.

So why not try to always json_encode, and if it fails fall back to a base64 encoded response?

That way users don't have anything to do.

[skyrpex]

So why not try to always json_encode, and if it fails fall back to a base64 encoded response?

That could negatively impact response time. Reading headers should be better, I believe.

[victormacko]

I think i'm encountering this as well when trying to output image data as a response.

The error which comes back is;
Fatal error: Uncaught Exception: Failed encoding Lambda JSON response: Malformed UTF-8 characters, possibly incorrectly encoded in /var/task/vendor/bref/bref/src/Runtime/LambdaRuntime.php:248
Stack trace:
#0 /var/task/vendor/bref/bref/src/Runtime/LambdaRuntime.php(175): Bref\Runtime\LambdaRuntime->postJson('http://127.0.0....', Array)
#1 /var/task/vendor/bref/bref/src/Runtime/LambdaRuntime.php(92): Bref\Runtime\LambdaRuntime->sendResponse('29edfd67-4f42-4...', Array)
#2 /opt/bootstrap(34): Bref\Runtime\LambdaRuntime->processNextEvent(Object(Closure))

I've tried to encode the $data var in LambdaRuntime.php:248 using utf8_encode (which is an array, so doesn't encode), and the output JSON (which is 'false' (bool) ... I guess due to not being able to encoded).

[mnapoli]

This is finally done!

Thanks @victormacko, @shrink, @Guillaume-Rossignol, @italo1983, @bubba-h57 and all that participated. This was one of our most common issues and I'm really happy that we finally have a solution :)

I'll tag a release tomorrow!

[ashish-dirkmedia-de]

Tried with the function validBase64($string)
still getting error
Using laravel with following code to download

$fileMimeType = Storage::disk('s3')->mimeType($fileLocation);
$headers = [
'Content-Type' => $fileMimeType,
'Content-Disposition' => 'attachment; filename="'.$fileName.'"'
];
Storage::disk('s3')->download($fileLocation, 200, $headers);

[ashish-dirkmedia-de]

Even I tried
$contents = Storage::disk('s3')->get($fileLocation);

		return response(base64_encode($contents), 200)
			  ->header('Content-Type', $fileMimeType)
			  ->header('Content-Disposition', 'attachment; filename="'.$fileName.'"')
			  ->header('Cache-Control', 'public');

then the downloaded file is corrupted.

[mnapoli]

@ashish-dirkmedia-de please, plenty of people were involved in this issue that has been closed for some time. You duplicated your messages on several issues, this is not a behavior that is appropriate towards other project contributors and users.

Let's cool it off for a day and please open a separate issue with complete details, and let's stick to that new issue.