SQS support

[pm-mck]

I've been working on adding SQS support to this library and I ran into a some code in generate.ml that seems to assume all service url's are of the format:

(app1 "Uri.of_string" (str ("https://" ^ service ^ ".amazonaws.com")))

There needs to be a bit more logic to this, as the endpoints aren't really consistent, documented here: https://docs.aws.amazon.com/general/latest/gr/rande.html

I see the _endpoints.json file in the input directory and I'm thinking that by grabbing an updated version from boto, it could be used to generate an Endpoint module which would allow the appropriate url creation logic to be applied per service/region.

I have been mucking around with fixing some issues in the enum generator and ran into this. If wanted, I can probably pop open a wip PR for both items in a day or so.

[seliopou]

Seems like a reasonable approach. @tmcgilchrist sounds good to you?

[tmcgilchrist]

Sounds great!

[pm-mck]

So "a day or so" turned into a "week or so" but I have an proof of concept approach that achieves these things:

Generation Updates

• Add the latest endpoints.json from botocore, which I should have done first, because they completely changed the format. It ends up being quite a bit simpler but it chewed up some time
• Import atdgen in order to simplify reading the endpoints json file
• Provide endpoints.atd
• Adds an endpoint_gen.ml which is modeled after aws_gen.ml and generate.ml. It is extremely simplistic and simply generates a giant match module called Endpoints in /lib/endpoints.ml
• Imports Endpoints in the Aws module to make them available to the services
• Modifies generate.ml in order to support "flattened": true shapes. In particular, I found that SQS Messages are contained in a "MessageList" which the parse function was attempting to dereference in the response XML from AWS, which actually only actually contains 0 or more "Message" elements.
• Modifies generate.ml to support extracting the endpoints for a given region and service (the original issue)
• Add _service2.json from SQS to the appropriate folder and add the appropriate line to the Makefile to generate its modules
• Updates the Query.to_query_hashtbl function to support key transformations. SQS requires query params to be named based on their string representation (can provide an example if needed)
• Added a to_string and of_string to all BaseTypes to support the key transformations

Runtime Service Updates

• Added support for the x-amz-content-sha256 header for v4 signatures
• I believe I fixed a minor bug in the signature validation where an additional newline is required between the canonical headers and the signed headers
• fixed Yojson deprecation warnings #51
• Updated both LWT and Async runtimes to support calling to_http with both region and service to support endpoint selection
• Adds aws_sqs.opam

Limitations/Current issues not addressed

• No support for HTTP, I hardcoded HTTPS schemes
• No support for GovCloud or AWS China. This wouldn't be hard to add but it would require rethinking the design in a way I am not comfortable with
• No support for v2 signatures
• I haven't done anything with the to_json or of_json parts of the generation, I didn't want to get too far
• I need to test this whole thing a lot more than I have. I really focused hard on SQS, but I will make sure S3 and EC2 continue to work

I opened a draft PR here: #55

More than happy to discuss changes in approach or suggestions to improve the code.

[tmcgilchrist]

Thanks @pm-mck having a read over the PR now.

[Leonidas-from-XIV]

I ran into the same problem, in #60, since there is no rekognition.amazonaws.com and the to_http function which constructs the URL does not have access to the region.

[Leonidas-from-XIV]

So I think this can be closed now, thanks @pm-mck and @tmcgilchrist 🎉