feat: Account for custom S3 compatible storages (add optional parameters to AWS adapter) #11
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -10,4 +10,4 @@ | |
| <ProjectReference Include="..\..\src\Estranged.Lfs.Api\Estranged.Lfs.Api.csproj" /> | ||
| </ItemGroup> | ||
|
|
||
| </Project> | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,102 @@ | ||
| AWSTemplateFormatVersion: "2010-09-09" | ||
| Parameters: | ||
| GitLfsUsername: | ||
| Type: String | ||
| Description: Username for authenticating against Git LFS endpoint | ||
| GitLfsPassword: | ||
| Type: String | ||
| Description: Password for authenticating against Git LFS endpoint | ||
| Outputs: | ||
| LfsEndpoint: | ||
| Description: The Git LFS endpoint to use | ||
| Value: !Sub 'https://${RestApi}.execute-api.${AWS::Region}.amazonaws.com/lfs' | ||
| Resources: | ||
| StorageBucket: | ||
| Type: AWS::S3::Bucket | ||
| DeletionPolicy: Retain | ||
| RestApi: | ||
| Type: AWS::ApiGateway::RestApi | ||
| Properties: | ||
| Body: | ||
| swagger: '2.0' | ||
| info: | ||
| description: 'Describes a proxy to a Lambda function to sign S3 requests.' | ||
| title: 'Git LFS REST API' | ||
| version: '1.0.0' | ||
| paths: | ||
| /{proxy+}: | ||
| x-amazon-apigateway-any-method: | ||
| produces: | ||
| - application/json | ||
| parameters: | ||
| - name: proxy | ||
| in: path | ||
| required: true | ||
| type: string | ||
| responses: {} | ||
| x-amazon-apigateway-integration: | ||
| responses: | ||
| default: | ||
| statusCode: 200 | ||
| uri: !Sub 'arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${SigningLambda}/invocations' | ||
| passthroughBehavior: when_no_match | ||
| httpMethod: POST | ||
| contentHandling: CONVERT_TO_TEXT | ||
| type: aws_proxy | ||
| Description: Git LFS endpoint | ||
| FailOnWarnings: true | ||
| Name: !Ref AWS::StackName | ||
| RestDeployment: | ||
| Type: AWS::ApiGateway::Deployment | ||
| Properties: | ||
| RestApiId: !Ref RestApi | ||
| StageName: lfs | ||
| SigningLambda: | ||
| Type: AWS::Lambda::Function | ||
| Properties: | ||
| Code: | ||
| S3Bucket: !Sub 'ae-infrastructure-${AWS::Region}' | ||
| S3Key: git-lfs/3.0.0/Estranged.Lfs.Hosting.Lambda.zip | ||
| Description: Generates S3 signed URLs for Git LFS | ||
| FunctionName: !Ref AWS::StackName | ||
| Handler: Estranged.Lfs.Hosting.Lambda::Estranged.Lfs.Hosting.Lambda.LambdaEntryPoint::FunctionHandlerAsync | ||
| MemorySize: 512 | ||
| Role: !GetAtt SigningLambdaRole.Arn | ||
| Runtime: dotnetcore3.1 | ||
| Timeout: 30 | ||
| Environment: | ||
| Variables: | ||
| LFS_BUCKET: !Ref StorageBucket | ||
| LFS_USERNAME: !Ref GitLfsUsername | ||
| LFS_PASSWORD: !Ref GitLfsPassword | ||
| GITHUB_ORGANISATION: !Ref GitOrganisation | ||
| GITHUB_REPOSITORY: !Ref GitHubRepositoryVariable | ||
| BITBUCKET_WORKSPACE: !Ref BitBucketWorkspaceVariable | ||
| BITBUCKET_REPOSITORY: !Ref BitBucketRepositoryVariable | ||
| S3Region: !Ref S3_REGION | ||
| S3ServiceURL: !Ref S3_SERVICE_URL | ||
| S3AccessKey: !Ref S3_ACCESS_KEY | ||
| S3AccessSecret: !Ref S3_ACCESS_SECRET | ||
| SigningLambdaGatewayPermission: | ||
| Type: AWS::Lambda::Permission | ||
| Properties: | ||
| Action: lambda:InvokeFunction | ||
| FunctionName: !GetAtt SigningLambda.Arn | ||
| Principal: apigateway.amazonaws.com | ||
| SourceArn: !Sub arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${RestApi}/* | ||
| SigningLambdaRole: | ||
| Type: AWS::IAM::Role | ||
| Properties: | ||
| AssumeRolePolicyDocument: | ||
| Version: "2012-10-17" | ||
| Statement: | ||
| - | ||
| Effect: Allow | ||
| Principal: | ||
| Service: | ||
| - lambda.amazonaws.com | ||
| Action: | ||
| - sts:AssumeRole | ||
| ManagedPolicyArns: | ||
| - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole | ||
| - arn:aws:iam::aws:policy/AmazonS3FullAccess |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -43,6 +43,8 @@ public async Task<SignedBlob> UriForDownload(string oid, CancellationToken token | |
| } | ||
| catch (AmazonS3Exception ex) | ||
| { | ||
| Console.WriteLine($"[ERROR] - {ex.StatusCode} - {ex.Message}"); | ||
| Console.WriteLine(ex.StackTrace); | ||
| return new SignedBlob | ||
| { | ||
| ErrorCode = (int)ex.StatusCode, | ||
|
|
@@ -62,12 +64,8 @@ public Task<SignedBlob> UriForUpload(string oid, long size, CancellationToken to | |
| { | ||
| return Task.FromResult(new SignedBlob | ||
| { | ||
|
Comment on lines
-67
to
-70
There was a problem hiding this comment. The S3 supports headers, if the API in use does not support headers, it's not S3 compatible There was a problem hiding this comment. It does support headers, but it's Why are use using it in the first place? I found this example that's use content type This small PR would open to many cloud providers which are compatible with S3. I'm sure the community will understand you give support mostly for AWS, and they'll contribute if other cloud providers bring problems (like I did). There was a problem hiding this comment. You said you couldn't test it, but maybe you could test it on AWS S3. There was a problem hiding this comment. Another use case that could be useful for people with this. It allows to use an AWS S3 bucket that's outside the stack, even in another region 😉 There was a problem hiding this comment. Hello @alanedwardes, have you been able to try testing as I suggested? |
||
| Uri = MakePreSignedUrl(oid, HttpVerb.PUT, null), | ||
| Expiry = config.Expiry | ||
| }); | ||
| } | ||
| } | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The point of these being packages on NuGet is so that this kind of granular customisation can happen outside of this repository (e.g. in your own application). I think this is too application specific to be useful in the sample application.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You are just using the default values, which only restrict to AWS S3.
It opens the configuration with optional parameters without breaking the existing AWS one.
With this solution, anyone could just clone the repo and is ready to go (required only to change the parameters to fit its cloud provider).