-
Notifications
You must be signed in to change notification settings - Fork 72
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
Browse the repository at this point in the history
* Allow Aurora multi-az deployments to be created * Add scripts to create Global Aurora cluster. Resolves #445 Automatic Aurora client failover with Route53 and Lambda * fix some issues * Add global aurora to rosa-cross-dc Taskfile * review updates * Fail aurora_delete_global_db script on errors * Add additional logging --------- Co-authored-by: Michal Hajas <[email protected]> Co-authored-by: Kamesh Akella <[email protected]>
- Loading branch information
1 parent
c424009
commit 2589900
Showing
15 changed files
with
817 additions
and
50 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
101 changes: 101 additions & 0 deletions
101
doc/kubernetes/modules/ROOT/pages/storage/aurora-global-postgres.adoc
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,101 @@ | ||
= Using Amazon Global Aurora PostgreSQL Cluster | ||
:description: An Amazon Global Aurora PostgreSQL Cluster can be used as the underlying database for Keycloak in either single, | ||
or multi-site configurations. Currently this is only supported with Keycloak deployments on ROSA. | ||
|
||
Aurora global database spans multiple AWS Regions, enabling disaster recovery from outages across an AWS Region. | ||
Aurora automatically handles replicating all data and updates from the primary AWS Region to each of the secondary Regions. | ||
|
||
== Deploying an Aurora Cluster | ||
|
||
Aurora clusters can be deployed across multiple AWS regions by executing `./provision/aws/rds/aurora_create_global_db.sh` with the | ||
following env: | ||
|
||
[source] | ||
---- | ||
AURORA_GLOBAL_REGIONS= # A list of AWS regions for the Aurora cluster to span. The first region in the list is where the Primary cluster is hosted. | ||
AURORA_GLOBAL_CLUSTER= # The name of the Global Aurora cluster | ||
AURORA_INSTANCES= # The number of Aurora db instances to create in each region, defaults to 1. | ||
---- | ||
|
||
This creates an Aurora cluster per region and associates then with the Global Aurora cluster `$AURORA_GLOBAL_CLUSTER`. | ||
The script waits until all regional clusters and their instance are available before returning. If the global cluster | ||
already exists, a message indicating this is displayed and the script will fail with exit code 1. | ||
|
||
An Aurora Global DB cluster consists of multiple regional clusters, each of which have their own dedicated Writer and Reader | ||
endpoints. In order to abstract this, we create a Route53 CNAME entry that Keycloak instances must utilise to connect to | ||
the database. The Route53 entry exposes the writer endpoint of the Aurora primary cluster at `$AURORA_GLOBAL_CLUSTER.aurora-global.keycloak-benchmark.com`. | ||
|
||
In order to ensure that the aforementioned Route53 entry reflects the writer endpoint of the Primary cluster after failover, | ||
we deploy an AWS Lambda function to each of the `$AURORA_GLOBAL_REGIONS`. This function is triggered on completion of a | ||
global-failover event, in the region of the new Primary cluster, and updates the CNAME entry to point to the latest writer | ||
endpoint. | ||
|
||
[NOTE] | ||
==== | ||
The specified `AURORA_GLOBAL_CLUSTER` must be unique per the AWS account and follow the conventions outlined for the | ||
https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.CreateInstance.html#Aurora.CreateInstance.Settings["DB cluster identifier"]. | ||
==== | ||
|
||
== Connecting ROSA cluster to Aurora Clusters | ||
|
||
A Peering Connection must be established between a ROSA cluster and each of the individual Aurora clusters. | ||
|
||
To configure such a connection run `./provision/aws/rds/aurora_create_global_peering_connections.sh` with the following environment: | ||
|
||
[source] | ||
---- | ||
AURORA_GLOBAL_CLUSTER= # The name of the Global Aurora cluster | ||
CLUSTER_NAME= # The name of the ROSA cluster to establish the peering connectin with | ||
---- | ||
|
||
== Deploying Keycloak | ||
|
||
When deploying Keycloak via the various task files, the following env variables must be set in order to ensure that the | ||
correct DB endpoint is configured. | ||
|
||
[source] | ||
---- | ||
AURORA_GLOBAL_CLUSTER= # The name of the Global Aurora cluster | ||
KC_DATABASE_URL=$AURORA_GLOBAL_CLUSTER.aurora-global.keycloak-benchmark.com | ||
KC_DATABASE=aurora-postgres | ||
---- | ||
|
||
== Simulating Cluster Failover | ||
It's possible to trigger a failover from the Primary to Secondary Aurora cluster by executing the link:https://awscli.amazonaws.com/v2/documentation/api/latest/reference/rds/failover-global-cluster.html[failover-global-cluster] command: | ||
|
||
[source] | ||
---- | ||
aws rds failover-global-cluster \ | ||
--global-cluster-identifier ${AURORA_GLOBAL_CLUSTER} \ | ||
--target-db-cluster-identifier ${AURORA_CLUSTER_IDENTIFIER} \ | ||
--allow-data-loss | ||
---- | ||
|
||
Where `AURORA_CLUSTER_IDENTIFIER` is the arn of the secondary cluster that you desire to become the Primary. The following command outputs arns for all members of the Global Aurora cluster: | ||
[source] | ||
---- | ||
aws rds describe-global-clusters \ | ||
--query "GlobalClusters[?GlobalClusterIdentifier=='${AURORA_GLOBAL_CLUSTER}'].GlobalClusterMembers[*].DBClusterArn" | ||
---- | ||
|
||
== Disconnecting ROSA cluster from Aurora Cluster | ||
|
||
To remove a Peering Connection between the ROSA and Aurora VPCS, execute `./provision/aws/rds/aurora_delete_global_peering_connection.sh` | ||
with the following env: | ||
|
||
[source] | ||
---- | ||
AURORA_GLOBAL_CLUSTER= # The name of the Global Aurora cluster | ||
CLUSTER_NAME= # The name of the ROSA cluster to remove the peering connection from | ||
---- | ||
|
||
== Deleting an Aurora Cluster | ||
Before deleting an Aurora cluster it's first necessary for all Peering Connections established with ROSA cluster(s) to | ||
be removed. | ||
|
||
To remove an Aurora cluster, execute `./provision/aws/rds/aurora_delete_global_db.sh` with the following env: | ||
|
||
[source] | ||
---- | ||
AURORA_GLOBAL_CLUSTER= # The name of the Global Aurora cluster | ||
---- |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.