Skip to content

Commit

Permalink
chore: refine the replica-setup-pg and its doc
Browse files Browse the repository at this point in the history
  • Loading branch information
NoyException committed Nov 20, 2024
1 parent fe8bb4d commit 6391e87
Show file tree
Hide file tree
Showing 9 changed files with 73 additions and 88 deletions.
33 changes: 33 additions & 0 deletions devtools/load-from-pgdump/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Loading Data from PostgreSQL to MyDuck using pg_dump

This guide walks you through the process of loading data from an existing PostgreSQL instance to MyDuck Server.

## Prerequisites

Before you begin, ensure that:

- **MyDuck Server** is installed and running on your server.
- A **pg_dump** file is available.

## Getting Started

To let MyDuck Server replicate data from an existing PostgreSQL instance, run the provided `setup.sh` script.

### Usage

```bash
bash setup.sh --pg_dump /path/to/pg_dump
```

### Parameters

- **`--pg_dump`**: The path to the PostgreSQL dump file.
- **`--myduck_port`**: The port on which MyDuck Server is listening connections.

### Example

```bash
bash setup.sh --pg_dump pg_dump.sql
```

This command loads data from the PostgreSQL dump file `pg_dump.sql` into MyDuck Server.
9 changes: 9 additions & 0 deletions devtools/load-from-pgdump/checker.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
#!/bin/bash

# Function to check if a command was successful
check_command() {
if [[ $? -ne 0 ]]; then
echo "Error: $1 failed."
exit 1
fi
}
Original file line number Diff line number Diff line change
Expand Up @@ -62,34 +62,13 @@ if [[ -z $MYDUCK_DB ]]; then
usage
fi

# Step 1: Check if psql exists, if not, install it
# Step 1: Check if psql exists
if ! command -v psql &> /dev/null; then
echo "psql not found, attempting to install..."
bash install_psql.sh
check_command "psql installation"
else
echo "psql is already installed."
echo "Error: psql is not installed."
exit 1
fi

# Step 2: Check if replication has already been started
#echo "Checking if replication has already been started..."
#check_if_myduck_has_replica
#if [[ $? -ne 0 ]]; then
# echo "Replication has already been started. Exiting."
# exit 1
#fi

# Step 3: Prepare MyDuck Server for replication
#echo "Preparing MyDuck Server for replication..."
#source prepare.sh
#check_command "preparing MyDuck Server for replication"

# Step 4: Establish replication
#echo "Starting replication..."
#source start_replication.sh
#check_command "starting replication"

# Step 5: Load the existing data from pg_dump file
# Step 2: Load the existing data from pg_dump file
if [[ -n "$PG_DUMP" ]]; then
echo "Loading the snapshot from pg_dump to MyDuck Server..."
source snapshot.sh
Expand Down
File renamed without changes.
33 changes: 0 additions & 33 deletions devtools/replica-setup-pg/README.md

This file was deleted.

26 changes: 0 additions & 26 deletions devtools/replica-setup-pg/checker.sh

This file was deleted.

3 changes: 0 additions & 3 deletions devtools/replica-setup-pg/install_psql.sh

This file was deleted.

6 changes: 5 additions & 1 deletion pgserver/connection_handler.go
Original file line number Diff line number Diff line change
Expand Up @@ -1044,7 +1044,11 @@ func (h *ConnectionHandler) sendError(err error) {
}

// convertQuery takes the given Postgres query, and converts it as an ast.ConvertedQuery that will work with the handler.
func (h *ConnectionHandler) convertQuery(query string) (ConvertedQuery, error) {
func (h *ConnectionHandler) convertQuery(query string, modifiers ...QueryModifier) (ConvertedQuery, error) {
for _, modifier := range modifiers {
query = modifier(query)
}

parsable := true
stmts, err := parser.Parse(query)
if err != nil {
Expand Down
22 changes: 22 additions & 0 deletions pgserver/query_modifier.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
package pgserver

import "regexp"

type QueryModifier func(string) string

func NewQueryReplacer(regex string, replacement string) QueryModifier {
r := regexp.MustCompile(regex)
return func(query string) string {
return r.ReplaceAllString(query, replacement)
}

}

func NewQueryRemover(regex string) QueryModifier {
return NewQueryReplacer(regex, "")
}

var (
removeLocaleProvider = NewQueryRemover(`(?i)LOCALE_PROVIDER = [^ ;]*`)
removeLocale = NewQueryRemover(`(?i)LOCALE = [^ ;]*`)
)

0 comments on commit 6391e87

Please sign in to comment.