Skip to content

Commit faeee35

Browse files
authored
Fix script for setting up postgres development database (#12035)
* Increase shared memory size for postgres docker container I recently started getting errors that look like ``` E dbt_common.exceptions.base.DbtDatabaseError: Database Error E could not resize shared memory segment "/PostgreSQL.3814850474" to 2097152 bytes: No space left on device ``` At first I thought this was a lack of memory, disk space, or ulimit file descriptors. However increasing all of those things did not solve the problem. I eventually found, by exec-ing into the container and running `df -h /dev/shm && ls -lh /dev/shm` that the container only had 64MB of memory available to it. This change increases the memory available to the container to 1GB, which resolved the issue. * Use `docker compose` instead of `docker-compose` The later was docker v1, and no longer works. Use `docker compose` instead. * Only run homebrew postgres in `setup_db.sh` if `SKIP_HOMEBREW` is not passed Our github actions use homebrew, but our local dev uses docker. When we were doing local development and running `make setup-db` suddenly there would be _two_ postgres instances running. One via homebrew, and another in docker. This was breaking the setup. Now when running `make setup-db` we skip the homebrew relevant portions of `setup_db.sh`. * Set more PG environment variables in `setup_db.sh`
1 parent bca5c4d commit faeee35

File tree

3 files changed

+38
-23
lines changed

3 files changed

+38
-23
lines changed

Makefile

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -97,8 +97,8 @@ interop: clean
9797
.PHONY: setup-db
9898
setup-db: ## Setup Postgres database with docker-compose for system testing.
9999
@\
100-
docker-compose up -d database && \
101-
PGHOST=localhost PGUSER=root PGPASSWORD=password PGDATABASE=postgres bash test/setup_db.sh
100+
docker compose up -d database && \
101+
PGHOST=localhost PGUSER=root PGPASSWORD=password PGDATABASE=postgres SKIP_HOMEBREW=true bash test/setup_db.sh
102102

103103
# This rule creates a file named .env that is used by docker-compose for passing
104104
# the USER_ID and GROUP_ID arguments to the Docker image.

docker-compose.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ version: "3.5"
77
services:
88
database:
99
image: postgres
10+
shm_size: 1gb
1011
environment:
1112
POSTGRES_USER: "root"
1213
POSTGRES_PASSWORD: "password"

test/setup_db.sh

Lines changed: 35 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,31 +1,33 @@
11
#!/bin/bash
22
set -x
33

4-
brew install postgresql@16
5-
brew link postgresql@16 --force
6-
export PATH="/opt/homebrew/opt/postgresql@16/bin:$PATH"
4+
if [ "${SKIP_HOMEBREW:-false}" = "false" ]; then
5+
brew install postgresql@16
6+
brew link postgresql@16 --force
7+
export PATH="/opt/homebrew/opt/postgresql@16/bin:$PATH"
78

8-
# Start PostgreSQL using the full command instead of brew services
9-
pg_ctl -D /opt/homebrew/var/postgresql@16 start
9+
# Start PostgreSQL using the full command instead of brew services
10+
pg_ctl -D /opt/homebrew/var/postgresql@16 start
1011

11-
echo "Check PostgreSQL service is running"
12-
i=10
13-
COMMAND='pg_isready'
14-
while [ $i -gt -1 ]; do
15-
if [ $i == 0 ]; then
16-
echo "PostgreSQL service not ready, all attempts exhausted"
17-
exit 1
18-
fi
19-
echo "Check PostgreSQL service status"
20-
eval $COMMAND && break
21-
echo "PostgreSQL service not ready, wait 10 more sec, attempts left: $i"
22-
sleep 10
23-
((i--))
24-
done
12+
echo "Check PostgreSQL service is running"
13+
i=10
14+
COMMAND='pg_isready'
15+
while [ $i -gt -1 ]; do
16+
if [ $i == 0 ]; then
17+
echo "PostgreSQL service not ready, all attempts exhausted"
18+
exit 1
19+
fi
20+
echo "Check PostgreSQL service status"
21+
eval $COMMAND && break
22+
echo "PostgreSQL service not ready, wait 10 more sec, attempts left: $i"
23+
sleep 10
24+
((i--))
25+
done
2526

26-
createuser -s postgres
27+
createuser -s postgres
2728

28-
env | grep '^PG'
29+
env | grep '^PG'
30+
fi
2931

3032
# If you want to run this script for your own postgresql (run with
3133
# docker-compose) it will look like this:
@@ -35,6 +37,18 @@ export PGUSER
3537
PGPORT="${PGPORT:-5432}"
3638
export PGPORT
3739
PGHOST="${PGHOST:-localhost}"
40+
export PGHOST
41+
PGDATABASE="${PGDATABASE:-postgres}"
42+
export PGDATABASE
43+
: "${PGPASSWORD:=password}"
44+
export PGPASSWORD
45+
46+
# Normalize localhost to IPv4 to avoid IPv6 (::1) surprises
47+
if [ "${PGHOST}" = "localhost" ]; then
48+
PGHOST="127.0.0.1"
49+
export PGHOST
50+
fi
51+
3852

3953
for i in {1..10}; do
4054
if pg_isready -h "${PGHOST}" -p "${PGPORT}" -U "${PGUSER}" ; then

0 commit comments

Comments
 (0)