Skip to content

feat: scale Postgres memory (config + container share) with instance RAM #100

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tonychang04 wants to merge 3 commits into
base: main
Choose a base branch
from
+49 −6
Open

feat: scale Postgres memory (config + container share) with instance RAM #100

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
419f78b
feat: scale Postgres memory settings with instance RAM
tonychang04 Jul 15, 2026
1e92c8f
feat: give Postgres the memory share on larger tiers, cap fixed-footp…
tonychang04 Jul 23, 2026
292f8a0
docs: fix stale shared_buffers rationale comment after rebalance
tonychang04 Jul 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 46 additions & 5 deletions auto-scale-memory.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,10 +55,30 @@ fi

echo "Scaling factor: ${SCALE_FACTOR}"

# Calculate new memory limits (rounded to nearest MB)
POSTGRES_MEM=$(awk "BEGIN {printf \"%.0f\", $POSTGRES_BASE * $SCALE_FACTOR}")
INSFORGE_MEM=$(awk "BEGIN {printf \"%.0f\", $INSFORGE_BASE * $SCALE_FACTOR}")
POSTGREST_MEM=$(awk "BEGIN {printf \"%.0f\", $POSTGREST_BASE * $SCALE_FACTOR}")
# Calculate new memory limits (rounded to nearest MB).
#
# Node (insforge) and PostgREST have ~fixed footprints — they do NOT use more
# RAM on a bigger box (measured: Node <100MB, PostgREST ~3MB at idle). Only
# Postgres benefits from more memory as the instance grows. So we cap those two
# at fixed ceilings and give Postgres the remainder, instead of scaling all
# three linearly. Linear scaling starved Postgres to ~43% of a large box: under
# memory-heavy load it pinned that cap and thrashed on swap while multiple GB of
# host RAM sat idle, and the Node container held GBs it never touched.
#
# The min() with the original linear share keeps small instances (nano/micro)
# unchanged — there the linear value is below the cap, so the cap never binds
# and behavior matches the load-tested small-tier config.
INSFORGE_CAP=512
POSTGREST_CAP=256
INSFORGE_LINEAR=$(awk "BEGIN {printf \"%.0f\", $INSFORGE_BASE * $SCALE_FACTOR}")
POSTGREST_LINEAR=$(awk "BEGIN {printf \"%.0f\", $POSTGREST_BASE * $SCALE_FACTOR}")
INSFORGE_MEM=$(( INSFORGE_LINEAR < INSFORGE_CAP ? INSFORGE_LINEAR : INSFORGE_CAP ))
POSTGREST_MEM=$(( POSTGREST_LINEAR < POSTGREST_CAP ? POSTGREST_LINEAR : POSTGREST_CAP ))
# Postgres gets everything left after the (capped) app + proxy — it's the
# workload that actually uses the memory. Never below its original linear share.
POSTGRES_LINEAR=$(awk "BEGIN {printf \"%.0f\", $POSTGRES_BASE * $SCALE_FACTOR}")
POSTGRES_MEM=$(( USABLE_MEM - INSFORGE_MEM - POSTGREST_MEM ))
if [ "$POSTGRES_MEM" -lt "$POSTGRES_LINEAR" ]; then POSTGRES_MEM=$POSTGRES_LINEAR; fi
Comment on lines +79 to +81

@coderabbitai coderabbitai Bot Jul 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Prevent rounded allocations from exceeding usable memory.

The three linear shares are rounded independently, so the PostgreSQL floor can over-allocate. For example, with TOTAL_MEM=384MB, the result is 152 + 51 + 152 = 355MB against 354MB usable. The later summary only prints this total.

Proposed fix 
-POSTGRES_LINEAR=$(awk "BEGIN {printf \"%.0f\", $POSTGRES_BASE * $SCALE_FACTOR}")
+POSTGRES_LINEAR=$(( USABLE_MEM - INSFORGE_LINEAR - POSTGREST_LINEAR ))
 POSTGRES_MEM=$(( USABLE_MEM - INSFORGE_MEM - POSTGREST_MEM ))
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
POSTGRES_LINEAR=$(awk "BEGIN {printf \"%.0f\", $POSTGRES_BASE * $SCALE_FACTOR}")
POSTGRES_MEM=$(( USABLE_MEM - INSFORGE_MEM - POSTGREST_MEM ))
if [ "$POSTGRES_MEM" -lt "$POSTGRES_LINEAR" ]; then POSTGRES_MEM=$POSTGRES_LINEAR; fi
POSTGRES_LINEAR=$(( USABLE_MEM - INSFORGE_LINEAR - POSTGREST_LINEAR ))
POSTGRES_MEM=$(( USABLE_MEM - INSFORGE_MEM - POSTGREST_MEM ))
if [ "$POSTGRES_MEM" -lt "$POSTGRES_LINEAR" ]; then POSTGRES_MEM=$POSTGRES_LINEAR; fi
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@auto-scale-memory.sh` around lines 79 - 81, Update the PostgreSQL allocation
logic around POSTGRES_LINEAR and POSTGRES_MEM so the rounded PostgreSQL floor
cannot make INSFORGE_MEM + POSTGREST_MEM + POSTGRES_MEM exceed USABLE_MEM. Cap
the PostgreSQL allocation at the remaining usable memory while preserving the
existing linear-floor minimum whenever it fits.

Comment on lines +80 to +81

@cubic-dev-ai cubic-dev-ai Bot Jul 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Near the minimum supported RAM sizes, rounding the linear values up makes the generated limits exceed USABLE_MEM, consuming part of the system reservation. Keep the remainder allocation when it is smaller than the rounded linear estimate.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At auto-scale-memory.sh, line 80:

<comment>Near the minimum supported RAM sizes, rounding the linear values up makes the generated limits exceed `USABLE_MEM`, consuming part of the system reservation. Keep the remainder allocation when it is smaller than the rounded linear estimate.</comment>

<file context>
@@ -55,10 +55,30 @@ fi
+# Postgres gets everything left after the (capped) app + proxy — it's the
+# workload that actually uses the memory. Never below its original linear share.
+POSTGRES_LINEAR=$(awk "BEGIN {printf \"%.0f\", $POSTGRES_BASE * $SCALE_FACTOR}")
+POSTGRES_MEM=$(( USABLE_MEM - INSFORGE_MEM - POSTGREST_MEM ))
+if [ "$POSTGRES_MEM" -lt "$POSTGRES_LINEAR" ]; then POSTGRES_MEM=$POSTGRES_LINEAR; fi
 # GHC heap cap for postgrest. Leave ~20MB for non-heap (binary, RTS internals,
</file context>
Suggested change
POSTGRES_MEM=$(( USABLE_MEM - INSFORGE_MEM - POSTGREST_MEM ))
if [ "$POSTGRES_MEM" -lt "$POSTGRES_LINEAR" ]; then POSTGRES_MEM=$POSTGRES_LINEAR; fi
POSTGRES_MEM=$(( USABLE_MEM - INSFORGE_MEM - POSTGREST_MEM ))

# GHC heap cap for postgrest. Leave ~20MB for non-heap (binary, RTS internals,
# thread stacks); floor at 20M to avoid pathological values on tiny instances.
POSTGREST_RTS_HEAP=$(( POSTGREST_MEM - 20 ))
Expand All @@ -80,6 +100,23 @@ else PG_MAX_CONNECTIONS=30; PGRST_DB_POOL=15
fi
echo "Connection scaling: PGRST_DB_POOL=${PGRST_DB_POOL}, PG_MAX_CONNECTIONS=${PG_MAX_CONNECTIONS} (RAM ${TOTAL_MEM}MB)"

# --- Scale Postgres memory settings with instance RAM ---------------------------
# postgresql.conf ships nano-safe values (shared_buffers=32MB etc.); without this
# block every tier runs them, so a large instance gets nano-tuned Postgres. Tiers
# mirror the connection scaling above. shared_buffers stays ~15-25% of the postgres
# CONTAINER limit (POSTGRES_MEMORY, ~35-40% of host RAM), not of host RAM, since the
# box is shared with the API. work_mem is per sort/hash node, so it is bounded per
# tier against worst-case work_mem * max_connections, not scaled linearly.
if [ "$TOTAL_MEM" -ge 30000 ]; then PG_SHARED_BUFFERS=2GB; PG_EFFECTIVE_CACHE_SIZE=8GB; PG_WORK_MEM=16MB; PG_MAINTENANCE_WORK_MEM=1GB # 2xl ~32G

@cubic-dev-ai cubic-dev-ai Bot Jul 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: Concurrent complex queries can exceed the PostgreSQL cgroup limit and be OOM-killed at the configured connection limits. Size PG_WORK_MEM from a bounded concurrent-query budget (and cap autovacuum_work_mem) rather than assigning 16MB to every potentially active backend.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At auto-scale-memory.sh, line 112:

<comment>Concurrent complex queries can exceed the PostgreSQL cgroup limit and be OOM-killed at the configured connection limits. Size `PG_WORK_MEM` from a bounded concurrent-query budget (and cap `autovacuum_work_mem`) rather than assigning 16MB to every potentially active backend.</comment>

<file context>
@@ -80,6 +100,25 @@ else                                  PG_MAX_CONNECTIONS=30;  PGRST_DB_POOL=15
+# there's headroom to raise it, but that's a follow-up load test, not a guess.
+# work_mem is per sort/hash node, so it's bounded per tier rather than scaled
+# linearly (worst-case work_mem * max_connections must stay within the container).
+if   [ "$TOTAL_MEM" -ge 30000 ]; then PG_SHARED_BUFFERS=2GB;   PG_EFFECTIVE_CACHE_SIZE=8GB;   PG_WORK_MEM=16MB; PG_MAINTENANCE_WORK_MEM=1GB    # 2xl ~32G
+elif [ "$TOTAL_MEM" -ge 15000 ]; then PG_SHARED_BUFFERS=1GB;   PG_EFFECTIVE_CACHE_SIZE=4GB;   PG_WORK_MEM=16MB; PG_MAINTENANCE_WORK_MEM=512MB  # xl ~16G
+elif [ "$TOTAL_MEM" -ge 7500  ]; then PG_SHARED_BUFFERS=512MB; PG_EFFECTIVE_CACHE_SIZE=2GB;   PG_WORK_MEM=8MB;  PG_MAINTENANCE_WORK_MEM=256MB  # large ~8G
</file context>

elif [ "$TOTAL_MEM" -ge 15000 ]; then PG_SHARED_BUFFERS=1GB; PG_EFFECTIVE_CACHE_SIZE=4GB; PG_WORK_MEM=16MB; PG_MAINTENANCE_WORK_MEM=512MB # xl ~16G
elif [ "$TOTAL_MEM" -ge 7500 ]; then PG_SHARED_BUFFERS=512MB; PG_EFFECTIVE_CACHE_SIZE=2GB; PG_WORK_MEM=8MB; PG_MAINTENANCE_WORK_MEM=256MB # large ~8G
elif [ "$TOTAL_MEM" -ge 3500 ]; then PG_SHARED_BUFFERS=256MB; PG_EFFECTIVE_CACHE_SIZE=1GB; PG_WORK_MEM=8MB; PG_MAINTENANCE_WORK_MEM=128MB # medium ~4G
elif [ "$TOTAL_MEM" -ge 1800 ]; then PG_SHARED_BUFFERS=128MB; PG_EFFECTIVE_CACHE_SIZE=512MB; PG_WORK_MEM=4MB; PG_MAINTENANCE_WORK_MEM=64MB # small ~2G
elif [ "$TOTAL_MEM" -ge 900 ]; then PG_SHARED_BUFFERS=64MB; PG_EFFECTIVE_CACHE_SIZE=256MB; PG_WORK_MEM=4MB; PG_MAINTENANCE_WORK_MEM=32MB # micro ~1G
else PG_SHARED_BUFFERS=32MB; PG_EFFECTIVE_CACHE_SIZE=128MB; PG_WORK_MEM=2MB; PG_MAINTENANCE_WORK_MEM=16MB # nano ~0.5G (= conf defaults)
fi
echo "Postgres memory scaling: shared_buffers=${PG_SHARED_BUFFERS}, effective_cache_size=${PG_EFFECTIVE_CACHE_SIZE}, work_mem=${PG_WORK_MEM}, maintenance_work_mem=${PG_MAINTENANCE_WORK_MEM}"

# Verify total doesn't exceed usable memory
TOTAL_ALLOCATED=$(( POSTGRES_MEM + POSTGREST_MEM + INSFORGE_MEM ))

Expand All @@ -99,7 +136,7 @@ ENV_FILE=".env"
cp "$ENV_FILE" "${ENV_FILE}.backup.$(date +%Y%m%d_%H%M%S)"

# Remove existing memory settings if present
sed -i.tmp '/^POSTGRES_MEMORY=/d; /^POSTGREST_MEMORY=/d; /^PGRST_DB_POOL=/d; /^PG_MAX_CONNECTIONS=/d; /^POSTGREST_RTS_HEAP=/d; /^INSFORGE_MEMORY=/d; /^DENO_MEMORY=/d; /^VECTOR_MEMORY=/d; /^NODE_EXPORTER_MEMORY=/d; /^# Auto-generated memory limits/d; /^# Total system memory:/d; /^# Usable memory:/d; /^# Scaling factor:/d' "$ENV_FILE"
sed -i.tmp '/^POSTGRES_MEMORY=/d; /^POSTGREST_MEMORY=/d; /^PGRST_DB_POOL=/d; /^PG_MAX_CONNECTIONS=/d; /^PG_SHARED_BUFFERS=/d; /^PG_EFFECTIVE_CACHE_SIZE=/d; /^PG_WORK_MEM=/d; /^PG_MAINTENANCE_WORK_MEM=/d; /^POSTGREST_RTS_HEAP=/d; /^INSFORGE_MEMORY=/d; /^DENO_MEMORY=/d; /^VECTOR_MEMORY=/d; /^NODE_EXPORTER_MEMORY=/d; /^# Auto-generated memory limits/d; /^# Total system memory:/d; /^# Usable memory:/d; /^# Scaling factor:/d' "$ENV_FILE"
rm -f "${ENV_FILE}.tmp"

# Append new memory settings
Expand All @@ -115,6 +152,10 @@ POSTGREST_RTS_HEAP=${POSTGREST_RTS_HEAP}M
INSFORGE_MEMORY=${INSFORGE_MEM}M
PGRST_DB_POOL=${PGRST_DB_POOL}
PG_MAX_CONNECTIONS=${PG_MAX_CONNECTIONS}
PG_SHARED_BUFFERS=${PG_SHARED_BUFFERS}
PG_EFFECTIVE_CACHE_SIZE=${PG_EFFECTIVE_CACHE_SIZE}
PG_WORK_MEM=${PG_WORK_MEM}
PG_MAINTENANCE_WORK_MEM=${PG_MAINTENANCE_WORK_MEM}
EOF

echo "Memory configuration updated in ${ENV_FILE}"
Expand Down
2 changes: 1 addition & 1 deletion docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ services:
image: ghcr.io/insforge/postgres:v15.13.4
container_name: insforge-postgres
restart: unless-stopped
command: postgres -c config_file=/etc/postgresql/postgresql.conf -c max_connections=${PG_MAX_CONNECTIONS:-100} -c app.encryption_key='${ENCRYPTION_KEY:-${JWT_SECRET:-dev-secret-please-change-in-production}}'
command: postgres -c config_file=/etc/postgresql/postgresql.conf -c max_connections=${PG_MAX_CONNECTIONS:-100} -c shared_buffers=${PG_SHARED_BUFFERS:-32MB} -c effective_cache_size=${PG_EFFECTIVE_CACHE_SIZE:-128MB} -c work_mem=${PG_WORK_MEM:-2MB} -c maintenance_work_mem=${PG_MAINTENANCE_WORK_MEM:-16MB} -c app.encryption_key='${ENCRYPTION_KEY:-${JWT_SECRET:-dev-secret-please-change-in-production}}'
deploy:
resources:
limits:
Expand Down