Debugging with PDB

How to DEBUG Galaxy API running on container

As galaxy_ng dev environment runs on containers and the API starts with gunicorn with multiple workers, it is difficult to run a pdb session in the default environment.

Debugging sometimes is crucial for API development, so there are some tweaks to get a running environment where you can debug normally using any PDB based tool.

Step 1. Running containers for debug

First step is to start all the dependency containers EXCEPT THE API (the one we want to debug)

./compose up resource-manager worker content-app

You can use ./compose up -d if you want it to run in background, it is recommended to have it running on a visible terminal so you can watch logs and messages.

NOTE: Wait at least until all TCP services are up, must be in 30 seconds or less.

Step 2. Running the API container in DEBUG mode

To enable the debugging mode we need to ensure API is served using the Django dev server with a single worker and also we want to install the debugging dependencies of our choice.

• ipdb
  Easier, all ipython features, no remote connection
• sdb
  recommended
  Feature full, has its own client, colorized, remote connection, SIGTRAP support to debug running processes
• remote_pdb
  Same as pdb, remote connection using nc, socat, telnet

---

Using ipdb

On a separate terminal start the django server with debugging enabled.

./compose run --rm --service-ports api /bin/bash -c "pip install ipython ipdb && ./entrypoint.sh manage runserver 0.0.0.0:8000"

Add the breakpoint

On the codeline you want to debug add:

...
__import__("ipdb").set_trace()
...

Save the code and make an api call to trigger the debugging session.

The debug console will open on the same console where you started the Django server.

---

Using sdb

On a separate terminal start the django server with debugging enabled.

./compose run --rm -p "5001:8000" -p "6899-6999:6899-6999" -e SDB_HOST=0.0.0.0 -e SDB_PORT=6899 -e "SDB_NOTIFY_HOST=$(ip -4 addr show docker0 | grep -Po 'inet \K[\d.]+')" api /bin/bash -c "pip install sdb && ./entrypoint.sh manage runserver 0.0.0.0:8000"

NOTE ON a Mac, If it doesn't work, try looking at ifconfig or ip addr | grep docker and set your docker host IP directly.

Listen for debug session on your local host using sdb-listen client.

Ia separate terminal on your local host.

$ pip install sdb
$ sdb-listen
listening for sdb notifications on :6899...

Add the breakpoint

On the codeline you want to debug add:

...
__import__("sdb").set_trace()
...

Save the code and make an api call to trigger the debugging session.

The debug session opens on the terminal where sdb-listen is running.

Use exit to stop the debugger.

BONUS

SIGTRAP allows the debugging of a running process without the need to specify a breakpoint.

On the top of the code you want to debug.

import sdb
sdb.sigtrap()

Then kill the process with -5 (do that starting an exec on the API container).

kill -5 <pid-of-process>

The debug session opens on the terminal where sdb-listen is running.

Other tips

sdb includes a few additional debugger aliases that make interactive debugging more pleasant:

• Prefix commands with an integer to repeat them. For example, 10n is the same as running next 10 times in a row.
• ? is the same as calling dir()
• ?? can be added to the end of a function call to view its source lines

---

Using remote_pdb (connect via nc, telnet or socat)

./compose run --rm -p "5001:8000" -p "4444:4444" -e REMOTE_PDB_HOST=0.0.0.0 -e REMOTE_PDB_PORT=4444 api /bin/bash -c "pip install remote-pdb && ./entrypoint.sh manage runserver 0.0.0.0:8000"

Add the breakpoint

...
__import__("remote_pdb").set_trace()
...

Save the code and make an api call to trigger the debugging session.

Connect using socat, telnet or nc

socat readline tcp:127.0.0.1:4444

or

nc -C 127.0.0.1 4444

Exit the debugger

To exit the debugger use exit or CTRL-D and ensure to press Ctrl-C after the main debugger exits to release the django server for subsequent calls.

---