LND ships with a few useful features for debugging, such as a built-in profiler and tunable logging levels. If you need to submit a bug report for LND, it may be helpful to capture debug logging and performance data ahead of time.
Logging is useful for security and operating purposes. LND logs can typically be found on Linux at ~/.lnd/logs/bitcoin/mainnet/lnd.log, on macOS at ~/Library/Application Support/Lnd/logs/bitcoin/mainnet or in their specified location using the--logdir flag at startup.
By default, LND will log 10MB worth of its history, and additionally keep three blocks of logs around, compressed with gzip as lnd.log.<i>.gz in the same directory.
You can adjust the location of your log files as well their maximum size (in MB) and how many historical log files you expect LND to keep, in your lnd.conf file.
logdir=~/.lnd/logs
maxlogfiles=3
maxlogfilesize=10
debuglevel=debug,PEER=info
Additionally, the debuglevel can be overridden and adjusted without requiring a restart using the command lncli debuglevel --level=
The available debug levels are, in order of descending detail: trace, debug, info, warn, error, critical, off
Example usage:
lncli debuglevel –-level=debug
Additionally, the log level can be adjusted for each individual subsystem. A list of the subsystem can also be obtained with the command lncli debuglevel --show
Varying debug levels for multiple subsystems can be chained together with commands.
Example usage:
debuglevel --level=BTCN=trace,LNWL=debug
Subsystems:
| LNWL | lnwallet | Lightning Wallet |
|---|---|---|
| DISC | discovery | Discovery |
| NTFN | chainntnfs | Chain Notifications |
| CHDB | channeldb | Channel database |
| HSWC | htlcswitch | HTLC Switch |
| CMGR | connmgr | Connection Manager |
| BTCN | neutrino | Neutrino |
| CNCT | contractcourt | Contract Court |
| UTXN | contractcourt | Contract Court |
| BRAR | contractcourt | Contract Court |
| SPHX | sphinx | Sphinx |
| SWPR | sweep | Sweep Transactions |
| SGNR | signrpc | Signature RPC |
| WLKT | walletrpc | Wallet RPC |
| ARPC | autopilotrpc | Autopilot RPC |
| INVC | invoices | Invoices |
| NANN | netann | Network Announcements |
| WTWR | watchtower | Watchtower |
| NTFR | chainrpc | Chain RPC |
| IRPC | invoicesrpc | Invoices RPC |
| CHNF | channelnotifier | Channel Notifier |
| CHBU | chanbackup | Channel backup |
| PROM | monitoring | Monitoring |
| WTCL | wtclient | Watch Tower Client |
| PRNF | peernotifier | Peer Notifier |
| CHFD | chanfunding | Channel Funding |
| PEER | peer | Peer |
| CHCL | chancloser | Channel Closer |
lnd has a built-in feature which allows you to capture profiling data at runtime using pprof, a profiler for Go. The profiler has negligible performance overhead during normal operations (unless you have explicitly enabled CPU profiling).
To enable this ability, start lnd with the --profile option using a free port or add profile=9736 to your lnd.conf.
⛰ lnd --profile=9736
Now, with lnd running, you can use the pprof endpoint on port 9736 to collect runtime profiling data. You can fetch this data using curl like so:
⛰ curl http://localhost:9736/debug/pprof/goroutine?debug=1
...
A CPU profile can be used to analyze LND's CPU usage. You can specify the time duration as a query parameter.
⛰ curl http://localhost:9736/debug/pprof/profile?seconds=10 > cpu.prof
The goroutine profile is very useful when analyzing deadlocks and lock contention.
⛰ curl http://localhost:9736/debug/pprof/goroutine?debug=2 > goroutine.prof
The heap profile is useful to analyze memory allocations.
⛰ curl http://localhost:9736/debug/pprof/heap > heap.prof
It can be hard to make sense of the profile dumps by just looking at them. The Golang ecosystem provides tools to analyze those profile dumps either via the terminal or by visualizing them. One of the tools is go tool pprof.
Assuming the profile was fetched via curl as in the examples above a nice svg visualization can be generated for the cpu profile like this:
⛰ go tool pprof -svg cpu.prof > cpu.svg