Ability to print and log script backtraces

[reduz]

• Changes how stack information is stored in GDScript to a reverse linked list.
• This makes it fast enough to leave it enabled all time time on debug.
• Added a new script function script_backtrace() to get a string with the current script backtrace.
• Added the script backtrace to the logger. Errors now include the full currently running script backtraces.
• Added script backtrace printing to console logs (Windows, MacOS, Linux) using blue color.
• Additionally, added a custom Logger class to allow users intercept the internal messages/warnings/error (supersedes Add user-facing log interception callback. #87576).

How it looks:

using script_backtrace()

Results in the following output:

Logger

Backtraces are now added to the loggers. Here is how it looks for ANSI/Windows console loggers:

[realcoloride]

Extremely wanted feature that could make it way easier to debug. Love this

[mihe]

Seeing as how the existing error/warning printing formats file-with-line as %s:%i (as seen in your own screenshot), rather than %s : %i, the spaces in " : " should probably be removed.

Suggested change

	trace += "\n" + itos(i) + ": " + stack[i].func + " (" + stack[i].file + " : " + itos(stack[i].line) + ")";
	trace += "\n" + itos(i) + ": " + stack[i].func + " (" + stack[i].file + ":" + itos(stack[i].line) + ")";

[reduz]

makes sense

[bruvzg]

Not sure how reliable it will be, but it might be worth calling it from the crash handler as well (tested on macOS only and seems to be working):

diff --git a/platform/macos/crash_handler_macos.mm b/platform/macos/crash_handler_macos.mm
index c370422bfa..91f76dc3b5 100644
--- a/platform/macos/crash_handler_macos.mm
+++ b/platform/macos/crash_handler_macos.mm
@@ -31,6 +31,7 @@
 #include "crash_handler_macos.h"
 
 #include "core/config/project_settings.h"
+#include "core/object/script_language.h"
 #include "core/os/os.h"
 #include "core/string/print_string.h"
 #include "core/version.h"
@@ -171,6 +172,9 @@ static void handle_crash(int sig) {
 	}
 	print_error("-- END OF BACKTRACE --");
 	print_error("================================================================");
+	print_error(ScriptServer::get_current_script_backtrace());
+	print_error("-- END OF SCRIPT BACKTRACE --");
+	print_error("================================================================");
 
 	// Abort to pass the error to the OS
 	abort();

[RedMser]

How does the new function relate to the existing @GDScript.get_stack() and print_stack()? I assume there are technical differences, but these are not really made clear enough to the average user (when should I use this new function vs the existing ones?).

[farfalk]

In addition to what RedMser asked, how does this relate to #87576 ?

[Calinou]

This can probably be removed, it's trivial to readd later if we need it:

Suggested change

//const char *blue_bold = tty ? "\E[1;34m" : "";

[reduz]

@RedMser

This new one works always, even if not running the debugger and it also is generic for all languages. print_stack() was created mostly for testing and should be deprecated and removed since its pretty useless.

@farfalk

Both PRs should work great together

[permelin]

Did a quick test script with a function that is recursively called 500 times inside a for loop. To my surprise, this branch is 5% faster than master.

(Pushing the call stack beyond a depth of 506 or 507 will SIGSEGV, but that happens in master too.)

[dalexeev]

Can script_backtrace() return data in a structured form (Array[Dictionary]), like get_stack()? It also looks like this PR will resolve godotengine/godot-proposals#105.

[reduz]

@dalexeev I honestly don't see a lot of use for this, since the main use case is to print it, but I think adding another function script_stack() may be good for this, that could also include more stack information if you want.

[mihe]

Not sure how big of a deal this is, but if someone uses auto-complete (or just copies the signature verbatim) to implement this virtual method in GDScript, they will trigger the "Shadowed Global Identifier" warning due to the script_backtrace parameter shadowing the global function by the same name.

Maybe renaming the script_backtrace function to get_script_backtrace is the better option?

[mihe]

Seeing as how _log_message will end up getting called on every print, you can fairly easily get yourself into a stack overflow by trying to print anything from _log_message.

Might be good to mention in the documentation at least.

EDIT: I would assume the same applies when trying to push_error from _log_error.

[mihe]

This whole thing with keeping a pointer to this call_level here seems to cause some trouble with coroutines (functions that await), as they're allowed to exit this method without calling GDScriptLanguage::exit_function first (causing this pointer to go stale) since that gets called as part of GDScriptFunctionState::resume instead.

[Calinou]

Tested locally, it works as expected.

extends Node2D

var iteration := 0

func _ready() -> void:
	recurse()


func recurse() -> void:
	iteration += 1
	if iteration < 5:
		recurse()
	else:
		print(script_backtrace())
		for i in 20:
			# Needed so the print shows up reliably in the Output panel.
			await get_tree().process_frame
		get_tree().quit()

Results in:

stack_language: GDScript
0: recurse (res://node_2d.gd : 14)
1: recurse (res://node_2d.gd : 12)
2: recurse (res://node_2d.gd : 12)
3: recurse (res://node_2d.gd : 12)
4: recurse (res://node_2d.gd : 12)
5: _ready (res://node_2d.gd : 6)

Some remarks:

• Should the method be renamed to get_script_backtrace(), since it's a getter and has no side effects in itself?
• Printed stack traces seem to go to stdout, but I'd generally expect them to go to stderr like in most other languages.
• The stacktrace could have a note of the form Traceback (most recent call last) on the first line, like in Python.
  • To avoid making the stack trace longer, we can format the first line as follows:
  • GDScript traceback (most recent call last):
• Colored printing in errors seems to be missing a reset color code at the end. Here, I use push_error() then manually print some text afterwards (which shouldn't appear colored as a result):

• In error traces, I'd indent each line with 4 spaces so the entire error message is indented (except its first line).

[Mickeon]

A bit out of the loop, does #87576 essentially supersede this PR?

[reduz]

@Mickeon No, the other way around.

[RadiantUwU]

Is the SIGSEGV gonna be fixed so i could try it out with #94364 ?

[kleonc]

Can script_backtrace() return data in a structured form (Array[Dictionary]), like get_stack()? It also looks like this PR will resolve godotengine/godot-proposals#105.

@dalexeev I honestly don't see a lot of use for this, since the main use case is to print it, but I think adding another function script_stack() may be good for this, that could also include more stack information if you want.

Maybe the main use case is indeed to print it the default way, but for sure it needs to allow customization. One might want to print it but not with the default formatting (users do use custom logging for nice/clear output up to their liking). E.g. one could make the outputted stack trace entries clickable within the Godot editor, using BBCode (so they would take you to the relevant code).

With backtrace provided as a string one would need to do some reverse engineer parsing to get the actual data from it, which is not user friendly and not reliable.

I would argue that what's fundamental here is providing the actual stack data (e.g. as Array[Dictionary]). An easy way to obtain the whole backtrace string (with an arbitrary default formatting) I would consider as a nice-to-have bonus on top of that.

So I'm suggesting that the base should be:

• Logger._log_error() instead of taking a preformatted string script_backtrace argument, it should take script_stack argument being the actual stack data.
• Global script_stack() function returning the actual data, not a string. Could be named differently of course.

Then I wouldn't really mind what would be the additional stuff on top of that:

• Some kind of script_stack_to_string(Array[Dictionary]) utility function for default formatting?
• Additional script_backtrace() function? (same as script_stack_to_string(script_stack()))
• Etc.

---

This kinda remainds me of Node.print_tree{_pretty}. I'm guessing these methods were initially added as "the main use case". Later users requested/added Node.get_tree_string{_pretty} as it makes sense to output same thing to some UI or file.
Note that these methods still do not handle cases when you want custom formatting, or to print custom data for each Node etc. But that's not a big problem, because Node tree data is available and thus everyone can manually traverse it outputting whatever/however they like.

In case of a script stack trace this would be a problem, because the actual stack trace data would be not obtainable in the first place.

[felipejfc]

Either this or #87576 need to be merged with high priority. Not being able to log unhandled exceptions to a remote system is a blocker to launch any serious mobile game with Godot.
For the SIGSEV that's happening here, could you fix it or give directions @reduz ?

[mihe]

I was asked by @reduz to try to take this across the finish line a while back, and was underway with diagnosing the SIGSEV at one point, but other priorities have kept getting in the way.

If I remember correctly the SIGSEV in the unit tests are largely the same issue I mentioned above with regards to stale pointers as a result of await, but I didn't have time to actually experiment with any fixes for it.

@reduz also expressed an interest in seeing some variation of #93648 merged into this PR, since you would then get not only the script backtrace as part of log_error, but also the native backtrace.

How that native backtrace holds up across the various platforms remains to be seen, but seeing as how symbolication will likely be impossible on mobile platforms I suspect the better option might be to have the backtraces be returned in some kind of structured format, rather than just a string, similar to what has been suggested in this thread by @kleonc already. That way you could at least get the stack frame pointers as part of the structured format and send those off to some service somewhere to be symbolicated, if you wanted to, or just pass the structured format into some function to be symbolicated locally (assuming you actually have the symbols and the means to do so) and then finally into some other function to be string-formatted.

Anyway, I'll try to get to this in a (hopefully) near future, if no one else beats me to it.

[juse4pro]

Glad you are taking care of this topic @mihe 😃
This issue is a huge show stopper for our online game since Godot 4 came out. Unhandled C# exceptions that don't crash the game are terrible to debug. So this with PR we could gracefully get our logs / Sentry report and be happy.

Compared to Godot 3 a missing "feature". 😄

Is there any estimation if this PR could be merged into 4.4?

[felipejfc]

Possibly into 4.3 given it's a very small change that's easy to backport and is super important to any production project

[AThousandShips]

This is an enhancement so it's not going to go into 4.3 unless it's considered really exceptional, we don't cherry-pick enhancements