How to ensure the consistency on the result of process_vm_readv

[BigBabolBee]

Hi,

It looks like this tool leverages process_vm_readv system call to get the running state of Python interpreter. I'm not sure whether the state we get is consistent.

While interpreter is running, PyInterpreterState is changing frome time to time. How to keep the state unchanged during the process of process_vm_readv without any program interrupt? I think it's possible that process_vm_readv is invoked in the middle of PyInterpreterState changing, which means the data we get is a mixture of old state and new state.

[P403n1x87]

I think it's possible that process_vm_readv is invoked in the middle of PyInterpreterState changing, which means the data we get is a mixture of old state and new state.

Indeed this is a very real possibility, but I think there isn't much we can do about it to completely solve the problem. I don't think pausing would yield any substantial improvements here because there is no guarantee that we would halt a thread at a time when everything is consistent.

Evidence seems to suggest that the data collected by Austin is sensible, despite this wild race condition. Besides, Austin tries to detect inconsistent stacks and report them. You would notice higher error rates if you are profiling very quick Python functions, but real-world applications generally have a mix of fast and slow functions.

Recent versions of Austin have added a heap parameter that can be used to tell Austin to allocate a chunk of memory that is used to read as many frames as possible with a single system call to attempt to reduce the error rate.

[P403n1x87]

The heap option is available since 3.2 (it was fixed and disabled by default in 3.3). It works best with Python 3.11, which chunks frame objects together, increasing the chances of grabbing as many frames as possible with a single system call 🙂

[BigBabolBee]

Great! I may not encounter the heap option part in code reading, just finished the procedure from main to py_proc__sample .

I've learned a lot from reading Austin and will proceed.

Thank you for your share.

[BigBabolBee]

Hi Gabriele, I'm still learning Austin and may have another confusion...

The sample procedure in Austin is PyInterpreterState->tstate_head->top_frame->next_thread->top_frame->........ I didn't find any code to check whether the current traversed thread is holding GIL.

I don't have enough knowledge about CPython, so apologies for that if this question is a silly one. I think only the thread who has GIL would have the right to occupy CPU. So no GIL, no running.

Austin will record all the top frames from all existing threads respectively. In some cases there may only one thread that is actually running on CPU and all others are just pending. I'm not sure Austin would distinguish this thread from others and mark it as HOT correctly.

ok... my english sucks just as my coding. Hope you could understand what I'm say.

Thanks

[P403n1x87]

The sample procedure in Austin is PyInterpreterState->tstate_head->top_frame->next_thread->top_frame->........ I didn't find any code to check whether the current traversed thread is holding GIL.

In general there is no need to check for the GIL, unless you really want to extract statistics about it. One exception is when we do memory profiling, since then we don't know which thread is running and likely to have changed the current RSS level. The code for that is here

austin/src/py_proc.c

Line 1295 in 4139481

current_thread = _py_proc__get_current_thread_state_raddr(self);

What we do is look at the interpreter state to get the current running thread.

I don't have enough knowledge about CPython, so apologies for that if this question is a silly one. I think only the thread who has GIL would have the right to occupy CPU. So no GIL, no running.

Austin will record all the top frames from all existing threads respectively. In some cases there may only one thread that is actually running on CPU and all others are just pending. I'm not sure Austin would distinguish this thread from others and mark it as HOT correctly.

That's right! When we do wall-time sampling, we don't care whether a thread is running or not. In CPU mode, we determine which thread is on CPU via the OS (e.g. on Linux we look at /proc/stat and determine the state of the thread

austin/src/linux/py_thread.h

Lines 36 to 72 in 4139481

	static int
	_py_thread__is_idle(py_thread_t * self) {
	with_resources;
	char file_name[64];
	char buffer[2048] = "";
	retval = -1;
	sprintf(file_name, "/proc/%d/task/" SIZE_FMT "/stat", self->proc->pid, self->tid);
	int fd = open(file_name, O_RDONLY);
	if (fd == -1) {
	log_d("Cannot open %s", file_name);
	return -1;
	}
	if (read(fd, buffer, 2047) == 0) {
	log_d("Cannot read %s", file_name);
	goto release;
	}
	char * p = strchr(buffer, ')');
	if (!isvalid(p)) {
	log_d("Invalid format for procfs file %s", file_name);
	goto release;
	}
	p+=2;
	if (*p == ' ')
	p++;
	retval = (*p != 'R');
	release:
	close(fd);
	released;
	}

What we see is that only one Python thread is running at a given time, and that's because of the GIL. Whilst we can use the OS information to conclude which thread currently holds the GIL, we can't go the other way, as a thread holding the GIL might be sleeping and hence pre-empted.

[BigBabolBee]

I've noticed _py_proc__get_current_thread_state_raddr and was curious about why it is only used in Memory Profiling mode. But now it is clear to me.

Whilst we can use the OS information to conclude which thread currently holds the GIL, we can't go the other way, as a thread holding the GIL might be sleeping and hence pre-empted.

I actually want to do some statistics about CPU time. I have spent some time trying to validate that gilstate in _PyRuntimeState is reliable to catch the running thread directly. But now I'm not sure if I need to proceed. Do you mean the GIL status may not truthfully report the running thhread?

In CPU mode, we determine which thread is on CPU via the OS (e.g. on Linux we look at /proc/stat and determine the state of the thread

Leveraging /proc/stat is an excellent idea! Got it!

Highly appreciated for your patience. Didn't expect you could elaborate all the details at the beginning... Apparently I didn't dig Austin carefully, sorry to disturb.

Thanks again