PyStack
Print the stack trace of a running Python process, or of a Python core dump.
PyStack is a tool that uses forbidden magic to let you inspect the stack frames of a running Python
process or a Python core dump, helping you quickly and easily learn what it's doing (or what it was
doing when it crashed) without having to interpret nasty CPython internals.
What PyStack can do
PyStack has the following amazing features:
- ๐ป Works with both running processes and core dump files.
- ๐งต Shows if each thread currently holds the Python GIL, is waiting to acquire it, or is currently
dropping it.
- ๐๏ธ Shows if a thread is running a garbage collection cycle.
- ๐ช Reports on multiple interpreters in the same process.
- ๐ Optionally shows native function calls, as well as Python ones. In this mode, PyStack prints
the native stack trace (C/C++/Rust function calls), except that the calls to Python callables are
replaced with frames showing the Python code being executed, instead of showing the internal C
code the interpreter used to make the call.
- ๐ Automatically demangles symbols shown in the native stack.
- ๐ Includes calls to inlined functions in the native stack whenever enough debug information is
available.
- ๐ Optionally shows the values of local variables and function arguments in Python stack frames.
- ๐ Safe to use on running processes. PyStack does not modify any memory or execute any code in a
process that is running. It simply attaches just long enough to read some of the process's
memory.
- โก Optionally, it can perform a Python stack analysis without pausing the process at all. This
minimizes impact to the debugged process, at the cost of potentially failing due to data races.
- ๐ Super fast! It can analyze core files 10x faster than general-purpose tools like GDB.
- ๐ฏ Even works with aggressively optimized Python interpreter binaries.
- ๐ Even works with Python interpreters' binaries that do not have symbols or debug information
(Python stack only).
- ๐ฅ Tolerates memory corruption well. Even if the process crashed due to memory corruption,
PyStack can usually reconstruct the stack.
- ๐ผ Self-contained: it does not depend on external tools or programs other than the Python
interpreter used to run PyStack itself.
What platforms are supported?
At this time only Linux is supported.
Building from source
If you wish to build PyStack from source, you need the following binary dependencies in your
system:
Note that sometimes both libraries are provided together as part of a distribution's elfutils
package.
Check your package manager on how to install these dependencies (e.g.,
apt-get install libdw-dev libelf-dev in Debian-based systems). Note that you may need to tell the
compiler where to find the header and library files of the dependencies for the build to succeed.
If pkg-config is available (e.g. apt-get install pkg-config on Debian-based systems), it will
automatically be used to locate the libraries and configure the correct build flags.
Check your distribution's documentation to determine the location of the header and library files
or for more detailed information. When building on Alpine Linux (or any other distribution that
doesn't use glibc) you'll need elfutils 0.188 or newer. You may need to build this from source if
your distribution's package manager doesn't have it.
Once you have these binary dependencies installed, you can clone the repository and follow the
typical build process for Python libraries:
git clone [email protected]:bloomberg/pystack.git pystack
cd pystack
python3 -m venv ../pystack-env/ # just an example, put this wherever you want
source ../pystack-env/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install -e . --group test --group extra
This will install PyStack in the virtual environment in development mode (the -e of the last
pip install command), and then install the Python libraries needed to test it, lint it, and
generate its documentation.
If you plan to contribute back, you should install the prek (pre-commit) hooks:
prek install
This will ensure that your contribution passes our linting checks.
Documentation
You can find the full documentation here.
Usage
PyStack uses distinct subcommands for analyzing running processes and core dump files.
usage: pystack [-h] [-v] [--no-color] {remote,core} ...
Get Python stack trace of a remote process
options:
-h, --help show this help message and exit
-v, --verbose
--no-color Deactivate colored output
commands:
{remote,core} What should be analyzed by PyStack (use <command> --help for a command-specific help section).
remote Analyze a remote process given its PID
core Analyze a core dump file given its location and the executable
Analyzing running processes
The remote command is used to analyze the status of a running (remote) process. The analysis is
always done in a safe and non-intrusive way, as no code is loaded in the memory space of the
process under analysis and no memory is modified in the remote process. This makes analysis using
PyStack a great option even for those services and applications that are running in environments
where the running process must not be impacted in any way (other than being temporarily paused,
though --no-block can avoid even that). There are several options available:
usage: pystack remote [-h] [-v] [--no-color] [--no-block] [--native] [--native-all] [--locals] [--exhaustive] pid
positional arguments:
pid The PID of the remote process
options:
-h, --help show this help message and exit
-v, --verbose
--no-color Deactivate colored output
--no-block do not block the process when inspecting its memory
--native Include the native (C) frames in the resulting stack trace
--native-all Include native (C) frames from threads not registered with the interpreter (implies --native)
--locals Show local variables for each frame in the stack trace
--exhaustive Use all possible methods to obtain the Python stack info (may be slow)
To use PyStack, you just need to provide the PID of the process:
$ pystack remote 112
Traceback for thread 112 [] (most recent call last):
(Python) File "/test.py", line 17, in <module>
first_func()
(Python) File "/test.py", line 6, in first_func
second_func()
(Python) File "/test.py", line 10, in second_func
third_func()
(Python) File "/test.py", line 14, in third_func
time.sleep(1000)
Analyzing core dumps
The core subcommand is used to analyze the status of a core dump file. Analyzing core files is
very similar to analyzing processes but there are some differences, as the core file does not
contain the totality of the memory that was valid when the program was live. In most cases, this
makes no difference, as PyStack will try to adapt automatically. However, in some cases, you will
need to specify extra command line options to help PyStack locate the information it needs. When
analyzing cores, there are several options available:
usage: pystack core [-h] [-v] [--no-color] [--native] [--native-all] [--locals] [--exhaustive] [--lib-search-path LIB_SEARCH_PATH | --lib-search-root LIB_SEARCH_ROOT] core [executable]
positional arguments:
core The path to the core file
executable (Optional) The path to the executable of the core file
options:
-h, --help show this help message and exit
-v, --verbose
--no-color Deactivate colored output
--native Include the native (C) frames in the resulting stack trace
--native-all Include native (C) frames from threads not registered with the interpreter (implies --native)
--locals Show local variables for each frame in the stack trace
--exhaustive Use all possible methods to obtain the Python stack info (may be slow)
--lib-search-path LIB_SEARCH_PATH
List of paths to search for shared libraries loaded in the core. Paths must be separated by the ':' character
--lib-search-root LIB_SEARCH_ROOT
Root directory to search recursively for shared libraries loaded into the core.
In most cases, you just need to provide the location of the core to use PyStack with core dump
files:
$ pystack core ./the_core_file
Using executable found in the core file: /usr/bin/python3.8
Core file information:
state: t zombie: True niceness: 0
pid: 570 ppid: 1 sid: 1
uid: 0 gid: 0 pgrp: 570
executable: python3.8 arguments: python3.8
The process died due receiving signal SIGSTOP
Traceback for thread 570 [] (most recent call last):
(Python) File "/test.py", line 19, in <module>
first_func({1: None}, [1,2,3])
(Python) File "/test.py", line 7, in first_func
second_func(x, y)
(Python) File "/test.py", line 12, in second_func
third_func(x, y)
(Python) File "/test.py", line 16, in third_func
time.sleep(1000)
License
PyStack is Apache-2.0 licensed, as found in the LICENSE file.
Code of Conduct
This project has adopted a Code of Conduct. If you have any concerns about the Code, or behavior
that you have experienced in the project, please contact us at [email protected].
Security Policy
If you believe you have identified a security vulnerability in this project, please send an email
to the project team at [email protected], detailing the suspected issue and any methods
you've found to reproduce it.
Please do NOT open an issue in the GitHub repository, as we'd prefer to keep vulnerability reports
private until we've had an opportunity to review and address them.
Contributing
We welcome your contributions to help us improve and extend this project!
Below you will find some basic steps required to be able to contribute to the project. If you have
any questions about this process or any other aspect of contributing to a Bloomberg open source
project, feel free to send an email to [email protected] and we'll get your questions
answered as quickly as we can.
Contribution Licensing
Since this project is distributed under the terms of an open source license,
contributions that you make are licensed under the same terms. For us to be able to accept your
contributions, we will need explicit confirmation from you that you are able and willing to provide
them under these terms, and the mechanism we use to do this is called a Developer's Certificate of
Origin (DCO). This is similar to the
process used by the Linux kernel, Samba, and many other major open source projects.
To participate under these terms, all that you must do is include a line like the following as the
last line of the commit message for each commit in your contribution:
Signed-Off-By: Random J. Developer <[email protected]>
The simplest way to accomplish this is to add -s or --signoff to your git commit command.
You must use your real name (sorry, no pseudonyms, and no anonymous contributions).
Steps
- Create an Issue, select 'Feature Request', and explain the proposed change.
- Follow the guidelines in the issue template presented to you.
- Submit the Issue.
- Submit a Pull Request and link it to the Issue by including "#" in the Pull Request
summary.
Please read CONTRIBUTING.md for more info.
