sys.monitoring is Python’s modern API for observing program execution with lower overhead and finer control than traditional tracing. It was designed for debuggers, profilers, coverage tools, performance analyzers, and observability systems that need specific execution events without receiving every possible trace notification.
This guide explains tool identifiers, event masks, callbacks, global and local monitoring, performance practices, error handling, testing, and production architecture.
Why sys.monitoring matters
Traditional tracing through sys.settrace can be expensive because a callback may run for many execution events. A tool that only needs function starts or returns still pays for broader tracing behavior. sys.monitoring allows tools to subscribe only to relevant events and, when appropriate, only for selected code objects.
This design is useful for code coverage, debugging, call counting, exception analysis, runtime diagnostics, and development tooling. Independent tools can coexist because each one uses its own tool identifier.
Reserve a tool identifier
A monitoring consumer first reserves an ID and associates it with a readable name. The ID isolates callbacks and event settings from other tools.
import sys
TOOL_ID = 3
sys.monitoring.use_tool_id(TOOL_ID, "my-monitor")
try:
pass
finally:
sys.monitoring.free_tool_id(TOOL_ID)
Always release the identifier. A context manager or dedicated class is a good way to guarantee cleanup when exceptions occur.
Select events carefully
The API exposes event constants that can be combined as a bit mask. Depending on the Python version, events cover Python function starts and returns, calls, jumps, instructions, exceptions, and other execution points.
events = sys.monitoring.events
mask = events.PY_START | events.PY_RETURN
sys.monitoring.set_events(TOOL_ID, mask)
Enable the smallest useful set. Monitoring every instruction when you only need function boundaries creates unnecessary work and larger data volumes.
Register callbacks
Callbacks are registered per event. Their arguments depend on the event, so verify the official documentation for the Python version you support.
def on_start(code, instruction_offset):
print("start:", code.co_name, instruction_offset)
sys.monitoring.register_callback(
TOOL_ID,
sys.monitoring.events.PY_START,
on_start,
)
Callbacks execute on the observed path. Keep them short. Prefer incrementing counters, writing compact records to a queue, or updating preallocated structures. Formatting logs, performing network calls, or running complex analysis inside a callback can distort the program being measured.
Global versus local monitoring
Global events affect a broad execution scope. Local events allow a tool to target specific code objects. Local monitoring is one of the most important techniques for reducing overhead.
A profiler for your application can monitor project modules while ignoring framework internals. A test tool can focus on selected functions. A debugger can activate detailed events only around a breakpoint or suspicious component.
A simple call counter
import sys
from collections import Counter
TOOL_ID = 3
calls = Counter()
def on_start(code, instruction_offset):
calls[code.co_name] += 1
sys.monitoring.use_tool_id(TOOL_ID, "call-counter")
sys.monitoring.register_callback(
TOOL_ID,
sys.monitoring.events.PY_START,
on_start,
)
sys.monitoring.set_events(
TOOL_ID,
sys.monitoring.events.PY_START,
)
# Run the application workload here.
sys.monitoring.set_events(TOOL_ID, 0)
sys.monitoring.free_tool_id(TOOL_ID)
print(calls)
A production implementation should use try/finally, avoid direct printing, and protect shared structures when threads are involved.
Performance guidelines
First, subscribe to minimal events. Second, filter modules and code objects. Third, keep callbacks constant-time whenever possible. Fourth, aggregate before exporting. Fifth, provide a configuration switch that disables monitoring completely.
Measure the monitor itself. Run representative benchmarks with instrumentation enabled and disabled. Observe CPU time, wall-clock latency, memory growth, and event volume. A tool that changes scheduling or response times too much may produce misleading conclusions.
Prevent reentrancy problems
A callback can execute Python code that is itself observable. Without care, the callback may trigger more callbacks. Use a thread-local or context-based guard to prevent accidental recursion. Keep callback dependencies small and avoid calling instrumented application functions.
Callbacks should also avoid raising exceptions into the monitored application. Catch internal errors, record a compact diagnostic, and disable the tool if its state is no longer trustworthy.
Threads and asynchronous programs
Shared counters need synchronization or thread-safe aggregation. For asynchronous services, associate observations with request context rather than a global variable. The article about Python contextvars explains context propagation across asyncio tasks. The guide to Python asyncio.Runner provides additional background on asynchronous execution lifecycle.
A useful architecture sends callback records to a bounded queue and processes them in a separate worker. Bounded queues prevent unlimited memory growth. If the queue is full, decide whether to drop samples, aggregate locally, or disable detailed monitoring.
Logs, metrics, and traces
Do not emit one log line for every event in a busy service. Aggregate calls, durations, exceptions, or samples. Structured records may include module name, function name, filename, line number, event type, and a correlation identifier.
For metrics, counters and histograms are usually more useful than raw streams. For traces, create spans only at meaningful boundaries. Filtering sensitive filenames, exception values, and user data is essential before export.
Testing a monitoring tool
Test normal functions, methods, generators, coroutines, nested calls, exceptions, recursion, and cancellation. Confirm that callbacks run for expected events, that local filters work, and that cleanup removes event registrations.
Tests should also cover reentrancy guards, concurrent updates, queue overflow, and partial initialization. Run a second test after shutdown to prove no callback leaked from the first test.
Version compatibility
sys.monitoring is unavailable in older Python releases. Reusable libraries should feature-detect it with hasattr(sys, "monitoring"). A fallback may use another tracing method, reduced functionality, or a clear compatibility error.
Read the official sys.monitoring documentation and PEP 669. For complementary introspection techniques, see Python inspect.signature and Python inspect.getmembers_static.
Recommended architecture
Separate configuration, collection, storage, and export. Configuration chooses events and filters. Collection callbacks create minimal records. Storage aggregates data. Export converts results into logs, metrics, files, or a user interface.
This separation makes callbacks easier to benchmark, storage easier to test, and exporters replaceable. It also supports multiple modes, such as lightweight always-on counters and temporarily enabled detailed diagnostics.
Security and privacy
Runtime instrumentation may expose paths, function names, exception details, and application structure. Treat collected data as operationally sensitive. Apply allowlists, redact secrets, define retention limits, and restrict access.
Do not turn monitoring callbacks into business logic. Application correctness should never depend on whether the observer is enabled. A monitoring failure should degrade observability, not the service itself.
When to choose another approach
Use ordinary logging when explicit domain events are enough. Use metrics when you need stable aggregate signals. Use sampling profilers when statistical performance data is sufficient. Choose sys.monitoring when execution-level events and selective instrumentation provide clear value.
Conclusion
sys.monitoring provides a strong foundation for modern Python tooling. Its selective events, tool isolation, and local monitoring controls can reduce overhead significantly. With minimal callbacks, careful filtering, cleanup, benchmarks, privacy controls, and robust tests, you can build useful profilers, debuggers, coverage tools, and diagnostics without making the observed application unreliable.
