Buckets:

download
raw
12.3 kB
"""Survivable per-worker diagnostics for CI test runs.
A test lane that *fails* explains itself: pytest prints the assertion and the
traceback. A lane that is *killed* explains nothing — the job hits its
``timeout-minutes``, the runner sends a signal, and whatever pytest was about to
say dies in a buffer. That is the case this module covers.
Two records are written per process, both continuously flushed so they are
complete at every instant rather than at exit:
``events-<worker>.jsonl``
One line per test start and finish. A ``start`` with no matching ``finish``
is the test that was in flight when the process died — see
``scripts/ci/diag_summary.py``.
``stacks-<worker>.txt``
Every thread's stack, sampled at a fixed interval for the whole session,
plus a traceback if the interpreter crashes outright. Sampling runs for the
whole session rather than around each test, so this is a periodic snapshot,
not a per-test stuck detector: a healthy long run leaves a handful of dumps,
and a wedged one leaves the same frames over and over, which is what "stuck"
looks like. Covering the whole session is deliberate — it keeps dumping
after the LAST test has finished, which is where a shutdown hang lives, and
a per-test timer would be cancelled by then and show nothing.
Both are gated on ``OSPREY_CI_DIAG_DIR``: unset (every local run) means this
module installs nothing at all.
Why a file rather than stderr
-----------------------------
pytest's built-in ``faulthandler_timeout`` does reach the job log even under
``-n`` — xdist relays worker stderr to the controller, verified against
pytest-xdist 3.8.0. What it cannot do is survive the case this module exists
for. Its dump lands only in the job log, which is exactly what a cancelled job
truncates; it fires once per test, so a stack that is genuinely stuck looks the
same as one that was merely slow; and it says nothing at all about which test
each worker was inside. Writing to a file fixes the first (the artifact is
uploaded whatever happens to the log), repeated sampling fixes the second
(identical frames dumped again and again is what "stuck" looks like), and the
event log fixes the third.
Why the sampler is a Python thread, and not ``dump_traceback_later``
--------------------------------------------------------------------
``faulthandler.dump_traceback_later`` is the obvious way to write this, and it
is not safe here. Its watchdog lives in C and walks every thread's frames
*without holding the GIL* — that is the whole point of it, since it has to work
when the GIL is deadlocked. The cost is that it reads frames the interpreter is
still mutating. Under this suite that is not a theoretical race: the config and
template tests spend much of their time inside PyYAML, ruamel and Jinja2, whose
recursive-descent parsers build and tear down frames continuously, and a sample
landing there walks a half-freed frame and takes the process down with it. The
worker dies with no traceback and no protocol shutdown, xdist reports ``node
down: Not properly terminated``, and the lane then fails in one of two ways
that look nothing like each other: xdist's loadscope scheduler raises
``KeyError`` on the replacement worker, or the controller parks forever in
``dsession.loop_once`` waiting for an event the dead worker will never send.
Both were observed, on consecutive runs, caused by this module.
Calling ``faulthandler.dump_traceback`` from an ordinary Python thread holds
the GIL for the walk, so no frame can move underneath it. What that gives up is
the case the C watchdog exists for: a hard GIL deadlock, or a C extension
spinning without releasing it, starves this thread and no dump appears. Every
hang this lane has actually produced — blocked on a queue, a socket, or a
subprocess — releases the GIL and samples fine. A dump that never comes is
itself the signature of the case we cannot sample.
``faulthandler.enable`` is kept. It installs a fatal-signal handler that only
runs once the process is already lost, so it carries none of this risk, and it
is what turns a segfault in a C extension into something with a stack on it.
The unit lane still passes no ``faulthandler_timeout`` — pytest's built-in is
implemented with ``dump_traceback_later``, so arming it re-introduces exactly
the crash described above, on a per-test timer.
The per-test cap that does exist, and why it is ``signal``
----------------------------------------------------------
The lane passes ``--timeout=600 --timeout-method=signal`` (pytest-timeout), so
a hung test ends in ten minutes with its own name on the failure instead of
running out the 40-minute step cap and taking the log with it (#743). That is a
kill, and it is not in tension with the paragraphs above: the objection there is
to a *watchdog walking frames without the GIL*, which is a crash risk, not to
bounding a test.
``--timeout-method=thread`` would be. It ends the timed-out test with
``os._exit`` of the xdist worker, which is precisely the ``node down: Not
properly terminated`` case described above — the lane would go from one hung
test to a scheduler ``KeyError`` or a controller parked forever. The ``signal``
method raises inside the test and leaves the worker alive to report it, which is
what keeps the failure readable.
What ``signal`` gives up is the same thing this module's sampler gives up: a
hang inside a C extension that holds the GIL never services the SIGALRM, so no
timeout fires. In that case the step cap is again the only backstop, and the
uploaded stacks are the only evidence — which is why they keep being written
whether or not a timeout is armed. The value is 600 s rather than something
tighter so that a cold container image pull in the ``xdist_group("docker")``
files, which legitimately takes minutes, is never mistaken for a hang.
"""
from __future__ import annotations
import faulthandler
import json
import os
import threading
import time
from pathlib import Path
from typing import Any
#: Enables the whole module, and names the directory both records are written
#: to. The CI job uploads that directory as an artifact.
ENV_DIR = "OSPREY_CI_DIAG_DIR"
#: Seconds between whole-process stack snapshots. ``0`` disables stack dumping
#: while leaving the event log on.
ENV_STACK_TIMEOUT = "OSPREY_CI_DIAG_STACK_TIMEOUT"
#: The sampling interval, not a deadline: nothing is ever killed on this timer.
#: Five minutes keeps a healthy half-hour lane down to a handful of dumps while
#: still landing several inside any hang worth investigating.
DEFAULT_STACK_TIMEOUT = 300.0
#: How long ``stop()`` waits for an in-flight sample to finish before closing
#: the file under it. A dump of a few dozen threads takes milliseconds; this is
#: only here so a wedged sampler can never hold up the end of a run.
SAMPLER_JOIN_TIMEOUT = 5.0
def worker_id() -> str:
"""This process's xdist worker name, or ``main`` when running unparallelised.
Every artifact is keyed on it. With ``-n 4`` a freeze produces four separate
files, which is what makes "all workers stopped at the same instant"
(a runner-level stall) distinguishable from "one worker wedged" (a real
deadlock in a test).
"""
return os.environ.get("PYTEST_XDIST_WORKER", "main")
def worker_index() -> int:
"""This process's xdist worker number — ``gw3`` → 3 — or 0 when unparallelised.
For anything a worker has to own exclusively while its siblings run: a
port block, a container name, a scratch directory. ``main`` (no xdist)
takes 0, the same slot as ``gw0``, which is fine because the two never
coexist in one run.
"""
worker = worker_id()
if worker.startswith("gw") and worker[2:].isdigit():
return int(worker[2:])
return 0
class DiagnosticsRecorder:
"""Writes the event log and arms the stack dumper for one process."""
def __init__(
self,
directory: Path | str,
worker: str,
stack_timeout: float | None = DEFAULT_STACK_TIMEOUT,
) -> None:
self.directory = Path(directory)
self.worker = worker
self.stack_timeout = stack_timeout
self._events: Any = None
self._stacks: Any = None
self._stopped = False
self._sampler: threading.Thread | None = None
self._stop_sampling = threading.Event()
# -- lifecycle ---------------------------------------------------------
def start(self) -> None:
self.directory.mkdir(parents=True, exist_ok=True)
# buffering=1 is line buffering: every record reaches the OS the moment
# its newline is written, with no dependence on a clean shutdown.
self._events = (self.directory / f"events-{self.worker}.jsonl").open(
"a", buffering=1, encoding="utf-8"
)
if self.stack_timeout:
self._stacks = (self.directory / f"stacks-{self.worker}.txt").open(
"a", buffering=1, encoding="utf-8"
)
# enable() covers a hard interpreter crash: a segfault inside a C
# extension leaves no Python-level trace otherwise. It only fires
# on a fatal signal, so it cannot itself destabilise a healthy
# process — unlike the C watchdog this sampler replaces. See the
# module docstring.
faulthandler.enable(file=self._stacks, all_threads=True)
self._sampler = threading.Thread(
target=self._sample_stacks,
name="osprey-ci-diag-stacks",
daemon=True,
)
self._sampler.start()
self.record("session_start", stack_timeout=self.stack_timeout)
def _sample_stacks(self) -> None:
"""Dump every thread's stack on an interval until ``stop()`` is called.
Runs as an ordinary Python thread so the walk happens under the GIL and
cannot read a frame mid-mutation. ``Event.wait`` rather than ``sleep``
so a finished run is not held up for a whole interval.
"""
assert self.stack_timeout is not None
while not self._stop_sampling.wait(self.stack_timeout):
try:
faulthandler.dump_traceback(file=self._stacks, all_threads=True)
except (OSError, ValueError, AttributeError):
# Closed handle or full disk. Diagnostics never break a run.
return
def stop(self) -> None:
if self._stopped:
return
self.record("session_end")
self._stopped = True
if self._stacks is not None:
# Wake the sampler and let any in-flight dump finish before the
# file goes out from under it.
self._stop_sampling.set()
if self._sampler is not None:
self._sampler.join(SAMPLER_JOIN_TIMEOUT)
self._sampler = None
faulthandler.disable()
self._stacks.close()
self._stacks = None
if self._events is not None:
self._events.close()
self._events = None
# -- writing -----------------------------------------------------------
def record(self, event: str, **fields: Any) -> None:
"""Append one event. Never raises — diagnostics must not break a run."""
if self._events is None:
return
payload = {
"event": event,
"worker": self.worker,
"pid": os.getpid(),
"t": time.time(),
**fields,
}
try:
self._events.write(json.dumps(payload, default=str) + "\n")
self._events.flush()
except (OSError, ValueError): # closed handle, full disk
pass
def recorder_from_env() -> DiagnosticsRecorder | None:
"""Build a recorder from the environment, or ``None`` when CI diag is off."""
directory = os.environ.get(ENV_DIR)
if not directory:
return None
raw = os.environ.get(ENV_STACK_TIMEOUT)
if raw is None:
stack_timeout: float | None = DEFAULT_STACK_TIMEOUT
else:
try:
stack_timeout = float(raw) or None
except ValueError:
stack_timeout = DEFAULT_STACK_TIMEOUT
return DiagnosticsRecorder(directory, worker=worker_id(), stack_timeout=stack_timeout)

Xet Storage Details

Size:
12.3 kB
·
Xet hash:
838349cbb23eda36488521e314e67be21f7f2816743efa035b1907cfa2e943ee

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.