Spaces:
Sleeping
feat: read per-product and per-region results from the filing's XBRL instance
Browse filescompanyfacts returns only undimensioned facts, which is why the brief has
never had a segment breakdown. The detail is in the inline-XBRL instance
document that ships inside the filing itself, where facts carry explicit
dimension members (srt:ProductOrServiceAxis,
us-gaap:StatementBusinessSegmentsAxis). One fetch per filing yields the
quarter and its prior-year comparable, so year-on-year costs no extra
request.
Two traps that would produce plausible wrong numbers, both covered by tests:
- A filing tags subtotals beside detail members (Apple's "Products" equals
iPhone + Mac + iPad + Wearables). Plotting both inflates the mix by 72%.
The subtotal is found by anchoring on consolidated revenue and taking the
subset equal to the excess, so no per-issuer member list is needed.
- The structural fallback enumerates subsets, and an early break on an
oversized sum is wrong: itertools.combinations is not ordered by sum.
analytics/impact.py adds the arithmetic that turns the breakdown into an
answer: contribution to group growth in percentage points (which must sum
to the group growth rate) and a mix-versus-rate margin split (which must
sum to the margin change). Both identities are asserted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- analytics/impact.py +195 -0
- analytics/segments.py +252 -0
- ingest.py +16 -0
- ingestion/segments.py +265 -0
- scripts/backfill_segments.py +102 -0
- storage/segments_db.py +154 -0
- tests/test_impact.py +164 -0
- tests/test_segments.py +286 -0
|
@@ -0,0 +1,195 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""How much a driver moved a number β computed, never estimated.
|
| 2 |
+
|
| 3 |
+
The question "what did the supply constraint cost in margin?" has three
|
| 4 |
+
possible answers and only two of them are allowed here.
|
| 5 |
+
|
| 6 |
+
**Computed.** Arithmetic on figures the company disclosed. A product line's
|
| 7 |
+
contribution to group growth is exactly ``(revenue_now - revenue_prior) /
|
| 8 |
+
group_revenue_prior``; the contributions sum to the group growth rate, which
|
| 9 |
+
is the checksum. A margin change splits exactly into a mix effect and a rate
|
| 10 |
+
effect. Both are reproducible from the filing and are computed here.
|
| 11 |
+
|
| 12 |
+
**Disclosed.** The company's own quantification β a guided margin range
|
| 13 |
+
against a reported one. The model may transcribe those figures under the
|
| 14 |
+
usual citation rules, and ``bridge()`` does the subtraction. It never
|
| 15 |
+
supplies a figure of its own.
|
| 16 |
+
|
| 17 |
+
**Estimated.** "Supply constraints probably cost 150bp." Not produced here,
|
| 18 |
+
not accepted from a model, and not renderable. A number with no source is
|
| 19 |
+
indistinguishable from a number that is wrong, and this product's whole
|
| 20 |
+
claim is that it does not print those.
|
| 21 |
+
"""
|
| 22 |
+
from __future__ import annotations
|
| 23 |
+
|
| 24 |
+
from typing import Optional
|
| 25 |
+
|
| 26 |
+
from ingestion.segments import PRODUCT_AXIS, SEGMENT_AXIS
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
def growth_contributions(ticker: str, axis: str = PRODUCT_AXIS) -> dict:
|
| 30 |
+
"""Each member's contribution, in percentage points, to group growth.
|
| 31 |
+
|
| 32 |
+
A line growing 29% matters less than one growing 12% if the first is a
|
| 33 |
+
tenth of the second's size. Contribution is the figure that settles it,
|
| 34 |
+
and the contributions add up to the group growth rate -- returned as
|
| 35 |
+
``total_pp`` so the arithmetic can be checked on screen.
|
| 36 |
+
"""
|
| 37 |
+
from analytics.segments import breakdown
|
| 38 |
+
|
| 39 |
+
view = breakdown(ticker, axis)
|
| 40 |
+
lines = [
|
| 41 |
+
line for line in (view.get("lines") or [])
|
| 42 |
+
if line.get("revenue") and line.get("prior_revenue")
|
| 43 |
+
]
|
| 44 |
+
if not lines:
|
| 45 |
+
return {}
|
| 46 |
+
|
| 47 |
+
prior_total = sum(line["prior_revenue"] for line in lines)
|
| 48 |
+
if not prior_total:
|
| 49 |
+
return {}
|
| 50 |
+
|
| 51 |
+
contributions = [
|
| 52 |
+
{
|
| 53 |
+
"label": line["label"],
|
| 54 |
+
"delta": line["revenue"] - line["prior_revenue"],
|
| 55 |
+
"contribution_pp": (line["revenue"] - line["prior_revenue"]) / prior_total * 100,
|
| 56 |
+
"yoy_pct": line.get("yoy_pct"),
|
| 57 |
+
"revenue": line["revenue"],
|
| 58 |
+
}
|
| 59 |
+
for line in lines
|
| 60 |
+
]
|
| 61 |
+
contributions.sort(key=lambda c: c["contribution_pp"], reverse=True)
|
| 62 |
+
return {
|
| 63 |
+
"axis": axis,
|
| 64 |
+
"period_end": view.get("period_end"),
|
| 65 |
+
"prior_period_end": view.get("prior_period_end"),
|
| 66 |
+
"lines": contributions,
|
| 67 |
+
"total_pp": sum(c["contribution_pp"] for c in contributions),
|
| 68 |
+
"source_url": view.get("source_url") or "",
|
| 69 |
+
}
|
| 70 |
+
|
| 71 |
+
|
| 72 |
+
def margin_decomposition(ticker: str) -> dict:
|
| 73 |
+
"""Split the segment margin change into mix and rate, exactly.
|
| 74 |
+
|
| 75 |
+
Two ways a group margin can move: the business earns more per dollar in
|
| 76 |
+
the places it already sells (rate), or the sales shift towards places
|
| 77 |
+
that already earned more (mix). Standard two-factor split --
|
| 78 |
+
|
| 79 |
+
mix = sum over members of (weight_now - weight_prior) * margin_prior
|
| 80 |
+
rate = sum over members of weight_now * (margin_now - margin_prior)
|
| 81 |
+
|
| 82 |
+
-- and the two add to the total change, which is returned so the identity
|
| 83 |
+
can be shown rather than asserted. Only the reportable-segment axis
|
| 84 |
+
carries operating income, so this is a regional read.
|
| 85 |
+
"""
|
| 86 |
+
from analytics.segments import breakdown
|
| 87 |
+
from storage import segments_db
|
| 88 |
+
|
| 89 |
+
view = breakdown(ticker, SEGMENT_AXIS)
|
| 90 |
+
period_end = view.get("period_end")
|
| 91 |
+
prior_end = view.get("prior_period_end")
|
| 92 |
+
if not period_end or not prior_end:
|
| 93 |
+
return {}
|
| 94 |
+
|
| 95 |
+
prior_rows = {
|
| 96 |
+
row["member"]: row
|
| 97 |
+
for row in segments_db.read_segments(ticker, SEGMENT_AXIS, prior_end)
|
| 98 |
+
}
|
| 99 |
+
pairs = []
|
| 100 |
+
for line in view.get("lines") or []:
|
| 101 |
+
was = prior_rows.get(line.get("member"))
|
| 102 |
+
if not was or not was.get("revenue") or not line.get("revenue"):
|
| 103 |
+
continue
|
| 104 |
+
if was.get("operating_income") is None or line.get("operating_income") is None:
|
| 105 |
+
continue
|
| 106 |
+
pairs.append((line, was))
|
| 107 |
+
if len(pairs) < 2:
|
| 108 |
+
return {}
|
| 109 |
+
|
| 110 |
+
total_now = sum(line["revenue"] for line, _was in pairs)
|
| 111 |
+
total_prior = sum(was["revenue"] for _line, was in pairs)
|
| 112 |
+
oi_now = sum(line["operating_income"] for line, _was in pairs)
|
| 113 |
+
oi_prior = sum(was["operating_income"] for _line, was in pairs)
|
| 114 |
+
if not total_now or not total_prior:
|
| 115 |
+
return {}
|
| 116 |
+
|
| 117 |
+
margin_now = oi_now / total_now * 100
|
| 118 |
+
margin_prior = oi_prior / total_prior * 100
|
| 119 |
+
|
| 120 |
+
members = []
|
| 121 |
+
mix_total = rate_total = 0.0
|
| 122 |
+
for line, was in pairs:
|
| 123 |
+
weight_now = line["revenue"] / total_now
|
| 124 |
+
weight_prior = was["revenue"] / total_prior
|
| 125 |
+
rate_prior = was["operating_income"] / was["revenue"] * 100
|
| 126 |
+
rate_now = line["operating_income"] / line["revenue"] * 100
|
| 127 |
+
mix = (weight_now - weight_prior) * rate_prior
|
| 128 |
+
rate = weight_now * (rate_now - rate_prior)
|
| 129 |
+
mix_total += mix
|
| 130 |
+
rate_total += rate
|
| 131 |
+
members.append({
|
| 132 |
+
"label": line["label"],
|
| 133 |
+
"margin_now_pct": rate_now,
|
| 134 |
+
"margin_prior_pct": rate_prior,
|
| 135 |
+
"mix_pp": mix,
|
| 136 |
+
"rate_pp": rate,
|
| 137 |
+
"total_pp": mix + rate,
|
| 138 |
+
})
|
| 139 |
+
|
| 140 |
+
members.sort(key=lambda m: m["total_pp"], reverse=True)
|
| 141 |
+
return {
|
| 142 |
+
"period_end": period_end,
|
| 143 |
+
"prior_period_end": prior_end,
|
| 144 |
+
"margin_now_pct": margin_now,
|
| 145 |
+
"margin_prior_pct": margin_prior,
|
| 146 |
+
"change_pp": margin_now - margin_prior,
|
| 147 |
+
"mix_pp": mix_total,
|
| 148 |
+
"rate_pp": rate_total,
|
| 149 |
+
"members": members,
|
| 150 |
+
"source_url": view.get("source_url") or "",
|
| 151 |
+
}
|
| 152 |
+
|
| 153 |
+
|
| 154 |
+
def bridge(impact: dict) -> Optional[dict]:
|
| 155 |
+
"""Turn one disclosed driver->metric claim into a computed delta.
|
| 156 |
+
|
| 157 |
+
*impact* carries only figures the company stated, transcribed by the
|
| 158 |
+
model under the citation rules: where the metric is now, and where the
|
| 159 |
+
company says it is going. The subtraction happens here so that no model
|
| 160 |
+
ever reports a delta it worked out itself.
|
| 161 |
+
|
| 162 |
+
Returns None when the claim is not arithmetically usable, which keeps a
|
| 163 |
+
half-filled impact off the page rather than rendering "from 50.1% to
|
| 164 |
+
None".
|
| 165 |
+
"""
|
| 166 |
+
if not isinstance(impact, dict):
|
| 167 |
+
return None
|
| 168 |
+
try:
|
| 169 |
+
current = float(impact["from_value"])
|
| 170 |
+
low = float(impact["to_low"])
|
| 171 |
+
high = float(impact.get("to_high", impact["to_low"]))
|
| 172 |
+
except (KeyError, TypeError, ValueError):
|
| 173 |
+
return None
|
| 174 |
+
if low > high:
|
| 175 |
+
low, high = high, low
|
| 176 |
+
|
| 177 |
+
# Basis points read better than "percentage points of a percentage" for
|
| 178 |
+
# a margin; anything else keeps the metric's own unit.
|
| 179 |
+
unit = str(impact.get("unit") or "%")
|
| 180 |
+
delta_low, delta_high = low - current, high - current
|
| 181 |
+
return {
|
| 182 |
+
"driver": str(impact.get("driver") or "").strip(),
|
| 183 |
+
"metric": str(impact.get("metric") or "").strip(),
|
| 184 |
+
"basis": str(impact.get("basis") or "disclosed"),
|
| 185 |
+
"unit": unit,
|
| 186 |
+
"from_value": current,
|
| 187 |
+
"to_low": low,
|
| 188 |
+
"to_high": high,
|
| 189 |
+
"delta_low_pp": delta_low,
|
| 190 |
+
"delta_high_pp": delta_high,
|
| 191 |
+
"delta_low_bp": delta_low * 100,
|
| 192 |
+
"delta_high_bp": delta_high * 100,
|
| 193 |
+
"is_range": abs(high - low) > 1e-9,
|
| 194 |
+
"fact_ids": [str(i) for i in (impact.get("fact_ids") or [])],
|
| 195 |
+
}
|
|
@@ -0,0 +1,252 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Turn stored segment rows into a display-ready breakdown.
|
| 2 |
+
|
| 3 |
+
Two problems sit between the database and a chart, and both are solved here
|
| 4 |
+
rather than in the renderer so they can be tested without Streamlit:
|
| 5 |
+
|
| 6 |
+
**Aggregates double-count.** A filing tags both the detail members and their
|
| 7 |
+
subtotals -- Apple reports iPhone, Mac, iPad and Wearables *and* a "Products"
|
| 8 |
+
line equal to their sum. Plotting all five inflates the total by 72%. The
|
| 9 |
+
subtotal is identified structurally, by being the sum of other members, so no
|
| 10 |
+
per-issuer list of member names is needed.
|
| 11 |
+
|
| 12 |
+
**Year-on-year needs the comparable period.** It comes free: a filing carries
|
| 13 |
+
its prior-year quarter in the same instance document, so the comparison is a
|
| 14 |
+
second read against a period end the store already holds.
|
| 15 |
+
"""
|
| 16 |
+
from __future__ import annotations
|
| 17 |
+
|
| 18 |
+
from itertools import combinations
|
| 19 |
+
from typing import Optional
|
| 20 |
+
|
| 21 |
+
from ingestion.segments import PRODUCT_AXIS, SEGMENT_AXIS
|
| 22 |
+
from storage import segments_db
|
| 23 |
+
|
| 24 |
+
# How close a member must be to a subtotal of others to count as one. Filers
|
| 25 |
+
# round to the million, so exact equality is too strict and anything looser
|
| 26 |
+
# starts swallowing real members.
|
| 27 |
+
_AGGREGATE_TOLERANCE = 0.005
|
| 28 |
+
|
| 29 |
+
# Enumerating subsets is exponential; a filer with more members than this is
|
| 30 |
+
# reporting at a granularity where subtotals are not the failure mode.
|
| 31 |
+
_MAX_MEMBERS_FOR_SUBSET_SCAN = 14
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
def _is_subtotal(value: float, others: list[float]) -> bool:
|
| 35 |
+
"""True when *value* equals the sum of two or more of *others*."""
|
| 36 |
+
if value <= 0 or len(others) > _MAX_MEMBERS_FOR_SUBSET_SCAN:
|
| 37 |
+
return False
|
| 38 |
+
# Every subset is tested. An early break on an over-target sum looks like
|
| 39 |
+
# a free prune but is wrong: itertools.combinations does not yield subsets
|
| 40 |
+
# in ascending-sum order, so breaking skips the matching subset that comes
|
| 41 |
+
# later at the same size -- which is exactly how Apple's "Products" line
|
| 42 |
+
# survived its first pass here.
|
| 43 |
+
for size in range(2, len(others) + 1):
|
| 44 |
+
for combo in combinations(others, size):
|
| 45 |
+
total = sum(combo)
|
| 46 |
+
if total and abs(total - value) / value <= _AGGREGATE_TOLERANCE:
|
| 47 |
+
return True
|
| 48 |
+
return False
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
def _smallest_subset_summing_to(rows: list[dict], target: float) -> list[dict]:
|
| 52 |
+
"""The fewest rows whose revenue sums to *target*, or []."""
|
| 53 |
+
priced = [r for r in rows if r.get("revenue")]
|
| 54 |
+
if len(priced) > _MAX_MEMBERS_FOR_SUBSET_SCAN:
|
| 55 |
+
return []
|
| 56 |
+
for size in range(1, len(priced) + 1):
|
| 57 |
+
for combo in combinations(priced, size):
|
| 58 |
+
total = sum(float(r["revenue"]) for r in combo)
|
| 59 |
+
if abs(total - target) / target <= _AGGREGATE_TOLERANCE:
|
| 60 |
+
return list(combo)
|
| 61 |
+
return []
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
def drop_aggregate_members(rows: list[dict], total: Optional[float] = None) -> list[dict]:
|
| 65 |
+
"""Remove members that restate the sum of other members.
|
| 66 |
+
|
| 67 |
+
With *total* -- the period's consolidated revenue -- the rule is exact:
|
| 68 |
+
whatever the members overshoot the company's own top line by is the
|
| 69 |
+
subtotal, so the rows summing to that excess are the ones to drop. This is
|
| 70 |
+
the path that runs in production.
|
| 71 |
+
|
| 72 |
+
Without it the fallback asks whether each member equals a sum of others.
|
| 73 |
+
That is enough to spot a subtotal but it can also fire on a genuine member
|
| 74 |
+
whose value happens to match a combination of its peers, so it is only
|
| 75 |
+
used when no consolidated figure is available to anchor against.
|
| 76 |
+
"""
|
| 77 |
+
priced = [(r, float(r["revenue"])) for r in rows if r.get("revenue")]
|
| 78 |
+
if not priced:
|
| 79 |
+
return list(rows)
|
| 80 |
+
|
| 81 |
+
if total:
|
| 82 |
+
excess = sum(v for _r, v in priced) - float(total)
|
| 83 |
+
if excess / float(total) <= _AGGREGATE_TOLERANCE:
|
| 84 |
+
return list(rows)
|
| 85 |
+
drop = {id(r) for r in _smallest_subset_summing_to(rows, excess)}
|
| 86 |
+
if drop:
|
| 87 |
+
return [r for r in rows if id(r) not in drop]
|
| 88 |
+
# The overshoot matches no combination of members: the mix and the
|
| 89 |
+
# top line disagree for a reason this code cannot name. Show every
|
| 90 |
+
# member rather than guessing which one to hide.
|
| 91 |
+
return list(rows)
|
| 92 |
+
|
| 93 |
+
keep = []
|
| 94 |
+
for row in rows:
|
| 95 |
+
value = row.get("revenue")
|
| 96 |
+
if not value:
|
| 97 |
+
keep.append(row)
|
| 98 |
+
continue
|
| 99 |
+
others = [v for r, v in priced if r is not row]
|
| 100 |
+
if not _is_subtotal(float(value), others):
|
| 101 |
+
keep.append(row)
|
| 102 |
+
return keep
|
| 103 |
+
|
| 104 |
+
|
| 105 |
+
def _consolidated_revenue(ticker: str, period_end: str) -> Optional[float]:
|
| 106 |
+
"""The period's top-line revenue from the metrics table, if stored.
|
| 107 |
+
|
| 108 |
+
``metrics.report_date`` holds a period END despite its name, which is what
|
| 109 |
+
makes it join cleanly against a segment period_end.
|
| 110 |
+
"""
|
| 111 |
+
try:
|
| 112 |
+
from storage import metrics_db
|
| 113 |
+
for row in metrics_db.get_all_metrics(ticker.upper()):
|
| 114 |
+
if row.get("report_date") == period_end and row.get("revenue"):
|
| 115 |
+
return float(row["revenue"])
|
| 116 |
+
except Exception:
|
| 117 |
+
return None
|
| 118 |
+
return None
|
| 119 |
+
|
| 120 |
+
|
| 121 |
+
def breakdown(ticker: str, axis: str) -> dict:
|
| 122 |
+
"""The latest quarter's lines for *axis*, with year-on-year and margin.
|
| 123 |
+
|
| 124 |
+
Returns ``{}`` when nothing is stored, so a caller can treat "no segment
|
| 125 |
+
data" and "no segment block" as the same case.
|
| 126 |
+
"""
|
| 127 |
+
period_end = segments_db.latest_period_end(ticker, axis)
|
| 128 |
+
if not period_end:
|
| 129 |
+
return {}
|
| 130 |
+
|
| 131 |
+
current = drop_aggregate_members(
|
| 132 |
+
segments_db.read_segments(ticker, axis, period_end),
|
| 133 |
+
total=_consolidated_revenue(ticker, period_end),
|
| 134 |
+
)
|
| 135 |
+
if not current:
|
| 136 |
+
return {}
|
| 137 |
+
|
| 138 |
+
prior_end = segments_db.prior_year_period_end(ticker, axis, period_end)
|
| 139 |
+
prior = {
|
| 140 |
+
r["member"]: r
|
| 141 |
+
for r in (segments_db.read_segments(ticker, axis, prior_end) if prior_end else [])
|
| 142 |
+
}
|
| 143 |
+
|
| 144 |
+
lines = []
|
| 145 |
+
for row in current:
|
| 146 |
+
revenue = row.get("revenue")
|
| 147 |
+
was = (prior.get(row["member"]) or {}).get("revenue")
|
| 148 |
+
operating_income = row.get("operating_income")
|
| 149 |
+
lines.append({
|
| 150 |
+
"label": row.get("label") or row.get("member"),
|
| 151 |
+
"member": row.get("member"),
|
| 152 |
+
"revenue": revenue,
|
| 153 |
+
"prior_revenue": was,
|
| 154 |
+
"yoy_pct": ((revenue - was) / was * 100) if revenue and was else None,
|
| 155 |
+
"operating_income": operating_income,
|
| 156 |
+
"margin_pct": (
|
| 157 |
+
operating_income / revenue * 100
|
| 158 |
+
if operating_income is not None and revenue else None
|
| 159 |
+
),
|
| 160 |
+
"source_url": row.get("source_url") or "",
|
| 161 |
+
})
|
| 162 |
+
|
| 163 |
+
total = sum(line["revenue"] for line in lines if line["revenue"])
|
| 164 |
+
for line in lines:
|
| 165 |
+
line["share_pct"] = (line["revenue"] / total * 100) if total and line["revenue"] else None
|
| 166 |
+
|
| 167 |
+
return {
|
| 168 |
+
"axis": axis,
|
| 169 |
+
"period_end": period_end,
|
| 170 |
+
"prior_period_end": prior_end,
|
| 171 |
+
"total_revenue": total or None,
|
| 172 |
+
"lines": lines,
|
| 173 |
+
"source_url": next((line["source_url"] for line in lines if line["source_url"]), ""),
|
| 174 |
+
}
|
| 175 |
+
|
| 176 |
+
|
| 177 |
+
def products(ticker: str) -> dict:
|
| 178 |
+
return breakdown(ticker, PRODUCT_AXIS)
|
| 179 |
+
|
| 180 |
+
|
| 181 |
+
def regions(ticker: str) -> dict:
|
| 182 |
+
return breakdown(ticker, SEGMENT_AXIS)
|
| 183 |
+
|
| 184 |
+
|
| 185 |
+
def outliers(view: dict) -> dict:
|
| 186 |
+
"""The two lines worth a sentence: fastest and slowest year-on-year.
|
| 187 |
+
|
| 188 |
+
Returned as ``{"fastest": line, "slowest": line}`` with either key absent
|
| 189 |
+
when there is no year-on-year to rank by. This is what makes the block
|
| 190 |
+
interpretable rather than merely accurate -- a reader should not have to
|
| 191 |
+
scan five percentages to find the one that diverges.
|
| 192 |
+
"""
|
| 193 |
+
ranked = [line for line in view.get("lines") or [] if line.get("yoy_pct") is not None]
|
| 194 |
+
if len(ranked) < 2:
|
| 195 |
+
return {}
|
| 196 |
+
ranked.sort(key=lambda line: line["yoy_pct"])
|
| 197 |
+
return {"slowest": ranked[0], "fastest": ranked[-1]}
|
| 198 |
+
|
| 199 |
+
|
| 200 |
+
def margin_spread(view: dict) -> Optional[dict]:
|
| 201 |
+
"""The widest gap in segment margin, when the axis carries margins.
|
| 202 |
+
|
| 203 |
+
Apple's Greater China runs six points below every other region; that gap
|
| 204 |
+
is a fact about the business the consolidated margin cannot show.
|
| 205 |
+
"""
|
| 206 |
+
ranked = [line for line in view.get("lines") or [] if line.get("margin_pct") is not None]
|
| 207 |
+
if len(ranked) < 2:
|
| 208 |
+
return None
|
| 209 |
+
ranked.sort(key=lambda line: line["margin_pct"])
|
| 210 |
+
return {
|
| 211 |
+
"lowest": ranked[0],
|
| 212 |
+
"highest": ranked[-1],
|
| 213 |
+
"spread_pct": ranked[-1]["margin_pct"] - ranked[0]["margin_pct"],
|
| 214 |
+
}
|
| 215 |
+
|
| 216 |
+
|
| 217 |
+
def _money(value: float) -> str:
|
| 218 |
+
for cutoff, unit in ((1e12, "T"), (1e9, "B"), (1e6, "M")):
|
| 219 |
+
if abs(value) >= cutoff:
|
| 220 |
+
return f"${value / cutoff:,.2f}{unit}"
|
| 221 |
+
return f"${value:,.0f}"
|
| 222 |
+
|
| 223 |
+
|
| 224 |
+
def digest_lines(ticker: str) -> list[dict]:
|
| 225 |
+
"""Segment results as sentences the narrative digest can cite.
|
| 226 |
+
|
| 227 |
+
These are the company's own tagged figures, so they enter the article's
|
| 228 |
+
evidence pool on the same footing as a verified filing fact -- and they
|
| 229 |
+
carry the product and region granularity nothing else in the brief has.
|
| 230 |
+
Every number the sentence states is in the sentence, which is what lets
|
| 231 |
+
the article quote it and still pass the number check.
|
| 232 |
+
"""
|
| 233 |
+
lines: list[dict] = []
|
| 234 |
+
for view, origin in ((products(ticker), "product"), (regions(ticker), "region")):
|
| 235 |
+
period_end = view.get("period_end") or ""
|
| 236 |
+
for line in view.get("lines") or []:
|
| 237 |
+
revenue = line.get("revenue")
|
| 238 |
+
if not revenue:
|
| 239 |
+
continue
|
| 240 |
+
parts = [f"{line['label']} revenue {_money(revenue)} in the quarter ended {period_end}"]
|
| 241 |
+
if line.get("yoy_pct") is not None:
|
| 242 |
+
parts.append(f"{line['yoy_pct']:+.1f}% year-on-year")
|
| 243 |
+
if line.get("share_pct") is not None:
|
| 244 |
+
parts.append(f"{line['share_pct']:.0f}% of total revenue")
|
| 245 |
+
if line.get("margin_pct") is not None:
|
| 246 |
+
parts.append(f"segment operating margin {line['margin_pct']:.1f}%")
|
| 247 |
+
lines.append({
|
| 248 |
+
"text": ", ".join(parts) + ".",
|
| 249 |
+
"origin": f"{origin} mix",
|
| 250 |
+
"source_url": line.get("source_url") or view.get("source_url") or None,
|
| 251 |
+
})
|
| 252 |
+
return lines
|
|
@@ -154,6 +154,22 @@ def ingest(ticker: str, full: bool = False) -> None:
|
|
| 154 |
**guidance_struct,
|
| 155 |
})
|
| 156 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 157 |
# Persist raw section text for cross-period text diffing (analysis/textdiff.py)
|
| 158 |
if edgar.mda_text:
|
| 159 |
upsert_section(edgar.ticker, edgar.period, edgar.form_type, "mda", edgar.mda_text)
|
|
|
|
| 154 |
**guidance_struct,
|
| 155 |
})
|
| 156 |
|
| 157 |
+
# Per-product and per-region results live in the filing's own XBRL
|
| 158 |
+
# instance, not in companyfacts -- one extra fetch per filing. Failure
|
| 159 |
+
# is non-fatal: the consolidated metrics above are the brief's
|
| 160 |
+
# backbone, the segment breakdown is an enhancement on top of them.
|
| 161 |
+
try:
|
| 162 |
+
from ingestion.segments import fetch_segments
|
| 163 |
+
from storage.segments_db import upsert_segments
|
| 164 |
+
cik = edgar.source_url.split("/data/")[1].split("/")[0]
|
| 165 |
+
primary_doc = edgar.source_url.rsplit("/", 1)[-1]
|
| 166 |
+
seg_facts, seg_url = fetch_segments(cik, edgar.accession, primary_doc)
|
| 167 |
+
written = upsert_segments(edgar.ticker, seg_facts, edgar.accession, seg_url)
|
| 168 |
+
if written:
|
| 169 |
+
print(f"[ingest] Segments: {written} product/region line(s)")
|
| 170 |
+
except Exception as exc:
|
| 171 |
+
print(f"[ingest] WARNING: segment extraction failed: {exc}")
|
| 172 |
+
|
| 173 |
# Persist raw section text for cross-period text diffing (analysis/textdiff.py)
|
| 174 |
if edgar.mda_text:
|
| 175 |
upsert_section(edgar.ticker, edgar.period, edgar.form_type, "mda", edgar.mda_text)
|
|
@@ -0,0 +1,265 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Per-product and per-region results, read from the filing's XBRL instance.
|
| 2 |
+
|
| 3 |
+
``data.sec.gov/api/xbrl/companyfacts`` returns only undimensioned facts, which
|
| 4 |
+
is why the rest of the ingestion sees a single consolidated revenue line. The
|
| 5 |
+
segment detail is in the inline-XBRL instance document that ships inside the
|
| 6 |
+
filing itself (``aapl-20260627_htm.xml``), where each fact points at a context
|
| 7 |
+
carrying explicit dimension members:
|
| 8 |
+
|
| 9 |
+
srt:ProductOrServiceAxis -> iPhone, Mac, iPad, Services...
|
| 10 |
+
us-gaap:StatementBusinessSegmentsAxis -> Americas, Europe, Greater China...
|
| 11 |
+
|
| 12 |
+
One fetch per filing yields the quarter, its prior-year comparable and the
|
| 13 |
+
year-to-date figures, so a year-on-year breakdown costs no extra request.
|
| 14 |
+
|
| 15 |
+
Runs offline, at ingest time. Nothing here is called from the app.
|
| 16 |
+
"""
|
| 17 |
+
from __future__ import annotations
|
| 18 |
+
|
| 19 |
+
import re
|
| 20 |
+
import xml.etree.ElementTree as ET
|
| 21 |
+
from dataclasses import dataclass
|
| 22 |
+
from datetime import date
|
| 23 |
+
from typing import Iterable, Optional
|
| 24 |
+
|
| 25 |
+
import httpx
|
| 26 |
+
|
| 27 |
+
from ingestion.edgar import HEADERS
|
| 28 |
+
|
| 29 |
+
_XBRLI = "{http://www.xbrl.org/2003/instance}"
|
| 30 |
+
_XBRLDI = "{http://xbrl.org/2006/xbrldi}"
|
| 31 |
+
|
| 32 |
+
# The two axes worth storing. Anything else in a filing (fair-value hierarchy,
|
| 33 |
+
# equity components, hedging designation) describes accounting structure, not
|
| 34 |
+
# where the money came from.
|
| 35 |
+
PRODUCT_AXIS = "srt:ProductOrServiceAxis"
|
| 36 |
+
SEGMENT_AXIS = "us-gaap:StatementBusinessSegmentsAxis"
|
| 37 |
+
_AXES = (PRODUCT_AXIS, SEGMENT_AXIS)
|
| 38 |
+
|
| 39 |
+
# A reportable-segment fact is tagged with the segment AND a consolidation
|
| 40 |
+
# item; the total for that segment is the OperatingSegments one. Corporate and
|
| 41 |
+
# elimination members are reconciling entries, not places revenue came from.
|
| 42 |
+
_OPERATING_SEGMENTS = "us-gaap:OperatingSegmentsMember"
|
| 43 |
+
|
| 44 |
+
_REVENUE_CONCEPTS = (
|
| 45 |
+
"RevenueFromContractWithCustomerExcludingAssessedTax",
|
| 46 |
+
"RevenueFromContractWithCustomerIncludingAssessedTax",
|
| 47 |
+
"Revenues",
|
| 48 |
+
"SalesRevenueNet",
|
| 49 |
+
)
|
| 50 |
+
_OPERATING_INCOME_CONCEPTS = ("OperatingIncomeLoss",)
|
| 51 |
+
|
| 52 |
+
# Fiscal quarters are 13 weeks but drift by a few days; year-to-date spans are
|
| 53 |
+
# what everything between a quarter and a year turns out to be.
|
| 54 |
+
_QUARTER_DAYS = range(80, 106)
|
| 55 |
+
_ANNUAL_DAYS = range(350, 381)
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
@dataclass(frozen=True)
|
| 59 |
+
class SegmentFact:
|
| 60 |
+
"""One (axis, member, period) result line from a filing."""
|
| 61 |
+
axis: str
|
| 62 |
+
member: str
|
| 63 |
+
label: str
|
| 64 |
+
period_start: str
|
| 65 |
+
period_end: str
|
| 66 |
+
period_kind: str # "quarter" | "ytd" | "annual"
|
| 67 |
+
revenue: Optional[float] = None
|
| 68 |
+
operating_income: Optional[float] = None
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
def _period_kind(start: str, end: str) -> str:
|
| 72 |
+
try:
|
| 73 |
+
days = (date.fromisoformat(end) - date.fromisoformat(start)).days
|
| 74 |
+
except ValueError:
|
| 75 |
+
return "unknown"
|
| 76 |
+
if days in _QUARTER_DAYS:
|
| 77 |
+
return "quarter"
|
| 78 |
+
if days in _ANNUAL_DAYS:
|
| 79 |
+
return "annual"
|
| 80 |
+
return "ytd"
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
_LABEL_FIXUPS = {
|
| 84 |
+
"IPhone": "iPhone",
|
| 85 |
+
"IPad": "iPad",
|
| 86 |
+
"IPod": "iPod",
|
| 87 |
+
"Service": "Services",
|
| 88 |
+
"Product": "Products",
|
| 89 |
+
"WearablesHomeandAccessories": "Wearables, Home & Accessories",
|
| 90 |
+
"RestOfAsiaPacificSegment": "Rest of Asia Pacific",
|
| 91 |
+
}
|
| 92 |
+
|
| 93 |
+
|
| 94 |
+
def member_label(member: str) -> str:
|
| 95 |
+
"""Turn ``aapl:GreaterChinaSegmentMember`` into ``Greater China``.
|
| 96 |
+
|
| 97 |
+
Keeps a small fixup table for the names that camel-case splitting gets
|
| 98 |
+
wrong ("IPhone", "Service") and falls back to splitting on case
|
| 99 |
+
boundaries, which handles every issuer's members without a per-issuer
|
| 100 |
+
table.
|
| 101 |
+
"""
|
| 102 |
+
bare = str(member or "").split(":")[-1]
|
| 103 |
+
bare = re.sub(r"Member$", "", bare)
|
| 104 |
+
if bare in _LABEL_FIXUPS:
|
| 105 |
+
return _LABEL_FIXUPS[bare]
|
| 106 |
+
bare = re.sub(r"Segment$", "", bare) or bare
|
| 107 |
+
if bare in _LABEL_FIXUPS:
|
| 108 |
+
return _LABEL_FIXUPS[bare]
|
| 109 |
+
spaced = re.sub(r"(?<=[a-z0-9])(?=[A-Z])", " ", bare)
|
| 110 |
+
spaced = re.sub(r"(?<=[A-Z])(?=[A-Z][a-z])", " ", spaced)
|
| 111 |
+
return spaced.strip() or bare
|
| 112 |
+
|
| 113 |
+
|
| 114 |
+
def _contexts(root: ET.Element) -> dict[str, tuple[str, str, tuple[str, ...]]]:
|
| 115 |
+
out: dict[str, tuple[str, str, tuple[str, ...]]] = {}
|
| 116 |
+
for ctx in root.iter(_XBRLI + "context"):
|
| 117 |
+
cid = ctx.get("id")
|
| 118 |
+
period = ctx.find(_XBRLI + "period")
|
| 119 |
+
if cid is None or period is None:
|
| 120 |
+
continue
|
| 121 |
+
start = period.findtext(_XBRLI + "startDate") or ""
|
| 122 |
+
end = period.findtext(_XBRLI + "endDate") or ""
|
| 123 |
+
if not start or not end:
|
| 124 |
+
continue # instants describe balances, not results
|
| 125 |
+
members = tuple(
|
| 126 |
+
f'{m.get("dimension")}={(m.text or "").strip()}'
|
| 127 |
+
for m in ctx.iter(_XBRLDI + "explicitMember")
|
| 128 |
+
)
|
| 129 |
+
out[cid] = (start, end, members)
|
| 130 |
+
return out
|
| 131 |
+
|
| 132 |
+
|
| 133 |
+
def _dimension_of(members: tuple[str, ...]) -> Optional[tuple[str, str]]:
|
| 134 |
+
"""Return (axis, member) when this context is exactly one segment cut.
|
| 135 |
+
|
| 136 |
+
A product context carries one member. A reportable-segment context carries
|
| 137 |
+
the segment plus ``OperatingSegmentsMember``; any other pairing is a
|
| 138 |
+
narrower slice (a segment within a product, say) that would double-count
|
| 139 |
+
if it were stored beside the totals.
|
| 140 |
+
"""
|
| 141 |
+
parsed = [tuple(m.split("=", 1)) for m in members if "=" in m]
|
| 142 |
+
wanted = [(a, v) for a, v in parsed if a in _AXES]
|
| 143 |
+
if len(wanted) != 1:
|
| 144 |
+
return None
|
| 145 |
+
extras = [(a, v) for a, v in parsed if a not in _AXES]
|
| 146 |
+
if extras and [v for _a, v in extras] != [_OPERATING_SEGMENTS]:
|
| 147 |
+
return None
|
| 148 |
+
return wanted[0]
|
| 149 |
+
|
| 150 |
+
|
| 151 |
+
def _facts(root: ET.Element, concepts: Iterable[str]) -> list[tuple[str, float]]:
|
| 152 |
+
wanted = {f"}}{c}" for c in concepts}
|
| 153 |
+
found: list[tuple[str, float]] = []
|
| 154 |
+
for el in root.iter():
|
| 155 |
+
tag = el.tag
|
| 156 |
+
if not any(tag.endswith(w) for w in wanted):
|
| 157 |
+
continue
|
| 158 |
+
ref, text = el.get("contextRef"), (el.text or "").strip()
|
| 159 |
+
if not ref or not text:
|
| 160 |
+
continue
|
| 161 |
+
try:
|
| 162 |
+
value = float(text)
|
| 163 |
+
except ValueError:
|
| 164 |
+
continue
|
| 165 |
+
scale = el.get("scale")
|
| 166 |
+
sign = el.get("sign")
|
| 167 |
+
if scale:
|
| 168 |
+
try:
|
| 169 |
+
value *= 10 ** int(scale)
|
| 170 |
+
except ValueError:
|
| 171 |
+
pass
|
| 172 |
+
if sign == "-":
|
| 173 |
+
value = -value
|
| 174 |
+
found.append((ref, value))
|
| 175 |
+
return found
|
| 176 |
+
|
| 177 |
+
|
| 178 |
+
def parse_segments(xml_bytes: bytes) -> list[SegmentFact]:
|
| 179 |
+
"""Extract every product / reportable-segment result line from an instance.
|
| 180 |
+
|
| 181 |
+
Pure: takes bytes, returns facts. No network, no database.
|
| 182 |
+
"""
|
| 183 |
+
root = ET.fromstring(xml_bytes)
|
| 184 |
+
contexts = _contexts(root)
|
| 185 |
+
|
| 186 |
+
merged: dict[tuple, dict] = {}
|
| 187 |
+
|
| 188 |
+
def _collect(concepts: Iterable[str], field: str) -> None:
|
| 189 |
+
# One concept at a time, in preference order: a filer tagging both
|
| 190 |
+
# `RevenueFromContractWithCustomer...` and the legacy `Revenues` must
|
| 191 |
+
# resolve to the first of the two, not to whichever the document
|
| 192 |
+
# happens to mention first.
|
| 193 |
+
for ref, value in [f for c in concepts for f in _facts(root, [c])]:
|
| 194 |
+
ctx = contexts.get(ref)
|
| 195 |
+
if not ctx:
|
| 196 |
+
continue
|
| 197 |
+
start, end, members = ctx
|
| 198 |
+
dim = _dimension_of(members)
|
| 199 |
+
if not dim:
|
| 200 |
+
continue
|
| 201 |
+
axis, member = dim
|
| 202 |
+
key = (axis, member, start, end)
|
| 203 |
+
row = merged.setdefault(key, {})
|
| 204 |
+
# The same figure is often tagged several times in one filing
|
| 205 |
+
# (statement, footnote, cover). First value wins; they agree.
|
| 206 |
+
row.setdefault(field, value)
|
| 207 |
+
|
| 208 |
+
_collect(_REVENUE_CONCEPTS, "revenue")
|
| 209 |
+
_collect(_OPERATING_INCOME_CONCEPTS, "operating_income")
|
| 210 |
+
|
| 211 |
+
facts = [
|
| 212 |
+
SegmentFact(
|
| 213 |
+
axis=axis,
|
| 214 |
+
member=member,
|
| 215 |
+
label=member_label(member),
|
| 216 |
+
period_start=start,
|
| 217 |
+
period_end=end,
|
| 218 |
+
period_kind=_period_kind(start, end),
|
| 219 |
+
revenue=values.get("revenue"),
|
| 220 |
+
operating_income=values.get("operating_income"),
|
| 221 |
+
)
|
| 222 |
+
for (axis, member, start, end), values in merged.items()
|
| 223 |
+
]
|
| 224 |
+
facts.sort(key=lambda f: (f.period_end, f.axis, -(f.revenue or 0), f.member))
|
| 225 |
+
return facts
|
| 226 |
+
|
| 227 |
+
|
| 228 |
+
# ββ Fetching βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 229 |
+
|
| 230 |
+
def instance_url(cik: str | int, accession: str, primary_doc: str = "") -> Optional[str]:
|
| 231 |
+
"""Locate the XBRL instance document inside a filing.
|
| 232 |
+
|
| 233 |
+
Inline-XBRL filings name it after the primary document (``x.htm`` ->
|
| 234 |
+
``x_htm.xml``); that guess is checked against the filing's own directory
|
| 235 |
+
listing rather than trusted, because a filer who breaks the convention
|
| 236 |
+
would otherwise yield a 404 with no explanation.
|
| 237 |
+
"""
|
| 238 |
+
base = f"https://www.sec.gov/Archives/edgar/data/{int(cik)}/{accession.replace('-', '')}"
|
| 239 |
+
guess = re.sub(r"\.htm$", "_htm.xml", primary_doc) if primary_doc else ""
|
| 240 |
+
try:
|
| 241 |
+
listing = httpx.get(f"{base}/index.json", headers=HEADERS, timeout=30)
|
| 242 |
+
listing.raise_for_status()
|
| 243 |
+
names = [item.get("name", "") for item in listing.json()["directory"]["item"]]
|
| 244 |
+
except Exception:
|
| 245 |
+
return f"{base}/{guess}" if guess else None
|
| 246 |
+
if guess and guess in names:
|
| 247 |
+
return f"{base}/{guess}"
|
| 248 |
+
for name in names:
|
| 249 |
+
if name.endswith("_htm.xml"):
|
| 250 |
+
return f"{base}/{name}"
|
| 251 |
+
return None
|
| 252 |
+
|
| 253 |
+
|
| 254 |
+
def fetch_segments(cik: str | int, accession: str, primary_doc: str = "") -> tuple[list[SegmentFact], str]:
|
| 255 |
+
"""Download and parse one filing's segment detail.
|
| 256 |
+
|
| 257 |
+
Returns ``(facts, source_url)``; an empty list means the filing carries no
|
| 258 |
+
segment dimensions, which is normal for a single-segment issuer.
|
| 259 |
+
"""
|
| 260 |
+
url = instance_url(cik, accession, primary_doc)
|
| 261 |
+
if not url:
|
| 262 |
+
return [], ""
|
| 263 |
+
resp = httpx.get(url, headers=HEADERS, timeout=90)
|
| 264 |
+
resp.raise_for_status()
|
| 265 |
+
return parse_segments(resp.content), url
|
|
@@ -0,0 +1,102 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Backfill per-product / per-region results for filings already ingested.
|
| 2 |
+
|
| 3 |
+
``ingest.py`` now pulls the segment detail as it goes, but every filing
|
| 4 |
+
ingested before that lives in ``metrics.db`` with only its consolidated
|
| 5 |
+
figures. This walks those rows, fetches each filing's XBRL instance once and
|
| 6 |
+
fills the ``segments`` table.
|
| 7 |
+
|
| 8 |
+
One HTTP request per filing, ~1 MB each, so it is rate-limited to stay inside
|
| 9 |
+
the SEC's fair-access policy.
|
| 10 |
+
|
| 11 |
+
Usage::
|
| 12 |
+
|
| 13 |
+
python -m scripts.backfill_segments AAPL
|
| 14 |
+
python -m scripts.backfill_segments --all
|
| 15 |
+
python -m scripts.backfill_segments AAPL --dry-run
|
| 16 |
+
"""
|
| 17 |
+
from __future__ import annotations
|
| 18 |
+
|
| 19 |
+
import argparse
|
| 20 |
+
import sys
|
| 21 |
+
import time
|
| 22 |
+
|
| 23 |
+
from ingestion.segments import PRODUCT_AXIS, SEGMENT_AXIS, fetch_segments
|
| 24 |
+
from storage import metrics_db, segments_db
|
| 25 |
+
|
| 26 |
+
# The SEC asks for no more than 10 requests a second; one filing per second is
|
| 27 |
+
# far inside that and keeps a full backfill polite.
|
| 28 |
+
_PAUSE_SECONDS = 1.0
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
def _cik_and_doc(source_url: str) -> tuple[str, str]:
|
| 32 |
+
"""Pull the CIK and primary document out of a stored filing URL."""
|
| 33 |
+
try:
|
| 34 |
+
cik = source_url.split("/data/")[1].split("/")[0]
|
| 35 |
+
except (IndexError, AttributeError):
|
| 36 |
+
return "", ""
|
| 37 |
+
return cik, source_url.rsplit("/", 1)[-1]
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
def backfill(ticker: str, *, dry_run: bool = False) -> int:
|
| 41 |
+
rows = metrics_db.get_all_metrics(ticker.upper())
|
| 42 |
+
filings = [
|
| 43 |
+
r for r in rows
|
| 44 |
+
if r.get("accession") and r.get("source_url") and r.get("period_basis") != "annual"
|
| 45 |
+
]
|
| 46 |
+
if not filings:
|
| 47 |
+
print(f"{ticker}: no ingested filings carrying an accession")
|
| 48 |
+
return 1
|
| 49 |
+
|
| 50 |
+
total = failures = 0
|
| 51 |
+
for row in filings:
|
| 52 |
+
cik, primary_doc = _cik_and_doc(row["source_url"])
|
| 53 |
+
if not cik:
|
| 54 |
+
print(f" {row['period']}: unparseable source_url, skipped")
|
| 55 |
+
failures += 1
|
| 56 |
+
continue
|
| 57 |
+
try:
|
| 58 |
+
facts, url = fetch_segments(cik, row["accession"], primary_doc)
|
| 59 |
+
except Exception as exc: # noqa: BLE001 - one bad filing must not stop the run
|
| 60 |
+
print(f" {row['period']}: fetch failed ({exc})")
|
| 61 |
+
failures += 1
|
| 62 |
+
continue
|
| 63 |
+
|
| 64 |
+
products = sum(1 for f in facts if f.axis == PRODUCT_AXIS)
|
| 65 |
+
regions = sum(1 for f in facts if f.axis == SEGMENT_AXIS)
|
| 66 |
+
print(f" {row['period']}: {len(facts)} line(s) β {products} product, {regions} region")
|
| 67 |
+
if not dry_run:
|
| 68 |
+
total += segments_db.upsert_segments(
|
| 69 |
+
ticker, facts, row["accession"], url
|
| 70 |
+
)
|
| 71 |
+
time.sleep(_PAUSE_SECONDS)
|
| 72 |
+
|
| 73 |
+
if dry_run:
|
| 74 |
+
print(f"{ticker}: dry run, nothing written")
|
| 75 |
+
else:
|
| 76 |
+
print(f"{ticker}: {total} row(s) written")
|
| 77 |
+
return 1 if failures and not total else 0
|
| 78 |
+
|
| 79 |
+
|
| 80 |
+
def main() -> int:
|
| 81 |
+
parser = argparse.ArgumentParser(description=__doc__)
|
| 82 |
+
parser.add_argument("ticker", nargs="?", help="Ticker to backfill.")
|
| 83 |
+
parser.add_argument("--all", action="store_true", help="Every ingested ticker.")
|
| 84 |
+
parser.add_argument("--dry-run", action="store_true", help="Fetch and report without writing.")
|
| 85 |
+
args = parser.parse_args()
|
| 86 |
+
|
| 87 |
+
if args.all:
|
| 88 |
+
tickers = metrics_db.list_tickers()
|
| 89 |
+
elif args.ticker:
|
| 90 |
+
tickers = [args.ticker.upper()]
|
| 91 |
+
else:
|
| 92 |
+
parser.error("give a ticker or --all")
|
| 93 |
+
|
| 94 |
+
failures = 0
|
| 95 |
+
for ticker in tickers:
|
| 96 |
+
print(f"\n== {ticker} ==")
|
| 97 |
+
failures += backfill(ticker, dry_run=args.dry_run)
|
| 98 |
+
return 1 if failures else 0
|
| 99 |
+
|
| 100 |
+
|
| 101 |
+
if __name__ == "__main__":
|
| 102 |
+
raise SystemExit(main())
|
|
@@ -0,0 +1,154 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Per-product and per-region results, stored beside the consolidated metrics.
|
| 2 |
+
|
| 3 |
+
Lives in ``data/metrics.db`` next to the ``metrics`` table: same file, same
|
| 4 |
+
ticker keys, one more table. Rows are keyed by the reporting period's own
|
| 5 |
+
start and end dates rather than by a derived label like "Q32026" -- a filing
|
| 6 |
+
carries its prior-year comparable in the same document, and inventing a label
|
| 7 |
+
for that comparable is exactly the kind of calendar guess that produced the
|
| 8 |
+
mislabelled backfill in the consensus archive.
|
| 9 |
+
|
| 10 |
+
``dashboard/segments_view.py`` reads it; ``ingestion/segments.py`` writes it.
|
| 11 |
+
"""
|
| 12 |
+
from __future__ import annotations
|
| 13 |
+
|
| 14 |
+
import sqlite3
|
| 15 |
+
from datetime import datetime, timezone
|
| 16 |
+
from pathlib import Path
|
| 17 |
+
from typing import Iterable, Optional
|
| 18 |
+
|
| 19 |
+
DB_PATH = Path("data/metrics.db")
|
| 20 |
+
|
| 21 |
+
_SCHEMA = """
|
| 22 |
+
CREATE TABLE IF NOT EXISTS segments (
|
| 23 |
+
ticker TEXT NOT NULL,
|
| 24 |
+
axis TEXT NOT NULL,
|
| 25 |
+
member TEXT NOT NULL,
|
| 26 |
+
label TEXT NOT NULL,
|
| 27 |
+
period_start TEXT NOT NULL,
|
| 28 |
+
period_end TEXT NOT NULL,
|
| 29 |
+
period_kind TEXT NOT NULL,
|
| 30 |
+
revenue REAL,
|
| 31 |
+
operating_income REAL,
|
| 32 |
+
accession TEXT NOT NULL DEFAULT '',
|
| 33 |
+
source_url TEXT NOT NULL DEFAULT '',
|
| 34 |
+
ingested_at TEXT NOT NULL DEFAULT '',
|
| 35 |
+
PRIMARY KEY (ticker, axis, member, period_start, period_end)
|
| 36 |
+
)
|
| 37 |
+
"""
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
def _connect() -> sqlite3.Connection:
|
| 41 |
+
DB_PATH.parent.mkdir(parents=True, exist_ok=True)
|
| 42 |
+
conn = sqlite3.connect(DB_PATH, timeout=10)
|
| 43 |
+
conn.row_factory = sqlite3.Row
|
| 44 |
+
return conn
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
def init_db() -> None:
|
| 48 |
+
with _connect() as conn:
|
| 49 |
+
conn.execute(_SCHEMA)
|
| 50 |
+
conn.execute(
|
| 51 |
+
"CREATE INDEX IF NOT EXISTS idx_segments_lookup "
|
| 52 |
+
"ON segments (ticker, axis, period_kind, period_end)"
|
| 53 |
+
)
|
| 54 |
+
|
| 55 |
+
|
| 56 |
+
def upsert_segments(
|
| 57 |
+
ticker: str,
|
| 58 |
+
facts: Iterable,
|
| 59 |
+
accession: str = "",
|
| 60 |
+
source_url: str = "",
|
| 61 |
+
) -> int:
|
| 62 |
+
"""Write *facts* for *ticker*. Returns the number of rows written.
|
| 63 |
+
|
| 64 |
+
Re-ingesting a filing overwrites its own rows and leaves every other
|
| 65 |
+
period alone, so a re-run is idempotent and never orphans history.
|
| 66 |
+
"""
|
| 67 |
+
init_db()
|
| 68 |
+
now = datetime.now(timezone.utc).isoformat()
|
| 69 |
+
rows = [
|
| 70 |
+
(
|
| 71 |
+
ticker.upper(), f.axis, f.member, f.label,
|
| 72 |
+
f.period_start, f.period_end, f.period_kind,
|
| 73 |
+
f.revenue, f.operating_income, accession, source_url, now,
|
| 74 |
+
)
|
| 75 |
+
for f in facts
|
| 76 |
+
]
|
| 77 |
+
if not rows:
|
| 78 |
+
return 0
|
| 79 |
+
with _connect() as conn:
|
| 80 |
+
conn.executemany(
|
| 81 |
+
"INSERT INTO segments (ticker, axis, member, label, period_start, "
|
| 82 |
+
"period_end, period_kind, revenue, operating_income, accession, "
|
| 83 |
+
"source_url, ingested_at) VALUES (?,?,?,?,?,?,?,?,?,?,?,?) "
|
| 84 |
+
"ON CONFLICT(ticker, axis, member, period_start, period_end) DO UPDATE SET "
|
| 85 |
+
"label=excluded.label, period_kind=excluded.period_kind, "
|
| 86 |
+
"revenue=excluded.revenue, operating_income=excluded.operating_income, "
|
| 87 |
+
"accession=excluded.accession, source_url=excluded.source_url, "
|
| 88 |
+
"ingested_at=excluded.ingested_at",
|
| 89 |
+
rows,
|
| 90 |
+
)
|
| 91 |
+
return len(rows)
|
| 92 |
+
|
| 93 |
+
|
| 94 |
+
def _table_exists() -> bool:
|
| 95 |
+
if not DB_PATH.exists():
|
| 96 |
+
return False
|
| 97 |
+
try:
|
| 98 |
+
with _connect() as conn:
|
| 99 |
+
return conn.execute(
|
| 100 |
+
"SELECT 1 FROM sqlite_master WHERE type='table' AND name='segments'"
|
| 101 |
+
).fetchone() is not None
|
| 102 |
+
except sqlite3.DatabaseError:
|
| 103 |
+
# A git-lfs pointer left unresolved reads as a corrupt database. The
|
| 104 |
+
# segment block is an enhancement; it must never take the page down.
|
| 105 |
+
return False
|
| 106 |
+
|
| 107 |
+
|
| 108 |
+
def latest_period_end(ticker: str, axis: str, period_kind: str = "quarter") -> Optional[str]:
|
| 109 |
+
if not _table_exists():
|
| 110 |
+
return None
|
| 111 |
+
with _connect() as conn:
|
| 112 |
+
row = conn.execute(
|
| 113 |
+
"SELECT MAX(period_end) FROM segments WHERE ticker=? AND axis=? AND period_kind=?",
|
| 114 |
+
(ticker.upper(), axis, period_kind),
|
| 115 |
+
).fetchone()
|
| 116 |
+
return row[0] if row and row[0] else None
|
| 117 |
+
|
| 118 |
+
|
| 119 |
+
def read_segments(
|
| 120 |
+
ticker: str,
|
| 121 |
+
axis: str,
|
| 122 |
+
period_end: str,
|
| 123 |
+
period_kind: str = "quarter",
|
| 124 |
+
) -> list[dict]:
|
| 125 |
+
"""Every member's line for one period, largest revenue first."""
|
| 126 |
+
if not _table_exists():
|
| 127 |
+
return []
|
| 128 |
+
with _connect() as conn:
|
| 129 |
+
rows = conn.execute(
|
| 130 |
+
"SELECT * FROM segments WHERE ticker=? AND axis=? AND period_end=? "
|
| 131 |
+
"AND period_kind=? ORDER BY revenue DESC",
|
| 132 |
+
(ticker.upper(), axis, period_end, period_kind),
|
| 133 |
+
).fetchall()
|
| 134 |
+
return [dict(r) for r in rows]
|
| 135 |
+
|
| 136 |
+
|
| 137 |
+
def prior_year_period_end(ticker: str, axis: str, period_end: str) -> Optional[str]:
|
| 138 |
+
"""The comparable period one year earlier, or None.
|
| 139 |
+
|
| 140 |
+
Fiscal calendars drift by a few days, so the match is the closest stored
|
| 141 |
+
period end in a 330-400 day window rather than an exact date minus 365.
|
| 142 |
+
"""
|
| 143 |
+
if not _table_exists():
|
| 144 |
+
return None
|
| 145 |
+
with _connect() as conn:
|
| 146 |
+
row = conn.execute(
|
| 147 |
+
"SELECT period_end, ABS(JULIANDAY(?) - JULIANDAY(period_end)) AS gap "
|
| 148 |
+
"FROM segments WHERE ticker=? AND axis=? AND period_kind='quarter' "
|
| 149 |
+
"AND period_end < ? "
|
| 150 |
+
"AND JULIANDAY(?) - JULIANDAY(period_end) BETWEEN 330 AND 400 "
|
| 151 |
+
"ORDER BY gap ASC LIMIT 1",
|
| 152 |
+
(period_end, ticker.upper(), axis, period_end, period_end),
|
| 153 |
+
).fetchone()
|
| 154 |
+
return row["period_end"] if row else None
|
|
@@ -0,0 +1,164 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Impact arithmetic: the identities that must hold, and the bridge's refusals.
|
| 2 |
+
|
| 3 |
+
Two of these tests are checksums rather than examples. Contributions must sum
|
| 4 |
+
to the group growth rate and a margin split must sum to the margin change --
|
| 5 |
+
if either drifts the page prints a decomposition that does not decompose
|
| 6 |
+
anything, which is worse than printing nothing.
|
| 7 |
+
"""
|
| 8 |
+
from __future__ import annotations
|
| 9 |
+
|
| 10 |
+
import pytest
|
| 11 |
+
|
| 12 |
+
from analytics.impact import bridge, growth_contributions, margin_decomposition
|
| 13 |
+
from ingestion.segments import PRODUCT_AXIS, SEGMENT_AXIS
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
@pytest.fixture
|
| 17 |
+
def fake_store(monkeypatch):
|
| 18 |
+
"""A two-period, three-member store with known arithmetic."""
|
| 19 |
+
now = [
|
| 20 |
+
{"member": "x:A", "label": "A", "revenue": 60.0, "operating_income": 30.0},
|
| 21 |
+
{"member": "x:B", "label": "B", "revenue": 30.0, "operating_income": 9.0},
|
| 22 |
+
{"member": "x:C", "label": "C", "revenue": 10.0, "operating_income": 2.0},
|
| 23 |
+
]
|
| 24 |
+
prior = [
|
| 25 |
+
{"member": "x:A", "label": "A", "revenue": 50.0, "operating_income": 20.0},
|
| 26 |
+
{"member": "x:B", "label": "B", "revenue": 25.0, "operating_income": 7.5},
|
| 27 |
+
{"member": "x:C", "label": "C", "revenue": 12.5, "operating_income": 3.0},
|
| 28 |
+
]
|
| 29 |
+
|
| 30 |
+
def _breakdown(ticker, axis):
|
| 31 |
+
return {
|
| 32 |
+
"axis": axis,
|
| 33 |
+
"period_end": "2026-06-27",
|
| 34 |
+
"prior_period_end": "2025-06-28",
|
| 35 |
+
"source_url": "https://sec.gov/x",
|
| 36 |
+
"lines": [
|
| 37 |
+
{
|
| 38 |
+
"label": row["label"], "member": row["member"],
|
| 39 |
+
"revenue": row["revenue"],
|
| 40 |
+
"prior_revenue": next(p["revenue"] for p in prior if p["member"] == row["member"]),
|
| 41 |
+
"operating_income": row["operating_income"],
|
| 42 |
+
"yoy_pct": None,
|
| 43 |
+
}
|
| 44 |
+
for row in now
|
| 45 |
+
],
|
| 46 |
+
}
|
| 47 |
+
|
| 48 |
+
import analytics.segments as seg
|
| 49 |
+
from storage import segments_db
|
| 50 |
+
monkeypatch.setattr(seg, "breakdown", _breakdown)
|
| 51 |
+
monkeypatch.setattr(segments_db, "read_segments", lambda t, a, e, **k: prior)
|
| 52 |
+
return None
|
| 53 |
+
|
| 54 |
+
|
| 55 |
+
# ββ growth contributions βββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 56 |
+
|
| 57 |
+
def test_contributions_sum_to_the_group_growth_rate(fake_store):
|
| 58 |
+
# 100.0 -> 87.5 prior, 100 now: group growth is (100-87.5)/87.5 = 14.2857%
|
| 59 |
+
result = growth_contributions("X", PRODUCT_AXIS)
|
| 60 |
+
|
| 61 |
+
assert result["total_pp"] == pytest.approx(14.2857, abs=1e-3)
|
| 62 |
+
assert sum(c["contribution_pp"] for c in result["lines"]) == pytest.approx(
|
| 63 |
+
result["total_pp"], abs=1e-9
|
| 64 |
+
)
|
| 65 |
+
|
| 66 |
+
|
| 67 |
+
def test_a_shrinking_member_contributes_negatively(fake_store):
|
| 68 |
+
result = growth_contributions("X", PRODUCT_AXIS)
|
| 69 |
+
|
| 70 |
+
shrinking = next(c for c in result["lines"] if c["label"] == "C")
|
| 71 |
+
assert shrinking["delta"] == pytest.approx(-2.5)
|
| 72 |
+
assert shrinking["contribution_pp"] < 0
|
| 73 |
+
|
| 74 |
+
|
| 75 |
+
def test_contributions_are_ranked_by_what_moved_the_group(fake_store):
|
| 76 |
+
result = growth_contributions("X", PRODUCT_AXIS)
|
| 77 |
+
|
| 78 |
+
# Not by size and not by growth rate: B grows faster than A in percentage
|
| 79 |
+
# terms here, but A moves the group more.
|
| 80 |
+
assert [c["label"] for c in result["lines"]] == ["A", "B", "C"]
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
def test_no_prior_period_means_no_contribution_table(monkeypatch):
|
| 84 |
+
import analytics.segments as seg
|
| 85 |
+
monkeypatch.setattr(seg, "breakdown", lambda t, a: {"lines": [
|
| 86 |
+
{"label": "A", "member": "x:A", "revenue": 10.0, "prior_revenue": None},
|
| 87 |
+
]})
|
| 88 |
+
|
| 89 |
+
assert growth_contributions("X", PRODUCT_AXIS) == {}
|
| 90 |
+
|
| 91 |
+
|
| 92 |
+
# ββ margin decomposition βββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 93 |
+
|
| 94 |
+
def test_mix_and_rate_sum_to_the_total_margin_change(fake_store):
|
| 95 |
+
result = margin_decomposition("X")
|
| 96 |
+
|
| 97 |
+
assert result["mix_pp"] + result["rate_pp"] == pytest.approx(result["change_pp"], abs=1e-9)
|
| 98 |
+
|
| 99 |
+
|
| 100 |
+
def test_the_decomposition_reports_the_margins_it_decomposed(fake_store):
|
| 101 |
+
result = margin_decomposition("X")
|
| 102 |
+
|
| 103 |
+
# prior 30.5/87.5 = 34.857%, now 41/100 = 41%
|
| 104 |
+
assert result["margin_prior_pct"] == pytest.approx(34.857, abs=1e-3)
|
| 105 |
+
assert result["margin_now_pct"] == pytest.approx(41.0, abs=1e-9)
|
| 106 |
+
|
| 107 |
+
|
| 108 |
+
def test_a_member_without_operating_income_is_excluded_rather_than_zeroed(monkeypatch):
|
| 109 |
+
"""Treating a missing operating income as zero would invent a loss."""
|
| 110 |
+
import analytics.segments as seg
|
| 111 |
+
from storage import segments_db
|
| 112 |
+
monkeypatch.setattr(seg, "breakdown", lambda t, a: {
|
| 113 |
+
"period_end": "2026-06-27", "prior_period_end": "2025-06-28",
|
| 114 |
+
"lines": [
|
| 115 |
+
{"label": "A", "member": "x:A", "revenue": 60.0, "operating_income": 30.0},
|
| 116 |
+
{"label": "B", "member": "x:B", "revenue": 30.0, "operating_income": None},
|
| 117 |
+
],
|
| 118 |
+
})
|
| 119 |
+
monkeypatch.setattr(segments_db, "read_segments", lambda t, a, e, **k: [
|
| 120 |
+
{"member": "x:A", "revenue": 50.0, "operating_income": 20.0},
|
| 121 |
+
{"member": "x:B", "revenue": 25.0, "operating_income": None},
|
| 122 |
+
])
|
| 123 |
+
|
| 124 |
+
# One usable pair is not a decomposition.
|
| 125 |
+
assert margin_decomposition("X") == {}
|
| 126 |
+
|
| 127 |
+
|
| 128 |
+
# ββ the disclosed bridge βββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 129 |
+
|
| 130 |
+
def test_a_guided_range_becomes_a_basis_point_delta():
|
| 131 |
+
result = bridge({
|
| 132 |
+
"driver": "Supply constraints", "metric": "gross margin",
|
| 133 |
+
"from_value": 50.1, "to_low": 47.0, "to_high": 48.0, "basis": "guided",
|
| 134 |
+
})
|
| 135 |
+
|
| 136 |
+
assert result["delta_high_bp"] == pytest.approx(-210.0, abs=1e-6)
|
| 137 |
+
assert result["delta_low_bp"] == pytest.approx(-310.0, abs=1e-6)
|
| 138 |
+
assert result["is_range"] is True
|
| 139 |
+
|
| 140 |
+
|
| 141 |
+
def test_a_single_point_target_is_not_reported_as_a_range():
|
| 142 |
+
result = bridge({"metric": "gross margin", "from_value": 50.1, "to_low": 48.0})
|
| 143 |
+
|
| 144 |
+
assert result["is_range"] is False
|
| 145 |
+
assert result["to_high"] == 48.0
|
| 146 |
+
|
| 147 |
+
|
| 148 |
+
def test_a_reversed_range_is_normalised_rather_than_rendered_backwards():
|
| 149 |
+
result = bridge({"metric": "m", "from_value": 50.0, "to_low": 48.0, "to_high": 47.0})
|
| 150 |
+
|
| 151 |
+
assert (result["to_low"], result["to_high"]) == (47.0, 48.0)
|
| 152 |
+
|
| 153 |
+
|
| 154 |
+
@pytest.mark.parametrize("payload", [
|
| 155 |
+
None,
|
| 156 |
+
"not a dict",
|
| 157 |
+
{},
|
| 158 |
+
{"from_value": 50.1}, # nowhere to go
|
| 159 |
+
{"to_low": 47.0}, # nowhere from
|
| 160 |
+
{"from_value": "n/a", "to_low": 47.0}, # unusable figure
|
| 161 |
+
])
|
| 162 |
+
def test_an_unusable_claim_yields_no_bridge(payload):
|
| 163 |
+
# Half a bridge on the page would read as "from 50.1% to None".
|
| 164 |
+
assert bridge(payload) is None
|
|
@@ -0,0 +1,286 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Segment extraction: the parts that would silently produce wrong numbers.
|
| 2 |
+
|
| 3 |
+
None of these touch the network. The XBRL fixture is a cut-down instance with
|
| 4 |
+
the same shape as a real one: dimensional contexts, a subtotal member, and a
|
| 5 |
+
prior-year comparable inside the same document.
|
| 6 |
+
"""
|
| 7 |
+
from __future__ import annotations
|
| 8 |
+
|
| 9 |
+
from analytics.segments import (
|
| 10 |
+
drop_aggregate_members,
|
| 11 |
+
margin_spread,
|
| 12 |
+
outliers,
|
| 13 |
+
)
|
| 14 |
+
from ingestion.segments import (
|
| 15 |
+
PRODUCT_AXIS,
|
| 16 |
+
SEGMENT_AXIS,
|
| 17 |
+
_period_kind,
|
| 18 |
+
member_label,
|
| 19 |
+
parse_segments,
|
| 20 |
+
)
|
| 21 |
+
|
| 22 |
+
INSTANCE = """<?xml version="1.0" encoding="UTF-8"?>
|
| 23 |
+
<xbrl xmlns="http://www.xbrl.org/2003/instance"
|
| 24 |
+
xmlns:xbrli="http://www.xbrl.org/2003/instance"
|
| 25 |
+
xmlns:xbrldi="http://xbrl.org/2006/xbrldi"
|
| 26 |
+
xmlns:us-gaap="http://fasb.org/us-gaap/2026"
|
| 27 |
+
xmlns:aapl="http://www.apple.com/20260627">
|
| 28 |
+
<xbrli:context id="Q_TOTAL">
|
| 29 |
+
<xbrli:period><xbrli:startDate>2026-03-29</xbrli:startDate><xbrli:endDate>2026-06-27</xbrli:endDate></xbrli:period>
|
| 30 |
+
</xbrli:context>
|
| 31 |
+
<xbrli:context id="Q_PHONE">
|
| 32 |
+
<xbrli:entity><xbrli:segment>
|
| 33 |
+
<xbrldi:explicitMember dimension="srt:ProductOrServiceAxis">aapl:IPhoneMember</xbrldi:explicitMember>
|
| 34 |
+
</xbrli:segment></xbrli:entity>
|
| 35 |
+
<xbrli:period><xbrli:startDate>2026-03-29</xbrli:startDate><xbrli:endDate>2026-06-27</xbrli:endDate></xbrli:period>
|
| 36 |
+
</xbrli:context>
|
| 37 |
+
<xbrli:context id="Q_MAC">
|
| 38 |
+
<xbrli:entity><xbrli:segment>
|
| 39 |
+
<xbrldi:explicitMember dimension="srt:ProductOrServiceAxis">aapl:MacMember</xbrldi:explicitMember>
|
| 40 |
+
</xbrli:segment></xbrli:entity>
|
| 41 |
+
<xbrli:period><xbrli:startDate>2026-03-29</xbrli:startDate><xbrli:endDate>2026-06-27</xbrli:endDate></xbrli:period>
|
| 42 |
+
</xbrli:context>
|
| 43 |
+
<xbrli:context id="Q_AMER">
|
| 44 |
+
<xbrli:entity><xbrli:segment>
|
| 45 |
+
<xbrldi:explicitMember dimension="us-gaap:StatementBusinessSegmentsAxis">aapl:AmericasSegmentMember</xbrldi:explicitMember>
|
| 46 |
+
<xbrldi:explicitMember dimension="srt:ConsolidationItemsAxis">us-gaap:OperatingSegmentsMember</xbrldi:explicitMember>
|
| 47 |
+
</xbrli:segment></xbrli:entity>
|
| 48 |
+
<xbrli:period><xbrli:startDate>2026-03-29</xbrli:startDate><xbrli:endDate>2026-06-27</xbrli:endDate></xbrli:period>
|
| 49 |
+
</xbrli:context>
|
| 50 |
+
<xbrli:context id="Q_PHONE_PRIOR">
|
| 51 |
+
<xbrli:entity><xbrli:segment>
|
| 52 |
+
<xbrldi:explicitMember dimension="srt:ProductOrServiceAxis">aapl:IPhoneMember</xbrldi:explicitMember>
|
| 53 |
+
</xbrli:segment></xbrli:entity>
|
| 54 |
+
<xbrli:period><xbrli:startDate>2025-03-30</xbrli:startDate><xbrli:endDate>2025-06-28</xbrli:endDate></xbrli:period>
|
| 55 |
+
</xbrli:context>
|
| 56 |
+
<xbrli:context id="Q_HEDGE">
|
| 57 |
+
<xbrli:entity><xbrli:segment>
|
| 58 |
+
<xbrldi:explicitMember dimension="srt:ProductOrServiceAxis">aapl:IPhoneMember</xbrldi:explicitMember>
|
| 59 |
+
<xbrldi:explicitMember dimension="us-gaap:HedgingDesignationAxis">us-gaap:DesignatedAsHedgingInstrumentMember</xbrldi:explicitMember>
|
| 60 |
+
</xbrli:segment></xbrli:entity>
|
| 61 |
+
<xbrli:period><xbrli:startDate>2026-03-29</xbrli:startDate><xbrli:endDate>2026-06-27</xbrli:endDate></xbrli:period>
|
| 62 |
+
</xbrli:context>
|
| 63 |
+
<xbrli:context id="BALANCE">
|
| 64 |
+
<xbrli:period><xbrli:instant>2026-06-27</xbrli:instant></xbrli:period>
|
| 65 |
+
</xbrli:context>
|
| 66 |
+
|
| 67 |
+
<us-gaap:RevenueFromContractWithCustomerExcludingAssessedTax contextRef="Q_TOTAL">109417000000</us-gaap:RevenueFromContractWithCustomerExcludingAssessedTax>
|
| 68 |
+
<us-gaap:RevenueFromContractWithCustomerExcludingAssessedTax contextRef="Q_PHONE">54250000000</us-gaap:RevenueFromContractWithCustomerExcludingAssessedTax>
|
| 69 |
+
<us-gaap:RevenueFromContractWithCustomerExcludingAssessedTax contextRef="Q_MAC">10350000000</us-gaap:RevenueFromContractWithCustomerExcludingAssessedTax>
|
| 70 |
+
<us-gaap:RevenueFromContractWithCustomerExcludingAssessedTax contextRef="Q_AMER">45780000000</us-gaap:RevenueFromContractWithCustomerExcludingAssessedTax>
|
| 71 |
+
<us-gaap:OperatingIncomeLoss contextRef="Q_AMER">21700000000</us-gaap:OperatingIncomeLoss>
|
| 72 |
+
<us-gaap:RevenueFromContractWithCustomerExcludingAssessedTax contextRef="Q_PHONE_PRIOR">44580000000</us-gaap:RevenueFromContractWithCustomerExcludingAssessedTax>
|
| 73 |
+
<us-gaap:RevenueFromContractWithCustomerExcludingAssessedTax contextRef="Q_HEDGE">999000000</us-gaap:RevenueFromContractWithCustomerExcludingAssessedTax>
|
| 74 |
+
<us-gaap:StockholdersEquity contextRef="BALANCE">107520000000</us-gaap:StockholdersEquity>
|
| 75 |
+
</xbrl>
|
| 76 |
+
"""
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
def _by(facts, member, end="2026-06-27"):
|
| 80 |
+
return next(f for f in facts if f.member.endswith(member) and f.period_end == end)
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
# ββ parse_segments βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 84 |
+
|
| 85 |
+
def test_dimensional_facts_are_extracted_with_their_axis_and_period():
|
| 86 |
+
facts = parse_segments(INSTANCE.encode())
|
| 87 |
+
|
| 88 |
+
phone = _by(facts, "IPhoneMember")
|
| 89 |
+
assert phone.axis == PRODUCT_AXIS
|
| 90 |
+
assert phone.label == "iPhone"
|
| 91 |
+
assert phone.revenue == 54_250_000_000
|
| 92 |
+
assert phone.period_kind == "quarter"
|
| 93 |
+
|
| 94 |
+
|
| 95 |
+
def test_the_prior_year_comparable_comes_from_the_same_document():
|
| 96 |
+
facts = parse_segments(INSTANCE.encode())
|
| 97 |
+
|
| 98 |
+
# This is the whole reason year-on-year costs no second request.
|
| 99 |
+
prior = _by(facts, "IPhoneMember", end="2025-06-28")
|
| 100 |
+
assert prior.revenue == 44_580_000_000
|
| 101 |
+
|
| 102 |
+
|
| 103 |
+
def test_a_reportable_segment_keeps_its_operating_income():
|
| 104 |
+
facts = parse_segments(INSTANCE.encode())
|
| 105 |
+
|
| 106 |
+
americas = _by(facts, "AmericasSegmentMember")
|
| 107 |
+
assert americas.axis == SEGMENT_AXIS
|
| 108 |
+
assert americas.label == "Americas"
|
| 109 |
+
assert americas.revenue == 45_780_000_000
|
| 110 |
+
assert americas.operating_income == 21_700_000_000
|
| 111 |
+
|
| 112 |
+
|
| 113 |
+
def test_the_undimensioned_total_is_not_a_segment():
|
| 114 |
+
facts = parse_segments(INSTANCE.encode())
|
| 115 |
+
|
| 116 |
+
# Consolidated revenue already lives in the metrics table; storing it
|
| 117 |
+
# here as a nameless member would double the segment total.
|
| 118 |
+
assert all(f.member for f in facts)
|
| 119 |
+
assert not any(f.revenue == 109_417_000_000 for f in facts)
|
| 120 |
+
|
| 121 |
+
|
| 122 |
+
def test_a_fact_cut_by_an_unrelated_axis_is_ignored():
|
| 123 |
+
facts = parse_segments(INSTANCE.encode())
|
| 124 |
+
|
| 125 |
+
# iPhone-and-hedging-designation is a slice of iPhone, not a peer of it.
|
| 126 |
+
# Admitting it would add 999m of phantom revenue to the product mix.
|
| 127 |
+
assert not any(f.revenue == 999_000_000 for f in facts)
|
| 128 |
+
|
| 129 |
+
|
| 130 |
+
def test_instant_contexts_are_ignored():
|
| 131 |
+
facts = parse_segments(INSTANCE.encode())
|
| 132 |
+
|
| 133 |
+
assert all(f.period_start for f in facts)
|
| 134 |
+
|
| 135 |
+
|
| 136 |
+
def test_an_instance_with_no_dimensions_yields_nothing_rather_than_raising():
|
| 137 |
+
minimal = (
|
| 138 |
+
'<?xml version="1.0"?><xbrl xmlns="http://www.xbrl.org/2003/instance" '
|
| 139 |
+
'xmlns:xbrli="http://www.xbrl.org/2003/instance"><xbrli:context id="C">'
|
| 140 |
+
"<xbrli:period><xbrli:startDate>2026-03-29</xbrli:startDate>"
|
| 141 |
+
"<xbrli:endDate>2026-06-27</xbrli:endDate></xbrli:period>"
|
| 142 |
+
"</xbrli:context></xbrl>"
|
| 143 |
+
)
|
| 144 |
+
|
| 145 |
+
assert parse_segments(minimal.encode()) == []
|
| 146 |
+
|
| 147 |
+
|
| 148 |
+
# ββ labels and period classification βββββββββββββββββββββββββββββββββββββββββ
|
| 149 |
+
|
| 150 |
+
def test_member_labels_are_readable():
|
| 151 |
+
assert member_label("aapl:IPhoneMember") == "iPhone"
|
| 152 |
+
assert member_label("aapl:GreaterChinaSegmentMember") == "Greater China"
|
| 153 |
+
assert member_label("aapl:RestOfAsiaPacificSegmentMember") == "Rest of Asia Pacific"
|
| 154 |
+
assert member_label("us-gaap:ServiceMember") == "Services"
|
| 155 |
+
assert member_label("aapl:WearablesHomeandAccessoriesMember") == "Wearables, Home & Accessories"
|
| 156 |
+
# An issuer this code has never seen must still get a sane label.
|
| 157 |
+
assert member_label("msft:IntelligentCloudMember") == "Intelligent Cloud"
|
| 158 |
+
|
| 159 |
+
|
| 160 |
+
def test_period_kind_separates_a_quarter_from_a_year_to_date_span():
|
| 161 |
+
assert _period_kind("2026-03-29", "2026-06-27") == "quarter"
|
| 162 |
+
assert _period_kind("2025-09-28", "2026-06-27") == "ytd"
|
| 163 |
+
assert _period_kind("2025-06-29", "2026-06-27") == "annual"
|
| 164 |
+
|
| 165 |
+
|
| 166 |
+
# ββ aggregate removal ββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 167 |
+
|
| 168 |
+
def _line(label, revenue, **extra):
|
| 169 |
+
return {"label": label, "member": f"x:{label}", "revenue": revenue, **extra}
|
| 170 |
+
|
| 171 |
+
|
| 172 |
+
def test_the_consolidated_total_identifies_the_subtotal_exactly():
|
| 173 |
+
rows = [
|
| 174 |
+
_line("Products", 78.67e9), # = iPhone + Mac + iPad + Wearables
|
| 175 |
+
_line("iPhone", 54.25e9),
|
| 176 |
+
_line("Services", 30.74e9),
|
| 177 |
+
_line("Mac", 10.35e9),
|
| 178 |
+
_line("Wearables", 7.88e9),
|
| 179 |
+
_line("iPad", 6.19e9),
|
| 180 |
+
]
|
| 181 |
+
|
| 182 |
+
kept = drop_aggregate_members(rows, total=109.417e9)
|
| 183 |
+
|
| 184 |
+
assert [r["label"] for r in kept] == ["iPhone", "Services", "Mac", "Wearables", "iPad"]
|
| 185 |
+
assert abs(sum(r["revenue"] for r in kept) - 109.41e9) < 0.02e9
|
| 186 |
+
|
| 187 |
+
|
| 188 |
+
def test_a_mix_that_already_matches_the_total_loses_nothing():
|
| 189 |
+
rows = [_line("Americas", 45.78e9), _line("Europe", 29.39e9),
|
| 190 |
+
_line("Greater China", 18.82e9), _line("Rest of Asia Pacific", 8.87e9),
|
| 191 |
+
_line("Japan", 6.55e9)]
|
| 192 |
+
|
| 193 |
+
assert len(drop_aggregate_members(rows, total=109.417e9)) == 5
|
| 194 |
+
|
| 195 |
+
|
| 196 |
+
def test_an_unexplained_overshoot_hides_nothing():
|
| 197 |
+
"""The members exceed the top line by an amount no combination matches.
|
| 198 |
+
|
| 199 |
+
Something is wrong with the filing or with this code; dropping a member
|
| 200 |
+
on a guess would present a mix that looks tidy and is wrong. Showing all
|
| 201 |
+
of them keeps the discrepancy visible.
|
| 202 |
+
"""
|
| 203 |
+
# Members sum to 130 against a 95 top line: the 35 excess matches no
|
| 204 |
+
# member and no combination of them (30, 40, 60, 70, 90, 100, 130).
|
| 205 |
+
rows = [_line("a", 60e9), _line("b", 40e9), _line("c", 30e9)]
|
| 206 |
+
|
| 207 |
+
assert len(drop_aggregate_members(rows, total=95e9)) == 3
|
| 208 |
+
|
| 209 |
+
|
| 210 |
+
def test_without_a_total_the_structural_fallback_still_catches_a_subtotal():
|
| 211 |
+
rows = [
|
| 212 |
+
_line("Products", 78.67e9),
|
| 213 |
+
_line("iPhone", 54.25e9),
|
| 214 |
+
_line("Services", 30.74e9),
|
| 215 |
+
_line("Mac", 10.35e9),
|
| 216 |
+
_line("Wearables", 7.88e9),
|
| 217 |
+
_line("iPad", 6.19e9),
|
| 218 |
+
]
|
| 219 |
+
|
| 220 |
+
assert [r["label"] for r in drop_aggregate_members(rows)] == [
|
| 221 |
+
"iPhone", "Services", "Mac", "Wearables", "iPad"
|
| 222 |
+
]
|
| 223 |
+
|
| 224 |
+
|
| 225 |
+
def test_the_subset_scan_does_not_stop_at_the_first_oversized_combination():
|
| 226 |
+
"""Regression: itertools.combinations is not ordered by sum.
|
| 227 |
+
|
| 228 |
+
The matching subset (8+2) is generated before several oversized triples,
|
| 229 |
+
but an early break on "sum exceeded" would abandon the scan and let the
|
| 230 |
+
subtotal through -- which is what shipped first and put Apple's "Products"
|
| 231 |
+
line back into the product mix.
|
| 232 |
+
"""
|
| 233 |
+
rows = [_line("total", 10.0), _line("a", 8.0), _line("b", 2.0)]
|
| 234 |
+
|
| 235 |
+
assert [r["label"] for r in drop_aggregate_members(rows)] == ["a", "b"]
|
| 236 |
+
|
| 237 |
+
|
| 238 |
+
def test_genuine_members_are_never_mistaken_for_subtotals():
|
| 239 |
+
rows = [_line("Americas", 45.78e9), _line("Europe", 29.39e9),
|
| 240 |
+
_line("Greater China", 18.82e9), _line("Japan", 6.55e9)]
|
| 241 |
+
|
| 242 |
+
assert len(drop_aggregate_members(rows)) == 4
|
| 243 |
+
|
| 244 |
+
|
| 245 |
+
def test_a_member_without_revenue_survives():
|
| 246 |
+
rows = [_line("a", 10.0), _line("b", None)]
|
| 247 |
+
|
| 248 |
+
assert len(drop_aggregate_members(rows)) == 2
|
| 249 |
+
|
| 250 |
+
|
| 251 |
+
# ββ derived reads ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 252 |
+
|
| 253 |
+
def test_outliers_name_the_two_lines_worth_a_sentence():
|
| 254 |
+
view = {"lines": [
|
| 255 |
+
{"label": "iPhone", "yoy_pct": 21.7},
|
| 256 |
+
{"label": "Mac", "yoy_pct": 28.7},
|
| 257 |
+
{"label": "iPad", "yoy_pct": -5.9},
|
| 258 |
+
]}
|
| 259 |
+
|
| 260 |
+
found = outliers(view)
|
| 261 |
+
|
| 262 |
+
assert found["fastest"]["label"] == "Mac"
|
| 263 |
+
assert found["slowest"]["label"] == "iPad"
|
| 264 |
+
|
| 265 |
+
|
| 266 |
+
def test_outliers_stay_silent_when_there_is_nothing_to_rank():
|
| 267 |
+
assert outliers({"lines": [{"label": "only", "yoy_pct": 3.0}]}) == {}
|
| 268 |
+
assert outliers({"lines": [{"label": "no yoy"}]}) == {}
|
| 269 |
+
|
| 270 |
+
|
| 271 |
+
def test_margin_spread_finds_the_gap_the_consolidated_margin_hides():
|
| 272 |
+
view = {"lines": [
|
| 273 |
+
{"label": "Americas", "margin_pct": 47.4},
|
| 274 |
+
{"label": "Greater China", "margin_pct": 39.4},
|
| 275 |
+
{"label": "Europe", "margin_pct": 45.7},
|
| 276 |
+
]}
|
| 277 |
+
|
| 278 |
+
spread = margin_spread(view)
|
| 279 |
+
|
| 280 |
+
assert spread["lowest"]["label"] == "Greater China"
|
| 281 |
+
assert spread["highest"]["label"] == "Americas"
|
| 282 |
+
assert round(spread["spread_pct"], 1) == 8.0
|
| 283 |
+
|
| 284 |
+
|
| 285 |
+
def test_margin_spread_is_absent_when_the_axis_carries_no_margins():
|
| 286 |
+
assert margin_spread({"lines": [{"label": "iPhone", "yoy_pct": 21.7}]}) is None
|