{/* Dot grid overlay */}

{/* Top edge glow */}

{/* ── TERMINAL TOPBAR ── */}

~/sortiq/docs/rl-inference.md

{/* ── BREADCRUMB ── */}

research / reinforcement-learning / decision-kernels

{/* ── HERO ── */}

v1.0.0 — stable build

Sortiq
RL Inference
Tabular Q-Learning Under Constraints

A pedagogical environment for studying RL fundamentals. Unlike standard classification, Sortiq forces an agent to learn the relationship between{" "} state management and{" "} long-term survival — physical constraints make naive policies fail immediately.

{/* ── QUICK STATS ── */}

{[ { val: "15K", label: "Training Episodes", accent: "#3b82f6" }, { val: "10", label: "Items per Batch", accent: "#a855f7" }, { val: "+8.0", label: "Max Reward / Sort", accent: "#22c55e" }, ].map(({ val, label, accent }) => (

{val}

{label}

))}

{/* ── 01. PURPOSE ── */}

Sortiq was built as a pedagogical RL environment. Unlike standard classification — where the only goal is accuracy — Sortiq introduces{" "} physical constraints, forcing the agent to manage a storage buffer and energy budget simultaneously. Naive "always store" policies crash immediately. The agent must learn{" "} survivability before it can learn accuracy.

{/* ── 02. ENVIRONMENT ── */}

{/* State space */}

State Space

{["variable", "range", "description"].map((h) => ( {h} ))}

{stateVars.map((s, i) => (

(e.currentTarget.style.background = "#080808")} onMouseLeave={(e) => (e.currentTarget.style.background = "transparent")} > {s.key} {s.range} {s.desc}

))}

{/* Action space */}

Action Space

{actions.map((a) => (

(e.currentTarget.style.borderColor = "#1e3a8a")} onMouseLeave={(e) => (e.currentTarget.style.borderColor = "#1a1a1a")} >

Action {a.id} {a.name}

{a.desc}

cost: {a.cost}

))}

{/* Constraints */}

⚠ Critical Constraints

Storage Bottleneck — Buffer is exactly 6 units. A 7th item triggers an immediate system crash with a −20.0 penalty.

Energy Budget — Agent starts with 4 units to process 10 items. It cannot simply store everything — every action must be deliberate.

{/* ── 03. REWARD ── */}

System death is penalized far more heavily than classification errors — teaching the agent that{" "} survival is the prerequisite for performance.

{["outcome", "reward", "description"].map((h) => ( {h} ))}

{rewardData.map((r, i) => (

(e.currentTarget.style.background = "#080808")} onMouseLeave={(e) => (e.currentTarget.style.background = "transparent")} > {r.outcome} {r.reward} {r.desc}

))}

// state = (storage_count, energy_left, item_type)

{/* ── 04. AGENT ARCHITECTURE ── */}

The agent uses Tabular Q-Learning — a 3D Q-Table where each cell holds the expected future reward of taking a specific action in a specific state. No neural networks, no approximation. Pure classical RL — every Q-value is looked up directly.

{/* Hyperparams */}

{hyperparams.map((h) => (

{h.name}

{h.val}

{h.desc}

{h.sym}

))}

{/* ── 05. DATASET ── */}

The "dataset" is a stochastic stream of items generated by the environment at runtime — not a static file. This ensures the agent encounters a diverse range of resource states such as high_storage / low_energy vs. low_storage / high_energy, testing the robustness of its policy across edge cases rather than memorising a fixed sequence.

{/* ── 06. PERFORMANCE ── */}

{/* Score comparison chips */}

{[ { label: "Random Baseline", val: "~−12.0", color: "#f87171", sub: "avg reward — frequent crashes" }, { label: "Trained Agent", val: "~+58.0", color: "#4ade80", sub: "avg reward — near-perfect policy" }, ].map((s) => (

{s.label}

{s.val}

{s.sub}

))}

{/* ── FOOTER ── */}