codestar-backend / data /problems /brier-score-probability-calibration.json
github-actions
Sync backend from GitHub d72a1ca
56bd691
Raw History Blame Contribute Delete
8.5 kB
{
"slug": "brier-score-probability-calibration",
"title": "Brier Score 多分类概率预测均方校准误差",
"difficulty": "Easy",
"difficulty_breakdown": {
"concept": 2,
"implementation": 2,
"math_derivation": 2,
"boundary_complexity": 2
},
"category": "ML",
"tags": [
"Calibration",
"BrierScore",
"Evaluation",
"TorchLeet"
],
"description_md": "### 题目描述\n计算多分类概率预测的 Brier Score (Brier, 1950):给定预测概率分布 `probs`(形状 `[B, C]`,每行和为 1)与真实类别索引 `targets`(形状 `[B]`,`int64`),将 `targets` 转为 one-hot 编码 $Y \\in \\{0, 1\\}^{B \\times C}$,计算每样本所有类别上的平方误差之和并求批次平均:$\\displaystyle \\text{BS} = \\frac{1}{B}\\sum_{b=1}^B \\sum_{c=1}^C (p_{b,c} - Y_{b,c})^2$。\n\n### 函数签名\n```python\ndef brier_score_multiclass(\n probs: torch.Tensor,\n targets: torch.Tensor,\n) -> torch.Tensor:\n```\n\n### 示例\n```python\n# 输入\nprobs = [[1.0, 0.0], [0.0, 1.0]]\ntargets = [0, 1]\n\n# 输出 (Tensor, shape=[], dtype=float32)\n0.0\n```\n\n### 参数规格与约束\n- **`probs`**:`torch.Tensor`,测试规格 `shape=[2, 2], dtype=float32`\n- **`targets`**:`torch.Tensor`,测试规格 `shape=[2], dtype=int64`\n- **返回值**:`torch.Tensor`,输出规格 `shape=[], dtype=float32`\n- **精度容差**:`atol=0.0001, rtol=0.0001`\n> 题目来源:改编自 [TorchLeet](https://github.com/Exorust/TorchLeet)(MIT License),已改写为函数式评测接口与中文题面。",
"entry_point": "brier_score_multiclass",
"starter_code": "import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport math\nfrom typing import List, Tuple, Dict, Optional\n\ndef brier_score_multiclass(\n probs: torch.Tensor,\n targets: torch.Tensor,\n) -> torch.Tensor:\n # 请在此处编写你的实现代码\n pass\n",
"reference_code": "import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport math\nfrom typing import List, Tuple, Dict, Optional\n\ndef brier_score_multiclass(probs: torch.Tensor, targets: torch.Tensor) -> torch.Tensor:\n y_onehot = F.one_hot(targets, num_classes=probs.shape[-1]).to(probs.dtype)\n return ((probs - y_onehot) ** 2).sum(dim=-1).mean()\n",
"test_code": "import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport math\nfrom typing import List, Tuple, Dict, Optional\n\ncases = [\n (\"100% 确定且全对时 BS=0\", torch.tensor([[1.0, 0.0], [0.0, 1.0]]), torch.tensor([0, 1])),\n (\"均匀二分类 [0.5, 0.5] -> BS=0.5\", torch.tensor([[0.5, 0.5], [0.5, 0.5]]), torch.tensor([0, 1])),\n (\"100% 确定但全错时 BS=2.0\", torch.tensor([[0.0, 1.0], [1.0, 0.0]]), torch.tensor([0, 1])),\n (\"3 分类随机概率 B=4,C=3\", F.softmax(torch.randn(4, 3), dim=-1), torch.tensor([0, 1, 2, 0])),\n]\nfor idx, (name, *args) in enumerate(cases, 1):\n t0 = time.time()\n try:\n exp = ref_obj(*args)\n act = user_obj(*args)\n match, err, max_diff = compare_outputs(act, exp, atol=atol, rtol=rtol)\n passed, msg = match, err\n except Exception as e:\n passed, msg = False, f\"Runtime Exception: {str(e)}\"\n act, exp, max_diff = None, None, None\n cases_results.append({\n \"case_id\": idx, \"case_name\": name, \"passed\": passed,\n \"time_ms\": round((time.time() - t0) * 1000, 2), \"message\": msg,\n \"user_output_summary\": summarize_obj(act) if act is not None else None,\n \"expected_summary\": summarize_obj(exp) if exp is not None else None,\n \"max_diff\": max_diff,\n })\n",
"test_cases": [
{
"name": "100% 确定且全对时 BS=0",
"input_desc": "真实参数契约用例 (probs, targets)",
"params": [
{
"name": "probs",
"type": "torch.Tensor",
"value": "torch.tensor([[1.0, 0.0], [0.0, 1.0]], dtype=torch.float32)"
},
{
"name": "targets",
"type": "torch.Tensor",
"value": "torch.tensor([0, 1])"
}
],
"raw_inputs": {
"probs": "torch.tensor([[1.0, 0.0], [0.0, 1.0]], dtype=torch.float32)",
"targets": "torch.tensor([0, 1])"
}
},
{
"name": "均匀二分类 [0.5, 0.5] -> BS=0.5",
"input_desc": "真实参数契约用例 (probs, targets)",
"params": [
{
"name": "probs",
"type": "torch.Tensor",
"value": "torch.tensor([[0.5, 0.5], [0.5, 0.5]], dtype=torch.float32)"
},
{
"name": "targets",
"type": "torch.Tensor",
"value": "torch.tensor([0, 1])"
}
],
"raw_inputs": {
"probs": "torch.tensor([[0.5, 0.5], [0.5, 0.5]], dtype=torch.float32)",
"targets": "torch.tensor([0, 1])"
}
},
{
"name": "100% 确定但全错时 BS=2.0",
"input_desc": "真实参数契约用例 (probs, targets)",
"params": [
{
"name": "probs",
"type": "torch.Tensor",
"value": "torch.tensor([[0.0, 1.0], [1.0, 0.0]], dtype=torch.float32)"
},
{
"name": "targets",
"type": "torch.Tensor",
"value": "torch.tensor([0, 1])"
}
],
"raw_inputs": {
"probs": "torch.tensor([[0.0, 1.0], [1.0, 0.0]], dtype=torch.float32)",
"targets": "torch.tensor([0, 1])"
}
}
],
"time_limit": 5.0,
"memory_limit_mb": 512,
"atol": 0.0001,
"rtol": 0.0001,
"forbidden_modules": [
"os",
"sys",
"subprocess",
"socket",
"shutil",
"pathlib",
"ctypes",
"multiprocessing"
],
"forbidden_calls": [
"eval",
"exec",
"compile",
"open",
"__import__",
"input",
"breakpoint"
],
"is_featured": false,
"editorial_md": "#### 💡 核心算法原理与数学推导\n**Brier Score 多分类概率预测均方校准误差**(`brier_score_multiclass`)聚焦于 **ML** 中的关键计算逻辑:\n\n计算多分类概率预测的 Brier Score (Brier, 1950):给定预测概率分布 `probs`(形状 `[B, C]`,每行和为 1)与真实类别索引 `targets`(形状 `[B]`,`int64`),将 `targets` 转为 one-hot 编码 $Y \\in \\{0, 1\\}^{B \\times C}$,计算每样本所有类别上的平方误差之和并求批次平均:$\\displaystyle \\text{BS} = \\frac{1}{B}\\sum_{b=1}^B \\sum_{c=1}^C (p_{b,c} - Y_{b,c})^2$。\n\n在实现中,函数接收参数 `probs` (`torch.Tensor`)、`targets` (`torch.Tensor`),目标是计算出类型与形状严格符合 `torch.Tensor` 契约的结果。\n\n#### 📐 输入输出张量变换与步骤拆解\n- **输入基准规格**:`probs` 规格为 `torch.Tensor(shape=[2, 2], dtype=float32)`,`targets` 规格为 `torch.Tensor(shape=[2], dtype=int64)`。\n- **期望输出规格**:`torch.Tensor(shape=[], dtype=float32)`。\n\n**具体推导与计算步骤如下:**\n1. 计算中间量 `y_onehot`:通过表达式 `F.one_hot(targets, num_classes=probs.shape[-1]).to(probs.dtype)` 完成张量或数值变换。\n2. 返回最终结果:输出 `((probs - y_onehot) ** 2).sum(dim=-1).mean()`,严格对齐函数签名要求的返回类型与维度。\n\n#### 📊 复杂度分析\n- **时间复杂度**:$O(N)$(线性于输入张量总元素个数 $\\prod d_i$),所有运算均为向量化逐元素算子或归约计算。\n- **空间复杂度**:$O(N)$,需要与输入或输出张量同规模的显存空间保存计算结果。\n\n#### ⚠️ 关键实现细节与易错点\n- **归约维度 (`dim`) 与广播对齐 (`keepdim`)**:调用归约算子时需明确指定目标轴(如 `dim=-1`),并在需要与原张量逐元素运算时开启 `keepdim=True`。\n- **边界输入鲁棒性**:注意单样本(`Batch=1`)、零值输入或极端超参数下的计算行为,禁止硬编码特定测试用例常量。\n- **向量化算子优先**:优先使用 PyTorch 原生向量化张量算子,避免低效的逐元素循环。\n\n#### 💻 官方标准参考实现\n```python\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport math\nfrom typing import List, Tuple, Dict, Optional\n\ndef brier_score_multiclass(probs: torch.Tensor, targets: torch.Tensor) -> torch.Tensor:\n y_onehot = F.one_hot(targets, num_classes=probs.shape[-1]).to(probs.dtype)\n return ((probs - y_onehot) ** 2).sum(dim=-1).mean()\n```"
}