Spaces:
Running on Zero
Running on Zero
File size: 6,282 Bytes
56bd691 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 | {
"slug": "alibi-slopes-head-generation",
"title": "手撕 ALiBi 多头线性斜率几何递减生成 (ALiBi Head Slopes)",
"difficulty": "Easy",
"category": "Attention",
"tags": [
"ALiBi",
"外推位置编码",
"几何序列"
],
"description_md": "### 题目描述\n为 $H$ 个注意力头（$H$ 为 2 的整数次幂）生成 ALiBi 的基准衰减斜率向量 $m \\in \\mathbb{R}^H$：$m_h = 2^{-\\frac{8 h}{H}}$（其中 $h = 1, 2, \\dots, H$）。\n\n### 函数签名\n```python\ndef alibi_slopes(num_heads: int) -> torch.Tensor:\n```\n\n### 示例\n```python\n# 输入\nnum_heads = 4\n\n# 输出 (Tensor, shape=[4], dtype=float32)\n[0.25, 0.0625, 0.0156, 0.0039]\n```\n\n### 参数规格与约束\n- **`num_heads`**：`int`\n- **返回值**：`torch.Tensor`，输出规格 `shape=[4], dtype=float32`\n- **精度容差**：`atol=0.0001, rtol=0.0001`",
"entry_point": "alibi_slopes",
"starter_code": "import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport math\nfrom typing import Optional, List, Tuple\n\ndef alibi_slopes(num_heads: int) -> torch.Tensor:\n \"\"\"\n 手撕 ALiBi 多头线性斜率几何递减生成 (ALiBi Head Slopes)\n \"\"\"\n # 请在此编写代码\n pass\n",
"reference_code": "import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport math\nfrom typing import Optional, List, Tuple\n\ndef alibi_slopes(num_heads: int) -> torch.Tensor:\n h = torch.arange(1, num_heads + 1, dtype=torch.float32)\n return torch.pow(2.0, -8.0 * h / num_heads)\n",
"forbidden_modules": [
"os",
"sys",
"subprocess",
"shutil",
"socket",
"requests"
],
"forbidden_calls": [],
"time_limit": 3.0,
"memory_limit_mb": 512,
"atol": 0.0001,
"rtol": 0.0001,
"test_code": "import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport math\n\ncases = [\n ('4头ALiBi斜率生成', 4),\n ('8头ALiBi斜率生成', 8),\n ('多维批次与参数泛化验证', 9),\n ('边界极值与稳定性验证', 32),\n]\nfor idx, (name, *args) in enumerate(cases, 1):\n t0 = time.time()\n try:\n exp = ref_obj(*args)\n act = user_obj(*args)\n match, err, max_diff = compare_outputs(act, exp, atol=atol, rtol=rtol)\n passed, msg = match, err\n except Exception as e:\n passed, msg = False, f\"Runtime Exception: {str(e)}\"\n act, exp, max_diff = None, None, None\n cases_results.append({\n \"case_id\": idx, \"case_name\": name, \"passed\": passed,\n \"time_ms\": round((time.time() - t0) * 1000, 2), \"message\": msg,\n \"user_output_summary\": summarize_obj(act) if act is not None else None,\n \"expected_summary\": summarize_obj(exp) if exp is not None else None,\n \"max_diff\": max_diff,\n })\n",
"is_featured": false,
"editorial_md": "#### 💡 核心算法原理与数学推导\n**手撕 ALiBi 多头线性斜率几何递减生成 (ALiBi Head Slopes)**（`alibi_slopes`）聚焦于 **Attention** 中的关键计算逻辑：\n\n为 $H$ 个注意力头（$H$ 为 2 的整数次幂）生成 ALiBi 的基准衰减斜率向量 $m \\in \\mathbb{R}^H$：$m_h = 2^{-\\frac{8 h}{H}}$（其中 $h = 1, 2, \\dots, H$）。\n\n在实现中，函数接收参数 `num_heads` (`int`)，目标是计算出类型与形状严格符合 `torch.Tensor` 契约的结果。\n\n#### 📐 输入输出张量变换与步骤拆解\n- **输入基准规格**：`num_heads` 规格为 `int` (示例值: `4`)。\n- **期望输出规格**：`torch.Tensor(shape=[4], dtype=float32)`。\n\n**具体推导与计算步骤如下：**\n1. 计算中间量 `h`：通过表达式 `torch.arange(1, num_heads + 1, dtype=torch.float32)` 完成张量或数值变换。\n2. 返回最终结果：输出 `torch.pow(2.0, -8.0 * h / num_heads)`，严格对齐函数签名要求的返回类型与维度。\n\n#### 📊 复杂度分析\n- **时间复杂度**：$O(N)$（线性于输入张量总元素个数 $\\prod d_i$），所有运算均为向量化逐元素算子或沿指定维度的归约计算。\n- **空间复杂度**：$O(N)$，需要与输入或输出张量同规模的显存空间保存计算结果。\n\n#### ⚠️ 关键实现细节与易错点\n- **除零保护与数值平滑 (`eps`)**：在执行分母归一化、标准差缩放或对数运算时，必须按公式位置准确加上 `eps`（例如在平方根内 `sqrt(var + eps)` 还是分母外 `+ eps`），防止全零或常量输入触发 `NaN` / `Inf`。\n- **边界输入鲁棒性**：注意单样本（`Batch=1`）、单步序列（`Seq=1`）、零值输入或极端超参数下的计算行为，不要在代码中硬编码特定测试用例的维度常数。\n- **纯向量化算子优先**：尽量使用 PyTorch 原生向量化张量算子替代低效的 Python 逐元素循环，以兼顾代码简洁性与执行性能。\n\n#### 💻 官方标准参考实现\n```python\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport math\nfrom typing import Optional, List, Tuple\n\ndef alibi_slopes(num_heads: int) -> torch.Tensor:\n h = torch.arange(1, num_heads + 1, dtype=torch.float32)\n return torch.pow(2.0, -8.0 * h / num_heads)\n```",
"difficulty_breakdown": {
"concept": 2,
"implementation": 1,
"math_derivation": 2,
"boundary_complexity": 1
},
"test_cases": [
{
"name": "4头ALiBi斜率生成",
"input_desc": "真实参数契约用例 (num_heads)",
"params": [
{
"name": "num_heads",
"type": "int",
"value": "4"
}
],
"raw_inputs": {
"num_heads": "4"
}
},
{
"name": "8头ALiBi斜率生成",
"input_desc": "真实参数契约用例 (num_heads)",
"params": [
{
"name": "num_heads",
"type": "int",
"value": "8"
}
],
"raw_inputs": {
"num_heads": "8"
}
},
{
"name": "多维批次与参数泛化验证",
"input_desc": "真实参数契约用例 (num_heads)",
"params": [
{
"name": "num_heads",
"type": "int",
"value": "9"
}
],
"raw_inputs": {
"num_heads": "9"
}
}
]
} |