codestar-backend / data /problems /causal-attention-mask.json
github-actions
Sync backend from GitHub d72a1ca
56bd691
Raw History Blame Contribute Delete
5.95 kB
{
"slug": "causal-attention-mask",
"title": "手撕自注意力因果掩码生成 (Causal Attention Mask)",
"difficulty": "Easy",
"category": "Attention",
"tags": [
"常见手撕",
"Mask",
"Transformer",
"Decoder"
],
"description_md": "### 题目描述\n自回归 Decoder-only 模型(如 GPT、LLaMA)需要使用因果下三角掩码,禁止当前 Token 关注未来的 Token:\n$$\\text{mask}[i, j] = \\begin{cases} 0 & i \\ge j \\\\ -\\infty & i < j \\end{cases}$$\n\n### 函数签名\n```python\ndef create_causal_mask(seq_len: int) -> torch.Tensor:\n```\n\n### 示例\n```python\n# 输入\nseq_len = 1\n\n# 输出 (Tensor, shape=[1, 1], dtype=float32)\n[[0.0]]\n```\n\n### 参数规格与约束\n- **`seq_len`**:`int`\n- **返回值**:`torch.Tensor`,输出规格 `shape=[1, 1], dtype=float32`\n- **精度容差**:`atol=0.0001, rtol=0.0001`",
"entry_point": "create_causal_mask",
"starter_code": "import torch\n\ndef create_causal_mask(seq_len: int) -> torch.Tensor:\n \"\"\"\n 生成因果注意力掩码 [seq_len, seq_len],未来位置填充 -inf\n \"\"\"\n # 请在此编写代码\n pass\n",
"reference_code": "import torch\n\ndef create_causal_mask(seq_len: int) -> torch.Tensor:\n mask = torch.full((seq_len, seq_len), -float(\"inf\"))\n mask = torch.triu(mask, diagonal=1)\n return mask\n",
"forbidden_modules": [
"os",
"sys",
"subprocess",
"shutil",
"socket",
"requests"
],
"forbidden_calls": [],
"time_limit": 3.0,
"memory_limit_mb": 512,
"atol": 0.0001,
"rtol": 0.0001,
"test_code": "import torch\ncases = [\n (\"最小单步因果掩码 (seq=1)\", 1),\n (\"小尺寸因果掩码 (seq=4)\", 4),\n (\"奇数序列长度掩码 (seq=7)\", 7),\n (\"较大因果掩码 (seq=16)\", 16),\n]\nfor idx, (name, s) in enumerate(cases, 1):\n t0 = time.time()\n try:\n exp = ref_obj(s); act = user_obj(s)\n match, err, max_diff = compare_outputs(act, exp, atol=atol, rtol=rtol)\n passed, msg = match, err\n except Exception as e:\n passed, msg, act, exp, max_diff = False, str(e), None, None, None\n cases_results.append({\n \"case_id\": idx, \"case_name\": name, \"passed\": passed,\n \"time_ms\": round((time.time() - t0) * 1000, 2), \"message\": msg,\n \"user_output_summary\": summarize_obj(act) if act is not None else None,\n \"expected_summary\": summarize_obj(exp) if exp is not None else None,\n \"max_diff\": max_diff,\n })\n",
"is_featured": true,
"editorial_md": "#### 💡 核心算法原理与数学推导\n**手撕自注意力因果掩码生成 (Causal Attention Mask)**(`create_causal_mask`)聚焦于 **Attention** 中的关键计算逻辑:\n\n自回归 Decoder-only 模型(如 GPT、LLaMA)需要使用因果下三角掩码,禁止当前 Token 关注未来的 Token:\n$$\\text{mask}[i, j] = \\begin{cases} 0 & i \\ge j \\\\ -\\infty & i < j \\end{cases}$$\n\n在实现中,函数接收参数 `seq_len` (`int`),目标是计算出类型与形状严格符合 `torch.Tensor` 契约的结果。\n\n#### 📐 输入输出张量变换与步骤拆解\n- **输入基准规格**:`seq_len` 规格为 `int` (示例值: `1`)。\n- **期望输出规格**:`torch.Tensor(shape=[1, 1], dtype=float32)`。\n\n**具体推导与计算步骤如下:**\n1. 计算中间量 `mask`:通过表达式 `torch.full((seq_len, seq_len), -float(\"inf\"))` 完成张量或数值变换。\n2. 计算中间量 `mask`:通过表达式 `torch.triu(mask, diagonal=1)` 完成张量或数值变换。\n3. 返回最终结果:输出 `mask`,严格对齐函数签名要求的返回类型与维度。\n\n#### 📊 复杂度分析\n- **时间复杂度**:$O(1)$,仅涉及标量代数运算或闭式公式求值,执行耗时与模型规模参数数值大小无关。\n- **空间复杂度**:$O(1)$,仅使用常数级标量局部变量,无额外张量显存分配。\n\n#### ⚠️ 关键实现细节与易错点\n- **掩码布尔极性与负无穷填充**:构造因果掩码或阈值过滤掩码时,需仔细区分“保留位置”与“屏蔽位置”的布尔条件,屏蔽处应填充 `float('-inf')` 而非 `0`,以确保后续 `softmax` 权重精确归零。\n- **边界输入鲁棒性**:注意单样本(`Batch=1`)、单步序列(`Seq=1`)、零值输入或极端超参数下的计算行为,不要在代码中硬编码特定测试用例的维度常数。\n- **纯向量化算子优先**:尽量使用 PyTorch 原生向量化张量算子替代低效的 Python 逐元素循环,以兼顾代码简洁性与执行性能。\n\n#### 💻 官方标准参考实现\n```python\nimport torch\n\ndef create_causal_mask(seq_len: int) -> torch.Tensor:\n mask = torch.full((seq_len, seq_len), -float(\"inf\"))\n mask = torch.triu(mask, diagonal=1)\n return mask\n```",
"difficulty_breakdown": {
"concept": 2,
"implementation": 2,
"math_derivation": 2,
"boundary_complexity": 1
},
"test_cases": [
{
"name": "最小单步因果掩码 (seq=1)",
"input_desc": "真实参数契约用例 (seq_len)",
"params": [
{
"name": "seq_len",
"type": "int",
"value": "1"
}
],
"raw_inputs": {
"seq_len": "1"
}
},
{
"name": "小尺寸因果掩码 (seq=4)",
"input_desc": "真实参数契约用例 (seq_len)",
"params": [
{
"name": "seq_len",
"type": "int",
"value": "4"
}
],
"raw_inputs": {
"seq_len": "4"
}
},
{
"name": "奇数序列长度掩码 (seq=7)",
"input_desc": "真实参数契约用例 (seq_len)",
"params": [
{
"name": "seq_len",
"type": "int",
"value": "7"
}
],
"raw_inputs": {
"seq_len": "7"
}
}
]
}