Spaces:
Running on Zero
Running on Zero
| { | |
| "slug": "allgather-comm-volume", | |
| "title": "手撕 All-Gather 分布式通信量估算", | |
| "difficulty": "Easy", | |
| "category": "AIInfra", | |
| "tags": [ | |
| "AllGather", | |
| "FSDP", | |
| "ZeRO-3" | |
| ], | |
| "description_md": "### 题目描述\n在包含 $N$ 个 GPU 的通信域中，每个 GPU 进行 All-Gather 传输的数据量为 $\\frac{N - 1}{N} S$。\n\n### 函数签名\n```python\ndef allgather_comm_volume(data_bytes: int, num_gpus: int) -> float:\n```\n\n### 示例\n```python\n# 输入\ndata_bytes = 1000000000\nnum_gpus = 8\n\n# 输出 (float)\n875000000.0\n```\n\n### 参数规格与约束\n- **`data_bytes`**：`int`\n- **`num_gpus`**：`int`\n- **返回值**：`float`，输出规格 `float` (示例值: `875000000.0`)\n- **精度容差**：`atol=0.0001, rtol=0.0001`", | |
| "entry_point": "allgather_comm_volume", | |
| "starter_code": "import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport math\nfrom typing import Optional, List, Tuple\n\ndef allgather_comm_volume(data_bytes: int, num_gpus: int) -> float:\n \"\"\"\n 手撕 All-Gather 分布式通信量估算\n \"\"\"\n # 请在此编写代码\n pass\n", | |
| "reference_code": "import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport math\nfrom typing import Optional, List, Tuple\n\ndef allgather_comm_volume(data_bytes: int, num_gpus: int) -> float:\n return float(((num_gpus - 1) / num_gpus) * data_bytes)\n", | |
| "forbidden_modules": [ | |
| "os", | |
| "sys", | |
| "subprocess", | |
| "shutil", | |
| "socket", | |
| "requests" | |
| ], | |
| "forbidden_calls": [], | |
| "time_limit": 3.0, | |
| "memory_limit_mb": 512, | |
| "atol": 0.0001, | |
| "rtol": 0.0001, | |
| "test_code": "import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport math\n\ncases = [\n ('8卡AllGather通信', 1000000000, 8),\n ('数值缩放与幅值扰动验证', 2000000000, 9),\n ('多维批次与参数泛化验证', 4000000000, 10),\n ('边界极值与稳定性验证', 8000000000, 11),\n]\nfor idx, (name, *args) in enumerate(cases, 1):\n t0 = time.time()\n try:\n exp = ref_obj(*args)\n act = user_obj(*args)\n match, err, max_diff = compare_outputs(act, exp, atol=atol, rtol=rtol)\n passed, msg = match, err\n except Exception as e:\n passed, msg = False, f\"Runtime Exception: {str(e)}\"\n act, exp, max_diff = None, None, None\n cases_results.append({\n \"case_id\": idx, \"case_name\": name, \"passed\": passed,\n \"time_ms\": round((time.time() - t0) * 1000, 2), \"message\": msg,\n \"user_output_summary\": summarize_obj(act) if act is not None else None,\n \"expected_summary\": summarize_obj(exp) if exp is not None else None,\n \"max_diff\": max_diff,\n })\n", | |
| "is_featured": false, | |
| "editorial_md": "#### 💡 核心算法原理与数学推导\n**手撕 All-Gather 分布式通信量估算**（`allgather_comm_volume`）聚焦于 **AIInfra** 中的关键计算逻辑：\n\n在包含 $N$ 个 GPU 的通信域中，每个 GPU 进行 All-Gather 传输的数据量为 $\\frac{N - 1}{N} S$。\n\n在实现中，函数接收参数 `data_bytes` (`int`)、`num_gpus` (`int`)，目标是计算出类型与形状严格符合 `float` 契约的结果。\n\n#### 📐 输入输出张量变换与步骤拆解\n- **输入基准规格**：`data_bytes` 规格为 `int` (示例值: `1000000000`)，`num_gpus` 规格为 `int` (示例值: `8`)。\n- **期望输出规格**：`float` (示例值: `875000000.0`)。\n\n**具体推导与计算步骤如下：**\n1. 返回最终结果：输出 `float(((num_gpus - 1) / num_gpus) * data_bytes)`，严格对齐函数签名要求的返回类型与维度。\n\n#### 📊 复杂度分析\n- **时间复杂度**：$O(1)$，仅涉及标量代数运算或闭式公式求值，执行耗时与模型规模参数数值大小无关。\n- **空间复杂度**：$O(1)$，仅使用常数级标量局部变量，无额外张量显存分配。\n\n#### ⚠️ 关键实现细节与易错点\n- **除零保护与数值平滑 (`eps`)**：在执行分母归一化、标准差缩放或对数运算时，必须按公式位置准确加上 `eps`（例如在平方根内 `sqrt(var + eps)` 还是分母外 `+ eps`），防止全零或常量输入触发 `NaN` / `Inf`。\n- **边界输入鲁棒性**：注意单样本（`Batch=1`）、单步序列（`Seq=1`）、零值输入或极端超参数下的计算行为，不要在代码中硬编码特定测试用例的维度常数。\n- **纯向量化算子优先**：尽量使用 PyTorch 原生向量化张量算子替代低效的 Python 逐元素循环，以兼顾代码简洁性与执行性能。\n\n#### 💻 官方标准参考实现\n```python\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport math\nfrom typing import Optional, List, Tuple\n\ndef allgather_comm_volume(data_bytes: int, num_gpus: int) -> float:\n return float(((num_gpus - 1) / num_gpus) * data_bytes)\n```", | |
| "difficulty_breakdown": { | |
| "concept": 1, | |
| "implementation": 1, | |
| "math_derivation": 2, | |
| "boundary_complexity": 1 | |
| }, | |
| "test_cases": [ | |
| { | |
| "name": "8卡AllGather通信", | |
| "input_desc": "真实参数契约用例 (data_bytes, num_gpus)", | |
| "params": [ | |
| { | |
| "name": "data_bytes", | |
| "type": "int", | |
| "value": "1000000000" | |
| }, | |
| { | |
| "name": "num_gpus", | |
| "type": "int", | |
| "value": "8" | |
| } | |
| ], | |
| "raw_inputs": { | |
| "data_bytes": "1000000000", | |
| "num_gpus": "8" | |
| } | |
| }, | |
| { | |
| "name": "数值缩放与幅值扰动验证", | |
| "input_desc": "真实参数契约用例 (data_bytes, num_gpus)", | |
| "params": [ | |
| { | |
| "name": "data_bytes", | |
| "type": "int", | |
| "value": "2000000000" | |
| }, | |
| { | |
| "name": "num_gpus", | |
| "type": "int", | |
| "value": "9" | |
| } | |
| ], | |
| "raw_inputs": { | |
| "data_bytes": "2000000000", | |
| "num_gpus": "9" | |
| } | |
| }, | |
| { | |
| "name": "多维批次与参数泛化验证", | |
| "input_desc": "真实参数契约用例 (data_bytes, num_gpus)", | |
| "params": [ | |
| { | |
| "name": "data_bytes", | |
| "type": "int", | |
| "value": "4000000000" | |
| }, | |
| { | |
| "name": "num_gpus", | |
| "type": "int", | |
| "value": "10" | |
| } | |
| ], | |
| "raw_inputs": { | |
| "data_bytes": "4000000000", | |
| "num_gpus": "10" | |
| } | |
| } | |
| ] | |
| } |