tianxuan / scripts /check_raw.ts
masterofthestars's picture
Tianxuan: full project + data (checkpoints, world_model, data without raw) with correct directory structure
db7bc06
Raw History Blame Contribute Delete
307 Bytes
import { loadRealFiles } from "../src/ml/data";
const files = loadRealFiles("data/raw", 30000);
console.log(`加载文件数: ${files.length}`);
let total = 0;
for (const f of files) {
console.log(`${f.name}: ${f.text.length} 字符`);
total += f.text.length;
}
console.log(`总计: ${total} 字符`);