| --- |
| language: |
| - zh |
| - en |
| - code |
| license: apache-2.0 |
| library_name: cpp |
| tags: |
| - neuroflow |
| - causal-lm |
| - sn |
| - ecn |
| - dmn |
| - memory-augmented |
| - transformer |
| - cpp |
| - llm |
| pipeline_tag: text-generation |
| --- |
| |
| # NeuroFlow C++ LLM |
|
|
| NeuroFlow is a **memory-augmented causal language model** implemented entirely in **C++17**, featuring a three-brain architecture (SN/ECN/DMN) inspired by cognitive neuroscience. Designed for efficient training and inference on consumer GPUs. |
|
|
| ## Model Architecture |
|
|
| | Parameter | Value | Description | |
| |-----------|-------|-------------| |
| | `d_model` | 512 | Model dimension | |
| | `hidden_dim` | 2048 | FFN hidden dimension | |
| | `memory_dim` | 512 | Memory dimension | |
| | `num_layers` | 12 | ECN layers | |
| | `memory_slots` | 64 | Memory slots | |
| | `num_associations` | 8 | DMN association heads | |
| | `vocab_size` | 128,000 | 128K multilingual BPE tokenizer | |
| | `max_seq_len` | 512 | Maximum sequence length | |
| | `causal_window_size` | 64 | Causal attention window | |
| | `lm_num_attn_layers` | 2 | Causal LM attention layers | |
| | `params` | ~120M | Total parameters | |
|
|
| ### Three-Brain Architecture |
| - **SN (Sensory Network)**: Input encoding and feature extraction |
| - **ECN (Executive Control Network)**: Core reasoning and processing layers |
| - **DMN (Default Mode Network)**: Memory-augmented association and retrieval |
|
|
| ## Files |
|
|
| | File | Size | Description | |
| |------|------|-------------| |
| | `output/checkpoint_step1000/model.nfv1` | 431 MB | Checkpoint at 1000 steps | |
| | `output/checkpoint_step2000/model.nfv1` | 431 MB | Checkpoint at 2000 steps | |
| | `output/lm_head_lmh1.nfv1` | 255 MB | LM head weights (native format v1) | |
| | `configs/config.json` | — | Model architecture configuration | |
| | `configs/tokenizer_128k.json` | — | 128K BPE tokenizer | |
| | `configs/huggingface/` | — | HuggingFace-compatible tokenizer files (vocab.json, merges.txt) | |
|
|
| ### Source Code |
| The full C++ source is included under `src/` and `include/` directories: |
| - **Core**: `tensor.hpp/cpp`, `model.hpp/cpp`, `tokenizer.hpp/cpp` |
| - **Architecture**: `causal_lm.hpp/cpp`, `generative_model.hpp/cpp`, `networks.hpp` |
| - **Training**: `train_lm.hpp/cpp`, `train_v2.cpp`, `sft_train.cpp`, `dpo_train.cpp` |
| - **CUDA**: `cuda_context.hpp/cpp`, `cuda_kernels.hpp`, `tensor_ops.cpp` |
| - **Optimizers**: `adamw.hpp/cpp`, `scheduler.hpp/cpp`, `grad_scaler.hpp/cpp` |
|
|
| ## Build & Train |
|
|
| ### Prerequisites |
| - CMake ≥ 3.15 |
| - C++17 compiler (GCC ≥ 9, MSVC 2019+) |
| - CUDA Toolkit ≥ 11.4 (optional, for GPU training) |
| - BLAS (OpenBLAS recommended) |
|
|
| ### Build |
| ```bash |
| # CPU only |
| mkdir build && cd build |
| cmake .. -DCMAKE_BUILD_TYPE=Release |
| make -j$(nproc) |
| |
| # With CUDA |
| mkdir build_cuda && cd build_cuda |
| cmake .. -DNEUROFLOW_USE_CUDA=ON -DCMAKE_BUILD_TYPE=Release |
| make -j$(nproc) |
| ``` |
|
|
| ### Train |
| ```bash |
| ./build_cuda/neuroflow_train_v2 \ |
| --config configs/config_distill.json \ |
| --data data/distill_train.txt \ |
| --output output \ |
| --epochs 20 \ |
| --batch-size 64 \ |
| --lr 0.0003 \ |
| --use-cuda --adam |
| ``` |
|
|
| ## Training Scripts |
|
|
| Key Python scripts in `scripts/`: |
| - `train_distill.py` — Knowledge distillation training pipeline |
| - `preprocess_distill.py` — Data preprocessing for distillation |
| - `deploy_dsw.sh` — One-click deployment for Alibaba Cloud DSW (A10 GPU) |
| - `train_optimized.sh` — Optimized multi-stage training |
|
|
| ## License |
|
|
| Apache 2.0 |
|
|
| ## Links |
|
|
| - [GitHub Repository](https://github.com/chenzhiwenhphp12-afk/neuroflow-model) |
| - [HuggingFace Mirror](https://hf-mirror.com/cwenzi/neuroflow-cpp) |
|
|
