| --- |
| license: gpl-3.0 |
| --- |
| |
|
|
| # JiRack GPT-2 Initial Weights |
|
|
| This file is strictly intended for saving the **initial weights (checkpoint)** of the JiRack GPT model. |
| The model is **"clean"**: it contains no data and has never undergone any pre-training. |
| - Powered by CMS Manhattan’s cutting-edge Vision-BERT architecture. |
|
|
| It is engineered to be a maximally safe and robust base for **training from scratch** for specialized, smaller models, such as: |
|
|
| - **SPAM Detection Systems** |
| - **FRAUD Detection Models** |
| - **Background Check (BG Check) Models** |
|
|
| _A product of CMS Manhattan._ |
|
|
| --- |
|
|
| ## Tokenizer Choices |
|
|
| - For English: **GPT-2 Hugging Face tokenizer** |
| - For multilingual use: **BERT tokenizer** from the Hugging Face library |
|
|
| --- |
|
|
| ## Model Architecture Details |
|
|
| ### GPT-2 Architecture (Classic, Transformer-like) |
|
|
| ``` |
| CustomEmbedding |
| FrozenSignatureLayer |
| LearnedPositionalEmbedding |
| [TransformerBlock] |
| ├── MultiHeadAttention |
| ├── LayerNorm |
| ├── LayerNorm |
| ├── FFN |
| ├── Linear |
| ├── Activation: GELU |
| └── Linear |
| LayerNorm |
| Linear |
| ``` |
|
|
| --- |
|
|
| ## Model Checkpoint File Explanations |
|
|
| ### **12-head Attention Model** |
|
|
| **Parameters:** |
| - `VOCAB_SIZE = 50257` |
| - `MODEL_DIM = 768` |
| - `NUM_HEADS = 12` |
| - `NUM_LAYERS = 6` |
| - `MAX_SEQ_LEN = 8192` |
| - `FFN_HIDDEN_DIM = 4 * MODEL_DIM` |
| - `HEAD_DIM = MODEL_DIM // NUM_HEADS` |
|
|
| **File:** |
| `JiRack_H12_L6_V50257_D768_MSL8192_FF768x4.pt` |
|
|
| --- |
|
|
| ### **6-head Attention Model** |
|
|
| **Parameters:** |
| - `VOCAB_SIZE = 50257` |
| - `MODEL_DIM = 768` |
| - `NUM_HEADS = 6` |
| - `NUM_LAYERS = 6` |
| - `MAX_SEQ_LEN = 8192` |
| - `FFN_HIDDEN_DIM = 4 * MODEL_DIM` |
| - `HEAD_DIM = MODEL_DIM // NUM_HEADS` |
|
|
| **File:** |
| `JiRack_H6_L6_V50257_D768_MSL8192_FF768x4.pt` |
|
|
|
|
|
|
| - So About PyTorch script . You can use Pytorch script for AI classification task . |
| - Do not Jit for Chatbot task . Use just state dict PyTorch for GPT (Chatbot) tasks |
|
|
|
|
| --- |
|
|
| See other models with same patterns for read parameters |
|
|
| # install tokenizer before run |
| --- |
| - mkdir -p tokenizer |
| - wget -O tokenizer/tokenizer.json https://huggingface.co/gpt2/resolve/main/tokenizer.json |
| - wget -O tokenizer/vocab.json https://huggingface.co/gpt2/resolve/main/vocab.json |
| - wget -O tokenizer/merges.txt https://huggingface.co/gpt2/resolve/main/merges.txt |
| - wget -O tokenizer/tokenizer_config.json https://huggingface.co/gpt2/resolve/main/tokenizer_config.json |
|
|
| --- |
| ### JiRack RAG System |
| - It is microservice architecture with API Gateway and Service Discovery |
| - Framework Spring boot and Google embeddings model for JiRack RAG System with Chatbot and JiRach model deployment with docker scipt |
| - video https://www.youtube.com/watch?v=vHClQu76kMc |
| - RAG System https://bitbucket.org/cmsmanhattan/rag/src/main/ |
|
|
|
|
| # Copyright Office |
| |
| - From: |
| - cop-rc@loc.gov |
| - To: |
| - konstantin.grabko@yahoo.com |
|
|
| - Mon, Dec 15 at 7:31 AM |
|
|
| - THIS IS AN AUTOMATED EMAIL. PLEASE DO NOT REPLY. |
|
|
| - Thank you for submitting your registration claim using the Electronic Copyright Office (ECO) System. |
|
|
| - The following files were successfully uploaded for service request 1-15058193231 |
|
|
| - File Name :jirack_gpt2_class_pytorch.zip |
| - File Size :2993 KB |
| - Date/Time :12/15/2025 7:27:48 AM |
| |
| - [THREAD ID: 1-6X1C895] |
| |
| - United States Copyright Office |
| |
| --- |
| Welcome to ask to design your corp model over 33B or 70B or more parameters |
| ## |
| |
| CMS Manhattan |
| Copyright © 2002–2026 |
