Upload README.md

README.md

@@ -1,3 +1,227 @@
 ---
-license: cc-by-4.0
+language:
+- en
+license: apache-2.0
+pipeline_tag: text-generation
+tags:
+- code
+- python
+- code-generation
+- programming-language-model
+- transformer
+- from-scratch
+- jumplander
 ---
+
+# JLCM-Python-100M-v0.1
+
+## JumpLander Code Language Models
+
+JLCM-Python-100M-v0.1 is an experimental Python-focused code language
+model created by JumpLander.
+
+This model is part of the **JumpLander Code Language Models (JLCM)**
+family. The goal of this family is to build specialized models for
+individual programming languages.
+
+Examples of future models:
+
+-   JLCM-JavaScript
+-   JLCM-Rust
+-   JLCM-C++
+-   JLCM-PHP
+-   JLCM-SQL
+
+------------------------------------------------------------------------
+
+# Model Summary
+
+JLCM-Python-100M is a decoder-only Transformer language model trained
+from scratch for Python programming.
+
+Key points:
+
+-   No pretrained weights were used.
+-   The model was initialized from random weights.
+-   The tokenizer was created specifically for this project.
+-   The training pipeline was built by JumpLander.
+-   The model focuses on Python code and technical English
+    understanding.
+
+This is a research and experimental model, not a replacement for large
+production coding models.
+
+------------------------------------------------------------------------
+
+# Architecture
+
+  Component           Value
+  ------------------- --------------------------
+  Model Type          Decoder-only Transformer
+  Parameters          97,536,768
+  Layers              12
+  Hidden Size         768
+  Attention Heads     12
+  Context Length      1024 Tokens
+  Vocabulary Size     16,384
+  Position Encoding   RoPE
+  Normalization       RMSNorm
+  Activation          SwiGLU
+  Training Method     From Scratch
+
+------------------------------------------------------------------------
+
+# Training
+
+The model was trained using causal language modeling.
+
+Training objective:
+
+Predict the next token based on previous tokens.
+
+The model learns:
+
+-   Python syntax
+-   Python functions
+-   Code completion patterns
+-   Technical English related to programming
+
+------------------------------------------------------------------------
+
+# Dataset
+
+The training pipeline used Python-focused datasets.
+
+## CodeParrot Clean
+
+Used for:
+
+-   Real Python source code
+-   Programming patterns
+-   Code structure learning
+
+Dataset processing:
+
+-   Syntax validation
+-   Duplicate filtering
+-   Generated code filtering
+-   Secret detection
+-   File size filtering
+
+## MBPP
+
+Used for:
+
+-   English-to-Python examples
+-   Function generation tasks
+-   Programming problem examples
+
+------------------------------------------------------------------------
+
+# Dataset Processing
+
+The pipeline performs:
+
+-   Python AST validation
+-   Duplicate removal
+-   Secret filtering
+-   Generated code filtering
+-   Dataset quality checks
+
+Final prepared dataset:
+
+-   Training samples: 35,855
+-   Validation samples: 389
+-   Test samples: 793
+
+------------------------------------------------------------------------
+
+# Tokenizer
+
+Tokenizer:
+
+Byte-Level BPE
+
+Vocabulary:
+
+16,384 tokens
+
+Optimized for:
+
+-   Python keywords
+-   Indentation
+-   Operators
+-   Function names
+-   Technical English
+
+------------------------------------------------------------------------
+
+# Hardware
+
+Training hardware:
+
+-   NVIDIA RTX 3060 12GB
+-   PyTorch
+-   Mixed Precision Training
+
+The project demonstrates training a small specialized code model on
+consumer hardware.
+
+------------------------------------------------------------------------
+
+# Intended Use
+
+Suitable for:
+
+-   Python code generation experiments
+-   Code completion research
+-   Local AI coding experiments
+-   Educational model training research
+
+------------------------------------------------------------------------
+
+# Limitations
+
+This model:
+
+-   Is a small language model.
+-   Can generate incorrect code.
+-   Does not guarantee executable solutions.
+-   Is not designed for production-critical software.
+-   Has limited general knowledge.
+
+------------------------------------------------------------------------
+
+# Model Family
+
+JumpLander Code Language Models:
+
+    JumpLander
+    |
+    └── JLCM
+        |
+        ├── JLCM-Python-100M
+        ├── JLCM-JavaScript
+        ├── JLCM-Rust
+        ├── JLCM-C++
+        └── Future language models
+
+Each model focuses on one programming language.
+
+------------------------------------------------------------------------
+
+# Future Work
+
+Planned:
+
+-   Instruction tuning
+-   Larger Python models
+-   More programming languages
+-   Better code evaluation
+-   Unit-test based training
+
+------------------------------------------------------------------------
+
+# Created by
+
+JumpLander AI Research