Create README.md

README.md

@@ -0,0 +1,337 @@
+---
+license: apache-2.0
+language:
+- es
+- en
+---
+
+PASITA v1
+
+PASITA is a compact generative language model specialized in plain-text to Markdown (GFM) transformation with an emphasis on preserving the information contained in the input.
+
+It is a decoder-only Transformer based on the Llama architecture, trained entirely from scratch without a pretrained base model. PASITA is designed as a specialized text transformation model rather than a general-purpose conversational or instruction-following language model.
+
+Overview
+
+PASITA targets the following transformation:
+
+Plain text
+    |
+    v
+PASITA
+    |
+    v
+Markdown (GFM)
+
+The primary objective is to introduce Markdown structure while minimizing changes to the source content.
+
+PASITA is intended for use cases where preserving the original information is more important than aggressively formatting the input.
+
+Model Specifications
+
+Property| Value
+Model| PASITA v1
+Task| Plain text → Markdown (GFM)
+Architecture| LlamaForCausalLM
+Parameters| 88,099,584 (~88M)
+Hidden size| 768
+Layers| 12
+Intermediate size| 2048
+Attention heads| 12
+Key/value heads| 4
+Attention| Grouped Query Attention (GQA)
+Head dimension| 64
+Activation| SiLU / SwiGLU
+Normalization| RMSNorm
+RMSNorm epsilon| 1e-5
+Positional encoding| RoPE
+RoPE theta| 100,000
+Maximum context| 2,048 tokens
+Vocabulary| 16,384 tokens
+Word embeddings| Tied
+Attention bias| Disabled
+MLP bias| Disabled
+Parameter dtype| bfloat16
+Weight format| safetensors
+Number of tensors| 110
+
+Architecture
+
+PASITA uses a 12-layer decoder-only Transformer with a hidden dimension of 768 and an intermediate dimension of 2048.
+
+The attention mechanism uses Grouped Query Attention with 12 query heads and 4 key/value heads. This reduces the number of key/value projections and associated cache requirements while retaining multiple query heads.
+
+The model uses:
+
+- RMSNorm
+- SiLU activation
+- SwiGLU feed-forward blocks
+- Rotary Position Embeddings (RoPE)
+- A maximum sequence length of 2,048 tokens
+- Tied input and output embeddings
+- No attention or MLP biases
+
+The model is implemented using "LlamaForCausalLM".
+
+Tokenizer
+
+PASITA uses a custom Byte-Level BPE tokenizer with a vocabulary of 16,384 pieces.
+
+The tokenizer includes:
+
+<pad>
+<s>
+</s>
+<unk>
+<think>
+</think>
+
+The ByteLevel pre-tokenizer and decoder have been verified using exact round-trip tests.
+
+Measured tokenization efficiency is approximately:
+
+- 4.0 characters per token for Spanish and English
+- 2.0 characters per token for code
+
+Training Data
+
+PASITA was trained on approximately 80 million BPE tokens.
+
+The underlying corpus consists of human-produced Markdown obtained from public sources, including:
+
+- GoodWiki English
+- Spanish Wikipedia dumps
+- StackExchange Q&A
+- WikiHow
+- Cosmopedia
+
+The Markdown documents were deterministically degraded to produce noisy plain-text inputs.
+
+The resulting training data contains approximately 37% Spanish and 63% English.
+
+Dataset Splits
+
+Split| Tokens / prompts| Description
+SFT| 58M / 47k| Supervised fine-tuning
+DPO| 16M / 9k pairs| Preference optimization
+GRPO| 6M / 10.4k prompts| Group-relative optimization
+Benchmark| 2.5M / 2k| Held-out evaluation
+
+The benchmark split is disjoint from the training data by hash.
+
+Dataset quality controls included Markdown validity checks, hash and ROUGE-based deduplication, context-length validation, and multi-judge verification.
+
+Training
+
+PASITA v1 was trained using bfloat16 on NVIDIA B300 and A100 hardware.
+
+The training pipeline consisted of three stages.
+
+1. Supervised Fine-Tuning
+
+SFT was performed for two epochs.
+
+Reported final metrics:
+
+- Loss: approximately 0.09
+- Token accuracy: 98.4%
+
+2. Direct Preference Optimization
+
+DPO was performed for one epoch with:
+
+β = 0.1
+
+The reported final preference margin was approximately 4.2.
+
+The original DPO run contained partially spurious signals in some rejected examples. A subsequent DPO-v2 dataset was developed with more minimal edits and harder negatives.
+
+3. Group Relative Policy Optimization
+
+GRPO was performed for 550 steps using groups of four generations.
+
+The reward design focused primarily on structural correctness and content fidelity.
+
+During the original training process, the fidelity component of the reward exhibited noise related to the decoder configuration, while structural rewards were learned more consistently.
+
+Evaluation
+
+Evaluation was performed greedily on held-out data.
+
+Bench-500
+
+Metric| Score
+GFM validity| 0.958
+Faithfulness| 0.930
+Semantic faithfulness| 0.880
+Table correctness| 1.000
+
+Bench-1000
+
+Metric| Score
+GFM validity| 0.956
+Faithfulness| 0.927
+Semantic faithfulness| 0.888
+Table correctness| 1.000
+
+Domain Results
+
+Domain| Score
+Code| 0.965
+OCR / transcripts| 0.953
+Documents| 0.946
+Mathematics| 0.981
+Tables| 0.945
+HTML| 1.000
+Control| 0.190
+
+The control category is the principal known weakness of the current version.
+
+Inference
+
+PASITA expects a prompt beginning with "CONVIERTE A MARKDOWN", followed by the source text and the "### Markdown:" separator.
+
+Example:
+
+prompt = (
+    "CONVIERTE A MARKDOWN:\n"
+    + text
+    + "\n\n### Markdown:\n"
+)
+
+A recommended deterministic generation configuration is:
+
+generate(
+    do_sample=False,
+    max_new_tokens=int(input_length * 1.3 + 128)
+)
+
+The model is intended for deterministic transformation rather than conversational generation.
+
+Model Size and Runtime
+
+The model contains approximately 88 million parameters.
+
+The stored model weights occupy approximately:
+
+176,211,400 bytes
+
+or approximately 168 MB.
+
+The weights are stored natively in the safetensors format.
+
+The model uses bfloat16 parameters, while CPU inference can be performed using a float32-loaded model when required by the runtime configuration.
+
+Approximate memory usage for the weights is around 180 MB, excluding runtime allocations and KV-cache requirements.
+
+The model is therefore small enough to be deployed on CPU systems and on GPUs with at least 2 GB of memory, subject to runtime overhead and the selected inference configuration.
+
+Weight Integrity
+
+The primary model file is:
+
+final_model/model.safetensors
+
+SHA-256 prefix:
+
+664665e7ee2b0002
+
+Full hashes should be obtained from the corresponding release artifacts when verifying a downloaded model.
+
+Intended Use
+
+PASITA is designed for applications that require deterministic or near-deterministic conversion of unstructured text into Markdown.
+
+Potential applications include:
+
+- Plain-text document formatting
+- Markdown normalization pipelines
+- Documentation preprocessing
+- Text-to-Markdown conversion
+- Local document transformation
+- CPU-based Markdown processing
+
+PASITA is not designed or evaluated as a general-purpose conversational assistant.
+
+Known Limitations
+
+PASITA v1 has several known limitations.
+
+Content Preservation
+
+Although faithfulness is substantially stronger than its Markdown validity score alone would indicate, the model can still:
+
+- Truncate numerical values
+- Omit secondary information
+- Echo headings unnecessarily
+- Continue generating after completing the transformation
+
+For example, numerical content such as "23.3" may occasionally be reduced to "23".
+
+Short Inputs
+
+The current model is less reliable on very short inputs, particularly inputs consisting of only one to three lines.
+
+Strict Formatting Instructions
+
+The control benchmark remains substantially weaker than the other evaluated domains. PASITA should therefore not be assumed to reliably satisfy arbitrary, highly specific formatting constraints.
+
+Generation Termination
+
+Under greedy generation, PASITA can occasionally continue generating after the correct Markdown transformation has already been produced.
+
+Applications should therefore use an appropriate generation limit and explicitly handle end-of-sequence behavior.
+
+License
+
+PASITA v1 is released under the Apache License 2.0.
+
+Copyright © 2026 OpceanAI.
+
+The Apache License 2.0 permits use, reproduction, modification, distribution, and creation of derivative works subject to the terms and conditions of the license.
+
+The complete license text is provided in the ""LICENSE"" (LICENSE) file.
+
+The Apache License 2.0 applies to the PASITA project and its released model artifacts. Third-party datasets and source material used during training may be subject to separate licenses, attribution requirements, or other terms.
+
+Users intending to redistribute or commercially deploy systems trained on the underlying datasets should independently review the applicable licenses and attribution requirements, including the requirements associated with Wikipedia-derived material.
+
+Data Provenance
+
+The model weights were produced from a random initialization and were not initialized from a pretrained language model.
+
+Training data was derived from publicly available sources, including GoodWiki, Wikipedia, StackExchange, WikiHow, and Cosmopedia.
+
+The licensing status of the resulting model does not supersede the licensing requirements applicable to third-party source material.
+
+Project Status
+
+PASITA v1 should be considered an experimental specialized language model.
+
+The current results demonstrate that a compact model trained specifically for plain-text-to-Markdown transformation can achieve high content-fidelity scores on held-out evaluation data while remaining small enough for practical local inference.
+
+Future development is focused on improving:
+
+- Termination behavior
+- Control and instruction-following performance
+- Content preservation
+- Robustness on short inputs
+- Markdown structural correctness
+- Preference and reinforcement-learning data quality
+
+Summary
+
+PASITA v1 is an approximately 88M-parameter decoder-only Transformer trained from scratch specifically for plain-text-to-Markdown transformation.
+
+Its current Bench-1000 results are:
+
+GFM validity       95.6%
+Faithfulness       92.7%
+Semantic faith.    88.8%
+Table correctness 100.0%
+
+The model's primary strength is its ability to perform structured Markdown transformation while preserving most of the source information.
+
+Its primary weaknesses are strict control tasks, very short inputs, and occasional post-completion generation.
+
+PASITA is therefore best understood as a small, specialized text-transformation model, rather than a general-purpose LLM.