| --- |
| language: |
| - en |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - csharp |
| - code |
| - causal-lm |
| - experimental |
| - base-model |
| - research |
| --- |
| |
| # Sharp Coder V0.1 |
|
|
| **Sharp Coder V0.1 is the first AI model released by Weinel Ventures.** |
|
|
| Sharp Coder V0.1 is an experimental **160,977,408-parameter causal language model** created during Weinel Ventures' early AI model research. |
|
|
| The model was trained primarily as an experiment in **C#, .NET, programming, technical text, and general language-model training**. |
|
|
| > **This model is published primarily for testing, research, archival, and educational purposes.** |
| > |
| > It may generate recognizable code or text in some situations, but its output is highly unreliable. It is not recommended for production use or important tasks. |
|
|
| Sharp Coder V0.1 is a **base pretrained model**. It is **not instruction-tuned and not chat-tuned**. |
|
|
| You can also access our **Official website here** at [Weinel Ventures AI](https://ai.weinelventures.com/), and the **Official model page here:** [Sharp Coder V0.1](https://ai.weinelventures.com/models/sharp-coder-v0.1) |
|
|
| --- |
|
|
| ## Model Details |
|
|
| - **Developer:** Weinel Ventures, led by Spencer Weinel |
| - **Model:** Sharp Coder V0.1 |
| - **Series:** V0 Experimental Series |
| - **Parameters:** 160,977,408 |
| - **Architecture:** GPT-2-compatible causal Transformer |
| - **Transformer layers:** 20 |
| - **Hidden size:** 768 |
| - **Attention heads:** 12 |
| - **Feed-forward size:** 3,072 |
| - **Vocabulary size:** 24,000 |
| - **Maximum context length:** 1,024 tokens |
| - **Model type:** Base causal language model |
| - **Instruction tuned:** No |
| - **Chat tuned:** No |
|
|
| --- |
|
|
| ## What Can It Do? |
|
|
| Sharp Coder V0.1 is unreliable, but testing shows that it has learned recognizable patterns from its training data. |
|
|
| Depending on the prompt, it may be able to: |
|
|
| - Continue short pieces of C# code |
| - Generate C#-like syntax |
| - Produce classes, methods, and test-like structures |
| - Reproduce some common .NET programming patterns |
| - Continue technical documentation-style text |
| - Generate Markdown and technical prose |
| - Continue general English text |
| - Demonstrate patterns learned during small-model pretraining |
| - Serve as a checkpoint for inference and conversion experiments |
| - Serve as a starting point for further training or fine-tuning experiments |
|
|
| The model has demonstrated recognition of patterns including: |
|
|
| - C# classes and methods |
| - `Console.WriteLine` |
| - Unit-test-style structures |
| - Compiler and diagnostic testing patterns |
| - Documentation formatting |
| - Markdown |
| - General technical prose |
|
|
| However, recognizing these patterns does **not** mean the model reliably understands or correctly completes programming tasks. |
|
|
| --- |
|
|
| ## Reliability |
|
|
| Sharp Coder V0.1 should be considered **highly unreliable**. |
|
|
| Observed behavior includes: |
|
|
| - Incorrect code |
| - Undefined variables |
| - Incomplete code |
| - Code that appears reasonable but does not compile |
| - Repetition loops |
| - Severe repetition of the same short phrases |
| - Nonsensical or gibberish output |
| - Sudden topic changes |
| - Mixing code with documentation |
| - Mixing unrelated training patterns |
| - Poor instruction following |
| - Poor conversational behavior |
| - Hallucinated text |
| - Output quality degrading during longer generations |
|
|
| In some cases, the model may enter repetition loops where the same word, phrase, or short token sequence is generated repeatedly. |
|
|
| A completion may begin with recognizable C# and later drift into: |
|
|
| - Unit tests |
| - Technical documentation |
| - Markdown |
| - HTML-like text |
| - Unrelated prose |
| - Repeated text |
| - Gibberish |
|
|
| Generated code should **never be assumed to be correct** without independent review and testing. |
|
|
| --- |
|
|
| ## Intended Use |
|
|
| Sharp Coder V0.1 is released primarily for: |
|
|
| - Language-model research |
| - Small-model experimentation |
| - Studying model failure modes |
| - Testing continued pretraining |
| - Fine-tuning experiments |
| - Comparing model checkpoints |
| - Testing Hugging Face Transformers inference |
| - Testing GGUF conversion |
| - Testing llama.cpp-compatible runtimes |
| - Local AI experimentation |
| - Educational purposes |
| - Preserving an early Weinel Ventures model-development checkpoint |
|
|
| ### Generation Use |
|
|
| No practical generation workload is currently recommended. |
|
|
| The model may still produce interesting or partially useful output during experimentation, particularly with short code-completion-style prompts, but results are inconsistent. |
|
|
| It should not be relied upon for: |
|
|
| - Production programming |
| - Security-sensitive code |
| - Factual answers |
| - Professional advice |
| - Important decision-making |
| - Autonomous software development |
| - Unsupervised code modification |
|
|
| --- |
|
|
| ## Base Model Behavior |
|
|
| Sharp Coder V0.1 is a **base language model**, not an AI assistant. |
|
|
| Its core task is predicting what text is likely to come next. |
|
|
| For example: |
|
|
| ```csharp |
| public static int Add(int a, int b) |
| { |
| return |
| ``` |
|
|
| The model attempts to predict text that could follow the prompt. |
|
|
| It does **not** inherently interpret the prompt as an instruction to correctly implement an addition function. |
|
|
| Similarly, conversational prompts such as: |
|
|
| ```text |
| What model are you? |
| ``` |
|
|
| should not be expected to produce reliable assistant-style responses. |
|
|
| The model was not trained using a chat or instruction-following format. |
|
|
| --- |
|
|
| ## Training |
|
|
| Sharp Coder V0.1 was trained from scratch as an experimental small language model. |
|
|
| The training corpus contained approximately **550 million tokens**. |
|
|
| Training material included a mixture of: |
|
|
| - C# source code |
| - .NET-related source code |
| - Programming documentation |
| - Technical text |
| - Programming-related discussion |
| - General English text |
|
|
| The model was trained with a maximum context length of **1,024 tokens**. |
|
|
| The best recorded validation checkpoint reached a validation loss of approximately: |
|
|
| ```text |
| 2.3805 |
| ``` |
|
|
| Sharp Coder V0.1 represents the selected checkpoint from this training run for public release. |
|
|
| --- |
|
|
| ## Architecture |
|
|
| Sharp Coder V0.1 uses a decoder-only causal Transformer architecture compatible with the Hugging Face GPT-2 implementation. |
|
|
| ```text |
| Vocabulary size: 24,000 |
| Context length: 1,024 |
| Hidden size: 768 |
| Transformer layers: 20 |
| Attention heads: 12 |
| FFN size: 3,072 |
| Parameters: 160,977,408 |
| ``` |
|
|
| The architecture includes: |
|
|
| - Learned token embeddings |
| - Learned positional embeddings |
| - Pre-layer normalization |
| - GELU activation |
| - Causal self-attention |
| - Tied input/output token embeddings |
|
|
| --- |
|
|
| ## Available Formats |
|
|
| This repository contains multiple formats of Sharp Coder V0.1. |
|
|
| ### Hugging Face / Transformers |
|
|
| The original exported model is available in Hugging Face-compatible format, including: |
|
|
| ```text |
| model.safetensors |
| config.json |
| tokenizer.json |
| tokenizer_config.json |
| ``` |
|
|
| This format is generally the better starting point for: |
|
|
| - Further training |
| - Fine-tuning |
| - Model inspection |
| - Research |
| - Transformers-based inference |
|
|
| ### GGUF |
|
|
| An F16 GGUF version is also provided: |
|
|
| ```text |
| Sharp-Coder-V0.1-F16.gguf |
| ``` |
|
|
| The GGUF version is intended for compatible software such as: |
|
|
| - llama.cpp |
| - LM Studio |
| - Other GGUF-compatible runtimes |
|
|
| The GGUF conversion was tested locally before release. |
|
|
| --- |
|
|
| ## Tokenizer |
|
|
| Sharp Coder V0.1 uses a custom **24,000-token Byte-Level BPE tokenizer**. |
|
|
| Special tokens include: |
|
|
| ```text |
| <|pad|> |
| <|unk|> |
| <|bos|> |
| <|eos|> |
| <|endoftext|> |
| ``` |
|
|
| The tokenizer distributed with the model should be used rather than substituting another GPT-2 tokenizer. |
|
|
| --- |
|
|
| ## Known Limitations |
|
|
| Sharp Coder V0.1 is an early experimental model with substantial limitations. |
|
|
| Its relatively small size and experimental training process mean that it does not provide the capabilities expected from modern large language models. |
|
|
| Known limitations include: |
|
|
| - Weak general reasoning |
| - Weak conversational ability |
| - Poor instruction following |
| - Limited context length |
| - Unreliable code generation |
| - Repetitive generations |
| - Topic drift |
| - Hallucination |
| - Gibberish generation |
| - Difficulty maintaining coherent long outputs |
|
|
| Sharp Coder V0.1 should not be interpreted as representing the expected quality of future Weinel Ventures models. |
|
|
| Some of these limitations are part of why the model is being published: V0.1 provides a public record of an early model-development experiment and a reference point for future research. |
|
|
| --- |
|
|
| ## V0 and Future Models |
|
|
| Sharp Coder V0.1 belongs to the **V0 experimental series**. |
|
|
| It will **not** be renamed or promoted into the V1 series. |
|
|
| The future **V1 series** is planned as a new generation of Weinel Ventures models using a different development approach. |
|
|
| One area of planned research is the creation of general English foundation models that can later be specialized through additional training. |
|
|
| Future experiments may explore models that first learn strong general English patterns and are then continued into areas such as: |
|
|
| - Programming |
| - Technical domains |
| - Specialized datasets |
| - Custom user-created models |
|
|
| Sharp Coder V0.1 serves as an experimental predecessor to that work. |
|
|
| --- |
|
|
| ## About This Release |
|
|
| Sharp Coder V0.1 is the **first publicly released AI model from Weinel Ventures**. |
|
|
| It represents the beginning of Weinel Ventures' public AI research and model-development work. |
|
|
| Future models may differ substantially in: |
|
|
| - Architecture |
| - Parameter count |
| - Tokenizer design |
| - Training data |
| - Training methodology |
| - Intended use |
| - General capability |
|
|
| --- |
|
|
| ## License |
|
|
| Licensing for Sharp Coder V0.1 is currently being reviewed because the training corpus contained material from multiple sources. |
|
|
| Until explicit license terms are published in this repository, users should **not assume** that the model is licensed under MIT, Apache-2.0, or another permissive open-source license. |
|
|
| Additional licensing and dataset-provenance information may be added as release documentation is completed. |
|
|
| --- |
|
|
| ## Developer |
|
|
| **Weinel Ventures, led by Spencer Weinel** |
|
|
| Sharp Coder V0.1 is the first AI model released by Weinel Ventures and represents the beginning of our public AI model research and development work. |
