Mingyin0312/Genome-Bench
Viewer • Updated • 3.33k • 83 • 8
Ilya
franchb
·
AI & ML interests
None yet
Recent Activity
liked a dataset 4 days ago
Mingyin0312/Genome-Bench liked a model 8 months ago
zai-org/GLM-OCR upvoted a paper 10 months ago
Souper-Model: How Simple Arithmetic Unlocks State-of-the-Art LLM PerformanceOrganizations
None yet
liked a
dataset 4 days ago
liked a
model 8 months ago
upvoted a paper 10 months ago
liked a
model 11 months ago
updated a
Space 12 months ago
published a
Space 12 months ago
reacted to codelion's post with 🚀 over 1 year ago
Post
2064
New Research: Theoretical Foundations for In-Context Learning in Transformers
I'm excited to share our latest theoretical work that formally proves an interesting property of large language models: base transformer models can approximate fine-tuned capabilities using only inference-time techniques like in-context learning.
The core question we investigated: Can specialized behaviors typically acquired through expensive supervised fine-tuning be elicited from base models without any parameter updates?
Our theoretical contribution: We provide a formal proof, grounded in the Turing completeness of transformers, showing that this is indeed possible under certain assumptions. The work establishes mathematical bounds on the minimal dataset sizes needed for approximation.
Key theoretical results:
- For text generation tasks: O(mV/ε²) examples suffice (where m = number of contexts, V = vocabulary size, ε = error tolerance)
- For linear classification: O(d/ε) examples (where d = input dimension)
- Extensions to finite context scenarios with practical bounds
This work helps explain why techniques like few-shot prompting, retrieval-augmented generation, and in-context learning work so effectively in practice. It bridges formal computer science theory with empirical observations about modern language models.
While the assumptions are idealized (unbounded computational resources, full dataset access), the results provide mathematical foundations for understanding inference-time adaptation strategies that are increasingly important in AI deployment.
Paper: Eliciting Fine-Tuned Transformer Capabilities via Inference-Time Techniques (2506.08060)
I'm excited to share our latest theoretical work that formally proves an interesting property of large language models: base transformer models can approximate fine-tuned capabilities using only inference-time techniques like in-context learning.
The core question we investigated: Can specialized behaviors typically acquired through expensive supervised fine-tuning be elicited from base models without any parameter updates?
Our theoretical contribution: We provide a formal proof, grounded in the Turing completeness of transformers, showing that this is indeed possible under certain assumptions. The work establishes mathematical bounds on the minimal dataset sizes needed for approximation.
Key theoretical results:
- For text generation tasks: O(mV/ε²) examples suffice (where m = number of contexts, V = vocabulary size, ε = error tolerance)
- For linear classification: O(d/ε) examples (where d = input dimension)
- Extensions to finite context scenarios with practical bounds
This work helps explain why techniques like few-shot prompting, retrieval-augmented generation, and in-context learning work so effectively in practice. It bridges formal computer science theory with empirical observations about modern language models.
While the assumptions are idealized (unbounded computational resources, full dataset access), the results provide mathematical foundations for understanding inference-time adaptation strategies that are increasingly important in AI deployment.
Paper: Eliciting Fine-Tuned Transformer Capabilities via Inference-Time Techniques (2506.08060)
- 1 reply
liked a
model over 1 year ago
liked 4
models over 2 years ago
reacted to zolicsaki's post with 🚀❤️ over 2 years ago
Post
2817
We posted new SOTA SambaLingo 70B parameter models for Arabic, Thai and Hungarian!
Check out the models here sambanovasystems/sambalingo-65e25770f2037c85ad35ca77
and our paper
https://arxiv.org/abs/2404.05829
Check out the models here sambanovasystems/sambalingo-65e25770f2037c85ad35ca77
and our paper
https://arxiv.org/abs/2404.05829
liked 2
models over 2 years ago
liked a
model almost 3 years ago
liked a
model over 3 years ago
