Instructions to use patdev/k3-a40-bootstrap with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use patdev/k3-a40-bootstrap with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf patdev/k3-a40-bootstrap:BF16 # Run inference directly in the terminal: llama cli -hf patdev/k3-a40-bootstrap:BF16
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf patdev/k3-a40-bootstrap:BF16 # Run inference directly in the terminal: llama cli -hf patdev/k3-a40-bootstrap:BF16
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf patdev/k3-a40-bootstrap:BF16 # Run inference directly in the terminal: ./llama-cli -hf patdev/k3-a40-bootstrap:BF16
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf patdev/k3-a40-bootstrap:BF16 # Run inference directly in the terminal: ./build/bin/llama-cli -hf patdev/k3-a40-bootstrap:BF16
Use Docker
docker model run hf.co/patdev/k3-a40-bootstrap:BF16
- LM Studio
- Jan
- Ollama
How to use patdev/k3-a40-bootstrap with Ollama:
ollama run hf.co/patdev/k3-a40-bootstrap:BF16
- Unsloth Desktop
- Docker Model Runner
How to use patdev/k3-a40-bootstrap with Docker Model Runner:
docker model run hf.co/patdev/k3-a40-bootstrap:BF16
- Lemonade
How to use patdev/k3-a40-bootstrap with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull patdev/k3-a40-bootstrap:BF16
Run and chat with the model
lemonade run user.k3-a40-bootstrap-BF16
List all available models
lemonade list
- Atomic Chat
Download connect.ps1 from patdev/k3-a40-bootstrap: direct link, hf CLI and curl.
- Browser
- Download file 4.51 kB
-
https://huggingface.co/patdev/k3-a40-bootstrap/resolve/main/connect.ps1
- Command line
-
hf download hf://patdev/k3-a40-bootstrap/connect.ps1
-
curl -L -o connect.ps1 https://huggingface.co/patdev/k3-a40-bootstrap/resolve/main/connect.ps1
4.51 kB
| <# | |
| .SYNOPSIS | |
| Branche Claude Code sur l'endpoint auto-heberge, apres verification. | |
| .DESCRIPTION | |
| Les variables sont posees dans la session PowerShell courante : elles | |
| disparaissent a sa fermeture, et n'affectent aucune autre fenetre. C'est | |
| voulu -- on veut pouvoir revenir a l'API officielle en fermant l'onglet. | |
| Le script VERIFIE l'endpoint avant de lancer quoi que ce soit. Sans ce | |
| controle, un pod arrete ou en cours de rechargement donne un Claude Code qui | |
| demarre puis echoue au premier message, ce qui se diagnostique mal. | |
| .EXAMPLE | |
| .\connect.ps1 | |
| .\connect.ps1 -NoLaunch # pose les variables sans demarrer Claude Code | |
| .\connect.ps1 -Model claude-kimi-k3 | |
| #> | |
| [CmdletBinding()] | |
| param( | |
| [string] $BaseUrl = "https://y26flhzhufrkb3-8080.proxy.runpod.net", | |
| [string] $Model = "claude-ornith[1m]", | |
| [switch] $NoLaunch | |
| ) | |
| $ErrorActionPreference = "Stop" | |
| Write-Host "Verification de $BaseUrl ..." -ForegroundColor Cyan | |
| try { | |
| $models = Invoke-RestMethod -Uri "$BaseUrl/v1/models" -TimeoutSec 30 ` | |
| -Headers @{ "anthropic-version" = "2023-06-01" } | |
| } catch { | |
| Write-Host "Endpoint injoignable : $($_.Exception.Message)" -ForegroundColor Red | |
| Write-Host "Le pod est peut-etre arrete, ou en cours de rechargement." -ForegroundColor Yellow | |
| exit 1 | |
| } | |
| $ids = $models.data | ForEach-Object { $_.id } | |
| if ($ids -notcontains $Model) { | |
| Write-Host "Modele '$Model' absent. Disponibles : $($ids -join ', ')" -ForegroundColor Red | |
| exit 1 | |
| } | |
| $ctx = ($models.data | Where-Object { $_.id -eq $Model }).context_window | |
| Write-Host " OK - $Model, contexte $('{0:N0}' -f $ctx) jetons" -ForegroundColor Green | |
| # ANTHROPIC_API_KEY, s'il traine, prend le pas sur AUTH_TOKEN et fait partir les | |
| # requetes vers l'API officielle : on obtient un vrai Claude en croyant tester | |
| # le modele local, et la facture arrive avec. | |
| if ($env:ANTHROPIC_API_KEY) { | |
| Write-Host " ANTHROPIC_API_KEY efface pour cette session (il aurait pris le dessus)" -ForegroundColor Yellow | |
| Remove-Item Env:\ANTHROPIC_API_KEY | |
| } | |
| $env:ANTHROPIC_BASE_URL = $BaseUrl | |
| $env:ANTHROPIC_AUTH_TOKEN = "x" # le pont n'authentifie pas, le champ doit exister | |
| $env:ANTHROPIC_MODEL = $Model | |
| # Fenetre de contexte. Deux reglages DOCUMENTES, l'un ne suffit pas sans l'autre. | |
| # | |
| # 1. Claude Code suppose 200 k pour tout identifiant qu'il ne reconnait pas, et | |
| # declenche l'auto-compactage a cette limite meme si le moteur accepte un | |
| # million. DISABLE_UNKNOWN_MODEL_WINDOW_ENFORCEMENT leve cette presomption : | |
| # il ne compacte alors qu'apres un refus explicite de l'API. | |
| # 2. AUTO_COMPACT_WINDOW fixe la fenetre reellement utilisee. La doc precise que | |
| # Claude Code la PLAFONNE a la fenetre supposee du modele -- d'ou l'inutilite | |
| # de la poser sans le point 1. | |
| $env:CLAUDE_CODE_DISABLE_UNKNOWN_MODEL_WINDOW_ENFORCEMENT = "1" | |
| $env:CLAUDE_CODE_AUTO_COMPACT_WINDOW = "1000000" | |
| # Ajoute l'entree au selecteur /model SANS validation d'identifiant. Inutile si | |
| # CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=1 est deja pose (le selecteur lit | |
| # alors /v1/models du pont), mais sans effet nefaste s'il l'est. | |
| $env:ANTHROPIC_CUSTOM_MODEL_OPTION = $Model | |
| $env:ANTHROPIC_CUSTOM_MODEL_OPTION_NAME = "Ornith-1.5-35B-A3B (pod A40 x2)" | |
| $env:ANTHROPIC_CUSTOM_MODEL_OPTION_DESCRIPTION = "Auto-heberge, contexte 1M, 2,82 sessions simultanees" | |
| # Claude Code appelle un modele "rapide" pour les taches secondaires (titres, | |
| # resumes). Sans ces alias il reclamerait un Haiku inexistant ici. Le pont les | |
| # route tous vers l'unique modele charge. | |
| $env:ANTHROPIC_SMALL_FAST_MODEL = "claude-3-5-haiku" | |
| $env:ANTHROPIC_DEFAULT_HAIKU_MODEL = "claude-3-5-haiku" | |
| $env:ANTHROPIC_DEFAULT_SONNET_MODEL = $Model | |
| $env:ANTHROPIC_DEFAULT_OPUS_MODEL = $Model | |
| Write-Host "" | |
| Write-Host "Variables posees pour CETTE session :" -ForegroundColor Cyan | |
| "ANTHROPIC_BASE_URL","ANTHROPIC_MODEL","CLAUDE_CODE_AUTO_COMPACT_WINDOW", | |
| "CLAUDE_CODE_DISABLE_UNKNOWN_MODEL_WINDOW_ENFORCEMENT" | | |
| ForEach-Object { " {0,-28} {1}" -f $_, (Get-Item "Env:\$_").Value } | |
| Write-Host "" | |
| Write-Host "N'utilisez PAS --permission-mode acceptEdits pour l'instant." -ForegroundColor Yellow | |
| Write-Host "Le garde-fou du pont bloque les ecritures abregees, mais il ne couvre" -ForegroundColor Yellow | |
| Write-Host "que ce mode de defaillance. Le modele n'a ete valide que sur un fichier." -ForegroundColor Yellow | |
| Write-Host "" | |
| if (-not $NoLaunch) { | |
| Write-Host "Lancement de Claude Code..." -ForegroundColor Cyan | |
| claude | |
| } | |