Instructions to use Frankenstein-Labs/cortex.6.sol with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Frankenstein-Labs/cortex.6.sol with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Frankenstein-Labs/cortex.6.sol")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Frankenstein-Labs/cortex.6.sol")
model = AutoModelForCausalLM.from_pretrained("Frankenstein-Labs/cortex.6.sol", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Frankenstein-Labs/cortex.6.sol with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Frankenstein-Labs/cortex.6.sol"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Frankenstein-Labs/cortex.6.sol",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/Frankenstein-Labs/cortex.6.sol

SGLang

How to use Frankenstein-Labs/cortex.6.sol with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Frankenstein-Labs/cortex.6.sol" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Frankenstein-Labs/cortex.6.sol",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Frankenstein-Labs/cortex.6.sol" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Frankenstein-Labs/cortex.6.sol",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use Frankenstein-Labs/cortex.6.sol with Docker Model Runner:
```
docker model run hf.co/Frankenstein-Labs/cortex.6.sol
```

cortex.6.sol / cortex_ai /adapters /base.py

asdfasdfqrqwer

feat(cortex-ai): agent engine, tool registry, OpenAI-compatible API, fine-tuning pipeline

c63bc31 about 1 month ago

Raw History Blame Contribute Delete

1.27 kB

"""Model adapters: a single interface over different inference backends.
The engine never talks to a model directly. It talks to a ``ModelAdapter``.
That keeps the agent loop testable without a GPU and lets the same code drive
a mock, a small local model, or the full 1.65T checkpoint.
"""

from __future__ import annotations

from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import Any


@dataclass
class GenerationResult:
    """Raw text produced by a model, plus usage counters."""

    text: str
    prompt_tokens: int = 0
    completion_tokens: int = 0


class ModelAdapter(ABC):
    """Interface every backend must implement."""

    #: Identifier reported by the API, e.g. ``Frankenstein-Labs/Cortex-ai``.
    model_id: str = "unknown"

    @abstractmethod
    def generate(
        self,
        prompt: str,
        *,
        max_new_tokens: int = 512,
        temperature: float = 0.6,
    ) -> GenerationResult:
        """Run one completion over an already-encoded prompt."""

    def count_tokens(self, text: str) -> int:
        """Approximate token count. Backends with a tokenizer should override."""
        return len(text.split())

    def close(self) -> None:
        """Release resources. Default: nothing to do."""

	"""Model adapters: a single interface over different inference backends.

	The engine never talks to a model directly. It talks to a ``ModelAdapter``.
	That keeps the agent loop testable without a GPU and lets the same code drive
	a mock, a small local model, or the full 1.65T checkpoint.
	"""

	from __future__ import annotations

	from abc import ABC, abstractmethod
	from dataclasses import dataclass
	from typing import Any


	@dataclass
	class GenerationResult:
	"""Raw text produced by a model, plus usage counters."""

	text: str
	prompt_tokens: int = 0
	completion_tokens: int = 0


	class ModelAdapter(ABC):
	"""Interface every backend must implement."""

	#: Identifier reported by the API, e.g. ``Frankenstein-Labs/Cortex-ai``.
	model_id: str = "unknown"

	@abstractmethod
	def generate(
	self,
	prompt: str,
	*,
	max_new_tokens: int = 512,
	temperature: float = 0.6,
	) -> GenerationResult:
	"""Run one completion over an already-encoded prompt."""

	def count_tokens(self, text: str) -> int:
	"""Approximate token count. Backends with a tokenizer should override."""
	return len(text.split())

	def close(self) -> None:
	"""Release resources. Default: nothing to do."""