Instructions to use ayjays132/Phillnet-2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ayjays132/Phillnet-2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ayjays132/Phillnet-2", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ayjays132/Phillnet-2", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("ayjays132/Phillnet-2", trust_remote_code=True, device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ayjays132/Phillnet-2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ayjays132/Phillnet-2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ayjays132/Phillnet-2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ayjays132/Phillnet-2

SGLang

How to use ayjays132/Phillnet-2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ayjays132/Phillnet-2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ayjays132/Phillnet-2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ayjays132/Phillnet-2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ayjays132/Phillnet-2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ayjays132/Phillnet-2 with Docker Model Runner:
```
docker model run hf.co/ayjays132/Phillnet-2
```

Phillnet-2 / memory_optimization /config.py

ayjays132

Upload 478 files

101858b verified 6 months ago

Raw History Blame Contribute Delete

1.19 kB

"""
Memory Optimization Configuration
Configuration for the unified memory management system.
"""
import torch
from dataclasses import dataclass
from typing import Optional

@dataclass
class MemoryOptimizationConfig:
    """Configuration for memory optimization system."""
    # Device settings
    device: str = "cuda" if torch.cuda.is_available() else "cpu"
    
    # Memory thresholds
    memory_threshold: float = 0.85  # 85% memory usage threshold
    cleanup_threshold: float = 0.75  # 75% threshold for proactive cleanup
    
    # Tensor pool settings
    max_pool_size: int = 50  # Reduced for RTX 3060
    max_tensor_size: int = 1000000  # Don't pool very large tensors
    
    # Cleanup settings
    cleanup_frequency: int = 25  # Cleanup every 25 operations
    
    # Memory optimization flags
    use_4bit_quantization: bool = True
    use_gradient_checkpointing: bool = True
    use_mixed_precision: bool = True
    
    # Shared model integration
    use_shared_model: bool = True  # Use shared Qwen model
    shared_model_name: str = "Qwen/Qwen3-0.6B"  # Default shared model
    
    # Lazy loading
    enable_lazy_loading: bool = True

	"""
	Memory Optimization Configuration
	Configuration for the unified memory management system.
	"""
	import torch
	from dataclasses import dataclass
	from typing import Optional

	@dataclass
	class MemoryOptimizationConfig:
	"""Configuration for memory optimization system."""
	# Device settings
	device: str = "cuda" if torch.cuda.is_available() else "cpu"

	# Memory thresholds
	memory_threshold: float = 0.85 # 85% memory usage threshold
	cleanup_threshold: float = 0.75 # 75% threshold for proactive cleanup

	# Tensor pool settings
	max_pool_size: int = 50 # Reduced for RTX 3060
	max_tensor_size: int = 1000000 # Don't pool very large tensors

	# Cleanup settings
	cleanup_frequency: int = 25 # Cleanup every 25 operations

	# Memory optimization flags
	use_4bit_quantization: bool = True
	use_gradient_checkpointing: bool = True
	use_mixed_precision: bool = True

	# Shared model integration
	use_shared_model: bool = True # Use shared Qwen model
	shared_model_name: str = "Qwen/Qwen3-0.6B" # Default shared model

	# Lazy loading
	enable_lazy_loading: bool = True