Spaces:

whassan
/

SensorflowStudio

Paused

SensorflowStudio / sensorflow /retro /inference /vllm_server /model.env

Deploy Sensorflow Studio

6b3874e verified 3 months ago

267 Bytes

# vLLM model config.
MODEL_NAME=${MODEL_NAME:-meta-llama/Llama-3.1-8B-Instruct}
MAX_MODEL_LEN=${MAX_MODEL_LEN:-8192}
# QUANTIZATION: none | awq | gptq | fp8  (support differs by GPU vendor;
# see docs/retro/compatibility-matrix.md)
QUANTIZATION=${QUANTIZATION:-none}

	# vLLM model config.
	MODEL_NAME=${MODEL_NAME:-meta-llama/Llama-3.1-8B-Instruct}
	MAX_MODEL_LEN=${MAX_MODEL_LEN:-8192}
	# QUANTIZATION: none \| awq \| gptq \| fp8 (support differs by GPU vendor;
	# see docs/retro/compatibility-matrix.md)
	QUANTIZATION=${QUANTIZATION:-none}