Instructions to use nextai-team/Moe-2x7b-QA-Code with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nextai-team/Moe-2x7b-QA-Code with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="nextai-team/Moe-2x7b-QA-Code")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("nextai-team/Moe-2x7b-QA-Code")
model = AutoModelForCausalLM.from_pretrained("nextai-team/Moe-2x7b-QA-Code", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use nextai-team/Moe-2x7b-QA-Code with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nextai-team/Moe-2x7b-QA-Code"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nextai-team/Moe-2x7b-QA-Code",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/nextai-team/Moe-2x7b-QA-Code

SGLang

How to use nextai-team/Moe-2x7b-QA-Code with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nextai-team/Moe-2x7b-QA-Code" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nextai-team/Moe-2x7b-QA-Code",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nextai-team/Moe-2x7b-QA-Code" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nextai-team/Moe-2x7b-QA-Code",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use nextai-team/Moe-2x7b-QA-Code with Docker Model Runner:
```
docker model run hf.co/nextai-team/Moe-2x7b-QA-Code
```

nextai-team commited on Feb 9, 2024

Commit

3a0e6d4

verified ·

1 Parent(s): 760e137

Update README.md

Browse files

Files changed (1) hide show

README.md +14 -5

README.md CHANGED Viewed

@@ -44,7 +44,17 @@ response = generate_resposne("How to learn coding .Please provide a step by step
 print(response)
 ```
-***Limitations and Bias***
 This model, like any other, has its limitations. It may exhibit biases inherent in the training data or struggle with questions outside its training scope. Users should critically assess the model's outputs, especially for sensitive or critical applications.
@@ -52,15 +62,14 @@ Training Data:
 The Moe-2x7b-QA-Code model was trained on a curated dataset comprising technical documentation, Stack Overflow posts, GitHub repositories, and other code-related content. This extensive training set ensures the model's proficiency in understanding and generating code-related content alongside general language understanding.
-***Training Procedure***
 The model was trained using a Mixture of Experts (MoE) approach, allowing it to dynamically leverage different subsets of parameters for different types of input data. This method enhances the model's capacity and efficiency, enabling it to excel in a wide range of QA and coding tasks.
-***Model Architecture**
 Moe-2x7b-QA-Code employs an advanced MoE architecture with 2x7 billion parameters, optimized for high performance in QA and coding tasks. This architecture enables the model to efficiently process and generate accurate responses to complex queries.
-***Contact**
 Https://nextai.co.in

 print(response)
 ```
+**Intended Use**
+This model is intended for developers, data scientists, and researchers seeking to integrate sophisticated natural language understanding and code generation functionalities into their applications. Ideal use cases include but are not limited to:
+Automated coding assistance Technical support bots Educational tools for learning programming Enhancing code review processes
+Model Architecture employs a Mixture of Experts (MoE) architecture, which allows it to efficiently manage its vast number of parameters for specialized tasks. This architecture facilitates the model's ability to discern subtle nuances in programming languages and natural language queries, leading to more accurate code generation and question answering performance.
+Performance demonstrates significant improvements in accuracy and relevance over its predecessor, particularly in complex coding scenarios and detailed technical queries. ***Benchmarks and performance metrics can be provided upon request.***
+**Limitations and Bias**
 This model, like any other, has its limitations. It may exhibit biases inherent in the training data or struggle with questions outside its training scope. Users should critically assess the model's outputs, especially for sensitive or critical applications.
 The Moe-2x7b-QA-Code model was trained on a curated dataset comprising technical documentation, Stack Overflow posts, GitHub repositories, and other code-related content. This extensive training set ensures the model's proficiency in understanding and generating code-related content alongside general language understanding.
+**Training Procedure**
 The model was trained using a Mixture of Experts (MoE) approach, allowing it to dynamically leverage different subsets of parameters for different types of input data. This method enhances the model's capacity and efficiency, enabling it to excel in a wide range of QA and coding tasks.
+**Model Architecture**
 Moe-2x7b-QA-Code employs an advanced MoE architecture with 2x7 billion parameters, optimized for high performance in QA and coding tasks. This architecture enables the model to efficiently process and generate accurate responses to complex queries.
+**Contact**
 Https://nextai.co.in