Instructions to use InfosysEnterprise/NT-Java-1.1B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use InfosysEnterprise/NT-Java-1.1B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="InfosysEnterprise/NT-Java-1.1B")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("InfosysEnterprise/NT-Java-1.1B")
model = AutoModelForCausalLM.from_pretrained("InfosysEnterprise/NT-Java-1.1B", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use InfosysEnterprise/NT-Java-1.1B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "InfosysEnterprise/NT-Java-1.1B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "InfosysEnterprise/NT-Java-1.1B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/InfosysEnterprise/NT-Java-1.1B

SGLang

How to use InfosysEnterprise/NT-Java-1.1B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "InfosysEnterprise/NT-Java-1.1B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "InfosysEnterprise/NT-Java-1.1B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "InfosysEnterprise/NT-Java-1.1B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "InfosysEnterprise/NT-Java-1.1B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use InfosysEnterprise/NT-Java-1.1B with Docker Model Runner:
```
docker model run hf.co/InfosysEnterprise/NT-Java-1.1B
```

rajabmondal commited on May 9, 2024

Commit

1d960c0

verified ·

1 Parent(s): da27032

updated intended use

Browse files

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -38,7 +38,7 @@ The Narrow Transformer (NT) model NT-Java-1.1B is an open-source specialized cod
 ### Intended use
-The model was trained on GitHub code. As such it is _not_ an instruction model and commands like "Write a function that computes the square root." do not work well. However, by using the [Tech Assistant prompt](https://huggingface.co/datasets/bigcode/ta-prompt) you can turn it into a capable technical assistant.
 **Feel free to share your generations in the Community tab!**

 ### Intended use
+Large code models require specialized hardware like GPUs for inference, highlighting the need for research into building small code models that can be deployed on developer desktops. This model addresses the gap by focusing on the development of a small Java code model and introducing a quantized version of NT-Java-1.1B, which performs comparably to open 1.1B models on MultiPL-E Java code benchmarks, making it ideal for desktop deployment.
 **Feel free to share your generations in the Community tab!**