updated quantization

README.md

@@ -82,6 +82,24 @@ inputs = tokenizer.encode("public class HelloWorld {\n    public static void mai
 outputs = model.generate(inputs)
 print(tokenizer.decode(outputs[0]))
 ```
+#### Quantized Versions through `bitsandbytes`
+* _Using 8-bit precision (int8)_
+
+```java
+# pip install bitsandbytes accelerate
+from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
+
+# to use 4bit use `load_in_4bit=True` instead
+quantization_config = BitsAndBytesConfig(load_in_8bit=True)
+
+checkpoint = "infosys/NT-Java-1.1B"
+tokenizer = AutoTokenizer.from_pretrained(checkpoint)
+model = AutoModelForCausalLM.from_pretrained(checkpoint, quantization_config=quantization_config)
+
+inputs = tokenizer.encode("public class HelloWorld {\n    public static void main(String[] args) {", return_tensors="pt").to("cuda")
+outputs = model.generate(inputs)
+print(tokenizer.decode(outputs[0]))
+```
 
 ### Attribution & Other Requirements