Update README.md
Browse files
README.md
CHANGED
|
@@ -48,9 +48,11 @@ print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
|
| 48 |
|
| 49 |
## Benchmark Results
|
| 50 |
|
| 51 |
-
Benchmarked on a single NVIDIA A100 40GB GPU using:
|
| 52 |
-
- Transformers backend (FP16 and INT8 variants).
|
| 53 |
-
- vLLM engine
|
|
|
|
|
|
|
| 54 |
---
|
| 55 |
|
| 56 |
| Metric | Transformers (INT8) | Transformers (FP16) | vLLM (INT8) |
|
|
|
|
| 48 |
|
| 49 |
## Benchmark Results
|
| 50 |
|
| 51 |
+
- Benchmarked on a single NVIDIA A100 40GB GPU using:
|
| 52 |
+
- Transformers backend (FP16 and INT8 variants). Note: The current implementation of `transformers` does not include optimized kernels for direct INT8 computation.
|
| 53 |
+
- vLLM engine with INT8 inference acceleration for improved performance.
|
| 54 |
+
|
| 55 |
+
|
| 56 |
---
|
| 57 |
|
| 58 |
| Metric | Transformers (INT8) | Transformers (FP16) | vLLM (INT8) |
|