|
Download README.md from fxmarty/donotdelete2: direct link, hf CLI and curl.
- Browser
- Download file 2.81 kB
-
https://huggingface.co/fxmarty/donotdelete2/resolve/main/README.md
- Command line
-
hf download hf://fxmarty/donotdelete2/README.md
-
curl -L -o README.md https://huggingface.co/fxmarty/donotdelete2/resolve/main/README.md
2.81 kB
| pipeline_tag: question-answering | |
| datasets: | |
| - squad | |
| metrics: | |
| - exact_match | |
| - f1 | |
| tags: | |
| - distilbert | |
| **task**: `question-answering` | |
| Fixed parameters: | |
| * **model_name_or_path**: `distilbert-base-uncased-distilled-squad` | |
| * **dataset**: | |
| * **path**: `squad` | |
| * **name**: `None` | |
| * **calibration_split**: `train` | |
| * **eval_split**: `validation` | |
| * **data_keys**: `{'question': 'question', 'context': 'context'}` | |
| * **ref_keys**: `['answers']` | |
| * **max_seq_length**: `128` | |
| * **node_exclusion**: `[]` | |
| * **per_channel**: `False` | |
| * **calibration**: | |
| * **method**: `minmax` | |
| * **num_calibration_samples**: `20` | |
| * **calibration_histogram_percentile**: `None` | |
| * **calibration_moving_average**: `None` | |
| * **calibration_moving_average_constant**: `None` | |
| * **framework**: `onnxruntime` | |
| * **framework_args**: | |
| * **opset**: `15` | |
| * **optimization_level**: `1` | |
| * **aware_training**: `False` | |
| Benchmarked parameters: | |
| * **quantization_approach**: `dynamic`, `static` | |
| * **operators_to_quantize**: `['Add', 'MatMul']`, `['Add']` | |
| ## Evaluation | |
| Below, time metrics for | |
| * Batch size: 8 | |
| * Input length: 128 | |
| | quantization_approach | operators_to_quantize | | latency_mean (original, ms) | latency_mean (optimized, ms) | | throughput (original, /s) | throughput (optimized, /s) | | exact_match (original) | exact_match (optimized) | | f1 (original) | f1 (optimized) | | |
| | :-------------------: | :-------------------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: | :-: | :--------------------: | :---------------------: | :-: | :-----------: | :------------: | | |
| | `dynamic` | `['Add', 'MatMul']` | \| | 247.65 | 75.80 | \| | 4.50 | 13.50 | \| | 90.000 | 85.000 | \| | 90.000 | 86.667 | | |
| | `dynamic` | `['Add']` | \| | 234.61 | 191.78 | \| | 4.50 | 5.50 | \| | 90.000 | 90.000 | \| | 90.000 | 90.000 | | |
| | `static` | `['Add', 'MatMul']` | \| | 238.04 | 131.81 | \| | 4.50 | 8.00 | \| | 90.000 | 50.000 | \| | 90.000 | 59.362 | | |
| | `static` | `['Add']` | \| | 241.84 | 257.85 | \| | 4.50 | 4.00 | \| | 90.000 | 70.000 | \| | 90.000 | 76.333 | | |