Buckets:
| import sys, numpy as np, pandas as pd | |
| sys.path.insert(0, "/home/user/RestoreKV") | |
| from prgf.evaluate import load_ruler | |
| exec(open("pertask_lb.py").read().split("v4 = ")[0]) | |
| split, a, b = sys.argv[1], sys.argv[2], sys.argv[3] | |
| df = load_ruler(split) | |
| S = {} | |
| for r in (a, b): | |
| n, p = r.split("=") | |
| S[n] = per_sample(pd.read_json(p, lines=True).set_index("index").predicted_answer.loc[df.index].values, df.answer.values, df.task.values) | |
| S = pd.DataFrame(S); S["task"] = df.task.values | |
| t = S.groupby("task").mean().round(1); t.loc["AVG"] = t.mean().round(2); print(t.to_string()) | |
| na, nb = list(S.columns[:2]); d = S[na].values - S[nb].values | |
| idx = [np.where(S.task.values == k)[0] for k in S.task.unique()] | |
| rng = np.random.default_rng(0) | |
| boots = [np.mean([d[rng.choice(v, len(v))].mean() for v in idx]) for _ in range(5000)] | |
| print(f"{na} - {nb}: {np.mean([d[v].mean() for v in idx]):+.2f} [{np.percentile(boots, 2.5):+.2f}, {np.percentile(boots, 97.5):+.2f}]") | |
Xet Storage Details
- Size:
- 959 Bytes
- Xet hash:
- f6c17ca6a334c956fe5c93dd6dbec40c027d017d47bc52067e7337186fda6170
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.