File size: 3,407 Bytes
a75faed
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
"""Subword BIO alignment and decoding with original Unicode offsets."""
import json
from pathlib import Path

import torch

ROOT = Path(__file__).resolve().parent

def rows(name):
    return [json.loads(line) for line in (ROOT/"data"/(name+".jsonl")).read_text().splitlines()]

def encode(tokenizer, row, label_ids=None):
    data = tokenizer(row["text"], return_offsets_mapping=True, truncation=True, max_length=256,
                     return_overflowing_tokens=True, stride=0)
    gold = sorted(row.get("gold", []), key=lambda e:e["start"])
    index = 0
    seen = set()
    for ids, offsets in zip(data["input_ids"],data["offset_mapping"]):
        item = {"input_ids":ids,"offsets":offsets,"id":row["id"]}
        if label_ids is not None:
            labels = []
            for start, end in offsets:
                if start == end:
                    labels.append(-100)
                    continue
                while index < len(gold) and gold[index]["end"] <= start:
                    index += 1
                if index < len(gold) and gold[index]["start"] < end:
                    tag = ("I-" if index in seen else "B-")+gold[index]["label"]
                    seen.add(index)
                else:
                    tag = "O"
                labels.append(label_ids[tag])
            item["labels"] = labels
        yield item

def batch(items, pad, device):
    length = ((max(len(i["input_ids"]) for i in items)+7)//8)*8
    def field(key, fill):
        values = [i[key]+[fill]*(length-len(i[key])) for i in items]
        return torch.tensor(values,device=device)
    ids = field("input_ids",pad)
    inputs = {"input_ids":ids, "attention_mask":torch.tensor([
        [1]*len(i["input_ids"])+[0]*(length-len(i["input_ids"])) for i in items],device=device)}
    if "labels" in items[0]:
        inputs["labels"] = field("labels",-100)
    return inputs

@torch.inference_mode()
def predict(model, tokenizer, documents, batch_size=24):
    windows = [w for r in documents for w in encode(tokenizer,r)]
    groups = {r["id"]:[] for r in documents}
    previous = {}
    model.eval()
    for begin in range(0,len(windows),batch_size):
        items = windows[begin:begin+batch_size]
        logits = model(**batch(items,tokenizer.pad_token_id,model.device)).logits
        probs, indices = logits.float().softmax(dim=-1).max(dim=-1)
        for w, confidence, tags in zip(items,probs.cpu().tolist(),indices.cpu().tolist()):
            rid = w["id"]
            for (start,end),score,idx in zip(w["offsets"],confidence,tags):
                if start == end:
                    continue
                tag = model.config.id2label[idx]
                if tag == "O":
                    previous[rid] = None
                    continue
                boundary,label = tag.split("-",1)
                if boundary == "I" and previous.get(rid) == label:
                    group = groups[rid][-1]
                    group["end"] = end
                    group["scores"].append(score)
                else:
                    groups[rid].append({"start":start,"end":end,"label":label,"scores":[score]})
                previous[rid] = label
    return groups

def apply_threshold(groups, threshold):
    return {rid:[{k:g[k] for k in ("start","end","label")} for g in spans
                 if sum(g["scores"])/len(g["scores"]) >= threshold] for rid,spans in groups.items()}