rafmacalaba commited on
Commit
80ad386
·
verified ·
1 Parent(s): a04db1c

Update model card (README.md) with usage instructions

Browse files
Files changed (1) hide show
  1. README.md +179 -181
README.md CHANGED
@@ -1,206 +1,204 @@
1
  ---
 
 
2
  base_model: fastino/gliner2-large-v1
3
- library_name: peft
4
  tags:
5
- - base_model:adapter:fastino/gliner2-large-v1
6
- - lora
7
- - transformers
 
 
 
 
8
  ---
9
 
10
- # Model Card for Model ID
11
 
12
- <!-- Provide a quick summary of what the model is/does. -->
13
 
 
14
 
 
15
 
16
- ## Model Details
17
-
18
- ### Model Description
19
-
20
- <!-- Provide a longer summary of what this model is. -->
21
-
22
-
23
-
24
- - **Developed by:** [More Information Needed]
25
- - **Funded by [optional]:** [More Information Needed]
26
- - **Shared by [optional]:** [More Information Needed]
27
- - **Model type:** [More Information Needed]
28
- - **Language(s) (NLP):** [More Information Needed]
29
- - **License:** [More Information Needed]
30
- - **Finetuned from model [optional]:** [More Information Needed]
31
-
32
- ### Model Sources [optional]
33
-
34
- <!-- Provide the basic links for the model. -->
35
-
36
- - **Repository:** [More Information Needed]
37
- - **Paper [optional]:** [More Information Needed]
38
- - **Demo [optional]:** [More Information Needed]
39
-
40
- ## Uses
41
-
42
- <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
43
-
44
- ### Direct Use
45
-
46
- <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
47
-
48
- [More Information Needed]
49
-
50
- ### Downstream Use [optional]
51
-
52
- <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
53
-
54
- [More Information Needed]
55
-
56
- ### Out-of-Scope Use
57
-
58
- <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
59
-
60
- [More Information Needed]
61
-
62
- ## Bias, Risks, and Limitations
63
-
64
- <!-- This section is meant to convey both technical and sociotechnical limitations. -->
65
-
66
- [More Information Needed]
67
-
68
- ### Recommendations
69
-
70
- <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
71
-
72
- Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
73
-
74
- ## How to Get Started with the Model
75
-
76
- Use the code below to get started with the model.
77
-
78
- [More Information Needed]
79
-
80
- ## Training Details
81
-
82
- ### Training Data
83
-
84
- <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
85
-
86
- [More Information Needed]
87
-
88
- ### Training Procedure
89
-
90
- <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
91
-
92
- #### Preprocessing [optional]
93
-
94
- [More Information Needed]
95
-
96
-
97
- #### Training Hyperparameters
98
-
99
- - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
100
-
101
- #### Speeds, Sizes, Times [optional]
102
-
103
- <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
104
-
105
- [More Information Needed]
106
-
107
- ## Evaluation
108
-
109
- <!-- This section describes the evaluation protocols and provides the results. -->
110
-
111
- ### Testing Data, Factors & Metrics
112
-
113
- #### Testing Data
114
-
115
- <!-- This should link to a Dataset Card if possible. -->
116
-
117
- [More Information Needed]
118
-
119
- #### Factors
120
-
121
- <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
122
-
123
- [More Information Needed]
124
-
125
- #### Metrics
126
-
127
- <!-- These are the evaluation metrics being used, ideally with a description of why. -->
128
-
129
- [More Information Needed]
130
-
131
- ### Results
132
-
133
- [More Information Needed]
134
-
135
- #### Summary
136
-
137
-
138
-
139
- ## Model Examination [optional]
140
-
141
- <!-- Relevant interpretability work for the model goes here -->
142
-
143
- [More Information Needed]
144
-
145
- ## Environmental Impact
146
-
147
- <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
148
-
149
- Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
150
-
151
- - **Hardware Type:** [More Information Needed]
152
- - **Hours used:** [More Information Needed]
153
- - **Cloud Provider:** [More Information Needed]
154
- - **Compute Region:** [More Information Needed]
155
- - **Carbon Emitted:** [More Information Needed]
156
-
157
- ## Technical Specifications [optional]
158
-
159
- ### Model Architecture and Objective
160
-
161
- [More Information Needed]
162
-
163
- ### Compute Infrastructure
164
-
165
- [More Information Needed]
166
-
167
- #### Hardware
168
-
169
- [More Information Needed]
170
-
171
- #### Software
172
-
173
- [More Information Needed]
174
 
175
- ## Citation [optional]
176
 
177
- <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
178
 
179
- **BibTeX:**
 
 
180
 
181
- [More Information Needed]
182
 
183
- **APA:**
 
184
 
185
- [More Information Needed]
 
 
 
186
 
187
- ## Glossary [optional]
188
 
189
- <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
 
 
 
 
 
 
 
 
 
190
 
191
- [More Information Needed]
192
 
193
- ## More Information [optional]
 
194
 
195
- [More Information Needed]
 
 
196
 
197
- ## Model Card Authors [optional]
 
 
 
 
198
 
199
- [More Information Needed]
200
 
201
- ## Model Card Contact
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
202
 
203
- [More Information Needed]
204
- ### Framework versions
205
 
206
- - PEFT 0.19.1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ library_name: gliner2
3
+ license: mit
4
  base_model: fastino/gliner2-large-v1
 
5
  tags:
6
+ - ner
7
+ - relation-extraction
8
+ - data-mention-extraction
9
+ - lora
10
+ - gliner2
11
+ - development-economics
12
+ - geography
13
  ---
14
 
15
+ # datause-extraction-v1
16
 
17
+ This is the official fine-tuned GLiNER2 LoRA adapter for extracting structured data mentions, their attributes, geographic coverage, and usage roles from economics and development research documents.
18
 
19
+ The model leverages a joint **entity + relation extraction schema** to detect mentions and link them to metadata (producers, acronyms, timeframes, and countries) without suffering from choices-based prefix collision.
20
 
21
+ ---
22
 
23
+ ## Usage Option 1: Using the `ai4data` Library Wrapper (Recommended)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
24
 
25
+ It is **highly recommended** to interact with this model using the official **`ai4data`** Python wrapper. The library handles markdown-aware document parsing, sliding context windows, overlap resolution, and entity-relation alignment automatically.
26
 
27
+ Install the library directly from GitHub:
28
 
29
+ ```bash
30
+ pip install git+https://github.com/rafmacalaba/monitoring_of_datause.git
31
+ ```
32
 
33
+ ### 1. Extract from Text
34
 
35
+ ```python
36
+ from ai4data import extract_from_text
37
 
38
+ text = """
39
+ We use the 2022 Demographic and Health Survey (DHS) conducted by the National Statistics Office in Uganda
40
+ to analyze child health outcomes. We complement this with population records from the Ministry of Health.
41
+ """
42
 
43
+ results = extract_from_text(text, include_confidence=True)
44
 
45
+ for dataset in results["datasets"]:
46
+ name = dataset["mention_name"]["text"]
47
+ acronym = dataset["acronym"]["text"] or "N/A"
48
+ producer = dataset["producer"]["text"] or "N/A"
49
+ geography = dataset["geography"]["text"] or "N/A"
50
+ usage = dataset["usage_context"]["text"]
51
+
52
+ print(f"Dataset: {name} ({acronym})")
53
+ print(f" Producer: {producer} | Geography: {geography} | Role: {usage}\n")
54
+ ```
55
 
56
+ ### 2. Extract from PDF
57
 
58
+ ```python
59
+ from ai4data import extract_from_document
60
 
61
+ # Extracts from a local path or a PDF URL
62
+ pdf_url = "https://pdf.usaid.gov/pdf_docs/PA00TB5D.pdf"
63
+ results = extract_from_document(pdf_url, pages=[0, 1, 2])
64
 
65
+ for page_data in results:
66
+ print(f"--- Page {page_data['page']} ---")
67
+ for dataset in page_data["datasets"]:
68
+ print(f"Found mention: {dataset['mention_name']['text']}")
69
+ ```
70
 
71
+ ---
72
 
73
+ ## Usage Option 2: Using raw `gliner2` Library (Without the wrapper)
74
+
75
+ If you prefer to integrate the model directly without using the wrapper library, you can use the raw `gliner2` package.
76
+
77
+ ### 1. Installation
78
+
79
+ ```bash
80
+ pip install gliner2 huggingface_hub
81
+ ```
82
+
83
+ ### 2. Code Example
84
+
85
+ ```python
86
+ from gliner2 import GLiNER2
87
+ from huggingface_hub import snapshot_download
88
+
89
+ BASE_MODEL = "fastino/gliner2-large-v1"
90
+ ADAPTER_ID = "ai4data/datause-extraction-v1"
91
+
92
+ # 1. Load model and adapter
93
+ model = GLiNER2.from_pretrained(BASE_MODEL)
94
+ model.load_adapter(snapshot_download(ADAPTER_ID))
95
+ model.eval()
96
+
97
+ # 2. Define schema
98
+ ENTITY_DEFS = {
99
+ "name": "The exact full name of the data source or dataset",
100
+ "acronym": "The acronym or abbreviation if any",
101
+ "producer": "The organization or entity that produced or published the data",
102
+ "timeframe": "The year or time period of the data such as 2019 or 2019 to 2020",
103
+ "datatype": "The type of data verbatim from text such as survey, report, census, program, system, or assessment",
104
+ "geography": "The country, region, or geographic area the data covers",
105
+ "specificity": "Whether this mention is named, descriptive, or vague",
106
+ "usage": "Whether this is primary, supporting, or background data",
107
+ }
108
+
109
+ RELATION_DEFS = {
110
+ "has_acronym": "The acronym of the dataset",
111
+ "has_producer": "The producer of the dataset",
112
+ "has_timeframe": "The timeframe of the dataset",
113
+ "has_datatype": "The data type of the dataset",
114
+ "has_geography": "The country or geographic coverage area of the dataset",
115
+ "has_specificity": "Whether this dataset is named, descriptive, or vague",
116
+ "has_usage": "Whether this dataset is primary, supporting, or background",
117
+ }
118
+
119
+ schema = model.create_schema()
120
+ schema.entities(ENTITY_DEFS)
121
+ schema.relations(RELATION_DEFS)
122
+
123
+ # 3. Add prompt prefix
124
+ LABEL_PREFIX = "specificity: named | descriptive | vague usage: primary | supporting | background |"
125
+ text = "We use the 2022 Demographic and Health Survey (DHS) conducted by the National Statistics Office in Uganda."
126
+ prefixed_text = f"{LABEL_PREFIX} {text}"
127
+
128
+ # 4. Extract
129
+ outputs = model.extract(prefixed_text, schema, threshold=0.3)
130
+ print(outputs)
131
+ ```
132
 
133
+ ---
 
134
 
135
+ ## Response Structure (Wrapper Output)
136
+
137
+ Each item in the returned `"datasets"` list from the `ai4data` library is structured as follows:
138
+
139
+ ```json
140
+ {
141
+ "mention_name": {
142
+ "text": "Demographic and Health Survey",
143
+ "confidence": 0.9998,
144
+ "start": 12,
145
+ "end": 41
146
+ },
147
+ "specificity_tag": {
148
+ "text": "named",
149
+ "confidence": 0.9998,
150
+ "start": 12,
151
+ "end": 41
152
+ },
153
+ "usage_context": {
154
+ "text": "primary",
155
+ "confidence": 0.9998,
156
+ "start": 12,
157
+ "end": 41
158
+ },
159
+ "typology_tag": {
160
+ "text": "survey",
161
+ "confidence": 0.9998,
162
+ "start": 12,
163
+ "end": 41
164
+ },
165
+ "acronym": {
166
+ "text": "DHS",
167
+ "confidence": 0.9996,
168
+ "start": 43,
169
+ "end": 46
170
+ },
171
+ "producer": {
172
+ "text": "National Statistics Office",
173
+ "confidence": 0.9992,
174
+ "start": 60,
175
+ "end": 86
176
+ },
177
+ "reference_year": {
178
+ "text": "2022",
179
+ "confidence": 0.9998,
180
+ "start": 7,
181
+ "end": 11
182
+ },
183
+ "geography": {
184
+ "text": "Uganda",
185
+ "confidence": 0.9997,
186
+ "start": 90,
187
+ "end": 96
188
+ }
189
+ }
190
+ ```
191
+
192
+ ### Attribute Fields
193
+
194
+ | Field | Type | Description |
195
+ |---|---|---|
196
+ | `mention_name` | String / Span | Verbatim name of the dataset mentioned in the text |
197
+ | `specificity_tag` | Choice / Span | Precision classification: `named` / `descriptive` / `vague` |
198
+ | `usage_context` | Choice / Span | Analytical role: `primary` (core dataset) / `supporting` (context/validation) / `background` (passing reference) |
199
+ | `is_used` | Boolean / Span | Derived field: `True` if `usage_context` is `primary`/`supporting`, `False` if `background` |
200
+ | `typology_tag` | Choice / Span | Derived/mapped data type: `survey` / `census` / `administrative` / `database` / `indicator` / `geospatial` / `microdata` / `report` / `other` |
201
+ | `acronym` | String / Span | Abbreviation or acronym linked to the dataset |
202
+ | `producer` | String / Span | Organizing body or agency that published/collected the data |
203
+ | `reference_year` | String / Span | Year or timeframe the data represents |
204
+ | `geography` | String / Span | Geographic coverage or region the data represents |