Spaces:
Runtime error
Runtime error
Dmitry Kirsanov commited on
Commit ·
c4d6264
1
Parent(s): 43e002c
debug
Browse files- docs/understander.md +4 -0
- understand.py +2 -0
docs/understander.md
CHANGED
|
@@ -82,6 +82,8 @@ The taxonomy of annotation classes is TBD, for now we foresee at least:
|
|
| 82 |
|
| 83 |
* **Gap annotations**: similarly, we prod the LLM to identify where it might feel like some cricial information or link is missing. We don't ask it to fill in this information right away, but we record the spots with LLM's comments.
|
| 84 |
|
|
|
|
|
|
|
| 85 |
* **Tone annotations**: We ask LLM to identify sentences or words that are out of place in the tone (too colloquial, too flowery, too scientific, etc). We record the spans the LLM indicates with its comments.
|
| 86 |
|
| 87 |
* **Understanding faults**: We ask LLM to identify places where it is not completely sure of the meaning of the sentence, and record such spans with its comments.
|
|
@@ -143,6 +145,8 @@ This approach has limitations. An LLM's context size may be insufficient to fit
|
|
| 143 |
|
| 144 |
While we give LLM as much context as possible, we instruct it to do as little as possible. While it's tempting to save on input/output tokens by packing several instructions into one prompt, LLM performance suffers when you ask it to do too many things. We can perhaps still give multiple instructions if they are the same in nature and focus on the same area of the document (for example, "do an actor/audience analysis of each of the 4 sentences in the first paragraph in section 1.2.3").
|
| 145 |
|
|
|
|
|
|
|
| 146 |
### Semantic markup
|
| 147 |
|
| 148 |
We generally feed LLM a flattened text-only version of the document. However, when we need to point out elements in the document we want LLM to look at, and to convey to it results of the analysis already done (when it's relevant to the task at hand), we will use semantic XML markup.
|
|
|
|
| 82 |
|
| 83 |
* **Gap annotations**: similarly, we prod the LLM to identify where it might feel like some cricial information or link is missing. We don't ask it to fill in this information right away, but we record the spots with LLM's comments.
|
| 84 |
|
| 85 |
+
* **Contradictions**: !!!
|
| 86 |
+
|
| 87 |
* **Tone annotations**: We ask LLM to identify sentences or words that are out of place in the tone (too colloquial, too flowery, too scientific, etc). We record the spans the LLM indicates with its comments.
|
| 88 |
|
| 89 |
* **Understanding faults**: We ask LLM to identify places where it is not completely sure of the meaning of the sentence, and record such spans with its comments.
|
|
|
|
| 145 |
|
| 146 |
While we give LLM as much context as possible, we instruct it to do as little as possible. While it's tempting to save on input/output tokens by packing several instructions into one prompt, LLM performance suffers when you ask it to do too many things. We can perhaps still give multiple instructions if they are the same in nature and focus on the same area of the document (for example, "do an actor/audience analysis of each of the 4 sentences in the first paragraph in section 1.2.3").
|
| 147 |
|
| 148 |
+
!!!output limit
|
| 149 |
+
|
| 150 |
### Semantic markup
|
| 151 |
|
| 152 |
We generally feed LLM a flattened text-only version of the document. However, when we need to point out elements in the document we want LLM to look at, and to convey to it results of the analysis already done (when it's relevant to the task at hand), we will use semantic XML markup.
|
understand.py
CHANGED
|
@@ -210,6 +210,8 @@ Output the line "ACTORS:" followed by your list of actors, one per line.
|
|
| 210 |
if f'<{span}>' in chunk:
|
| 211 |
text = chunk.replace(f'<{span}>', '')
|
| 212 |
self.fad.add_span_mark(span, t, t + len(text))
|
|
|
|
|
|
|
| 213 |
t += len(text)
|
| 214 |
|
| 215 |
def annotate_sentences_with_actors(self, src="document", dst="document with actors"):
|
|
|
|
| 210 |
if f'<{span}>' in chunk:
|
| 211 |
text = chunk.replace(f'<{span}>', '')
|
| 212 |
self.fad.add_span_mark(span, t, t + len(text))
|
| 213 |
+
else:
|
| 214 |
+
text = chunk
|
| 215 |
t += len(text)
|
| 216 |
|
| 217 |
def annotate_sentences_with_actors(self, src="document", dst="document with actors"):
|