Dmitry Kirsanov commited on
Commit
c4d6264
·
1 Parent(s): 43e002c
Files changed (2) hide show
  1. docs/understander.md +4 -0
  2. understand.py +2 -0
docs/understander.md CHANGED
@@ -82,6 +82,8 @@ The taxonomy of annotation classes is TBD, for now we foresee at least:
82
 
83
  * **Gap annotations**: similarly, we prod the LLM to identify where it might feel like some cricial information or link is missing. We don't ask it to fill in this information right away, but we record the spots with LLM's comments.
84
 
 
 
85
  * **Tone annotations**: We ask LLM to identify sentences or words that are out of place in the tone (too colloquial, too flowery, too scientific, etc). We record the spans the LLM indicates with its comments.
86
 
87
  * **Understanding faults**: We ask LLM to identify places where it is not completely sure of the meaning of the sentence, and record such spans with its comments.
@@ -143,6 +145,8 @@ This approach has limitations. An LLM's context size may be insufficient to fit
143
 
144
  While we give LLM as much context as possible, we instruct it to do as little as possible. While it's tempting to save on input/output tokens by packing several instructions into one prompt, LLM performance suffers when you ask it to do too many things. We can perhaps still give multiple instructions if they are the same in nature and focus on the same area of the document (for example, "do an actor/audience analysis of each of the 4 sentences in the first paragraph in section 1.2.3").
145
 
 
 
146
  ### Semantic markup
147
 
148
  We generally feed LLM a flattened text-only version of the document. However, when we need to point out elements in the document we want LLM to look at, and to convey to it results of the analysis already done (when it's relevant to the task at hand), we will use semantic XML markup.
 
82
 
83
  * **Gap annotations**: similarly, we prod the LLM to identify where it might feel like some cricial information or link is missing. We don't ask it to fill in this information right away, but we record the spots with LLM's comments.
84
 
85
+ * **Contradictions**: !!!
86
+
87
  * **Tone annotations**: We ask LLM to identify sentences or words that are out of place in the tone (too colloquial, too flowery, too scientific, etc). We record the spans the LLM indicates with its comments.
88
 
89
  * **Understanding faults**: We ask LLM to identify places where it is not completely sure of the meaning of the sentence, and record such spans with its comments.
 
145
 
146
  While we give LLM as much context as possible, we instruct it to do as little as possible. While it's tempting to save on input/output tokens by packing several instructions into one prompt, LLM performance suffers when you ask it to do too many things. We can perhaps still give multiple instructions if they are the same in nature and focus on the same area of the document (for example, "do an actor/audience analysis of each of the 4 sentences in the first paragraph in section 1.2.3").
147
 
148
+ !!!output limit
149
+
150
  ### Semantic markup
151
 
152
  We generally feed LLM a flattened text-only version of the document. However, when we need to point out elements in the document we want LLM to look at, and to convey to it results of the analysis already done (when it's relevant to the task at hand), we will use semantic XML markup.
understand.py CHANGED
@@ -210,6 +210,8 @@ Output the line "ACTORS:" followed by your list of actors, one per line.
210
  if f'<{span}>' in chunk:
211
  text = chunk.replace(f'<{span}>', '')
212
  self.fad.add_span_mark(span, t, t + len(text))
 
 
213
  t += len(text)
214
 
215
  def annotate_sentences_with_actors(self, src="document", dst="document with actors"):
 
210
  if f'<{span}>' in chunk:
211
  text = chunk.replace(f'<{span}>', '')
212
  self.fad.add_span_mark(span, t, t + len(text))
213
+ else:
214
+ text = chunk
215
  t += len(text)
216
 
217
  def annotate_sentences_with_actors(self, src="document", dst="document with actors"):