Dmitry Kirsanov commited on
Commit
a9fdece
·
1 Parent(s): eb2c877

pre-chew into sentences to make it easier for gpt

Browse files
Files changed (2) hide show
  1. requirements.txt +2 -1
  2. understand.py +8 -4
requirements.txt CHANGED
@@ -4,4 +4,5 @@ bs4
4
  transformers
5
  anthropic
6
  google-generativeai
7
- tiktoken
 
 
4
  transformers
5
  anthropic
6
  google-generativeai
7
+ tiktoken
8
+ nltk
understand.py CHANGED
@@ -1,4 +1,5 @@
1
  import re
 
2
 
3
  import parse
4
  import llm
@@ -190,6 +191,9 @@ Output the line "ACTORS:" followed by your list of actors, one per line.
190
 
191
 
192
  def annotate_actors(self):
 
 
 
193
  ret = self.query(
194
  f"""Below is a document inside <document></document>. It is a {self.values.get("document type")}.
195
 
@@ -202,17 +206,17 @@ A reader of this document has compiled a list of actors (parties) mentioned or i
202
 
203
  {newline.join(["- " + x for x in self.values.get("actors, generalized")])}
204
 
205
- Your goal is to annotate each sentence of this document with its intended audience, by picking parties from this list. For each sentence, ask yourself: which of the parties from the list would benefit from this sentence's information? Use only the parties from the list; add the relevant partes, separated by commas, after the end of each sentence in [[double brackets]].
206
 
207
- For example, suppose you have a document describing the game of Sudoku, and your parties are "Players" and "Creators". Here are example sentences with audience markup:
208
 
209
  <example>
210
- Sudoku is a popular logic puzzle with numbers. Its rules are quite simple, so even beginners can handle the simple levels.[[Creators,Players]] A presented puzzle with more than one possible solution would usually be considered defective.[[Creators]] If you have completed finding all positions for a number, draw an X over that number in the list outside.[[Players]]
211
  </example>
212
 
213
  Output just the document with the audience markup after each sentence, but otherwise unchanged:
214
 
215
- <document>{self.values.get("FAD")}</document>""")
216
  annotated = self.parse_document(ret)
217
  self.values.put("FAD with actors", annotated)
218
 
 
1
  import re
2
+ from nltk.tokenize import sent_tokenize
3
 
4
  import parse
5
  import llm
 
191
 
192
 
193
  def annotate_actors(self):
194
+ sents = sent_tokenize(self.values.get("FAD"))
195
+ markup = "".join([f'<sentence actors="">{x}</sentence>' for x in sents])
196
+
197
  ret = self.query(
198
  f"""Below is a document inside <document></document>. It is a {self.values.get("document type")}.
199
 
 
206
 
207
  {newline.join(["- " + x for x in self.values.get("actors, generalized")])}
208
 
209
+ Your goal is to annotate each sentence of this document, inside a <sentence></sentence>, with its intended audience by picking one or more parties from this list. For each sentence, ask yourself: which of the parties from the list would want to know this sentence's information? Use only the parties from the list.
210
 
211
+ Add the relevant partes, separated by commas, inside the "actors" atribute of each <sentence> tag. For example, suppose you have a document describing the game of Sudoku, and your parties are "Players" and "Creators". Here are example sentences with audience markup:
212
 
213
  <example>
214
+ <sentence actors="Creators,Players">Sudoku is a popular logic puzzle with numbers with simple rules that even beginners can easily master.</sentence> <sentence actors="Creators">A presented puzzle with more than one possible solution would usually be considered defective.</sentence> <sentence actors="Players">If you have completed finding all positions for a number, draw an X over that number in the list outside.</sentence>
215
  </example>
216
 
217
  Output just the document with the audience markup after each sentence, but otherwise unchanged:
218
 
219
+ <document>{markup}</document>""")
220
  annotated = self.parse_document(ret)
221
  self.values.put("FAD with actors", annotated)
222