| --- |
| license: mit |
| language: |
| - en |
| - fr |
| - es |
| --- |
| |
| # 🔥 Classifiers of FinTOC 2022 Shared task winners (ISPRAS team) 🔥 |
|
|
| Classifiers of texual lines of English, French and Spanish financial prospects in PDF format for the [FinTOC 2022 Shared task](https://wp.lancs.ac.uk/cfie/fintoc2022/). |
|
|
| ## 🤗 Source code 🤗 |
|
|
| Training scripts are available in the repository https://github.com/ispras/dedoc/ (see `scripts/fintoc2022` directory). |
|
|
| ## 🤗 Task description 🤗 |
|
|
| Lines are classified in two stages: |
| 1. Binary classification title/not title (title detection task). |
| 2. Classification of title lines into title depth classes (TOC generation task). |
|
|
| There are two types of classifiers according to the stage: |
| 1. For the first stage, **binary classifiers** are trained. They return `bool` values: `True` for title lines and `False` for non-title lines. |
| 2. For the second stage, **target classifiers** are trained. They return `int` title depth classes from 1 to 6. More important lines have a lesser depth. |
|
|
| ## 🤗 Results evaluation 🤗 |
|
|
| The training dataset contains English, French, and Spanish documents, so three language categories are available ("en", "fr", "sp"). |
| To obtain document lines, we use [dedoc](https://dedoc.readthedocs.io) library (`dedoc.readers.PdfTabbyReader`, `dedoc.readers.PdfTxtlayerReader`), so two reader categories are available ("tabby", "txt_layer"). |
| |
| To obtain FinTOC structure, we use our method described in [our article](https://aclanthology.org/2022.fnp-1.13.pdf) (winners of FinTOC 2022 Shared task!). |
| The results of our method (3-fold cross-validation on the FinTOC 2022 training dataset) for different languages and readers are given in the table below (they slightly changed since the competition finished). |
| As in the FinTOC 2022 Shared task, we use two metrics for results evaluation (metrics from the [article](https://aclanthology.org/2022.fnp-1.12.pdf)): |
| **TD** - F1 measure for the title detection task, **TOC** - harmonic mean of Inex F1 score and Inex level accuracy for the TOC generation task. |
| |
| <table border="1" class="dataframe"> |
| <thead> |
| <tr style="text-align: left;"> |
| <th></th> |
| <th>TD 0</th> |
| <th>TD 1</th> |
| <th>TD 2</th> |
| <th>TD mean</th> |
| <th>TOC 0</th> |
| <th>TOC 1</th> |
| <th>TOC 2</th> |
| <th>TOC mean</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <th>en_tabby</th> |
| <td>0.811522</td> |
| <td>0.833798</td> |
| <td>0.864239</td> |
| <td>0.836520</td> |
| <td>56.5</td> |
| <td>58.0</td> |
| <td>64.9</td> |
| <td>59.800000</td> |
| </tr> |
| <tr> |
| <th>en_txt_layer</th> |
| <td>0.821360</td> |
| <td>0.853258</td> |
| <td>0.833623</td> |
| <td>0.836081</td> |
| <td>57.8</td> |
| <td>62.1</td> |
| <td>57.8</td> |
| <td>59.233333</td> |
| </tr> |
| <tr> |
| <th>fr_tabby</th> |
| <td>0.753409</td> |
| <td>0.744232</td> |
| <td>0.782169</td> |
| <td>0.759937</td> |
| <td>51.2</td> |
| <td>47.9</td> |
| <td>51.5</td> |
| <td>50.200000</td> |
| </tr> |
| <tr> |
| <th>fr_txt_layer</th> |
| <td>0.740530</td> |
| <td>0.794460</td> |
| <td>0.766059</td> |
| <td>0.767016</td> |
| <td>45.6</td> |
| <td>52.2</td> |
| <td>50.1</td> |
| <td>49.300000</td> |
| </tr> |
| <tr> |
| <th>sp_tabby</th> |
| <td>0.606718</td> |
| <td>0.622839</td> |
| <td>0.599094</td> |
| <td>0.609550</td> |
| <td>37.1</td> |
| <td>43.6</td> |
| <td>43.4</td> |
| <td>41.366667</td> |
| </tr> |
| <tr> |
| <th>sp_txt_layer</th> |
| <td>0.629052</td> |
| <td>0.667976</td> |
| <td>0.446827</td> |
| <td>0.581285</td> |
| <td>46.4</td> |
| <td>48.8</td> |
| <td>30.7</td> |
| <td>41.966667</td> |
| </tr> |
| </tbody> |
| </table> |
| |
| ## 🤗 See also 🤗 |
|
|
| Please see our article [ISPRAS@FinTOC-2022 shared task: Two-stage TOC generation model](https://aclanthology.org/2022.fnp-1.13.pdf) |
| to get more information about the FinTOC 2022 Shared task and our method of solving it. |
| We will be grateful, if you cite our work (see citation in BibTeX format below). |
|
|
| ``` |
| @inproceedings{bogatenkova-etal-2022-ispras, |
| title = "{ISPRAS}@{F}in{TOC}-2022 Shared Task: Two-stage {TOC} Generation Model", |
| author = "Bogatenkova, Anastasiia and |
| Belyaeva, Oksana Vladimirovna and |
| Perminov, Andrew Igorevich and |
| Kozlov, Ilya Sergeevich", |
| editor = "El-Haj, Mahmoud and |
| Rayson, Paul and |
| Zmandar, Nadhem", |
| booktitle = "Proceedings of the 4th Financial Narrative Processing Workshop @LREC2022", |
| month = jun, |
| year = "2022", |
| address = "Marseille, France", |
| publisher = "European Language Resources Association", |
| url = "https://aclanthology.org/2022.fnp-1.13", |
| pages = "89--94" |
| } |
| ``` |
