It may also assign points for a match with the reference which is unsupported by the table.

it can give a wrong estimate of both precision and recall and should be complemented with a human evaluation if two similar performing models are compared.


These models may highly rely on manual feature engineering, which makes them laborious and time-consuming and are difficult to adapt to new domains.

more and more research (Manning and Eric, 2017; Sukhbaatar et al., 2015; Dodge et al., 2016; Serban et al., 2016; Bordes et al., 2017; Eric and Manning, 2017) dedicated to building end-to-end dialogue systems, in which all their components are trained entirely from the utterances themselves without the need to assume domains or dialog state structure, so it is easy to automatically extend to new domains and free it from manually designed pipeline modules.


Then we use a memory network to encode the results retrieved from KBs.

we can access KBs more efficiently and achieve a high task success rate.


Intuitively, the more columns are subject to variations, the more diverse the records are.

fewer records will match the query when more columns are subject to variations.


By employing a subsystem, including a Dialogue State Tracker and a SQL Generator, AirConcierge can issue a precise SQL query at the right time during a dialogue and retrieve relevant data from KBs.

airConcierge can handle large-scale KBs efficiently, in terms of shorter processing time and less memory consumption.


Unique to our problem, however, is the fact that we have an open set of relation types in the graphs.

we propose a novel graph-conditioned sparse transformer, in which the relation information is embed-ded directly into the self-attention grid.


From the analysis of our in-house search log, more than 95% of the queries have only one or two nodes, thus a scenario in which more than one edit operation applied is unlikely.

the instances in Modified MSCOCO and GCC are constructed with one edit operation.


Since our task takes a source graph and a modification query as inputs, we need two encoders to model the graph and text information separately.

there are four main components in our model: the query encoder, the graph encoder, the edge decoder and the node decoder.


To efficiently encode a graph, we need to encode the information not only from these constituent components, but also their interactions, namely the node-edge association and connectivity.

we incorporate the information from all the edges to the nodes from which these edges are originated.


Getting annotation from users is expensive, especially for a complex task like our graph modification problem.

we explore the possibility of augmenting the user-generated data with synthetic data in order to train a better model.


Using Design I for the annotation task, we noticed that workers were not motivated to identify and select different physical entities in motion.

the majority of labeled entities were animate, and few were inanimate entities in motion.


It is difficult to reuse sentence-level templates in new tasks that usually have different requirements.

people usually break long templates into smaller template units (TUs).


Since the vocabulary of the connection phrases is limited, we automatically generate text stitch training data by dropping certain words in free texts with simple rules.

we can train a high-quality text stitch model in a self-supervised paradigm.


Note that it only inserts connection phrases and preserves all the contents in the input.

compared with traditional encoder-decoder frameworks that generate texts from scratch, edition-based methods are better fits for this setting.


We find tokens with POS tags adp, aux, cconj, part, punct, sconj, verb are often parts of a connection phrase.

for each pair of adjacent sentences, we consider each token tok i with these POS tags as an indicator of potential segmentation of two TU instantiations.


In contrast, TS2 only needs 3 templates to constrain the TU orders and does not need to consider the connection phrases.

significant human efforts are reduced in template design and Q4 is partially answered.


METEOR calculates the precision and recall of the matched words between the generated and reference texts after alignment by taking paraphrases into account.

it is less sensitive to expression variations and content orders than other automatic metrics.


Additionally, generation methods are not suitable for non-English text owing to a lack of training data because they are heavily dependent on in-language supervision (Ponti et al., 2019).

we adopted the sequence labeling method to maximize scalability by using (multilingual) BERT (Devlin et al., 2019) and multi-head attention (Vaswani et al., 2017).


Although some studies have demonstrated the potential of multilingual open IE (Faruqui and Kumar, 2015; Gamallo and Garcia, 2015; White et al., 2016), most approaches are based on shallow patterns, resulting in low precision (Claro et al., 2019).

we introduce a multilingual-BERTbased open IE system.


The corpus strives for a general understanding of ST&WR and its textual material should be as diverse as possible.

we opted to use shorter excerpts from multiple texts rather than longer, complete texts and also tried to represent many different authors, newspapers and magazines.


As mentioned above, our annotation system shows similarities to the system defined in the influential narratological theory of Genette (2010), and also to that defined by Leech and Short (2013), both fairly formal systems that incorporate linguistic features in their definitions.

they were particularly suited to be adapted for annotation guidelines and also well suited to our other task of developing automatic ST&WR recognizers.


weDH is mainly the user interface for the DHTK library conceived and proposed by Picca and Egloff (2017)

if the main purpose of the library is to facilitate the exploitation of textual repositories such as Gutenberg.org along with of LOD resources such as DBpedia, wikidata and VIAF, the web interface has been conceived in order to be exploited by students and practitioners in the human science field with no or few coding skills.


In dialogue-heavy works, quoted speech can often exceed 50% of a novel’s text (Elson and McKeown, 2010).

quotations are an important structural component of literary texts.


VGG was designed with the aim of reconstructing the "polyphony" of the languages of Italy at war: the official voice of propaganda and the voice of soldiers, the voice of newspapers and the voice of letters, the voice of the elite of intellectuals and the popular voice, the voice of consensus and the voice of dissent, male voices and female voices.

the final corpus is balanced along various dimensions, corresponding to the textual genre, the language variety used, the author type (e.g., sex, education, profession, political orientation etc.


As a methodological note, in the original corpus the pronoun "I" has been resolved to the speaker's name in the process of annotating propositions from locutions (e.g., for the sentence "I believe Americans do have the ability to give their kids a better future", "I believe" has been replaced with "O'MALLEY believes") (Jo et al., 2019).

it is difficult to tell whether the source of a reported speech proposition is indeed the speaker or not.


However, they use reported speech often, partly because their discussions occurred after the debates had occurred.

these texts often refer back to speech from the debates themselves and the reported speech of the candidates.


Note that the CKY phase is designed to simplify the original RvNN framework that calculates discourse representations recursively according to the tree structure.

bERT, along with its pretrained linguistic knowledge, can learn the underlying discourse structure itself with raw text segments as inputs.


It is known that under micro F1 evaluation, different paragraphs are weighted in proportion to the number of their nodes in the discourse trees, while each paragraph is equally weighted under macro evaluation.

we can infer that Our-R takes advantage of predicting local structures.


In particular, during discussions with teachers where we visualized the collaboration annotations in the corpus that came from their particular classrooms, we found that teachers were very curious about whether students were introducing new information into the discussion or building off of what was previously said.

we experimented on how well a classifier could distinguish student turns labeled 'New' from the other collaboration annotations.


Additionally, the example illustrates that the connective has no relation sense assigned to it.

in our 2.2 version, we add this relation sense according to the PDTB 3.0 sense hierarchy.


Because of its complexity, it is hard to get annotated data for training statistical models to perform SDP.

our goal is to produce high-quality annotations other than the standard corpus used in the field-the Penn Discourse Treebank (PDTB) (Prasad et al., 2008)-in order to improve performance on explicit argument extraction.


In contrast, the neural-network-based model predicted low scores for all columns.

the neural-network-based model is robust against an unexpected essay.


We confirmed the robustness of the BERT model with three essays: an essay with a high/low score and one written with only one character.

we found that the BERT model is more robust against unexpected inputs than the feature-based models.


They could create a corpus with 16,000 humorous one-liners in English, collected from the Web, while, towards the development of a model of humour recognition, much negative data was available.

four sets of negative examples were gathered, namely: news titles from Reuters; proverbs on the Web; sentences from the British National Corpus (BNC); and sentences from the Open Mind Common Sense project.


Yet, a system for humour recognition should not be restricted to a single style of humour.

to complement the collected text in the one-liners style, we targeted another kind of short humorous texts: humorous headlines.


The subtitles feature, which refers to the intermediate titles throughout the text, was excluded because none of the news contained intermediate titles in their text structure.

the remaining 165 features were computed for each text in the corpus.


As our current approach is supervised, direct comparison is not possible.

we compare them indirectly, assuming that the average accuracy over 2000 article pairs from the unsupervised approach roughly corresponds to the average accuracy in a 10fold cross-validation with 10 repetitions over the same 2000 article pairs.


To better assess the universality of our systems, knowing that in real-world scenario some text will not have paragraph division, we wanted to explore how much paragraph organization influences the results of our systems on both classification tasks.

we compared classification performances on three different feature sets: surface + shallow, deep, and combination of all three types of features, in two scenarios: using all features of the corresponding type, and excluding the features that require paragraph information, e.g.


In order to be able to compare the same parts of different texts (which very often have different sizes), our model breaks the aspect flows in a fixed number of frames.

regardless of the number of sentences in a text, the first frame will represent the first part of the text, for example, which we can compare to another text's first part.


As we can notice in Equations (2) and (6), to calculate MCR and Energy Entropy correctly, the flow must contain, at least, 2 sentences per frame.

this minimum requirement must be considered during the definition of the number of frames and the K parameter (as it requires, at least, one sentence per subframe).


Legitimate and fake news contain an average of 21 and 14 sentences per document, respectively.

we decided to split the flows into 3 frames, resulting into 7 and 4.67 sentences per frame, on average, for the legitimate and fake news, respectively.


We observe that many annotations contain punctuation marks, which are considered as separate tokens by spaCy.

we perform the same calculations while ignoring the punctuation tokens.


Although our findings generally fit with theorized patterns of emotional development, we cannot say for certain whether the current results re-flect changes in felt emotion, emotion vocabulary or, more broadly, changes in the ability to use abstract language.

we are more inclined to cautiously interpret our results as reflecting developmental trends in the distribution of emotion words.


The main goals of call center conversations are either to pursue a person to sign a contract, or to solve some technical or financial problems.

the question of the evolution of frustration or satisfaction (called satisfaction dimension in the following) along the conversation, is crucial.


For ethical and commercial reasons the agent channel was discarded.

the corpus contains callers' voice only without any overlapping speech.


In order to do so, having clues about the satisfaction dimension of the caller can be beneficial.

we define a task of satisfaction dimension prediction throughout the conversation.


On the contrary, if satisfaction is completely differently rated by the three annotators, the CCC computed on each pair is close to 0.

the gold annotation, defined as the mean of the 3 annotation's values, is not consistent.


Training from the human labeling result, the evaluation model learns which generative models is better in each dialog context.

it can be used for system developers to compare the fine-tuned models over and over again without the human labor.


However, if an evaluation is fully automatic, then it can be incorporated into a generation system.

it can always generate sentences with higher evaluation value.


We consider emotions in poetry as they are elicited in the reader, rather than what is expressed in the text or intended by the author.

we conceptualize a set of aesthetic emotions that are predictive of aesthetic appreciation in the reader, and allow the annotation of multiple labels per line to capture mixed emotions within their context.


Kant (2001) already spoke of a "feeling of beauty", and it should be noted that it is not a 'merely pleasing emotion'.

in our pilot annotations, Beauty and Joy were separate labels.


Furthermore, our pilot annotations revealed, while Beauty is the more dominant and frequent feeling, both labels regularly accompany each other, and they often get confused across annotators.

we add Joy to form an inclusive label Beauty/Joy that increases consistency.


For the experts, we aggregate their emotion labels on stanza level, then perform the same strategy for selection of emotion labels.

for s, both crowds and experts have 1 or 2 emotions.


None of our models was able to learn this label for German.

we omit it, leaving us with eight proper labels.


However, those studies (described in detail below) focus on their particular application rather than addressing the underlying, abstract learning problem (formalized in Section 3.).

previously proposed methods have not been quantitatively compared.


For MLFFN, we built on the implementation2 and hyperparameter choices Buechel et al. (2018) used for the Empathic Reactions dataset.

mLFFN has two hidden layers (256 and 128 units, respectively) with ReLU activation.


Accordingly, evaluating the accuracy of automatically detected communicative functions reduces to the evaluation of what is captured in sentence representations.

we propose a task of ranking sentence representations according to a given communicative function.


Formulaic expressions are extracted from the example expressions by hand, but because they can be very specific or sometimes contain irrelevant content, some queries return no results.

we simplify and shorten the formulaic expressions and obtain what we call the core FEs to retrieve more sentences.


Each sentence has a sentence ID that corresponds to the sentence ID in AASC.

the surrounding context of each sentence can be easily retrieved if a classifier needs it.


The accuracy indicates how likely evaluators were to choose the correct answers, while the agreement indicates the degree to which they made the same choice.

if the sentence selection in the process of creating the dataset fails to make pairs of sentences with the same communicative functions, the accuracy will be low but the agreement will be high.


We note that 64.7% of the data showed 100% accuracy, and the accuracy for 84.4% of the data is greater than 75%, which implies that the majority of the quizzes are easy to answer.

the task of detecting the communicative functions of sentences is not too difficult for humans.


For P Score the correction performance of NONE is multipled with the averaged performance over all error categories, due to the fact that we want a changed output for all error categories instead of the NONE category where we explicitely do not want any changed output.

the correction parameter for NONE can be interpreted as a penality parameter.


Although some document types may have structured tabular data, we still need a general approach to detect tables on different kinds of documents.

large-scale endto-end deep learning models make it feasible for achieving better performance.


For instance, we found that models trained on data similar to Figure 1a 1b 1c would not perform well on Figure 1d because the table layouts and colors are so different.

enlarging the training data should be the only way to build open-domain table analysis models with deep learning.


Latex documents are different from Word documents because they need other resources to be compiled into PDF files.

we cannot only crawl the '.tex' file from the internet.


Although these methods perform well on some documents, they require extensive human efforts to figure out better rules, while sometimes failing to generalize to documents from other sources.

it is inevitable to leverage statistical approaches in table detection.


The absence of significant success in ad-hoc IR using deep learning approaches is mainly due to the complexity of solving the ranking task using only unlabelled data (Dehghani et al., 2017).

the availability of large amount of labelled data is crucial to develop effective DNNs for ad-hoc IR.


Such datasets are suitable for feature-based learning-to-rank models but not for DNNs that require the original content of queries and documents.

most of the deep learning model for ad-hoc IR that have been proposed recently are developed using one of the following approaches: (1) Using large amounts of data collected from commercial search engines that are not publicly available (Yang et al., 2019a;Mitra et al., 2017).


They rely on a first ranking stage made by an efficient model such as BM25 and only re-rank the top-k documents for a given query in order to have an efficient search.

wIKIR can be used to run BM25 and save the top-k documents for each query.


The normalization of taxonomic mentions of bacteria is not much of a challenge for the BioNLP community because the nomenclature is complete, variations are relatively standardized, and synonymy is rare (except in some special cases such as strain names).

string matching with basic variations yields decent results (Grouin, 2016).


We observed no impact of the window size (see Table 1).

we set a short symmetrical window of two tokens in all subsequent experiments.


It can mean that the concepts mentioned in the annotated corpus (whole training and development set) are largely those mentioned in the annotated test corpus.

both training sets have the same order of training examples for concepts mentioned in the test corpus, which could explain the similar performance.


In addition, from the perspective of EL solutions, it is observed that (1) EL on short text tends to require excessive hand-crafted features specific to a certain kind of application, which makes it not necessarily applicable to others; and (2) short-text oriented corpus finds itself inappropriate for evaluating the cluster of EL methods based on collective schemes, since short text is unable to supply enough contextual mentions.

long-text oriented corpora are considered to be at least of equal, if not greater, significance to verifying the effectiveness and robustness of EL methods.


The underlying reason is that mentions in these datasets are not even ambiguous, since most of them were derived from text with high clarity.

there is a pressing need to construct a corpus with a certain level of difficulty, so as to better examine various EL methods.


Also, we used the referential task for participants fMRI scanning.

it was demonstrated that annotated multichannel corpora like RUPEX can be an important resource for experimental research in interdisciplinary fields.


As a result, it was demonstrated that annotated multichannel corpora like RUPEX can be an important resource for experimental research in interdisciplinary fields.

different aspects of communication can be explored through the prism of brain activation.


Authors are highly advised to link news articles to Wikinews categories, to allow effective information organization in Wikinews, and do so only when a category is strongly related to the written article.

the categories can be viewed as salience annotations and entities corresponding to these categories as salient entities.


For pages that receive enough traffic, reliable user click statistics can be obtained and used to derive entity salience labels.

a dataset called Microsoft Document Aboutness (MDA), was constructed.


However, human annotated salience labels rely on crowdsourcing, which is usually very expensive.

it is preferred to derive salience labels using automated methods.


For example, in the example article, the entity mention Kim Jong-un is representing an entity and has corresponding Wikinews category Kim Jong-un.

a wikilink is added to refer to the Wikinews category Kim Jong-un.


However, we observe that basic statistics show major differences between news articles in different years.

we choose to split the dataset on a monthly basis, i.e., all articles up to a threshold month are placed in the training set, while the remaining articles are placed in the test set.


However, comments are noise for an evaluation dataset because automatic evaluation methods utilizing corrected sentences typically rely on the matching rate between the system output and the corrected sentences to calculate a score.

in this study, we manually corrected the learner sentences extracted from the Lang-8 corpus using consistent rules and created a highly reliable evaluation corpus for the correction of grammatical errors in Japanese.


The Lang-8 corpus has often only one corrected sentence per learner sentence, which is not enough for evaluation.

we ensured that our evaluation corpus has multiple references.


The types of errors differ between handwritten sentences and typewritten sentences.

the JPDB and NAIST Misuse Corpus are not suitable as evaluation datasets for correcting grammatical errors in typewritten sentences.


The Japanese portion of the Lang-8 corpus has a wide coverage.

these techniques can also be applied to JSL texts.


Many of the articles in the Lang-8 corpus are written as if the learner writes a diary.

we designed the annotation rules considering the local and global contexts dedicated to Lang-8's register (writing a blog).


Overall, 11,437 out of 14,671 (78%) time points are indeterminate for abstract TDT timelines and 10,023 out of 14,671 (70%) are indeterminate for full TDT timelines, compared with 8,769 out of 15,623 (56%) for TimeML timelines.

even full TDTs increase temporal indeterminacy significantly compared to TimeML graphs.


Indexing minimizes the amount of expensive pattern matching that must take place at runtime.

the runtime system matches a syntax-based graph traversal in 2.8 seconds in a corpus of over 134 million sentences, nearly 150,000 times faster than its predecessor.


Their proposed method automatically learns which parts are relevant for a given classication.

their proposed method gives best results without any external help.


This arbitration is made possible by the hypothesis that Financial Markets might not be as efficient as they are supposed to be in neoclassic economic theory (Fama, 1970), because of information asymmetry.

it is possible to have an unbiased algorithm digging deep into the mass of accessible documents to yield indications about future performances of a company.


If a company voluntarily hides a vital information for the market's wealth, they fall under the Financial Authority it depends on and might become the subject of a lawsuit for Statement Fraud.

aRs are the most comprehensive public document to evaluate a company potential and strategy.


We observe that in the first one, the risk section is easily detectable whereas it is not trivial for the second one.

we cannot only use "risque" word to isolate the Risk Section in Annual Reports, which can be cause by some risk sentences not containing risk vocabulary but markers of uncertainty.


We also built a Metadata table containing, for each company, information about its market capitalization and sector of activity, allowing modularity of the corpus for various tasks.

the DoRe corpus can be used for cross-country/dialect, cross industry, cross-size analysis, fraud detection, document segmentation and risk factor extraction but also for pre-training or fine-tuning Language Models on Financial and Economic specific domains for French language and French dialects.


It is important to note that the relations we have annotated are long-distance relations, connecting often concepts that are mentioned in different sections of the EEG report, or concepts that are in separate sentences.

we believe that by releasing these annotations, we will enable other researchers to make use of the data for many possible downstream applications, not only in the biomedical field, but also in application that rely on information extraction.


Table 2 defines each of the 16 attributes of EEG activities and illustrates the possible values each of these attributes.

any identification of EEG activities in EEG reports amounts to recognizing the 16 attributes listed in Table 2 along with the polarity and modality attributes.


In EEG reports, certain words tend to be more ambiguous than others, suggesting that computing the encodings of these words requires additional processing to correctly capture their meaning from the contexts in which they appear.

the TNE leverages Adaptive Computation Time (Graves, 2016) to dynamically allocate more computational resources for the encoding of some words compared to others in the same EEG report.


Knowing the semantic properties of the knowledge that is needed to connect argument units that are -for example -adjacent vs. those that are not, can guide the process of extracting knowledge for filling these gaps.

we want to investigate whether the distribution of the semantic properties we annotated for the inserted sentences -commonsense relation types and semantic clause types -respectively differs depending on the internal structure of an argument.


We then form the test set (train, development) by collecting all datapoints whose (subject,relation), (object,relation) is in the test( train or development) set pairs.

the test set has (subject,relation), (object,relation) which are not seen during training.


The existing automatic evaluation methods for these are limited (Lin et al., 2011;Pitler et al., 2010;Ellouze et al., 2017), usually do not take into account the complex and subjective nature of the linguistic quality factors.

we do not focus on these automatic quality measurement tools in this paper.


However, the crowd is typically composed of people with unknown and very diverse abilities, skills, interests, personal objectives, and technological resources, which lead to several challenges related to lack of control on participants and consistency of output quality in crowdsourcing (Hossfeld et al., 2014).

outputs produced by the crowd must be checked for quality, and so the quality of crowd-based NLP annotations has been repeatedly questioned (Lloret et al., 2018).


To our knowledge, there is no best practice guideline for summary quality evaluation regarding the optimal number of repetitions per item in crowdsourcing studies used in MOS.

we explore the relationship between the number of repetitions and the correlation coefficient between crowdsourcing and laboratory results to provide a best practice guideline regarding the optimal repetition number in MOS.


Furthermore, this work does not include any special data cleaning or annotation aggregation method other than the calculating mean values over 24 different judgments for a single item.

further analysis needs to be performed in order to find out the optimal aggregation method along with the corresponding optimal repetition number, such that comparable results to the laboratory can be obtained in a reliable and cost-effective way.


And, we randomized the order of the judgments five times to lower the effect of lurking variable, so k = 5 where the number 5 was selected arbitrarily.

we got a set of correlation coefficients for unrandomized judgments C measured , and C 1 , C 2 , C 3 , C 4 , and C 5 for five randomizations.


Minor differences in morpheme choice are scored as badly as mistranslating a whole word.

in addition to BLEU, we have evaluated the MT systems into Inuktitut using YiSi-0 (Lo, 2019), a word-level metric that incorporates character-level information, which has been shown to correlate better with human judgment than BLEU on translation quality into agglutinative languages, such as Finnish and Turkish, at both sentence level and document level (Ma et al., 2018;Ma et al., 2019).


The pdf format is not machine friendly so it is tricky for researchers to work with it.

we convert the original pdf files into plain text files step by step so that they can be used for machine translation or any other computational tasks.


In preliminary experiments we observed that for the full Wikipedia corpora, relatively few words in the evaluation dataset (discussed in Section 4.) were OOVs, yet OOVs are required for our experimental setup

following Adams et al. (2017) we carried out experiments in which we learned cross-lingual embeddings, but downsized the size of the corpora


Note that English has the largest corpus among the selected languages, and that we always use the full corpus for the target language, but a sample for the source language.

we expect the embeddings for English as the target language to be higher quality than those for English as the source language, which could explain why the accuracy is higher when English is used as the target language than as the source language.


However, the actual label of such a statement would be positive because of the author's intention.

the annotators were advised to strictly avoid making annotation decisions based on their own point of view (e.g -personal prejudices) when it comes to such sentences.


First, despite the third issue mentioned above 50% of the "no" answers and 80.8% of the "yes" answers of the students to closed questions were indeed correct.

there are good reasons to believe that if the learners were more conservative in their answers to closed questions, fewer dubious "yes" answers would be collected and the third issue could be solved.


Table 3 provides additional insights into the properties of the wordnet, and also compares it to Princeton WordNet, the most complete such resource.

our USGW has its fair share of verbs but is relatively poor in adjectives with respect to the PWN (that has 15% of adjectives), which should be addressed in future work.


Usually, these phrases are called idiomatic expressions, which is suitable when they are used in an idiomatic sense, but not so much when they are used in a literal sense.

we propose a new term: Potentially Idiomatic Expressions, or PIEs for short.


Because the PMB contains many very short documents, there might not be enough context to disambiguate the PIEs.

we extract only PIEs with at least 50 characters of context, i.e. the document should be at least 50 characters longer than the span of the PIE.


Given the set of candidate instances from the pre-extraction system, the challenge is to get high-quality sense annotations at manageable costs of time and money using the FigureEight platform.

we strive to make the task as easy as possible for crowdworkers.


The balance between idiomatic and other usages of PIEs does not mean much if the overall number of PIEs and/or tokens in the genre is very small.

we also look at the frequency of PIEs relative to the number of tokens.


As other ASR-systems, Elpis makes use of dictionaries to specify the pronunciation of words.

it cannot be used in the earliest stages of language documentation when most lexical data is still missing.


Time and resources for fieldwork are often very limited.

it makes sense to start on a phonemic level when lexical data are still sparse.


Due to the long-term and large-scale approach it can be expected that additional collections of possibly relevant primary data will appear over the entire project runtime.

it is unforeseeable which data types they will contain and at what point in time they might become relevant for research conducted along with the project.


To approach both problems in a sustainable way, the decisions in the area of data modeling that will have to be made will deal with the tension between unification of data sets and vocabularies on the one side and maximum openness to future resources and research queries on the other side.

as one solution it was decided to identify standardizable connecting information in the existing project resources and to make sure that connections to internal and external resources of any type can be represented and resolved in the future.


Most native languages in the country have adopted several loanwords given the extended contact with Spanish.

we need to carefully identify and clean the best sentences to build an NLP resource.


We can find different punctuation marks, bullet entries with or without enumeration, titles/subtitles/headers without any delimiter punctuation in the raw text, among others.

we must be able to handle different kinds of noise to obtain as many correct sentences as possible.


Similar to the findings of Buck et al. 2014, we noticed that even when the expected language is known, some files contain instructions or entries written in a different language than our targets.

we use a Peruvian language identification tool, developed by Espichan-Linares and Oncevay-Marcos (2017), to label each sentence and drop the texts identified as written in languages out of our scope.


Following the language identification step, we perform a manual inspection of part of the output and identified specific issues.

we propose different heuristics to clean and prepare a higher-quality corpus.


There are specific sentences with a large number of duplicated tokens (e.g. writing exercises for children in language guides or workbooks).

we compute the ratio per sentence of token types V and number of tokens N to identify the duplication process.


During the plain text conversion, some of the captured mathematical expressions are incorrectly located within a sentence, which loose its original meaning (see Figure 9 in the Appendix).

we establish a rule where we look for sequences of numbers and operators inside a sentence and remove them from our final corpus.


In the near future, there will be a demand to extend this to workplace-specific tasks and procedures.

a method of gathering crowdsourced dialogue data is needed that ensures compliance with such procedures, whilst providing coverage of a wide variety of dialogue phenomena that could be observed in deployment of a trained dialogue system.


Emergency response is clearly a high-stakes situation, which is difficult to emulate in a lab or crowdsourced data collection environment.

in order to foster engagement and collaboration, the scenario was gamified with a monetary reward given for task success.


However, for scenarios such as ours, the role playing requires a certain expertise and it is questionable whether the desired behaviour would be achieved simply by letting two non-experts converse with free text.

in recent data collections, there have been a number of attempts to control the data quality in order to produce a desired behaviour.


It is quite common for an application to be available in five to ten languages simultaneously.

to minimize the effort for new applications, the framework offers a Resource Grammars Library (RGL) for many languages (Ranta, 2009).


It is a small police office which was originally found only in Japan, although now variations exist elsewhere, such as in Singapore, where it is called a neighbourhood police post.

it needed a hyponym relation to the English synset for police station: 03977678n.


Both of these projects do not make use of the full range of etymological relationships present in Wiktionary.

we were motivated to develop our own Wiktionary parser that is both comprehensive and extensible: it can extract the etymological information and many other types of information annotated in Wiktionary, and it is easy to use and extend for further research.


Performance on Japanese (ja) beats the high-performing baseline because of a feature of the Japanese writing system: foreign words are written in katakana, while native words are written in hiragana or kanji.

foreign words are easily distinguished as borrowing due to differences in the script.


Because of this, the simpler models give an incorrect birth year, while the curve fitting model correctly identifies the start of a period of exponential grow around 1960.

the curve-fitting model works well as a model for word emergence.


We assume that the amount of samples is more important.

we investigate this hypothesis by selecting subsets of a 160k generated sample pairs from TRANSLIT.


The second way a surface word edit can occur is when a compound word is split up into its components; this does not occur a great deal in English and French, but is common in Swedish, German, Icelandic and Dutch.

for example in Kallocain (Swedish), where the action takes place in an imagined totalitarian society, there are many compounds with "polis" = "police" ("polischef" = "police chief", "polissekreterare" = "police secretary"...), "tjÃƒÂ¤nst" = "service" ("tjÃƒÂ¤nsteplikt" = "service duty", "offertjÃƒÂ¤nst" = "sacrifice service"...), etc.


A pattern in the library consists of a list of words, each marked by typecase as being either a surface word or a lemma.

for example in English the phrasal verb "catch up" is entered as CATCH up, indicating that "catch" can be inflected but not "up".


This last feature was designed intentionally to leverage the power of word embedding techniques, i.e., with the words mapped to an embedding space and the appropriate distance measures, we can easily capture semantically related words to the ones in the lexicons.

we do not need to build comprehensive vocabularies and can focus on the most representative words for each lexicon dimension.


Raters are supposed to perform impartial and objective evaluations, and they must enter specific comments in order to ground their scores.

for each essay in our dataset we also have the corresponding rater comments.


More specifically, we learn word embeddings (Mikolov et al., 2013c) for the Portuguese language, and then we employed the Word Mover's Distance function (Kusner et al., 2015) between a comment and the five subjectivity lexicons.

each comment is finally represented by a five-dimensional subjectivity vector, where each dimension corresponds to the amount of a specific type of subjectivity.


Over the years, thousands of public IT systems have been developed, but they are not communicating in a common language, and there has not previously been a common public plan for how IT systems securely and efficiently can exchange data and become part of a coherent process.

the government, municipalities and regions have agreed that, as part of the public digitization strategy for 2016-20, a common public architecture for secure and efficient data sharing and the development of processes that connect public services must be established.


It occurs when researchers share data on personal websites that become obsolete after time.

we attempt to mitigate that by sharing our corpus on LREC repository.


Moreover, segmenting Hadith components is a domain-specific task that can be even tricky for the non-specialist.

automating it ensures consistency in segmentation.


One of their current projects is studying forged Hadiths attributed to the prophet to understand the political views at a specific time in history.

forged Hadiths are being discovered and might keep emerging, which indicates a Hadith segmentation tool is not dealing with a closed set of data.


KPIs are not only considered useful for measuring progress, but also for collecting feedback on the strategy of a research infrastructure.

in 2018 CLARIN ERIC started work on a framework for KPIs that would help the CLARIN community to describe the progress in developing and operating the research infrastructure for language resources in quantitative terms.


Because our strings contain logographic material which may not have a clear connection to the phonological representation, alignment is challenging.

we opted for using models based on the neural encoderdecoder architecture.


for a coincidentally similar looking lexeme.

cleaning and more careful selection of the training data would likely have a positive impact on the results.


Also in syllabic renderings, many transcriptions map unambiguously to one phonological representation.

it would be useful to minimize ambiguity by first using a large dictionary lookup (e.g. consisting of the whole corpus of a given dialect in Oracc), and then trying to predict the correct phonological rendering only if the transcription is clearly ambiguous


Arguably, a test corpus sampled uniformly over a one year period might be more representative.

we first reserve the 2018 fillings (450k sentence pairs) for validation and test.


In this experiment, we are interested in measuring the impact of train/test overlapping on NMT performance.

we compare the performance of models trained on randomly picked 2M sentences from sedar-train and tested on 2 variants of the held-out datasets: before and after removing overlapping sentences.


In most of the work, the focus relies on the models without interpreting the data which performs much better on our own test set rather than on general translated sentences.

it is essential to analyses, correct and cleans the data before using it for the experiments.


However, to the best of our knowledge, no comparative analysis of MT errors output by PBSMT and NMT has been done for the English and Brazilian Portuguese language pair.

in this article we present the first error analysis of a NMT system's output for Brazilian Portuguese.


The authors provide multiple analyses of NMT outputs compared to PB-SMT ones, considering different characteristics, such as fluency and reordering.

Toral and Sanchez-Cartagena (2017) state that translations performed by NMt systems tend to be more fluent, however they also observed that quality degrades faster with the sentence length.


Although tens of thousands of original Japanese documents are disclosed every year (i.e., over 79,000 documents in 2018), the availability of English disclosure documents is limited.

there is a strong demand for machine translation on both listed companies and global investors since Japanese to English translation needs to be done in a timely manner.


Furthermore, most investors require TSE-listed companies to disclose both Japanese and English documents simultaneously.

it is not easy to meet the demand for the English translation of timely disclosure documents using manual translation only.


However, the subword tokenization solves the problem only if a rare word can be translated as constitutive words.

even using the subword tokenization, the NMT systems often are often unable to translate neither numbers with many digits nor constitutive proper nouns.


Taking news domain for example, one entity word usually needs to keep consistent translation across the whole document in newswire.

the gains mainly come from better translation consistency contributed by document context.


These root, prefix, and suffix combinations are decompositions of the gold standard words contained in the treebank.

our vocabulary is larger than that of the gold standard because of the additional patterns applicable to the prefix-root-suffix combinations.


For instance, a NOUN in PADT UD may be a noun, proper name or function word as it contains words such as myrAv (inheritance) "noun", dwlAr (dollar) "proper name", and kl (all) "function word".

a word that is analyzed as a noun in PADT UD and analyzed as a function word in our model is marked as a function word and a match occurs.


The analysis length method depends on a heuristic that shorter analyses are more probable than long ones.

complex analyses having many tags will have larger weight (less probability) than simple/short ones.


As can be seen from this example, some of the subtokens such as H, el,, l are not valid English words, whereas some such as world are.

the effect of subtokenisation on downstream NLP tasks that require the semantics of the original input string to be retained remains unclear.


Compared to BPE, which produces short and nonsensical subwords, Morfessor is a conservative segmenter that captures longer subwords.

morfessor reports the best performance on entity typing.


As observed in Table 2, among the different composition methods, SIF method reported the best results across languages.

we use SIF for creating word embeddings from subword embeddings in this experiment.


Incorporating characterlevel embeddings via LSTMs has shown to improve performance for named entity recognition tasks (Zhai et al., 2018).

applying more sophisticated supervised composition methods such as a recurrent neural network might help to create word embeddings from subtoken embeddings under such situations.


Because of the corpus' conversational style (Khalifa et al., 2016a), some portions of the text had particularly long sentences.

we split sentences in a cascading fashion with a length of 200 words and a buffer of 10 words at the beginning of the new sentence from the previous one to maintain contextual integrity.


A limitation of this research paradigm, of course, is that systems that are directly designed to identify morpheme boundaries do not provide more information than the morpheme itself.

it is not possible to tell the exact morphological structure of complex words, including how these complex words are derived from simpler words and what kind of morphological features the morphemes are corresponding to such as prefixes, suffixes, etc.


Domain changes will certainly impact the neighborhoods in the embedding space.

a comparison of words relative distances in two embedding spaces can be used to measure their degree of domain shift (Kulkarni et al., 2015;Asgari and Mofrad, 2016).


GP is designed to be uniform across languages with respect to the amount of data per language, the audio quality (microphone, noise, channel), the collection scenario (task, setup, speaking style), as well as the transcription and phone set conventions (IPA-based naming of phones in all pronunciation dictionaries).

gP supplies an excellent basis for research in the areas of (1) multilingual ASR, (2) rapid deployment of speech processing systems to yet unsupported languages, (3) language identification tasks, (4) speaker recognition in multiple languages, (5) multilingual speech synthesis, as well as (6) monolingual ASR.


However, texts used in our analysis are random sentences selected from different sources, mainly from newspapers.

we computed average TTR (ATTR) for the training transcription based on k disjunct 1000-utterance subsets of the training set instead of MATTR.


On the other hand, developing large-scale language resources is not economically viable.

alternative approaches need to be used to make Ethiopians benefit from speech and language processing tools.


We also gather participants' meta-data, including their gender, age, education, occupation, perceptions about CS as well as their personality traits, gathered through the Big-5 Personality Test.

this corpus serves as a useful resource in multiple fields, including NLP applications (mainly designed for ASR systems), linguistic analysis of the CS phenomenon, as well as sociolinguistic and psycholinguistic analyses.


It is claimed that dialectal Arabic was used as a tool to heighten the distinctiveness and distance of Egypt from the rest of the Arab world.

mSA was facing a threat posed by both, the foreign and dialectal languages.


It can be seen that 40% of the speakers are within the normal 140-160 range, 30% are below the rate and 30% are above.

for 30% of the interviews, accurate speech recognition for humans and ASR systems would be more challenging


In order to avoid having the interviewers as common speakers across all sets, their utterances have been placed in the train set.

the train set contains utterances from 4 female and 8 male interviewees, in addition to 1 female and 1 male interviewer.


Information about participants is also collected, including gender, age, educational background, perceptions about CS and personality traits.

the corpus serves as a useful resource for multiple research directions.


As mentioned before, the maximum size is 20 times higher than the minimum.

during 1 epoch for 1 sample in, for example, Russian as the target language, the model sees 20 samples in Odia.


We observed that users tend to create ambiguous mentions in tweets when they employ any expression (for example first or last name) other than Twitter screen names to mention other users in their posts (see Section 1).

we have drawn inspiration from this usage to elaborate a simple process for both candidate entity and ambiguous mention generations.


The Twitter Search  API 5 returns a collection of relevant tweets matching the specified query.

for each entity in the KB, i) we set its screen name (@user) as the query search; ii) we collected all the retrieved tweets, and iii) we filtered out tweets without images.


After a long study, we assumed that a thematic boundary can only be in the vicinity of a slide change during the course.

for each change of slide, a human expert annotated: 1. If there is a topic shift. 2. The exact moment of the topic shift defined as being positioned between two words. 3. The granularity of the topic shift (1 or 2) or if the segment type is an interruption.


In all, 134 of these occur with images showing flooding (IF), the majority (110) with articles about current flooding or the aftermath of recent flooding where the keyword is also found in the headline of the article.

there are 62 articles about current flooding with a flooding image and 48 articles about the aftermath of a recent flooding, where both the headline and the caption contain one of the keywords.


However, these same features mean that movies and TV are not representative of day-today life.

these datasets cannot be used to evaluate how well models perform when applied to realistic videos of day-to-day life.


However, unlike them, our proposed dataset relies on video clips that were recorded naturally by people, without predefined scripts.

understanding videos requires overcoming challenges such as environmental noise, camera movements, lighting conditions, and naturally occurring dialogues.


We interpret this behaviour as follows: For compounds with low-compositional modifiers the semantic relatedness compound-modifier is low, and here the strength of semantic relatedness compound-head (which is effectively WORD2) correlates with the degree of compositionality of the phrase.

in cases with low compound-modifier relatedness the degree of compositionality of the compound phrase and the compound-head pair are similar in their ranks across compounds.


BPE tokenization has become a de-facto standard way for processing sub-words in the era of BERT (Devlin et al., 2019) and BERT-like models.

we decided to draw a comparison between BPE tokenization and simpler character-level models, frequently used for segmentation in Chinese (Xue and Shen, 2003) or Arabic (Samih et al., 2017).


Despite some public pre-trained embedding vectors being already available for Portuguese (Bojanowski et al., 2017;Hartmann et al., 2017;Santos et al., 2019), the highly technical oil and gas vocabulary presents a challenge to Natural Language Processing applications, in which some terms may assume a completely different meaning compared to the general-context domain.

there are consistent evidences that generating embedding models from a domain-specific corpus can significantly increase the quality of their semantic representation and, hence, the performance of NLP applications on specialized downstream tasks on the same domain (Gomes et al., 2018;Nooralahzadeh et al., 2018;Lai et al., 2016).


However, it is hard to guarantee that each representation contains only the corresponding information.

reconstruction from these two parts directly might cause confliction in both content and sentiment aspects, which leads to poor performance in content preser-vation.


Note that the sentiment representations should be suitable for the semantic content of the template.

the modification should be combined with the contextual information of the template with target sentiment information.


Since most research focuses on shorter sentence-level texts, it is not clear whether these models can form sufficiently long range dependencies to be useful as a substitute for genuine training data.

we believe that applying NLG approaches to medical text for augmentation purposes is a worthwhile research area in order to ascertain its viability.


We are still not at a stage yet where we can release de-identified patient data publicly, whether this genuine or synthetic, due to their unquantified susceptibility to re-identification attacks.

even if we develop a model which shows strong performance in generating synthetic clinical notes, these notes cannot be easily shared with the wider research community.


Although EIEC is annotated with four entity types (Location, Person, Organization and Miscellaneous), the Miscellaneous class is rather sparse, occurring only in a proportion of 1 to 10 with respect to the other three classes.

in the training data there are 156 entities annotated as Miscellaneous whereas for each of the other three classes it contains around 1200 entities.


was first collected in order to make a cross-cultural comparison of smiling during humorous productions between American English and French (Priego-Valverde et al., 2018).

it was recorded following the American protocol, as closely as possible, especially concerning the tasks given to the participants (see section 2.2).


They show that, even if smiling is present in the whole conversation, its intensity is higher during humor productions.

the authors hypothesize that an increase in smiling intensity is a more significant way to frame an exchange as humorous than the sole presence of smiling.


Laughter is far less frequent and, unsurprisingly, displaying a neutral face is rare.

observing smiling in a binary way (presence/absence), tends to confirm the assumption that smiling is a marker of humor.


On the one hand, smiling is the most present facial behavior both in successful and in failed humor.

it does not seem to reduce the risk of failure.


Using BLEU, these two sentences would not receive the same score since BLEU is based on orthographic similarity.

bLEU inherently penalizes use of approved terminology even when it is used appropriately.


The overall architecture of the ontology follows the guiding principles and complies with the data standards for cataloguing cultural objects (Baca, 2006).

it could be easily incorporated in larger projects for ontological modelling of knowledge on cultural and historical heritage.


The object property spokenIn relates a dialect with an administrative or geographic location where this dialect can be found.

each dialect is related to at least one place, but multiple relations of this type are also conceivable.


In addition, there is also an interface package (IBGDialectsOntology) which contains all the methods that might be used to extract information from the ontology, and defines the parameters that could be sent by the graphical user interface.

we enable users to choose among different criteria for filtering and extracting information.


The method is public and implements the method getIndividualsInSignature() of the org.semanticweb.owlapi.model.OWLOntology class.

it returns a list of the type OWLNamedIndividual, which can then be handled by the BgDialectsOnto methods.


Although the features encoded in the ontology seem rather specific for the historical development of the Bulgarian language and its dialects, the overall design of the hierarchies of classes and properties can be easily adapted and applied to other languages.

it offers scope for valuable contribution that goes beyond national and regional borders.


The taxonomy is a tree structure with the majority of nodes positioned near the bottom of the tree.

as there are only a handful of nodes near the top, each time the random walk restarts, it is far more likely to start the random walk at a leaf node somewhere at the bottom of the taxonomy, rather than at the top.


Among the collected reviews, we worked on 6,287 reviews, which were segmented into sentences.

the dataset consists of 17,268 sentences whose length is about 10 tokens on average.


Online reviews are unstructured information and may contain various types of noise.

the process of cleaning and normalization of text is essential for the analysis.


Since the adversative conjunction guide the reader to more salient information, the argumentative force is combined with the latter statement.

we can categorize the first statement as POS OPINION and the second one as NEG OPINION, and yet attribute more weights on the latter.


The proposed scheme was specific to restaurants, but it can also be applied to other places such as hotels, vacation spots, shop- ping malls, theaters, etc., particularly considering that getting the visitors to revisit should be one of their primary goals.

a natural extension of this work would be to study the possibility of application of our scheme in other places and different languages.


Nowadays Personal Assistants (PAs) are available in multiple environments and become increasingly popular to use via voice.

we aim to provide proactive PA suggestions to car drivers via speech.


Especially, when the user drives or is busy with another task at home (e.g., cooking), the interaction with a PA is only the secondary task.

user experience designers need to focus on the user's cognitive load in such settings, too.


In binary classification framework, we regrouped the reviews as proposed in (Nabil et al., 2014): the reviews associated with one or two stars compose the negative class and those with four or five stars represent the positive class.


These embeddings are created by taking into account the morphology of words, instead of treating them as distinct units.

using this method, each word is represented as a sum of the representations of the character N-grams constituting it, giving us the opportunity to experiment with models that focus on N-grams in a sample to perform the classification.


This is not the case for fastText embeddings, as they are non-contextual by nature.

the non-contextual linguistic features may not add the same amount of advantage to the models using fastText embeddings in comparison to the models that use BERT and CamemBERT embeddings.


Due to this factor, and also because tweets in our collection are not longer than 140 characters11, most of the time only one false assertion is present in the text

the syntactic head of the only false assertion present (or false reported speech) has been easily marked, thus lowering the possibility of disagreement.


Because of this, occasionally during the voting period, we manually deprioritized the tweets that got three or more negative votes, to keep in the pool only the tweets that still had a chance of being considered positive.

the corpus contains some tweets that do not have five votes, mainly the non-humorous ones.


We manually inspected all pairs, clustered them into equivalence classes, and took one example from each class discarding the others from the corpus.

we pruned 1,278 tweets from the corpus, most of them were humorous.


At the same time, finding jokes within in-the-wild long texts can be problematic since you have to account for its boundaries concerning non-humorous content.

we collect jokes from Twitter, supposing a tweet is either completely humorous or not at all.


A low degree of morphological ambiguity is a planned design feature of Esperanto and, together with its regular inflection and affixation system, meant to make the language easy to learn.

automatic annotation is very reliable at this level, and few ambiguity classes exist, with little need for human revision.


The only systematic POS ambiguity is between proper nouns and other word classes because of upper-casing (especially in sentence-initial position), and in connection with tokenization errors.

the otherwise reliable vowel coding for POS (e.g. -o = noun, -a = adjective, -i = infinitive, -e = adverb) breaks down in the face of foreign names in (a) and (b).


However, we try to avoid unnecessary tag complexity by not introducing different syntactic tags, where POS already contains the distinction.

phrase-level modifiers are only attachment-tagged as prenominals (@>N) and postnominals (@NA)1 and post-adjects (@A<), not for what the modifier itself is (e.g. hypothetical @nmod for a modifier that is a nouns), because that would just be duplicated information.


It should be noted that some of the NE categories are intentionally vague and express "semantic (lexical) form" rather than "semantic function", leaving the latter to subsequent disambiguation at the semantic role level.

and can fill both agent and location slots, i.e. go to war or raise taxes on the one hand, and be lived in or traveled to on the other.


German and Danish, because compounding is more transparent in Esperanto than in languages with a lot of idiomatic traits.

the word 'bag' in German ('Tasche') or Danish ('taske') can occur as second part in compounds that do not denote a container, e.g. German 'Plaudertasche' (chatter box) and Danish 'havtaske' (monk fish).


One way to elicit these senses during linguistic data revision was to look for compounds with 'fonto' where the first part can help to disambiguate the sense of the second.

'akvofonto' (spring) is classified as <Lwater>, 'monfonto' (funding) as <Labs> (abstract source), 'interretfonto' (online sources) as <Lsem> and 'petrolfonto' (oil well) as <Lh> (human functional place).


In a breakdown of individual categories (table 2) ppattachment problems left their predictable mark, with postnominal pp's (PRP @N

19.8% of attachment errors and 26.8% of function errors involved the postnominal category, and 90% of cases were pp's.


Table 2 contains only the major categories, and it lumps all clause functions into only two groups, finite and nonfinite, but it clearly shows what is difficult for function tagging and for attachment tagging, respectively.

coordinators (@CO) and, to a lesser degree, adverbials (@ADVL) are more an attachment than a labeling problem, while copula complements (@SC) and subjects (@SUBJ) are more a labeling than an attachment problem.


The resulting value tells us, how much a category is over-represented among errors as compared to its share among running tokens.

the most unreliable categories in terms of function labeling are @N<PRED and @NPHR (9 times over-represented), while all clauselevel categories with the exception of complements, i.e subjects, objects and adverbials are safe (underrepresented).


In addition, heavy modifier material seems to be moved to the right.

all modifier clauses, including participle clauses, were placed to the right, as well as half of the coordinated adjectival modifiers.


Coverage is measured by breaking down the LFG scores according to whether the parser yields full parses or non-full parses (i.e. FRAGMENT, SKIMMED, or SKIMMED+FRAGMENT parses).

coverage is defined as the percentage of parsed sentences in relation to the full corpus, and to measure it, an evaluation was conducted against the 2364 test sentences.


A weakness with the coverage metric is that it does not guarantee that the assigned parse is indeed the correct one (Carroll et al., 1998).

accuracy was used in addition to assess parsing quality.


The experiments showed that the discourse type should be considered while training and interpreting the results.

the results of dependency parsing are more relevant to integrate in CRF model for the monologue in which the utterances are long and dependencies relations are more present.


Especially if a short video is given to users, most fragmentary facts within the scope of previous tasks can be easily perceived by themselves even before asking questions.

video question answering is expected to provide answers to more complicated non-factoid questions beyond the simple facts.


The other issue is that many videos with sufficient amounts of information, which a user is likely to pose questions on, have much longer lengths than the video clips in the existing datasets.

in practice, the most relevant part of a whole video needs to be determined prior to each an-swer generation phase.


Its goal is to distinguish the requests for information, which are sent to the QA system, from all other utterances including greetings and chitchat, which are sent to the chatbot.

instead of a using a complex dialogue act hierarchy (Bunt et al., 2012), our controller only needs to make a binary decision, which can be carried out using context-free classifiers (Clark and Popescu-Belis, 2004).


Another option is to use a large set of questions that jointly test the performance of the controller, the QA component and the chatbot -however, to the best of our knowledge, such a data set does not exist.

we use several component-level evaluation methods along with our own subjective testing, to assess the overall system's performance.


Open-domain tasks are usually more explored than domainspecific ones, mainly because of the ease of access to datasets and support from a wider research community.

open-domain methods and datasets are frequently applied to closed domains that lack sufficiently large datasets.


Modeling deep learning methods usually requires a large amount of training data.

we use 3 different large machine comprehension datasets for our analysis.


The authors of the original paper mentioned that word n-grams and word embeddings are not suitable for cross-language classification.

the considered features are: baseline, domain features, POS n-grams and dependency n-grams.


The lack of sufficient training data becomes the bottleneck for automatic Chinese irony detection.

it is of big importance to introduce a reasonably large and more diverse dataset suited for training machine learning algorithms as well as serving as a more reliable gold standard for Chinese irony detection.


But the contextual information for identifying the speaker's intention in this case is missing.

insufficient evidence is given in this case.


Since wikiHow is a collaborative online community, anyone can contribute and not all edits are relevant to the article content.

it is necessary to identify and filter irrelevant edits.


However, we found it difficult for annotators to pinpoint differences between versions where edits modify or provide new information, in contrast to providing only stylistic changes.

we designed the task such that annotators had to formulate questions on the presented information whenever possible.


We analyze a random sample of revision groups for which this pattern holds and observe that: 1) early edits mostly fix spelling mistakes, case, or simple grammatical errors, which are easy to detect; 2) intermediate versions are most difficult to distinguish, because changes made in them are mostly due to the addition or deletion of information for stylistic reasons or due to the need for subtle semantic refinements; 3) final edits usually improve clarity by resolving ambiguities and other potential issues.

they can be slightly easier to rank than some of the intermediate versions.


Furthermore, this is consistent with the hypothesis stated in Section 3.2: adopting only the vector of the first sense of each word harms the performance.

we have confirmed that the lower performance of the multi-sense models on previous datasets was not a result of the models' inability to handle non-major senses, but a result of the biases toward single-sense words and major senses in those datasets.


A semantic attribute of a concept, such as "an apple is red", explicitly dictates a semantic aspect of the concept.

given an appropriate set of attributes, a concept can be represented by a bundle of attributes.


No linguistic indication is given as to how UBs are specifically attributed, and no linguistic test is provided to discriminate between UBs.

we had to clarify and refine WordNet definitions of UBs, which may have resulted in modifications in supersense extensions.


Plant-denoting nouns denote natural objects, and had to be distinguished from closely related supersenses such as Object (see Table 1, line 5).

we refined definitions so as to avoid overlaps in supersense extensions.


High-quality user attributes are, however, hard to obtain since the information in social networks such as Facebook and Twitter is often sparsely populated (Li et al., 2014).

exploiting unstructured data sources to obtain structured user attributes is a challenging research direction.


According to the dual coding theory (Clark and Paivio, 1991), the formation of mental images aids in learning, although this advantage may not be available for abstract words as these are less likely to have corresponding word-to-image referential connections.

abstract words can be more challenging to learn and memorise.


While participants were strongly encouraged to wear a headset, not all of them followed this rule.

the audio quality varies between speakers.


In the beginning of the interaction, the robot commented on being assembled in Stockholm, Sweden, and having traveled to France in previous years.

those two countries were added to the initial knowledge base of the agent.


Our experiments show that the distribution of tweets in our collection is similar to the distribution of tweets in samples collected randomly using the Twitter "sample" API.

we can consider that we do not introduce any bias in our dataset (some events being more represented than others, for example) linked to the collection method.


The annotation interface has been designed to take advantage of the annotators' intelligence and avoid repetitive tasks.

it seemed unnecessary to make them annotate tweets that contained exactly the same text.


We conducted a manual analysis of 216 disagreement cases with the aim to extract the most common patterns, which contribute to the difficulty of the annotation task.


Besides, the lifetime of a troll account can be very short.

detecting troll tweets only using text should be considered.


Character and word n-grams can capture important morphological properties and discover useful inter-word and inter-phrase features.

they have been used as the core of many authorship analysis systems (Jankowska et al., 2014; Schwartz et al., 2013; Layton et al., 2010).


Using the following irony-related hashtags: ("#irony"), ("#ironical"), ("#mockery"), and #sarcasm, we built a dataset of 5,358 Arabic messages posted on Twitter.

of individual analysis and follow-up discussions, our annotators have labeled 4,809 (89.75%) as "Ironic", 435 (8.12%) as "Not Ironic", and 114 (2.13%) as "Ambiguous".


In the following experiments, we study whether the improved robustness through knowledge transfer of the English model is due to weight transfer, as we expect, or due to better i-vector extractor trained on English data instead of German.

we compare each combination of using German or English trained i-vector extractors each with a random model initialization or initialization of hidden layers with the English trained model when training on the 1000 hour German broadcast data.


Thereby we have shown that the direct adaptation of LF-MMI acoustic models from one language to another leads to good results, even using only very little training data from the target domain.

this observation can contribute to the ongoing research on speech recognition for under-resourced language.


Using the English i-vector extractor instead of the German one with random acoustic model initialization only leads to slightly better results on three test sets and slightly worse results on two test sets.

we conclude that, in deed, weight transfer leads to better results-and not a bet-ter i-vector extractor trained on English data.


One syntactic structure can also be organized within several prosodic periods.


The extracts represent different types of speech in terms of conversational environment, relationships between the speakers, etc.

this pilot corpus contains transcriptions of preparing a meal, meetings, conferences, radio transmissions, interviews, etc.


Segmentation of periods applies 4 criteria: 1) pause lasting for at least 300 milliseconds; 2) difference in height between the mean value of fundamental frequency over all the signal before the pause and the last value of fundamental frequency before the pause; 3) difference in height between the last value of fundamental frequency before the pause and the first one after the pause; 4) absence of hesitation (« euh ») just before or after the pause.

analor considers period as a prosodic segment between two pauses with its own melodic shape.


It is also worth noting that the Malayalam corpus contains significantly more vocabulary, with lesser mean word frequency.

we argue that this language would pose a bigger challenge for the TTS models.


There are 16 dialects in Japan, and these dialects have been found to effect the accuracy of speech recognition (Kudo, 1996).

when selecting recording areas for our corpus, we sought to include a balanced distribution of Japanese dialects.


We also did not consider the gender ratio of our participants.

currently the total number male participants in our corpus is 70, and the total number of female participants is 151 (table 1).


The shortcoming of the original implementation is that the first and the last heads do not interact as they are assumed not to be adjacent.

we assume that considering the heads' sub-spaces periodically, we can increase the model's effectiveness by applying circular convolution to the second dimension.


The vectors at the overlapping positions are averaged (by summing and dividing by the number of overlapping vectors).

we have the matrix of embeddings with the shape of the hidden size times the length of the text.


The quantity of overlap shows how much the summary and source texts contain similar concepts but it can only be a first hint as to whether the information in the sources is sufficient to re-create the abstractive summary given a particular user-query.

in addition to the overlap, we create extractive summaries from the selected candidate sources based on the abstractive summary.


In addition, at this large a scale, due to slow download rates from the free archive.org/web service, scraping all possible urls is unfeasible.

detecting noisy (poor quality) urls can help reduce the risk of wasting download time on unusable articles.


By contrast, because the extractive fragments are short, density will indicate extractive-ness.

combining density with coverage allows one to identify summaries that are mixed extractive and abstractive (so-called "mixed summaries") that compose abstractivelike summaries from short sequences of text found in the article.


In this paper we introduce this multimodal corpus which includes neural data from functional magnetic resonance imaging (fMRI), physiological data (blood flow pulse and respiration), transcribed conversational data, as well as face and eye-tracking recordings.

we present a unique corpus to study human conversations including neural, physiological and behavioral data.


We recorded three minutes of conversation per conversational agent in each block (6 minutes total in each block).

for each participant twenty-four minutes of conversation were recorded.


Data analysis of this corpus aims to integrate the multimodal markers of conversation to comprehensively characterize human conversation and conversation with an artificial agent.

the presented corpus allows, for the first time, to include multimodal conversational data in the investigation of bidirectional interaction.


Scored items compete with pretest items for exam space, the scarcity of which can create a bottleneck.

it is sometimes not possible to pretest as many new items as needed and some exam programs may not be able to afford pretesting at all.


It shows that NLI models fit the artificial patterns in the training set very well, which makes them fragile to the adversarial examples (hard set) which are against these patterns.

we assume the artificial patterns contributes to the hypothesis-only bias.


To train fact checking systems, we also need claims that are refuted by the evidence documents to serve as negative examples.

we generated claims that are refuted by the evidence.


from unlabeled data by itself and generate representations in a way that helps detect word boundaries.

the representations from the LM have shown to have a positive impact on the word segmentation performance.


This means that the classification was made qualitatively.

it is not certain whether or not the proposed classification is appropriate.


On the other hand, since it is not necessary to deeply consider the content of narratives for uttering responses with low concreteness like "response 5," the degree of empathy shown by these responses tends to be low.

the concreteness of responses is thought to reflect the degree of empathy they express.


For uttering these responses in appropriate timings, it is necessary to deeply understand the content of narrative.

contrary to the versatility, the degree of empathy shown by these responses to narrative tends to become high.


Furthermore, it is considered that back-channel responses can be uttered regardless of contents of narrative.

the timings when these responses can be uttered are diverse and the versatility of these responses is high.


For measuring ent, information on co-occurrence of responses is necessary.

the start time of response was mapped onto the nearest position of the following one in the narrative speech, considering that responses tend to be uttered right after a linguistic or phonetic boundary, namely: 1. clause boundary (linguistic) 2. pause longer than 200 milliseconds (phonetic)


Even with these additions, the parser can still match overlapping phrases.

the user can specify which phrases to keep.


The Estonian Reference Corpus (ERC) was missing information about paragraph boundaries, which is crucial for sentence segmentation.

we replaced the ERC with the original version by exporting texts from (Laur, 2018), in which paragraph endings are marked by double newlines.


For morphological analysis, we grouped an-notations by words, so that overlapping and differing morphological annotations of a word formed a single unit of evaluation.

situations arose when, despite the difference, "both tools are correct", because ESTNLTK can leave morphological analysis ambiguous.


In addition to the more traditionally used features (Dell'Orletta et al., 2011), this work incorporates discursive features as well as the relatively newly developed morphological complexity index.

it aims to push the boundaries of text difficulty classification systems in two respects: 1) by linking the automatic assessment of a text to Italian CEFR levels; 2) by incorporating the linguistic features previously found in separate studies into a single study.


For most cases however, from the perspective of the NLP and Machine Learning communities, all these RDF or graph representations just display one more format to be dealt with before relevant data can be used in respective pipelines.

each data source tends to be handled by a task-specific "python wrapper" 9 and then injected to the actual NLP workflow.


Real user behavior can be unpredictable and vary widely from user to user.

it is difficult for simulated dialogues to capture all aspects of real use, especially the relative frequencies of different kinds of issues.


Each button corresponds to a Wizard action which is then transformed into a sentence (through template-based natural language generation) and provided to the user.

the human user receives system output and responds accordingly.


The features of the profile serve as a guide for the Wizard.

the Wizard uses buttons that generate appropriate language for conversational vs. formal/direct behavior and explicit vs. implicit system responses.


The system policy and the SU do not have access to each other's internal representations and can deal with misunderstandings and missing information by using techniques such as reference resolution.

both the system policy and the SU are very realistic and can work with other users and system policies respectively.


Children with ASD prefer more limited social interaction compared to children without ASD, hence measurement of eye gaze as a screening tool may be an important contribution in this area (Vargas-Cuentas et al., 2017).

the development of innovative assistive technologies can alleviate current challenges and improve diagnostic accuracy.


Each book of hours was a customized devotional object dedicated to a specific patron for whom the content was adapted according to their gender, geographical location, preferred saints and prayers (Clark, 2003), etc.

each book of hours is unique with regard to its content and structure, which make their study highly complex since within the same structure the content may slightly or drastically differ, depending partially on the commissioner, on the crafts(wo)men, on the intended use, and on other phenomena yet to be discovered.


Due to the inclusion of the "biographies" section, the dynastic histories in the corpus contain not just purely historical data but also information on many aspects of everyday life in China, including family stories, where data on gender relations could be found.

it seems natural to do a gender analysis and consider what information about gendered terms could be extracted.


About 8% of the context terms are connected to over 95 target terms for sentences and more than 2000 such terms (around 20%) for paragraphs

at the paragraph level, there is a large group of characters that could be in the same context for male and female terms.


Despite often being thought of as adding complexity, processes like vowel harmony and finalobstruent devoicing improve the predictabil-ity of subsequent segments by constraining the number of well-formed forms.

they reduce complexity measured in bits per phoneme.


We wish to validate our models in a controlled setting, quantifying the contribution of specific linguistic phenomena to our complexity measure.

developing artificial languages, which only differ with respect to one phonological property, is useful.


Note that all Sound Pattern of English (SPE)-style rules may be so encoded (Kaplan and Kay, 1994).

the complexity of the phonotactics could be said to be related to the number of SPE-style rules that operate.


We briefly note that the van Son and Pols (2003) study did not make use of a train/dev/test split of their data, but rather simply analyzed raw relative frequency over their Dutch corpus.

all positions beyond any word onset that is unique in their corpus would have probability 1, leading to a more extreme position effect than we would observe using regularization and validating on unseen forms.


This ability, to compose and decompose questions, lies at the heart of human language (Pelletier, 1994) and allows us to tackle previously unseen problems.

better question understanding models should improve performance and generalization in tasks that require multi-step reasoning or that do not have access to substantial amounts of data.


To estimate expert judgment of correctness, we manually reviewed a random sample of 500 QDMRs from BREAK.

we classified 93.8% of the samples in C G and another 3.6% in C. 97.4% of the samples constitute a correct decomposition of the original question.


QDMR abstracts away from a concrete KB schema by assuming an underlying ''idealized'' KB, which contains all of its arguments.

qDMR can be viewed as an intermediate representation between a natural language question and an executable query.


We rolled out the experiments on AMT over several weeks and prevented users from doing more than one experiment.

a disjoint group of annotators performed each experiment.


It should be noted that there is no single fully reliable metric for evaluating syntactic paraphrase generation.

we evaluate on the following metrics to showcase the efficacy of syntactic paraphrasing models.


Accordingly, the generation script samples from the entire set of verbs and generates the required arguments on-the-fly.

the structure of the sentence then depends on whether the sampled verb is transitive, clauseembedding, raising, and so forth, but that same verb phrase and its arguments are used in both pairs in the paradigm.


As a threshold of inclusion in BLiMP, the majority of validators needed to agree with BLiMP on at least 4/5 examples from each paradigm.

all 67 paradigms in the public version of BLiMP passed this validation; only two additional paradigms were rejected on this criterion.


However, they are not guaranteed to be broad reproducible, because the generalizability of the results might be restricted due to fixed collections of hyperparameter configurations, the variance associated with multiple runs, and the unknown best representative set of MT data.

in this work, we should be careful to not make general conclusions from the observations, but to show how the dataset can be potentially used in facilitating HPO research.


In order to obtain reasonable results, such grammars need to be lexicalized because otherwise the independence assumptions of the PCFG are violated because of semantic relations, for example, between a verb and its subject.

the realizability assumption the approach relies on is dramatically false.


However, the low novelty of the generated objects showed that it could still be difficult for GPT-2 to generate commonsense texts solely based on its implicit knowledge.

we target integrating external knowledge into GPT-2 for generating more reasonable commonsense stories.


GPT-2 network is pretrained on a large-scale corpus but still suffers from many issues such as lack of necessary knowledge for commonsense story generation as aforementioned.

in this work we improve GPT-2 for generating more reasonable stories with external commonsense knowledge.


Although the proposed model outperforms the state-of-the-art baselines, it needs to be noted that there are still many unreasonable stories losing to other models in manual evaluation.

we analyzed error types by manually checking all lost stories in pairwise comparisons between our model and two strong baselines including Fusion and GPT-2 (Fine-tune) to reveal the factors that affect the performance.


Finally, to alleviate the annotation burden, we find possible parallel groups of sentences by matching their document titles and subsection titles, which denote medical PCIO elements, such as the Diagnosis and Symptoms.

we first disambiguate the internal links by matching the document title and its accompanied ICD-9 code.


Style accuracy remains similar among these medical PCIO elements, but there are significant differences among the models in their performance for preserving content.

models perform well for those sentences about treatment, but perform poorly for evaluation, because this type of sentences usually involve many rare terms, challenging understanding.


HiAGM-LA extracts the inductive label-wise text features while HiAGM-TP generates hybrid information in a deductive manner.

hiAGM-LA updates the label embedding across the holistic hierarchy and then employs node outputs as the hierarchy-aware label representations.


Our approaches empirically achieve significant and consistent improvement on three distinct datasets, especially on the low-frequency labels.

both variants outperform the state-of-the-art model on the RCV1-V2 benchmark dataset.


Like in the DGA, we model the stem and affix group forms by means of the associative relationships they create in the mental lexicon.

we predict links without semantic information.


We have split 5366 pairs of HS-CN for training and the rest (1288 pairs) for testing.

the original HS-CN pairs, one HS paraphrase, and the pairs translated from FR and IT were kept for training while the other HS paraphrases were used for testing.


We opted for a scale of 0-3, rather than a CE binary response, since it allows us to study various thresholds for better data selection.

the meanings of the scores are as follows: 0 is not suitable; 1 is suitable with small modifications, such as grammar or semantic; 2 is suitable; and 3 is extremely good as a CN.


Since our proposed methodology is agnostic towards the chosen word emotion model, we will re-use models from the literature.

we will rely on the multi-task learning feed-forward network (MTLFFN) worked out by Buechel and Hahn (2018b).


The annotators were asked to rate each sentence from 0-100 according to the perceived translation quality.

the 0-10 range represents an incorrect translation; 11-29, a translation with few correct keywords, but the overall meaning is different from the source; 30-50, a translation with major mistakes; 51-69, a translation which is understandable and conveys the overall meaning of the source but contains typos or grammatical errors; 70-90, a translation that closely preserves the semantics of the source sentence; and 90-100, a perfect translation.


It deploys a distancebased non-parametric classifier to generate the probability distribution of a slot-value and minimizes the log-likelihood of these values for all slot-types and dialogue turns.

their model includes four main parts: the BERT (Devlin et al., 2019) language model which encodes slot names, slot values, and utterance pairs, a multi-head attention module that computes an attention vector between slot and utterance representations, a RNN state tracking module, and a discriminative classifier which computes the probability of each slot value.


We first attend to some words in the source text, then translate them into the target language, and summarize to get the final summary.

we first employ the encoder-decoder attention distribution to attend to the source words.


This strategy aims to select the correct translation candidates by source-side context.

we first limit the number of translation candidates to at most m, which is consistent with the two strategies above.


The Equal strategy performs worst, but its advantage over the Naive strategy is that it is not affected by the prior probability in probabilistic bilingual lexicon.

the Equal strategy only makes use of the corresponding relationship between source language words and target language words, making it effective even if there is only a bilingual vocabulary dictionary.


We will show that about 30-40% of source documents do not entail their headlines under the widely-used experimental settings.

the current task setting is inappropriate for abstractive summarization.


Although this use of the ROUGE metric is unconventional, the intention here is to measure how many words in a generated headline originate from the input document.

if all words in a generated headline are covered by its source document (truthful), the score is 100; if none of the words in a generated headline originate from its source document (untruthful), the score is 0.


Hence, we hypothesize that the source documents under these task settings contain insufficient information for generating headlines.

headline generation models might be faced with supervision data where headlines cannot be generated from source documents and learned to be untruthful, i.e., producing pieces of information that are not mentioned in source documents.


In this work, we propose a Transformer-based model to enhance the copy mechanism.

we identify the importance of each source word based on the degree centrality with a directed graph built by the self-attention layer in the Transformer.


We utilize the centrality score as guidance for copy distribution.

we extend the dotproduct attention to a centrality-aware function.


We also propose an automatic evaluation model CMADE (Comparison Model for Automatic Dialog Evaluation) that automatically cleans self-reported user ratings as it trains on them.

we first use a self-supervised method to learn better dialog feature representation, and then use KNN and Shapley to remove confusing samples.


The previous works learn the embedding from intra-sentence within a single space, which is not enough for dialog systems.

the semantic correlation beyond a single sentence in the conversation pair is missing.


Our experiment only shows the beam widths up to five because the scores of worse sequences are already higher than that of the correct sequence in some cases.

the larger beam widths seem to be unnecessary.


Motivated by this, we propose P 2 BOT, a transmitter-receiver based framework with the aim of explicitly modeling understanding.

p 2 BOT incorporates mutual persona perception to enhance the quality of personalized dialogue generation.


Note that the transition system is sound and complete, which means there is always a sequence of transitions to sort any sequence into any reordering.

the transition system on its own could also linearize the tree by taking a random permutation as input.


Finally, we incorporate the coverage mechanisms (Tu et al., 2016) to encourage the decoder to utilize diverse components of the input document.

at each step, we maintain a coverage vector cov t , which is the sum of attention distributions over all previous decoder steps.


The two versions of our model -P1 and P2 -consistently outperform all other baselines in BLEU.

our model with DP-based semantic graph (P2) achieves an absolute improvement of 2.05 in BLEU-4 (+15.2%), compared to the document-level QG model which employs gated self-attention and has been enhanced with the same decoder as ours (B5).


Despite their success, previous works on extracting overlapping triples still leave much to be desired.

they all treat relations as discrete labels to be assigned to entity pairs.


It is designed to learn deep representations by jointly conditioning on both left and right context of each word, and it has recently been proven surprisingly effective in many downstream tasks (Zhong et al., 2019).

it is composed of a stack of N identical Transformer blocks.


It can be seen that the performance of most baselines on Normal, EPO and SEO presents a decreasing trend, reflecting the increasing difficulty of extracting relational triples from sentences with different overlapping patterns.

among the three overlapping patterns, Normal class is the easiest pattern while EPO and SEO classes are the relatively harder ones for baseline models to extract.


It reveals that most relations for the entity pairs in extracted triples are correctly identified while some extracted entities fail to form a valid relational triple.

it implies that identifying relations is somehow easier than identifying entities for our model.


We also adopt early stopping mechanism to prevent the model from over-fitting.

we stop the training process when the performance on validation set does not gain any improvement for at least 7 consecutive epochs.


Therefore, we propose a novel TransS-driven joint learning architecture to address the issues.

based on the multi-level encoder, we 1) translate discourse relations in low-dimensional embedding space (called TransS), which could mine the latent geometric structure information of argumentrelation instances; 2) further exploit the semantic features of arguments to assist discourse understanding; 3) jointly learn 1) and 2) to mutually reinforce each other to obtain the better argument representations, so as to improve the performance of the task.


To enrich the discourse argument representations, we exploit multi-level encoder shown in Figure 2 to learn the argument representations at the different levels.

the higher-level states of multilevel encoder could capture context-dependent aspects of words while the lower-level states could model aspects of syntax (Peters et al., 2018).


In our experiments, we train Transformer models (Vaswani et al., 2017), which often achieve state-of-the-art performance on MT for various language pairs.

we rely on the Py-Torch (Paszke et al., 2019) re-implementation of the Transformer model in the fairseq toolkit (Ott et al., 2019).


In our experiments, we train a Transformer model (Vaswani et al., 2017), which achieves state-of-the-art performance on a multitude of language pairs.

we rely on the PyTorch re-implementation of the Transformer model in the Fairseq toolkit (Ott et al., 2019).


It explicitly models the shared semantic space for all languages and acts as a bridge between the Encoder and Decoder network.

we first introduce a language embedding to represent unique characteristics of each language and an interlingua embedding to capture the common semantics across languages.