On one hand, assuming that the parallel sentences have the same meaning, many datasets find the aligned sentences to have higher string overlap (as measured by BLEU).

the two sentences should have different styles, and may vary a lot in expressions: and thus leading to a lower BLEU.


Recent solutions (Gidaris and Komodakis, 2018) leverage a memory component to maintain models' learning experience, e.g., by finding from a supervised stage the content that is similar to the unseen classes, leading to the state-of-the-art performance.

the memory weights are static during inference and the capability of the model is still limited when adapted to new classes.


A larger KES leads to predict more unique keyphrases, append less absolutely incorrect keyphrases and improve the chance to output more accurate keyphrases.

generating more unique keyphrases may also lead to more incorrect predictions, which will degrade the F 1 @M scores since F 1 @M considers all the unique predictions without a fixed cutoff.


For instance, all the baselines produce the keyphrase "debugging" at least three times.

our ExHiRD-h only generates it once, which demonstrates that our proposed method is more powerful in avoiding duplicated keyphrases.


Those models mainly focus on improving decoders based on the constraint of hierarchical paths.

we propose an effective hierarchy-aware global model, HiAGM, that extracts label-wise text features with hierarchy encoders based on prior hierarchy information.


Previous global models classify labels upon the original textual information and improve the decoder with predefined hierarchy paths.

we construct a novel end-to-end hierarchy-aware global model (HiAGM) for the mutual interaction of text features and label correlations.


As for the amount of operators' effort, we observed a slight decrease in HTER with the increase of pre-filtering conditions, indicating an improvement in the quality of candidates.

hTER scores were all between 0.1 and 0.2, much below the 0.4 acceptability threshold defined by Turchi et al.


Finally, we observe that despite reducing the ouput diversity and novelty, the reduction of expert effort by Reviewer≥2 in terms of the percentage of the obtained pairs is not attainable by a machine yet.

automatic filtering (Reviewer machine ) is a viable solution since (i) it helps the NGO operators save time better than human filter ≥1, (ii) it preserves diversity and novelty better than Reviewer≥2 and in line with Reviewer≥1 .


In this scenario, automation strategies, such as natural language generation, are necessary to help NGO operators in their countering effort.

these automation approaches are not mature yet, since they suffer from the lack of sufficient amount of quality data and tend to produce generic/repetitive responses.


However, gold data for the target language (stage) is usually inaccessible, often preventing evaluation against human judgment.

we here propose several alternative evaluation set-ups as an integral part of our methodology.


Words, such as German Sonnenschein for which a translational equivalent already exists in the Source ("sunshine"; see Figure 1), mainly rely on translation, while the prediction step acts as an optional refinement procedure.

the prediction step is crucial for words, such as Erdbeben, whose translational equivalents ("earthquake") are missing in the Source.


We want to point out that not every single entry should be considered meaningful because of noise in the embedding vocabulary caused by typos and tokenization errors.

choosing the "best" size for an emotion lexicon necessarily translates into a quality-coverage trade-off for which there is no general solution.


Neural MT (NMT) approaches have certainly made significant progress in this direction.

the diversity of possible outcomes makes it harder to evaluate MT models.


Dreyer and Marcu (2012) showed that if multiple human translations are used, any automatic MT evaluation metric achieves a substantially higher correlation with human judgments.

multiple translations are hardly ever available in practice due to the cost of collecting them.


The distinction between intended and perceived sarcasm, also referred to as encoded and decoded sarcasm, respectively, has been pointed out in previous research (Kaufer, 1981;Rockwell and Theriot, 2001).

it has not been considered in a computational context thus far when building datasets for textual sarcasm detection.


Text summarization has recently received increased attention with the rise of deep learning-based endto-end models, both for extractive and abstractive variants.

so far, only single-document summarization has profited from this trend.


Apparently, a summarization method is desirable to achieve a ROUGE score of 100, i.e., a system output is identical to the reference.

this is an unrealistic goal for the task setting on the Gigaword dataset.


They addressed the problem where an abstractive model made mistakes in facts (e.g., tuples of subjects, predicates, and objects).

we also regularly see examples where the abstractive model generates unexpected words.


For JNC, we use the pretrained BERT model for Japanese text (Kikuta, 2019).

no large-scale Japanese corpus for semantic inference (counterpart to MultiNLI) is available.


We could confirm the improvements from the support scores, entailment ratio, and human judgments.

the ROUGE scores between system and reference headlines did not indicate a clear difference.


Likert-score based self-reported user rating is widely adopted by social conversational systems, such as Amazon Alexa Prize chatbots.

selfreported user rating suffers from bias and variance among different users.


ELMo (Peters et al., 2018), BERT (Devlin et al., 2019), and XLNet (Yang et al., 2019) have achieved great success in many NLP tasks.

it is difficult to apply them in the industrial dialog system due to their low computational efficiency.


Previous work has demonstrated that neural encoders capture a rich hierarchy of syntactic and semantic information (Jawahar et al., 2019;Clark et al., 2019).

reasoning capability and commonsense knowledge are not captured sufficiently (Young et al., 2018).


BERT-MC and RoBERTa-MC obtain similar results with BERT and RoBERTa, respectively.

even RoBERTa is far behind human performance 23 points on R@1, indicating that MuTual is indeed a challenging dataset, which opens the door for tackling new and complex reasoning problems in multi-turn conversations.


The language score is evaluated individually, without considering the discourse coherence.

a reasonable response should establish links in meaning with context, which is also an important aspect of humanlike responses.


In practice, the solver can always find a solution to linearize the subtrees with the constraints.

it sometimes cannot find any solution to directly linearize the full tree within the time limit (1-10% of the cases depending on the treebank), because there are more nodes and more constraints in the full tree.


However, these approaches only consider sentence-level QG.

our work focus on the challenge of generating deep questions with multi-hop reasoning over document-level contexts.


Among them, "Semantic Error", "Redundant", and "Unanswerable" are noticeable errors for all models.

we find that baselines have more unreasonable subject-predicate-object collocations (semantic errors) than our model.


Extracting relational triples from unstructured text is crucial for large-scale knowledge graph construction.

few existing works excel in solving the overlapping triple problem where multiple relational triples in the same sentence share the same entities.


To reduce manual work, recent studies have investigated neural network-based methods, which deliver state-of-the-art performance.

most existing neural models like (Miwa and Bansal, 2016) achieve joint learning of entities and relations only through parameter sharing but not joint decoding.


Therefore, the object tagger for relation "Work in" will not identify the span of "Washington", i.e., the output of both start and end position are all zeros as shown in Figure 2.

the relation "Birth place" holds between "Jackie R. Brown" and "Washington", so the corresponding object tagger outputs the span of the candidate object "Washington".


Such inconsistent data distribution of two datasets leads to a comparatively better performance on NYT and a worse performance on WebNLG for all the baselines, exposing their drawbacks in extracting overlapping relational triples.

the CASREL model and its variants (i.e., CASREL random and CAS-REL LST M ) all achieve a stable and competitive performance on both NYT and WebNLG datasets, demonstrating the effectiveness of the proposed framework in solving the overlapping problem.


In other words, it implies that identifying relations is somehow easier than identifying entities for our model.

to NYT, for WebNLG, the performance gap between (E1, E2) and (E1, R, E2) is comparatively larger than that between (E1, R, E2) and (E1, R)/(R, E2).


We also find that there is only a trivial gap between the F1-score on (E1, E2) and (E1, R, E2), but an obvious gap between (E1, R, E2) and (E1, R)/(R, E2).

it reveals that most relations for the entity pairs in extracted triples are correctly identified while some extracted entities fail to form a valid relational triple


Information Extraction (IE), and specifically Relation Extraction (RE), can improve the access to central information for downstream tasks (Santos et al., 2015;Zeng et al., 2014;Jiang et al., 2016;Miwa and Bansal, 2016;Luan et al., 2018a).

the focus of current RE systems and datasets is either too narrow, i.e., a handful of semantic relations, such as 'USED-FOR' and 'SYNONYMY', or too broad, i.e., an unbounded number of generic relations extracted from large, heterogeneous corpora (Niklaus et al., 2018), referred to as Open IE (OIE) (Etzioni et al., 2005;Banko et al., 2007).


It has been shown that scientific texts contain many unique relation types and, therefore, it is not feasible to create separate narrow IE classifiers for these (Groth et al., 2018).

oIE systems are primarily developed for the Web and news-wire domain and have been shown to perform poorly on scientific texts.


Lei et al. (2017) conduct word pair interaction score to capture both linear and quadratic relation for argument representation.

these methods utilize the pre-trained embeddings for mining the interaction features and ignore the geometric structure information entailed in discourse arguments and their relation.


Xu et al. (2019) propose a topic tensor network (TTN) to model the sentence-level interactions and topic-level relevance among arguments for this task.

few studies model discourse relations by translating them in the low-dimensional embedding space as we do in this work.


With the increasing of the number of encoder layers, the model could capture the richer semantic information.

the results imply that with the more encoder layers considered, the model could incur the over-fitting problem due to adding more parameters.


Based on BLEU (Papineni et al., 2002) scores, previous work (Belinkov et al., 2017) suggests that translating into morphologically rich languages, such as Hungarian or Finnish, is harder than translating into morphologically poor ones, such as English.

a major obstacle in the crosslingual comparison of MT systems is that many automatic evaluation metrics, including BLEU and METEOR (Banerjee and Lavie, 2005), are not cross-lingually comparable.


Reliable reference-free evaluation metrics, directly measuring the (semantic) correspondence between the source language text and system translation, would remove the need for human references and allow for unlimited MT evaluations: any monolingual corpus could be used for evaluating MT systems.

the proposals of referencefree MT evaluation metrics have been few and far apart and have required either non-negligible supervision (i.e., human translation quality labels) (Specia et al., 2010) or language-specific preprocessing like semantic parsing (Lo et al., 2014;Lo, 2019), both hindering the wide applicability of the proposed metrics.


Position encoding (PE), an essential part of self-attention networks (SANs), is used to preserve the word order information for natural language processing tasks, generating fixed position indices for input sequences.

in cross-lingual scenarios, e.g., machine translation, the PEs of source and target sentences are modeled independently.


The filtering step as performed by Grave et al. (2018) consisted in only keeping the lines exceeding 100  bytes in length.

considering that Common Crawl is a mutilingual UTF-8 encoded corpus, this 100-byte threshold creates a huge disparity between ASCII and non-ASCII encoded languages.


Considering the discussion above, we believe an interesting follow-up to our experiments would be training the ELMo models for more of the languages included in the OSCAR corpus.

training ELMo is computationally costly, and one way to estimate this cost, as pointed out by Strubell et al. (2019), is by using the training times of each model to compute both power consumption and CO2 emissions.


Even though it would have been interesting to replicate all our experiments and computational cost estimations with state-of-the-art fine-tuning models such as BERT, XLNet, RoBERTa or AL-BERT, we recall that these transformer-based architectures are extremely costly to train, as noted by the BERT authors on the official BERT GitHub repository, 21 and are currently beyond the scope of our computational infrastructure.

we believe that ELMo contextualized word embeddings remain a useful model that still provide an extremely good trade-off between performance to training cost, even setting new state-of-the-art scores in parsing and POS tagging for our five chosen languages, performing even better than the multilingual mBERT model.


All neural models achieve a SG score significantly greater than a random baseline (dashed line).

the range within neural models is notable, with the bestperforming model (GPT-2-XL) scoring over twice as high as the worst-performing model (LSTM).


The TTG ap-proach cannot make use at run-time of an En QE model without translating the caption back to English and thus again requiring perfect translation in order not to ruin the predicted quality score.

the PLuGS approach appears to be best suited for leveraging an existing En QE model, due to the availability of the generated bilingual output that tends to maintain consistency between the generated EN-& X-language outputs, with respect to accuracy; therefore, directly applying an English QE model appears to be the most appropriate scalable solution.


They read one sentence at a time and provided a suspense judgement using the fivepoint scale consisting of Big Decrease in suspense (1% of the cases), Decrease (11%), Same (50%), Increase (31%), and Big Increase (7%).

to prior work (Delatorre et al., 2018), a relative rather than absolute scale was used.


One of our main hypotheses is that modeling edit sequences is better suited for this task than generating comments from scratch.

a counter argument could be that a comment generation model could be trained from substantially more data, since it is much easier to obtain parallel data in the form (method, comment), without the constraints of simultaneous code/comment edits.


Most of these approaches focus on code summarization or comment generation which only require single code-NL pairs for training and evaluation as the task entails generating a natural language summary of a given code snippet.

our proposed task requires two code-NL pairs that are assumed to hold specific parallel relationships with one another.


FACTEDITOR has a larger number of correct editing (CQT) than ENCDECEDITOR for fact-based text editing.

eNCDeCeDITOR often includes a larger number of unnecessary rephrasings (UPARA) than FACTeDITOR.


ENCDECEDITOR often generates the same facts multiple times (RPT) or facts in different relations (DREL).

fACTE-DITOR can seldomly make such errors.


ENCDECEDITOR cannot effectively eliminate the description about an unsupported fact (in orange) appearing in the draft text.

fACTEDI-TOR can deal with the problem well.


Gehrmann et al. (2018) utilized a data-efficient content selector, by aligning source and target, to restrict the model’s attention to likely-to-copy phrases.

we use the content selector to find domain knowledge alignment between source and target.


Such data-driven approaches achieve good performance on several benchmarks like E2E challenge (Novikova et al., 2017), WebNLG challenge (Gardent et al., 2017) and WIKIBIO (Lebret et al., 2016).

they rely on massive amount of training data.


Ma et al. (2019) propose low-resource table-to-text generation with 1,000 paired examples and large-scale target-side examples.

in our setting, only tens to hundreds of paired training examples are required, meanwhile without the need for any target examples.


This spillover is potentially sensitive only to Levenshtein distance.

confusability is sensitive to fine-grained perceptual structure.


For this case, IG, IC, and subjectivity all have overlapping confidence intervals, so we conclude that there is no evidence that one is better than the other.

we do have evidence that IG and IC are more accurate than PMI when estimated based on clusters.


The idea of adding the NOTA option to a candidate set is also widely used in other language technology fields like speaker verification (Pathak and Raj, 2013).

the effect of adding NOTA is rarely introduced in dialog retrieval research problems.


D-GPT gets the wrong answer 18% of the time (option a and c), because the input answer predicted by the CoQA baseline is also incorrect 17% of the time.

with oracle answers, it is able to generate correct responses 77% of the times (option e).


Question Answering Using crowd-sourcing methods to create QA datasets (Rajpurkar et al., 2016;Bajaj et al., 2016;Rajpurkar et al., 2018), conversational datasets (Dinan et al., 2018), and ConvQA datasets (Choi et al., 2018;Reddy et al., 2019;Elgohary et al., 2018;Saha et al., 2018) has largely driven recent methodological advances.

models trained on these ConvQA datasets typically select exact answer spans instead of generating them (Yatskar, 2019b).


As shown in Table 1, the hidden dimension of each building block is only 128.

we introduce two linear transformations for each building block to adjust its input and output dimensions to 512.


One may either only use the bottlenecks for MobileBERT (correspondingly the teacher becomes BERT LARGE ) or only the invertedbottlenecks for IB-BERT (then there is no bottleneck in MobileBERT) to align their feature maps.

when using both of them, we can allow IB-BERT LARGE to preserve the performance of BERT LARGE while having MobileBERT sufficiently compact.


This has enabled ever-increasing performance on benchmark data sets.

one thing has remained relatively constant: the softmax of a dot product as the output layer.


Recently Graph Neural Network (GNN) has shown to be powerful in successfully tackling many tasks.

there has been no attempt to exploit GNN to create taxonomies.


We get better performance if we tune the thresholds.

we chose a harder task and proved our model has better performance than others even we simply use 0.5 as the threshold.


The model identifies words related to community ("kids," "neighborhood," "we") as strong negative signals for depression, supporting that depressed language reflects detachment from community.

the model only focuses on these semantic themes in responses to generic backchannel categories.


Thanks to the increased complexity of deep neural networks and use of knowledge transfer from the language models pretrained on large-scale corpora (Peters et al., 2018; Devlin et al., 2019; Dong et al., 2019), the state of-the-art QA models have achieved human-level performance on several benchmark datasets (Rajpurkar et al., 2016, 2018)

what is also crucial to the success of the recent data-driven models, is the availability of large-scale QA datasets


Some of the recent works resort to semi-supervised learning, by leveraging large amount of unlabeled text (e.g. Wikipedia) to generate synthetic QA pairs with the help of QG systems (Tang et al., 2017; Yang et al., 2017; Tang et al., 2018; Sachan and Xing, 2018).

existing QG systems have overlooked an important point that generating QA pairs from a context consisting of unstructured texts, is essentially a one-to-many problem.


They should be semantically consistent, such that it is possible to predict the answer given the question and the context.

neural QG or QAG models often generate questions irrelevant to the context and the answer (Zhang and Bansal, 2019) due to the lack of the mechanism enforcing this consistency.


For example, when both models are trained with 1% of the Yelp dataset, the accuracy gap is around 9%.

as we increases the amount of training data to 90%, the accuracy gap drops to within 2%.


Upon further investigation, we find that experiments which use probabilities with image based features have an inter-quartile range of 0.05 and 0.1 for EBG and BLOG respectively whereas for experiments using probabilities with binning based features, this range is 0.32 for both datasets.

inter-quartile range for exper-iments using ranks with image based features is 0.08 and 0.05 for EBG and BLOG whereas for experiments using ranks with binning based features, this range is 0.49 and 0.42 respectively.


If we have a large number of training samples, the architecture is capable of learning how to discriminate correctly between classes only with the original training data.

in less-resourced scenarios, our proposed approaches with external knowledge integration could achieve a high positive impact.


These offer obvious benefits to users in terms of immediacy, interaction and convenience.

it remains challenging for application providers to assess language content collected through these means.


We make our code publicly available for others to use for benchmarking and replication experiments.

to feature-based scoring, we instead train neural networks on ASR transcriptions which are labeled with proficiency scores assigned by human examiners, and guide the networks with objectives that prioritize language understanding.


These results were not significantly better than the single-task POS prediction model, though we did not explore tuning the alpha weighting values for the combination models.

bERT only receives a significant improvement in grading ability when using the L1 prediction task.


Figure 4 shows, as expected, that training a speech grader with data from an ASR system with lower word error rates produces better results.

it is interesting to note that this holds true even when evaluating with data from inferior ASR systems.


This is because SciBERT, like other pretrained language models, is trained via language modeling objectives, which only predict words or sentences given their in-document, nearby textual context.

we propose to incorporate citations into the model as a signal of inter-document relatedness, while still leveraging the model's existing strength in modeling language.


The candidate program should adhere to the grammatical specification of the target language.

since incorporating the complete set of C++ grammatical constraints would require significant engineering effort, we instead restrict our attention to the set of "primary expressions" consisting of high-level control structures such as if, else, for loops, function declarations, etc.


For example, when there is only one statement within an if statement, the programmer can optionally include a curly brace.

the pseudocode does not contain such detailed information about style.


This error can be ruled out by SymTable constraint if variable A is undeclared.

symTable constraints do not preclude all errors related to declarations.


Prior works regarded SQG as a dialog generation task and recurrently produced each question.

they suffered from problems caused by error cascades and could only capture limited context dependencies.


We expect that this category is rare because the premise is not text.

since there are some textual elements in the tables, the hypothesis could paraphrase them.


Finally, research into multimodal or multi-view deep learning (Ngiam et al., 2011; Li et al., 2018) offers insights to effectively combine multiple data modalities or views on the same learning problem.

most work does not directly apply to our problem: i) the audio-text modality is significantly under-represented, ii) the models are typically not required to work online, and iii) most tasks are cast as document-level classification and not sequence labeling (Zadeh et al., 2018).


Current ASR approaches rely solely on utilizing audio input to produce transcriptions.

the wide availability of cameras in smartphones and home devices acts as motivation to build AV-ASR models that rely on and benefit from multimodal input.


Even for datasets with dialogue taking place in a similar domain as improv, they naturally contain only a small proportion of yes-ands.

the relatively large sizes of these datasets still make them useful for dialogue systems.


Their model uses an exemplar sentence as a syntactic guide during generation; the generated paraphrase is trained to incorporate the semantics of the input sentence while emulating the syntactic structure of the exemplar (see Appendix D for examples).

their proposed approach depends on the availability of such exemplars at test time; they manually constructed these for their test set (800 examples).


Recent work on controlled generation aims at controlling attributes such as sentiment (Shen et al., 2017), gender or political slant (Prabhumoye et al., 2018), topic (Wang et al., 2017), etc.

these methods cannot achieve fine-grained control over a property like syntax.


In the context of translation, there is often a canonical reordering that should be applied to align better with the target language; for instance, head-final languages like Japanese exhibit highly regular syntax-governed reorderings compared to English.

in diverse paraphrase generation, there doesn't exist a single canonical reordering, making our problem quite different.


Still, dialogue research papers tend to report scores based on word-overlap metrics from the machine translation literature (e.g. BLEU (Papineni et al., 2002), METEOR (Denkowski and Lavie, 2014)).

word-overlap metrics aggressively penalize the generated response based on lexical differences with the ground truth and correlate poorly to human judgements (Liu et al., 2016).


Lowe et al. (2017) propose a learned referenced metric named ADEM, which learns an alignment score between context and response to predict human score annotations.

since the score is trained to mimic human judgements, it requires collecting large-scale human annotations on the dataset in question and cannot be easily applicable to new datasets (Lowe, 2019).


Recently, Tao et al. (2017) proposed a hybrid referenced-unreferenced metric named RUBER, where the metric is trained without requiring human responses by bootstrapping negative samples directly from the dataset.

referenced metrics (including RUBER, as it is part referenced) are not feasible for evaluation of dialogue models in an online setting-when the model is pitched against a human agent (model-human) or a model agent (model-model)-due to lack of a reference response.


All models achieve high scores on the semantic positive samples when only trained with syntactical adversaries.

training only with syntactical negative samples results in adverse effect on detecting semantic negative items.


It has shown that response timings vary based on the semantic content of dialogue responses and the preceding turn (Levinson and Torreira, 2015), and that listeners are sensitive to these fluctuations in timing (Bogels and Levinson, 2017).

the question of whether certain response timings within different contexts are considered more realistic than others has not been fully investigated.


While Step-By-Step uses heuristic string matching to extract plans from the referenced sentences, other methods (GRU and transformer), as well as ours, use plans provided in the enriched WebNLG dataset (Castro Ferreira et al., 2018).

step-By-step reported worse BLEU results on these plans.


GCN does not perform well on Coverage, which demonstrates that the structural gap between encoding and decoding indeed makes generation more difficult.

it has the smallest difference between Coverage and Faithfulness among all the baselines, indicating that the fidelity of generation can benefit from the encoding of graph-level structural information.


These existing neural models have achieved encouraging results.

when a new condition is added (e.g., a new topic for categorical generation), they require a full retraining or finetuning.


Many great works have attempted to solve various subtasks like dialogue generation (Li et al., 2016), poetry generation (Yi et al., 2018) and story generation (Fan et al., 2018) and new techniques keep emerging (Bowman et al., 2016;Yu et al., 2017;Zhou et al., 2020).

due to the blackbox nature of neural networks, the recent proposed generic models suffer the problem of lacking interpretability and controllability.


Previous methods (Kingma et al., 2014; Hu et al., 2017) learn the joint conditional space by jointly considering all conditions.

once the model is trained, it is not possible to add a new condition without a full retraining.


These methods collect user feedback after the model-predicting stage and treat user feedback as additional offline training data to improve the model.

our model leverages user interaction to increase prediction performance.


Pretrained autoregressive models such as GPT (Radford et al., 2018, 2019) are especially capable of generating fluent and coherent text that highly resembles human-written text

unidirectional attention brings two limitations.


This can result in different computational complexity.

since a typical Graphics Processing Unit (GPU) computes matrices in parallel, the actual difference in inference time is not that significant.


The Wikitext103 dataset is more similar to the pretraining datasets, containing long articles.

the One-Billion Words dataset contains only single sentences, roughly half of which contain less than 24 tokens.


Besides, the results show that there are few differences between relative positional embedding and absolute positional embedding for u-PMLM.

although BERT supports generation in arbitrary word order as well, the PPL for BERT is significantly worse than our proposed u-PMLM for both "sequential" and "random" settings, demonstrating the effectiveness of the proposed probabilistic masking scheme.


We show more cases of text generation in random order for u-PMLM-A and BERT in Appendix B.

for PPL on One-Billion Words, the performances of u-PMLM and BERT are not satisfactory in comparison with GPT.


For GPT, the input text can only be placed in the beginning and the generation process become uncontrollable, resulting in generating sentences with topic drift.

u-PMLM allows manually placing anchor sentences in the middle or end of the generated text to guide the topic of the generated text.


Existing uses of pretrained MLMs in sequenceto-sequence models for automatic speech recognition (ASR) or neural machine translation (NMT) involve integrating their weights (Clinchant et al., 2019) or representations (Zhu et al., 2020) into the encoder and/or decoder during training.

we train a sequence model independently, then rescore its n-best outputs with an existing MLM.


As the MLM gets stronger, the improvement from adding scores from GPT-2 goes to zero, suggesting that their roles overlap at the limit.

unlike recent work (Shin et al., 2019) but like previous work (Chen et al., 2017), we found that interpolating with a unidirectional LM remained optimal, though our models are trained on different datasets and may have an ensembling effect.


In the IID setting, large pretrained Transformer models can attain near human-level performance on numerous tasks (Wang et al., 2019)

high IID accuracy does not necessarily translate to OOD robustness for image classifiers (Hendrycks and Dietterich, 2019), and pretrained Transformers may embody this same fragility.


The recent work of Pruthi et al. (2019), which uses a typo-corrector to defend against adversarial typos, is such a reusable defense: it is trained once, then reused across different tasks.

we find that current typo-correctors do not perform well against even heuristic attacks, limiting their applicability.


During training, each occurrence of "at" and "abet" is replaced with z.

since "at" is much more frequent, classifiers treat z similarly to "at in order to achieve good overall performance.


This data consists of approximately 2 million instances constructed using the abstract and body structure of Wikipedia.

our ap-proach to pre-training can generate data in unlimited quantity from any text source without assuming a particular document structure.


Both improve with multiple iterations, though the improvement is much larger with CMLM.

even with  10 iterations, ENGINE is comparable to CMLM on DE-EN and outperforms it on RO-EN


As in EWISE, in EWISER logits are computed by a dot product between a matrix of hidden scores and output synset embeddings.

we do not train our own synset embeddings: rather, we employ off-the-shelf vectors.


Consequently, the general-language and domain-specific contexts are maximally similar in these cases.

we assume that the contexts will vary more strongly for basic terms, and for non-terms we do not expect to find domain-specific sentences in the generallanguage corpus at all.


Recent research on fairness has primarily focused on racial and gender biases within distributed word representations (Bolukbasi et al., 2016), coreference resolution (Rudinger et al., 2018), sentence encoders (May et al., 2019), and language models .

we posit that there exists a significant potential for linguistic bias that has yet to be investigated, which is the motivation for our work.


In the context of question answering, SpanBERT appears to be slightly more robust than vanilla BERT when comparing overall performance on the two SQuAD datasets.

the difference becomes significant if we look only at the SQuAD 2.0-fine-tuned models' performance on answerable questions (7% difference).


Existing adversarial training approaches have shown that retraining the model on the augmented training set improves robustness (Belinkov and Bisk, 2018; Eger et al., 2019; Jin et al., 2019).

this requires substantial compute resources.


As depicted in Figure 1, we are interested in identifying clusters of subtly distinctive glyph shapes as these correspond to distinct metal stamps in the type-cases used by printers.

other sources of variation (inking, for example, as depicted in Figure 1) are likely to dominate conventional clustering methods.


For example, Gulcehre et al. (2014) explored the influence of selecting different pooling norms on the performance of different image classification tasks.

the norms in their method are manually tuned, which are usually very time-consuming and may not be optimal.


Multi-task learning (MTL) and transfer learning (TL) are techniques to overcome the issue of data scarcity when training state-of-theart neural networks.

finding beneficial auxiliary datasets for MTL or TL is a time-and resource-consuming trial-and-error approach.


This is reasonable as the "O" labels by far make up the majority of all labels in NER datasets.

this does not help to find similar dataset in other cases, because there is no meaningful ordering of the entropy values when comparing any of the POS samples with all the other samples.


The incorporation of pseudo-tags is a standard technique widely used in the NLP community, (Rico et al., 2016; Melvin et al., 2017).

to the best of our knowledge, our approach is the first attempt to incorporate pseudo-tags as an identification marker of virtual models within a single model.


As displayed in Table 2, SINGLEENS surpassed SINGLE by 0.44 and 0.14 on CoNLL-2003 and CoNLL-2000, respectively, for TFM:ELMO with the same parameter size.

nORMALEnS produced the best results in this setting.


Multi-modal neural machine translation (NMT) aims to translate source sentences into a target language paired with images.

dominant multi-modal NMT models do not fully exploit fine-grained semantic correspondences between semantic units of different modalities, which have potential to refine multi-modal representation learning.


Non-autoregressive neural machine translation (NAT) predicts the entire target sequence simultaneously and significantly accelerates inference process.

nAT discards the dependency information in a sentence, and thus inevitably suffers from the multi-modality problem: the target tokens may be provided by different possible translations, often causing token repetitions or missing.


Therefore, the decoder has richer target-side information to detect and recover from such errors.

it is non-trivial to train the model to learn such behaviour while maintaining a reasonable speedup.


The study of calibration on classification tasks has a long history, from statistical machine learning (Platt et al., 1999;Niculescu-Mizil and Caruana, 2005) to deep learning (Guo et al., 2017).

calibration on structured generation tasks such as neural machine translation (NMT) has not been well studied.


Modern neural networks have been found to be miscalibrated on classification tasks in the direction of overestimation (Guo et al., 2017).

nMT models also suffer from under-estimation problems.


For instance, if the spam candidate mutates at the critical positions, the label of the augmented text is likely to change.

normal candidates are less likely to be affected by this situation.


Since manual coding is very laborious and prone to errors, many methods have been proposed for the automatic ICD coding task.

most of existing methods independently predict each code, ignoring two important characteristics: Code Hierarchy and Code Co-occurrence.


(2) The effectiveness of hyperbolic representations: Our proposed model and the CNN+Attention can both correctly predict the code "518.81".

the CNN+Attention model gives contradictory predictions.


For example, label "movie" should have the largest scalar projection onto a document about "movie".

even the learned label representation of "music" can be distinguished from "movie", it may also have a large scalar projection onto the document.


(Zhao et al., 2019) improves the scalability of capsule networks for MLC.

they only use CNN to construct capsules, which capture local contextual information (Wang et al., 2016).


The typical MLC method SLEEC takes advantage of label correlations by embedding the label co-occurrence graph.

sLEEC uses TF-IDF vectors to represent documents, thus word order is also ignored.


REGGNN is generally superior to both of them as it combines the local and global contextual information dynamically and takes label correlations into consideration using a regularized loss.

the two capsulebased methods NLP-CAP and HYPERCAPS consistently outperform all the other methods owing to dynamic routing, which aggregates the fine-grained capsule features in a label-aware manner.


In the hospitals, the doctors will make a comprehensive analysis mainly based on CC, HPI, PE, TR and the basic information, and make a diagnosis.

it is very hard for computers to automatically understand all the diverse sections and capture the key information before making an appropriate diagnosis.


Zhang et al. (2017) combines the variational auto-encoder and the variational recurrent neural network together to make diagnosis based on laboratory test data.

laboratory test data are not the only resources considered in this paper.


Although ECNN also outputs a probability distribution over all diseases, the result is not interpretable due to its end-to-end nature.

the interpretability is very important in the CDS to explain how the diagnosis is generated by machines.


Remarkable success has been achieved when sufficient labeled training data is available.

annotating sufficient data is labor-intensive, which establishes significant barriers for generalizing the stance classifier to the data with new targets.


The aspect-opinion pairs can provide a global profile about a product or service for consumers and opinion mining systems.

traditional methods can not directly output aspect-opinion pairs without given aspect terms or opinion terms.


As the example sentence shown in Figure 1, (service, great), (prices, great) and (atmosphere, nice friendly) are three aspect-opinion pairs.

the co-extraction methods can only output the AT set {service, prices, atmosphere} and the OT set {great, nice friendly} jointly.


Most of the previous AT and OT extraction methods formulate the task as a sequence tagging problem (Wang et al., 2016, 2017; Wang and Pan, 2018; Yu et al., 2019), specifically using a 5-class tag set: {BA (beginning of aspect), IA (inside of aspect), BP (beginning of opinion), IP (inside of opinion), O (others)}.

the sequence tagging methods suffer from a huge search space due to the compositionality of labels for extractive ABSA tasks, which has been proven in (Lee et al., 2017b;Hu et al., 2019).


Motivated by the correlations between the two tasks, SRL has been utilized to help the ORL task by many previous studies (Ruppenhofer et al., 2008; Marasovic and Frank, 2018; Zhang et al., 2019b).

when opinion expressions and arguments compose complicated syntactic structures, it is difficult to correctly recognize the opinion arguments even with shallow semantic representation like SRL (Marasovic and Frank, 2018).


Specifically, the pipeline way first trains the dependency parser and then fixes the parser components during training the ORL model.

the MTL way trains both the parser and the ORL model at the same time.


As a baseline, Figure 2-(c) shows the most common MTL method, which shares a common encoder and uses multiple task-specific output layers, known as the hard-parameter-sharing MTL (Ruder, 2017; Marasovic and Frank, 2018).

this approach is not suitable for our scenario where the auxiliary parsing task has much more labeled data than the main ORL task, since the shared encoder is very likely to bias toward to parsing performance (Xia et al., 2019a).


For example, Stab and Gurevych (2017) introduced Argument Annotated Essays (hereafter, Essay), and researchers attempted to predict tree arguments in the corpus (Eger et al., 2017;Potash et al., 2017;Kuribayashi et al., 2019).

these techniques lack the capability of dealing with more flexible arguments such as reason edges where a proposition can have several parents.


Potash et al. (2017) developed a pointer network architecture to predict edges.

we cannot simply utilize them for non-tree arguments because these models were built upon the assumption that an argument forms a tree structure.


However, if one wants to apply WSD to some specific corpus, additional annotated training data might be required to meet the similar performance as ours, which defeats the purpose of a weakly supervised setting.

our contextualization, building upon (Devlin et al., 2019), is adaptive to the input corpus, without requiring any additional human annotations.


In micro average, all the span predictions are aggregated together and then compared with the gold spans to get the precision and recall.

macro average is obtained by calculating the F1 score for each individual sentence and then take an average over all the sentences.


Multi-threading is employed since sentences are mutually independent.

we find that using more than 4 threads does not further improve the speed.


We can see that the performance gap is quite steady when we gradually reduce the number of training sentences.

the gap clearly becomes larger when each training sentence has less annotated dependencies.


In BiLSTM-CRF, the CRF layer models the relation between neighbouring labels which leads to better results than simply predicting each label separately based on the BiLSTM outputs.

the CRF structure models the label sequence globally with the correlations between neighboring labels, which increases the difficulty in distilling the knowledge from the teacher models.


In particular, discarding the conversion matrix in the ESD module also leads to the performance drop, which indicates the usefulness of capturing the label correspondence between the auxiliary module and our main MNER task.

as the main contribution of  our MMI module, Image-Aware Word Representations (WR) demonstrates its indispensable role in the final performance due to the moderate performance drop after removal.


These neural approaches have been shown to achieve the state-of-the-art performance on different benchmark datasets based on formal text (Yang et al., 2018).

when applying these approaches to social media text, most of them fail to achieve satisfactory results.


Despite not being exposed to explicit syntactic supervision, neural language models (LMs), such as recurrent neural networks, are able to generate fluent and natural sentences, suggesting that they induce syntactic knowledge about the language to some extent.

it is still under debate whether such induced knowledge about grammar is robust enough to deal with syntactically challenging constructions such as long-distance subjectverb agreement.


We expect that the main reason for lower performance for object RCs is due to frequency, and with our augmentation the accuracy will reach the same level as that for subject RCs.

for both all and animate cases, accuracies are below those for subject rCs (Figure 2).


Moreover, Huang et al. (2019) improve TextGCN by introducing the message passing mechanism and reducing the memory consumption

there are two major drawbacks in these graph-based methods.


A concurrent work (Warstadt et al., 2019b) facilitates diagnosing language models by creating linguistic minimal pairs datasets for 67 isolate grammatical paradigms in English using linguistcrafted templates.

we do not rely heavily on artificial vocabulary and templates.


Such attention weights measure the relative importance of the token within a specific input sequence.

the attention score a j captures the absolute importance of the token.


After that, both of them directly use the word representation of two languages to retrieve the initial bilingual lexicons by computing the cosine distances of source and target word representations.

directly finding word alignments from scratch has some demerits.


Unsupervised neural machine translation (UNMT) has recently achieved remarkable results for several language pairs.

it can only translate between a single language pair and cannot produce translation results for multiple language pairs at the same time.


For example, Xu et al. (2019) and Sen et al. (2019) proposed a multilingual scheme that jointly trains multiple languages with multiple decoders.

the performance of their MUNMT is much worse than our re-implemented individual baselines (shown in Tables 2 and 3) and the scale of their study is modest (i.e., 4-5 languages).


Moreover, the MUNMT model could alleviate the poor performance achieved with low-resource language pairs, such as En-Lt and En-Lv.

the performance of MUNMT is slightly worse than SM in some language pairs.


The standard training algorithm in neural machine translation (NMT) suffers from exposure bias, and alternative algorithms have been proposed to mitigate this.

the practical impact of exposure bias is under debate.


Previous work has sought to reduce exposure bias in training (Bengio et al., 2015; Ranzato et al., 2016; Shen et al., 2016; Wiseman and Rush, 2016; Zhang et al., 2019).

the relevance of error propagation is under debate: Wu et al. (2018) argue that its role is overstated in literature, and that linguistic features explain some of the accuracy drop at higher time steps.


Advanced pre-trained models for text representation have achieved state-of-the-art performance on various text classification tasks.

the discrepancy between the semantic similarity of texts and labelling standards affects classifiers, i.e. leading to lower performance in cases where classifiers should assign different labels to semantically similar texts.


In general, AAN achieved greater performance than AM.

their effectiveness turned out to be task-dependent.


Previous studies aimed to improve multiple tasks; hence, they required multiple sets of annotated datasets.

our method does not require any extra labelled datasets and is easily applicable to various classification tasks.


On top of it, Crosslingual training (or bilingual, denoted by "Cross") obtains marginal improvements for moderately lowresource languages.

the performance drops dramatically for two extremely low-resource languages, i.e., JA from 0.740 to 0.711 and EL from 0.702 to 0.684.


Previous studies in multimodal sentiment analysis have used limited datasets, which only contain unified multimodal annotations.

the unified annotations do not always reflect the independent sentiment of single modalities and limit the model to capture the difference between modalities.


An intuitive idea is that the greater the difference between inter-modal representations, the better the complementarity of intermodal fusion.

it is not easy for existing late-fusion models to learn the differences between different modalities, further limits the performance of fusion.


The CHEAVD (Li et al., 2017) is also a Chinese multimodal dataset, but it only contains two modalities (vision and audio) and one unified annotation.

sIMs has three modalities and unimodal annotations except for multimodal annotations for each clip.


However, these existing multimodal datasets only contain a unified multimodal annotation for each multimodal corpus.

sIMs contains both unimodal and multimodal annotations.


Hierarchical FM performs better than MLP+CNN by incorporating additional attributes that provide the visual semantic information and generating better feature representations via a hierarchical fusion framework.

these multimodal baselines pay more attention to the fusion of multimodal features.


Thus, their performances are worse than MIARN, which focuses on textual context to model the contrast information between individual words and phrases.

due to the nature of short text, relying on textual information is often insufficient, especially in multimodal tweets where cross-modality context relies the most important role.


As interpretability is important for understanding and debugging the translation process and particularly to further improve NMT models, many efforts have been devoted to explanation methods for NMT (Ding et al., 2017;Alvarez-Melis and Jaakkola, 2017;Li et al., 2019;Ding et al., 2019;.

little progress has been made on evaluation metric to study how good these explanation methods are and which method is better than others for NMT.


In terms of i), Word Alignment Error Rate (AER) can be used as a metric to evaluate an explanation method by measuring agreement between human-annotated word alignment and that derived from the explanation method.

aER can not measure explanation methods on those target words that are not aligned to any source words according to human annotation.


On one hand, the real data distribution of c t is unknowable, making it impossible to exactly define the expectation with respect to an unknown distribution.

the domain of a proxy model Q is not bounded, and it is difficult to minimize a model Q within an unbounded domain.


This backpropagation through the generated data, combined with adversarial learning instabilities, has proven to be a compelling challenge when applying GANs for discrete data such as text.

it remains unknown if this is also an issue for feature matching networks since the effectiveness of GFMN for sequential discrete data has not yet been studied.


An interesting comparison would be between Se-qGFMN and GANs that use BERT as a pre-trained discriminator.

gANs fail to train when a very deep network is used as the discriminator Moreover, SeqgFMN also outperforms gAN generators even when shallow word embeddings (glove / FastText) are used to perform feature matching.


Extractive MRC requires a model to extract an answer span to a question from reference documents, such as the tasks in SQuAD (Rajpurkar et al., 2016) and CoQA (Reddy et al., 2019).

non-extractive MRC infers answers based on some evidence in reference documents, including Yes/No question answering (Clark et al., 2019), multiple-choice MRC (Lai et al., 2017; Khashabi et al., 2018; Sun et al., 2019), and open domain question answering (Dhingra et al., 2017b).


RL methods can indeed train a better extractor without evidence labels.

they are much more complicated and unstable to train, and highly dependent on model pre-training.


As the innermost ring shows, about 80% of the evidence predicted by BERT-HA (iter 0) was incorrect.

the proportion of wrong instances reduced to 60% after self-training (iter 3).


Earlier studies have attempted to perform the MWP task via statistical machine learning methods (Kushman et al., 2014; Hosseini et al., 2014; Mitra and Baral, 2016; Roy and Roth, 2018) and semantic parsing approaches (Shi et al., 2015; KoncelKedziorski et al., 2015; Roy and Roth, 2015; Huang et al., 2017).

these methods are nonscalable as tremendous efforts are required to design suitable features and expression templates.


To enrich the representation of a quantity, the relationships between the descriptive words associated with a quantity need to be modeled.

such relationships cannot be effectively modeled using recurrent models, which are commonly used in the existing MWP deep learning methods.


While all of these methods are bag-of-words models, Liu et al. (2019a) recently proposed an architecture based on context2vec (Melamud et al., 2016).

in contrast to our work, they (i) do not incorporate surface-form information and (ii) do not directly access the hidden states of context2vec, but instead simply use its output distribution.


For this reason, previous works used an in-house mapping between BabelNet versions to make them up to date.

in this process, several gold instances were lost making the datasets smaller than the original ones.


Word-Net provides information about sense frequency that is either manually-annotated or derived from SemCor (Miller et al., 1993), i.e., a corpus where words are manually tagged with WordNet meanings.

neither WordNet nor SemCor have been updated in the past 10 years, thus making their information about sense frequency outdated.


The method of applying REINFORCE to the discriminative parser is straightforward because sampling trees from the discriminative parser is easy.

that is not the case for the generative model from which we have to sample both trees and sentences at the same time.


In this task, models are expected to make predictions with the semantic information rather than with the demographic group identity information (e.g., "gay", "black") contained in the sentences.

recent research points out that there widely exist some unintended biases in text classification datasets.


In other words, a non-discrimination model should perform similarly across sentences containing different demographic groups.

"perform similarly" is indeed hard to define.


Similar to results on Toxicity Comments, we find that both Weight and Supplement perform significantly better than Baseline in terms of IPTTS AUC and FPED, and the results of Weight and Supplement are comparable.

we notice that Weight and Supplement improve FNED slightly, while the differences are not statistically significant at confidence level 0.05.


Current approaches define interpretation in a rather ad-hoc manner, motivated by practical usecases and applications.

this view often fails to distinguish between distinct aspects of the interpretation's quality, such as readability, plausibility and faithfulness (Herman, 2017).


For example, Serrano and Smith (2019) and Jain and Wallace (2019) show that high attention weights need not necessarily correspond to a higher impact on the model's predictions and hence they do not provide a faithful explanation for the model's predictions.

wiegreffe and Pinter (2019) argues that there is still a possibility that attention distributions may provide a plausible explanation for the predictions.


Our main goal is to show that our proposed models provide more faithful and plausible explanations for their predictions.

before we go there we need to show that the predictive performance of our models is comparable to that of a vanilla LSTM model and significantly better than non-contextual models.


We observe that randomly permuting the attention weights in the Diversity and Orthogonal LSTM model results in significantly different outputs.

there is little change in the vanilla LSTM model's output for several datasets suggesting that the attention weights are not so meaningful.


Several other works (Shao et al., 2019; Martins and Astudillo, 2016; Malaviya et al., 2018; Niculae and Blondel, 2017; Maruf et al., 2019; Peters et al., 2018) focus on improving the interpretability of attention distributions by inducing sparsity.

the extent to which sparse attention distributions actually offer faithful and plausible explanations haven't been studied in detail.


The plotted eight words are gathered together, and it can be seen that hidden representations of the same word gather in the same place regardless of correctness.

fine-tuned BERT produces a vector space that demonstrates correct and incorrect words on different sides, showing that hidden representations take grammatical errors into account when fine-tuned on GEC corpora.


For instance, DKN (Wang et al., 2018) learns knowledge-aware news representation via multi-channel CNN and gets a representation of a user by aggregating her clicked news history with different weights.

these methods (Wu et al., 2019b; Zhu et al., 2019; An et al., 2019) usually focus on news contents, and seldom consider the collaborative signal in the form of high-order connectivity underlying the user-news interactions.


Wang et al. (2019) explored the GNN to capture high-order connectivity information in user-item graph by propagating embeddings on it, which achieves better performance on recommendation.

existing news recommendation methods focus on, and rely heavily on news contents.


after obtaining the latent variables {r d,k }, we can find an estimate of z u,k by aggregating information from the clicked news, which is computed as Eq.


Most existing methods usually learn the representations of users and news from news contents for recommendation.

they seldom consider highorder connectivity underlying the user-news interactions.


It supposes that comparing to less important roles, the roles with bigger impact are expected to appear at more places and are more evenly distributed over the story.

this assumption ignores actions of roles (denoted as behavioral semantic information), which may be a key factor that estimates their impacts in legalcontext scenarios.


Position or frequency information does not effectively reflect the status of a role in such samples.

our method captures this information by the cooperation mode feature between Yin and Zhao, with the help of verb "instructed".


As we can see from Figure 2(a), the plain nets suffer from the degradation problem, which is not caused by overfitting, as they exhibit lower training BLEU.

the 72-layer MSC exhibits higher training BLEU than the 36-layer counterpart and is generalizable to the validation data.


Figure 4A) shows, as first identified by Kozlowski et al. (2019), that much of this is due to the variance of the survey data along that dimension; the correlation between variance and the coefficients in Figure 3 is 0.91

as discussed above, Kozlowski et al. (2019) study more general concepts on more general dimensions, and note that they have no easy way to connect their observations to any critical social processes.


On the other hand, from the "bias" per-spective, this suggests that a vast array of social biases are encoded in embeddings.

we also find that some beliefs-specifically, extreme beliefs on salient dimensions -are easier to measure than others.


Attention mechanisms learn to assign soft weights to (usually contextualized) token representations, and so one can extract highly weighted tokens as rationales.

attention weights do not in general provide faithful explanations for predictions (Jain and Wallace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019; Zhong et al., 2019; Pruthi et al., 2020; Brunner et al., 2020; Moradi et al., 2019; Vashishth et al., 2019).


Original rationale annotations were not necessarily comprehensive; we thus collected comprehensive rationales on the final two folds of the original dataset (Pang and Lee, 2004).

to most other datasets, the rationale annotations here are span level as opposed to sentence level.


In general, the rationales we have for tasks are sufficient to make judgments, but not necessarily comprehensive.

for some datasets we have explicitly collected comprehensive rationales for at least a subset of the test set.


Since DeFormer retains much of the original structure, we can initialize this model with the pre-trained weights of the original Transformer and fine-tune directly on downstream tasks.

deFormer looses some information in the representations of the lower layers.


The upper layers can learn to compensate for this during finetuning.

we can go further and use the original model behavior as an additional source of supervision.


This is an orthogonal approach that can be combined with our decomposition idea.

for the paired-input tasks we consider, pruning heads only provides limited speedup.


This is because the candidate justifications are coming from a relatively small numbers of paragraphs in MultiRC; thus even shorter queries (= 2 words) can retrieve relevant justifications.

the number of candidate justifications in QASC is much higher, which requires longer queries for disambiguation (>= 4 words).


Access to such data can greatly facilitate investigation of phonetic typology at a large scale and across many languages.

it is nontrivial and computationally intensive to obtain such alignments for hundreds of languages, many of which have few to no resources presently available.


In addition to its coverage, the CMU Wilderness corpus is unique in two additional aspects: cleanly recorded, read speech exists for all languages in the corpus, and the same content (modulo translation) exists across all languages.

this massively multilingual speech corpus is challenging to work with directly.


Since our greedy opportunistic decoding doesn't change the final output, there is no difference in BLEU compared with normal decoding, but the latency is reduced.

by applying beam search, we can achieve 3.1 BLEU improvement and 2.4 latency reduction on wait-7 policy.


Thanks to the wealth of high-quality annotated images available in popular repositories such as ImageNet, multimodal language-vision research is in full bloom.

events, feelings and many other kinds of concepts which can be visually grounded are not well represented in current datasets.


On one hand, the inclusion of NC concepts would be an important step towards wide-coverage image semantic understanding.

it also goes in the same direction as recent multimodal language-vision approaches, e.g., monoand cross-lingual Visual Sense Disambiguation (Barnard and Johnson, 2005; Loeff et al., 2006; Saenko and Darrell, 2008; Gella et al., 2016, 2019).


Our experiments show that both systems are reliable on our task, achieving precision and F1 scores that are over 70% on all the splits (see Table 2).

the F-VLP model proves to be the most stable for the task.


This accordance to some extent verifies that the neurons found through influence paths are functionally important.

the t-values shown in Table 1 show that both neuron 125 and 337 are influential regardless of the subject number, whereas Lakretz et al. assign a subject number for each of these two neurons due to their disparate effect in lowering accuracy in ablation experiment.


A slightly worse NA task performance (Lakretz et al., 2019) in cases of attractors (SP, PS) indicates that they interfere with prediction of the correct verb.

we also observe that helpful nouns (SS, PP) contribute positively to the correct verb number (although they should not from a grammar perspective).


In particular, in the largest setting with N = 1M, the BERT-24 embeddings distilled from the best-performing layer for each dataset drastically outperform both Word2Vec and GloVe.

this can be seen as an unfair comparison given that we are selecting specific layers for specific datasets.


As such, the validity of treating the resulting static embeddings as reliable proxies for the original contextualized model still remains open.

human language processing has often been conjectured to have both context-dependent and context-independent properties (Barsalou, 1982; Rubio-Fernandez, 2008; Depraetere, 2014, 2019).


We find that each of the tested word features can be encoded in contextual embeddings for other words of the sentence, often with perfect or nearperfect recoverability.

we see substantial variation across encoders in how robustly each information type is distributed to which tokens.


CheckList provides a framework for such techniques to systematically evaluate these alongside a variety of other capabilities.

checkList cannot be directly used for non-behavioral issues such as data versioning problems (Amershi et al., 2019), labeling errors, annotator biases (Geva et al., 2019), worst-case security issues (Wallace et al., 2019), or lack of interpretability (Ribeiro et al., 2016).


These experimental results show the critical role of triggers in dialogue-based relation extraction.

trigger identification is perhaps as difficult as relation extraction, and it is labor-intensive to annotate large-scale datasets with triggers.


RST Graph is constructed from RST parse trees over EDUs of the document.

coreference Graph connects entities and their coreference clusters/mentions across the document.


As observed in Louis et al. (2010), the RST tree structure already serves as a strong indicator for content selection.

the agreement between rhetorical relations tends to be lower and more ambiguous.


BERT is originally trained to encode a single sentence or sentence pair.

a news article typically contains more than 500 words, hence we need to make some adaptation to apply BERT for document encoding.


Because of the similarity to our task, we use a BERT-based neural network as the architecture for the coverage model.

the coverage task differs from MLM in two ways.


Just as with the Summarizer, by using a standardized architecture and model size, we can make use of pretrained models.

it is important for Fluency to fine tune the language model on the target domain, so that the Summarizer is rewarded for generating text similar to target content.


EMONET was conceived as a multiclass classification task for Plutchik-8 emotions (Abdul-Mageed and Ungar, 2017).

we introduce binary classification tasks, one for each Plutchik-8 emotion.


In our work, we raise similar concerns but through a different angle by highlighting issues with the evaluation procedure used by several recent methods. Chandrahas et al. (2018) analyze the geometry of KG embeddings and its correlation with task performance while Nayyeri et al. (2019) examine the effect of different loss functions on performance.

their analysis is restricted to non-neural approaches.


Several recently proposed methods report high performance gains on a particular dataset.

their performance on another dataset is not consistently improved.


Recently many reading comprehension datasets requiring complex and compositional reasoning over text have been introduced, including HotpotQA (Yang et al., 2018), DROP (Dua et al., 2019), Quoref , and ROPES (Lin et al., 2019).

models trained on these datasets (Hu et al., 2019;Andor et al., 2019) only have the final answer as supervision, leaving the model guessing at the correct latent reasoning.


Recent proposed approaches have made promising progress in dialogue state tracking (DST).

in multi-domain scenarios, ellipsis and reference are frequently adopted by users to express values that have been mentioned by slots from other domains.


Open vocabulary models show the promising performance in multidomain DST.

ellipsis and reference phenomena among multi-domain slots are still less explored in existing literature.


Some proposed solutions rely on leveraging knowledge distillation in the pre-training step, e.g., (Sanh et al., 2019), or used parameter reduction techniques (Lan et al., 2019) to reduce inference cost.

the effectiveness of these approaches varies depending on the target task they have been applied to.


This may be seen as a generalization of the ST approach, where the student needs to learn a simpler task than the teacher.

our approach is significantly different from the traditional ST setting, which our preliminary investigation showed to be not very effective.


These two dialogue-specific LM approaches, ULM and UOP, give very marginal improvement over the baseline models, that is rather surprising.

they show good improvement when combined with UID, implying that pre-training language models may not be enough to enhance the performance by itself but can be effective when it is coupled with an appropriate fine-tuning approach.


Recently, pretrained language representation models (Kocijan et al., 2019;Radford et al., 2019;Liu et al., 2019) have demonstrated significant improvements in both unsupervised and supervised settings.

as these approaches treat the concept 'commonsense knowledge' as a black box, we are not clear about why they can do better (e.g., can these models understand commonsense or they just capture the statistical bias of the dataset) and do not know how to further improve them.


For evaluation purposes, we may have labeled documents in the target language.

they are only used during the test period.


For the last layer before softmax, even though XLM-FT also generates reasonable representations to separate positive and negative reviews, the data points are scattered randomly.

our model's output in the lower right panel of Figure 3 shows two more obvious clusters with corresponding labels that can be easily separated.


This rating method, also known as Likert scale or Mean Opinion Score, is known to have two major drawbacks (Ye and Doermann, 2013): (1) Absolute rating is often treated as if it produces data on an interval scale.

assessors rarely perceive labels as equidistant, thus producing only ordinal data.


Overall, the Bradley-Terry model appears to be a promising candidate for our purposes: its robustness and statistical properties have been studied in great detail (Hunter, 2004), and it can be efficiently computed (Chen et al., 2013).

an alternative offline sampling method has to be formulated, which we introduce in the following section.


This can be generalized to higher step sizes s: for instance, if s = 2, all items that are separated by two positions around the ring are compared.

this strategy suffers from the major drawback that for some step sizes, the resulting graph has multiple unconnected components, thus violating the restriction that the comparison matrix must form a strongly connected graph.


Using a higher temperature yields a softer attention distribution.

a sharper attention distribution might be more suitable for NER because only a few tokens in the sentence are named entities.


In this work, we follow Chen et al. (2019) and use exactly the same functions.

as shown in 7 (c), understanding this statement requires the function of difference time, which is not covered by the current set.


Word-level attacking, which can be regarded as a combinatorial optimization problem, is a well-studied class of textual attack methods.

existing word-level attack models are far from perfect, largely because unsuitable search space reduction methods and inefficient optimization algorithms are employed.


Nonetheless, its attack validity rates against BiLSTM and BERT on SST-2 dramatically fall to 59.5% and 56.5%.

ours are 70.5% and 72.0%, and their differences are significant according to the results of significance tests in Appendix D. In this section, we conduct detailed decomposition analyses of different word substitution methods (search space reduction methods) and different search algorithms, aiming to further demonstrate the advantages of our sememe-based word substitution method and PSO-based search algorithm.


Mixed counting models that use the negative binomial distribution as the prior can well model over-dispersed and hierarchically dependent random variables; thus they have attracted much attention in mining dispersed document topics.

the existing parameter inference method like Monte Carlo sampling is quite time-consuming.


On the one hand, NVI-based models are fast and easy to estimate but hard to interpret.

document modeling via mixed counting models is easy to interpret but difficult to infer.


Such a framework learns the distribution of input data well, enabling it to combine with the traditional probability graphical models (e.g., LDA) and infer model parameters quickly (Srivastava and Sutton, 2017).

how to effectively integrate the distributed dependencies in mixed counting models into the framework of variational inference is still quite a challenging problem.


From these results, we can observe that the proportion of topics obtained by NB-NTM is close to the topic-word number distribution.

gNB-NTM obtains more dispersed proportions of topics than NB-NTM.


Everything was a point in a vector space, and everything about the nature of language could be learned from data.

most computational linguists had linguistic theories and the poverty-of-thestimulus argument.


And deep learning allows many aspects of these structured representations to be learned from data.

successful deep learning architectures for natural language currently still have many handcoded aspects.


(Alex et al., 2007) propose several different modeling techniques (layering and cascading) to combine multiple CRFs for nested NER.

their approach cannot handle nested entities of the same entity type.


Here, the two central entities (both indicate the city Liverpool) have similar sizes of neighborhoods and three common neighbors.

the three common neighbors (indicate United Kingdom, England and Labour Party (UK), respectively) are not discriminative enough.


Most similar to our work are (Chen et al., 2019)  and (Ghosh et al., 2019), as both studies are considered using the concept of question answering to address NLVL.

both studies do not explain the similarity and differences between NLVL and traditional span-based QA, and they do not adopt the standard span-based QA framework.


By treating input video as text passage, the above frameworks are all applicable to NLVL in principle.

these frameworks are not designed to consider the differences between video and text passage.


Figure 9 shows that VSLBase tends to predict longer moments, e.g., more samples with length error larger than 4 seconds in Charades-STA or 30 seconds in Activ-ityNet.

constrained by QGH, VSLNet tends to predict shorter moments, e.g., more samples with length error smaller that -4 seconds in Charades-STA or -20 seconds in ActivityNet Caption.


A very recent work makes use of attention over spans instead of syntactic distance to inject inductive bias to language models (Peng et al., 2019).

the time complexity of injecting supervision is much higher than distancebased approach (O(n 2 ) VS O(n) ).


For example, all punctuation symbols are removed, all characters are lower-cased, the vocabulary size is truncated at 10,000 and all sentences are concatenated.

this version of PTB discards the parse tree structures, which makes it unsuitable for comparing sequential language models with those utilizing tree structures.


Because of the sequential and parallel nature of our model, it can directly inherit and benefit from this set of tricks.

it is non-trivial to use them for RNNG and URNNG.


In particular, Kim et al. (2019a) report that unsupervised URNNG achieves 45.4 WSJ F1 in a similar setting, while another URNNG that finetunes a supervised RNNG model gives a much better F1 of 72.8, leading a 27.4 F1 improvement.

the F1 of our structure prediction trees is 61.3 in unbiased algorithm.


But this strategy does not consider that it is properly difficult to acquire a dictionary with high quality for a brand new domain.

we develop a simple and efficient strategy to perform domain-specific words mining without any predefined dictionaries.


In some languages, predicting declension class is argued to be easier if we know the noun's phonological form (Aronoff, 1992;Dressler and Thornton, 1996) or lexical semantics (Carstairs-McCarthy, 1994;Corbett and Fraser, 2000).

semantic and phonological clues are, at best, only very imperfect hints as to class (Wurzel, 1989;Harris, , 1992Aronoff, 1992;Halle and Marantz, 1994;Corbett and Fraser, 2000;Aronoff, 2007).


Until now, we have assumed a one-to-one mapping between paradigm slots and surface form changes.

different EDIT TREES may indeed represent the same inflection.


Our system, thus, needs to learn to apply the correct transformation for a combination of lemma and paradigm slot.

mapping lemmas and paradigm slots to inflected forms corresponds exactly to the morphological inflection task, which has been the subject of multiple shared tasks over the last years (Cotterell et al., 2018).


Systems for supervised or semi-supervised paradigm completion are commonly being evaluated using word-level accuracy (Dreyer and Eisner, 2011; Cotterell et al., 2017).

this is not possible for our task because our system cannot access the gold data paradigm slot descriptions and, thus, does not necessarily produce one word for each ground-truth inflected form.


On the one hand, applying more than one iteration of additional lemma retrieval impacts the results only slightly, as those lemmas are assigned very small weights.

we see performance differences > 2% between PCS-III-C and PCS-III-H for DEU, MLT, and SWE.


All these approaches first find a set of semantically similar sentences.

finding isolated similar sentences are not enough to construct a dialog utterances' paraphrase.


Under the guidance of the reasoning chain, we learn a neural QG model to make the result satisfy the logical correspondence with the answer.

the neural model is data-hungry, and the scale of training data mostly limits its performance.


Recently, the character-word lattice structure has been proved to be effective for Chinese named entity recognition (NER) by incorporating the word information.

since the lattice structure is complex and dynamic, most existing lattice-based models are hard to fully utilize the parallel computation of GPUs and usually have a low inference-speed.


For example, queries "red nike running shoes", "running nike shoes, red" and "red running shoes nike" all refer to the same general product, despite differing in structure.

item titles are structured, with brand, size, color, etc. all mentioned in a long sequence, which is also not how a conventional sentence is structured.


In this case, Q gen is similar to Q in that the item is somewhat related to Q gen , and there's a chance that I may be matched to Q gen due to keyword stuffing by sellers, or poor semantic matching.

another mismatched query Q gen = pizza cutter is not a good candidate to generate, since it's highly unlikely that a reasonable search engine will show shoes for a query about pizza cutters.


Wu el al. (2019) applied dynamic convolutions using shared softmax-normalized filters of depth-wise on GLU-regulated inputs within a fixed reception field rather than global contexts, challenging the common self-attention-dominated intuition.

all of the models, as mentioned earlier, adopt stacked CNNs rather than self-attention networks (SAN) to attend to the global contexts.


It can be also quantified by other manners, e.g. estimating the data likelihood with Monte Carlo approximation (Der Kiureghian and Ditlevsen, 2009) or validating the translation distribution using a well-trained NMT model (Zhang et al., 2018).

to these time-consuming techniques, LM marginally increases the computational cost and easy to be applied, conforming to the original motivation of CL.


Since the speaker information is indispensable for coreference resolution, previous methods (Wiseman et al., 2016;Lee et al., 2017;Joshi et al., 2019a) usually convert the speaker information into binary features indicating whether two mentions are from the same speaker.

we use a straightforward strategy that directly concatenates the speaker's name with the corresponding utterance.


Comparing with existing models (Lee et al., 2017Joshi et al., 2019b), the proposed question answering formalization has the flexibility of retrieving mentions left out at the mention proposal stage.

since we still have the mention proposal model, we need to know in which situation missed mentions could be retrieved and in which situation they cannot.


The one-one target-source alignment 2(a) is the ideal condition of the projection.

there could be many-to-one cases for the given words, leading to semantic role conflicts at the target language words.


Metrics which measure the word-level overlap like BLEU (Papineni et al., 2002) have been widely used for dialogue evaluation.

these metrics do not fit into our setting well as we would like to diversify the response generation with an external corpus, the generations will inevitably differ greatly from the ground-truth references in the original conversational corpus.


The second class seeks to bring in extra information into existing corpus like structured knowledge (Zhao et al., 2018;Ghazvininejad et al., 2018;Dinan et al., 2019), personal information (Li et al., 2016b;Zhang et al., 2018a) or emotions (Shen et al., 2017b;Zhou et al., 2018).

corpus with such annotations can be extremely costly to obtain and is usually limited to a specific domain with small data size.


The user requests to book one ticket in the second example, yet both HDSA and Human Response ask about the number once again.

our model directly answers the questions with correct information.


One limitation of ReGAT (Li et al., 2019) lies in the fact that it solely consider the relations between objects in an image while neglect the importance of text information.

our DC-GCN simultaneously capture visual relations in an image and textual relations in a question.


Figure 6a shows that in terms of validation perplexity, MDR and FB perform very similarly across target rates.

figure 6b shows that at the end of training the difference between the target rate and the validation rate is smaller for MDR.


Previous conversational QA datasets provide the relevant document or passage that contain the answer of a query.

in many real world scenarios such as FAQs, the answers need to be searched over the whole document collection.


For instance, all the span-based QA datasets, except CQ (Bao et al., 2016), contain more than 100k samples.

the data size of most existing MCQA datasets are far less than 100k (see Table 1), and the smallest one only contains 660 samples.


On the one hand, technical proposals as pre-trained embeddings, finetuning, and end-to-end modeling, have advanced NLP greatly.

neural advances often overlook MRL complexities, and disregard strategies that were proven useful for MRLs in the past.


Document-level information extraction requires a global understanding of the full document to annotate entities, their relations, and their saliency.

annotating a scientific article is timeconsuming and requires expert annotators.


One common characteristic of most of the tasks is that the texts are not restricted to some rigid formats when generating.

we may confront some special text paradigms such as Lyrics (assume the music score is given), Sonnet, SongCi (classical Chinese poetry of the Song dynasty), etc.


Our model can still generate high quality results on the aspects of format, rhyme as well as integrity.

for corpus Sonnet, even though the model can generate 14 lines text, the quality is not as good as SongCi due to the insufficient training-set (only 100 samples).


Our relevance framework is partially inspired by the local components matching which we apply here to model the relevance of the components of the model's inputs.

our work differs in several significant ways.


VisualBERT and CMR have a similar cross-modality alignment approach.

visualBERT only uses the Transformer representations while CMR uses the relevance representations.


As we increase the number of layers in the visual Transformer and the cross-modality Transformer, it tends to improve accuracy.

the performance becomes stable when there are more than five layers.


An agent needs to perform a functional communication task in a natural language (in this work, English).

examples of linguistic communication about this functional task are not available -the only natural language data that can be used consist of examples of generic natural language, which are not grounded in the functional task.


Single-headed cross attention speeds up decoding: Despite removing learned self-attention from both the encoder and decoder, we did not observe huge efficiency or speed gains.

reducing the source attention to just a single head results in more significant improvements.


We do find that the largest improvement in WinoMT accuracy consistently corresponds to the model predicting male and female entities in the closest ratio (see Appendix A).

the best ratios for models adapted to these datasets are 2:1 or higher, and the accuracy improvement is small.


Ribeiro et al. (2018) test for comprehension of minimally modified sentences in an adversarial setup while trying to keep the overall semantics the same.

we investigate large changes of meaning (negation) and context (mispriming).


Clustering of such short text streams has thus gained increasing attention in recent years due to many real-world applications like event tracking, hot topic detection, and news recommendation (Hadifar et al., 2019).

due to the unique properties of short text streams such as infinite length, evolving patterns and sparse data representation, short text stream clustering is still a big challenge (Aggarwal et al., 2003;Mahdiraji, 2009).


The similarity-based text clustering approaches usually follow vector space model (VSM) to represent the cluster feature space (Din and Shao, 2020).

a topic needs to be represented as the subspace of global feature space.


Human judges show surprisingly inferior performance on user profiling tasks, grounding their judgement in topical stereotypes (Carpenter et al., 2017).

albeit more accurate thanks to capturing stylistic variation elements, statistical models are prone to stereotype propagation as well (Costa-jussa et al., 2019; Koolen and van Cranenburgh, 2017).


For the foreseeable future, legal decision-making will be the province of lawyers, not AI.

one plausible use for MRC in a legal setting is as a screening tool for helping non-lawyers determine whether a case has enough merit to bother bringing in a lawyer.


Recent work has shown gains by improving the distribution of masked tokens , the order in which masked tokens are predicted (Yang et al., 2019), and the available context for replacing masked tokens (Dong et al., 2019).

these methods typically focus on particular types of end tasks (e.g. span prediction, generation, etc.), limiting their applicability.


We aim, as much as possible, to control for differences unrelated to the pre-training objective.

we do make minor changes to the learning rate and usage of layer normalisation in order to improve performance (tuning these separately for each objective).


Bidirectional encoders are crucial for SQuAD As noted in previous work (Devlin et al., 2019), just left-to-right decoder performs poorly on SQuAD, because future context is crucial in classification decisions.

bART achieves similar performance with only half the number of bidirectional layers.


Unsurprisingly, model output is fluent and grammatical English.

outputs are also highly abstractive, with few copied phrases.


One of the issues in the original ON-LSTM is that the master gates and the model-based importance score for each word are only conditioned on the word itself and the left context encoded in the previous hidden state.

in order to infer the importance for a word in the overall sentence effectively, it is crucial to have a view over the entire sentence (i.e., including the context words on the right).


On the one hand, as GCN is directly dependent on the syntactic structures of the input sentences, it would not be able to learn effective representations for the sentences with new structures in the GCN-failure examples for RE.

as CEON-LSTM only exploits a relaxed general form of the tree structures (i.e., the importance scores of the words), it will be able to generalize better to the new structures in the GCN-failure examples where the general tree form is still helpful to induce effective representations for RE.


ELMo down-samples the outputs of its convolutional layers by max-pooling over the feature maps.

this operation is not ideal to adapt to new morphological patterns from other languages as the model tends to discard patterns from languages other than English.


The soft gazetteer features we propose instead take advantage of existing limited gazetteers and English knowledge bases using lowresource EL methods.

to typical binary gazetteer features, the soft gazetteer feature values are continuous, lying between 0 and 1.


In addition, CGExpan-NoCN outperforms most baseline models, meaning that the pre-trained LM itself is powerful to capture entity similarities.

it still cannot beat CGExpan-NoFilter model, which shows that we can properly guide the set expansion process by incorporating generated class names.


HINT proposes a ranking loss between humanbased importance scores (Das et al., 2016) and the gradient-based sensitivities.

sCR does not require exact saliency ranks.


As observed by Selvaraju et al. (2019) and as shown in Fig. 2, we observe small improvements on VQAv2 when the models are fine-tuned on the entire train set.

if we were to compare against the improvements in VQA-CPv2 in a fair manner, i.e., only use the instances with visual cues while fine-tuning, then, the performance on VQAv2 drops continuously during the course of the training.


In the original paper, Das et al. (2017) find that models which structurally encode dialog history, such as Memory Networks (Bordes et al., 2016) or Hierarchical Recurrent Encoders (Serban et al., 2017) improve performance.

"naive" history modelling (in this case an encoder with late fusion/concatenation of current question, image and history encodings) might actually hurt performance.


In this paper, we show that competitive results on VisDial can indeed be achieved by replicating the top performing model for VQA (Yu et al., 2019b) -and effectively treating visual dialog as multiple rounds of question-answering, without taking history into account.

we also show that these results can be significantly improved by encoding dialog history, as well as by fine-tuning on a more meaningful retrieval metric.


Other visual dialog tasks, such as GuessWhich? (Chattopadhyay et al., 2017) and GuessWhat?! (De Vries et al., 2017) take place in a goal-oriented setting, which according to Schlangen (2019), will lead to data containing more natural dialog phenomena.

there is very limited evidence that dialog history indeed matters for these tasks (Yang et al., 2019).


The combined features approach models the relationships between the different features explicitly, but the large target spaces for morphologically rich languages further increase sparsity.

separate feature modeling guarantees smaller target spaces for the individual features, but the hard separation between the features prevents modeling any interfeature dependencies.


The results are lower, both for MSA and EGY.

the result for MSA is very close to the (Zalmout and Habash, 2017) baseline, which uses separate feature models (with the analyzer).


Although increases exist across all domains, these are most prominent in domains like TC (+5.36) that have a low density of named entities and where indomain models have access to limited amounts of data.

the in-domain performance is better than the pooled method of training, which shows consistent drops in performance on some domains (-8.69 on WB, -6.77 on BC, -1.98 on CoNLL), where information from other domains did not benefit the model.


Our proposed model architecture takes 0.15 ms (33% increase) longer for inference than InDomain or PoolDomain models, which is a result of more model parameters.

our proposed architecture is still 0.19 ms faster than using the InDomain+DomainClassifier approach.


State-of-the-art approaches for attribute value extraction (Zheng et al., 2018; Xu et al., 2019; Rezk et al., 2019) have employed deep learning to capture features of product attributes effectively for the extraction purpose.

they are all designed without considering the product categories and thus cannot effectively capture the diversity of categories across the product taxonomy.


The CT and CAT models learn to map source code and natural language tokens into a joint embedding space such that semantically similar code-natural language pairs are projected to vectors that are close to each other.

these two representations interact only in the final step when the global similarity of the sequence embeddings is calculated, but not during the first step when each sequence is encoded into its corresponding embedding.


Currently, the TClda are trained on student essays, while the TCpr only works on the source article.

TCattn uses both student essays and the source article for TC generation.


Entity linking systems consider three sources of information: 1) similarity between mention strings and names for the KB entity; 2) comparison of the document context to information about the KB entity (e.g. entity description); 3) information contained in the KB, such as entity popularity or inter-entity relations

to the dense KBs in entity linking, concept linking uses sparse ontologies, which contain a unique identifier (CUI), title, and links to synonyms and related concepts, but rarely longform text.


We ran experiments that padded the names with synonyms or other forms of available text within the knowledge base.


Depending on the application, a less accurate but faster linker might be a better choice (e.g. for all clinical notes at a medical institution).

a more complex linker, such as ours, maybe a better option for specific subsets of notes that require better accuracy (e.g., the results of specific clinical studies).


In natural language processing, Natural Language Inference (NLI)-a task whereby a system determines whether a pair of sentences instantiates in an entailment, a contradiction, or a neutral relation-has been useful for training and evaluating models on sentential reasoning.

linguists and philosophers now recognize that there are separate semantic and pragmatic modes of reasoning (Grice, 1975; Clark, 1996; Beaver, 1997; Horn and Ward, 2004; Potts, 2015), and it is not clear which of these modes, if either, NLI models learn


For example, the Independent model will produce identical scores for each output label, if it chooses to completely ignore the input explanations.

the model is still free to learn a different kind of bias which is an outcome of the fact that natural language explanations convey ideas through both content and form.


The results demonstrate a much weaker link between NILE-NS's predictions and associated explanations.


This led folk wisdom to suggest that modeling higher-order features in a neural parser would not bring additional advantages, and nearly all recent research on dependency parsing was restricted to first-order models (Dozat and Manning, 2016; Smith et al., 2018a). Kulmizev et al. (2019) further reinforced this belief comparing transition and graph-based decoders (but none of which higher order); Falenska and Kuhn (2019) suggested that higher-order features become redundant because the parsing models encode them implicitly.

there is some evidence that neural parsers still benefit from structure modeling.


This approach, named adversarial training (AT), has been reported to be highly effective on image classification (Goodfellow et al., 2015), text classification (Miyato et al., 2017), as well as sequence labeling (Yasunaga et al., 2018).

aT is limited to a supervised scenario, which uses the labels to compute adversarial losses.


To apply the conventional VAT on a model with CRF, one can calculate the KL divergence on the label distribution of each token between the original examples and adversarial examples.

it is sub-optimal because the transition probabilities are not taken into account.


VAT achieved state-of-the-art performance for image classification tasks (Miyato et al., 2019), and proved to be more efficient than traditional semi-supervised approaches, such as entropy minimization (Grandvalet and Bengio, 2004) and selftraining (Yarowsky, 1995), from a recent study (Oliver et al., 2018).

despite the successful applications on text classification (Miyato et al., 2017), VAT has not shown great benefits to semi-supervised sequence labeling tasks, due to its incompatibility with CRF.


With multiple layers, SpellGCN can aggregate the information in more hops and therefore, achieve better performance.

the F1score drops when the number of layers is larger than 3.


The evident way to construct a corpus with NL questions and their corresponding OT queries would consist of two main parts: first, collect a set of NL questions, and then create the corresponding OT queries to these questions.

this approach is very time-consuming and has a major issue.