Note that we follow the suggestion of Yu et al. (2018) to set α to 0.7

the parse trees annotated in CDTB-14 are not always binary.


As shown in Figure 1, all responses follow the same topic, but only the first one is appropriated.

tF-IDF achieves 54.98% R@1 score on the Ubuntu Corpus, showing our dataset is more difficult to get the correct answer by text overlap.


Among them, "Semantic Error", "Redundant", and "Unanswerable" are noticeable errors for all models.

To represent multiple relations in the edge, we base our model on the multi-relation Gated Graph Neural Network (GGNN) (Li et al., 2016), which provides a separate transformation matrix for each edge type.


We evaluate the framework on two public datasets NYT (Riedel et al., 2010) and WebNLG (Gardent et al., 2017).

first, we detect subjects from the input sentence.


This validates the utility of the proposed cascade decoder that adopts a novel binary tagging scheme.

it is composed of a stack of N identical Transformer blocks.


The SCIENCEIE dataset focuses on the extraction of 3 types of key-phrases, rather than Named Entities, and hyponymy and synonymy relations between these (Augenstein et al., 2017).

a gap remains in quickly comprehending the central information in a text, e.g., the biological mechanisms that are used to manipulate a trade-off.


to SORE reduces the number of OIE extractions by 65%, while increasing the relative amount of informative extractions by 5.75%.


To enrich the discourse argument representations, we exploit multi-level encoder shown in Figure 2 to learn the argument representations at the different levels.

in this section, we give an overview of the TransSdriven joint learning framework, which consists of four parts: embedding layer, multi-level encoder, latent geometric structure learning, and semantic feature learning, as shown in Figure 1.


Parent ignoring Due to the lack of parallel corpora with gold-standard parses, we rely on noisy annotations from an external parser.

to compare with previous work, we train our models on WMt16 En-De and WAt En-Ja tasks, removing sentences in incorrect languages from WMt16 data sets.


Unless otherwise specified, we first tokenize the data with Moses (Koehn et al., 2007) and remove sentences longer than 80 tokens in either source or target side.

the performance of syntactic parsers drops abruptly when evaluated on out-of-domain data (Dredze et al., 2007).


Compared with the individual models, our model is slightly better for Fr/Es-En in many-to-one scenario.

then we use the two embeddings to augment the self-attention mechanism which transforms the Encoder representation into the shared semantic space.


Compared with the INTL baseline, the REC training objective can further improve the translation quality of both supervised and zero-shot language pairs.

so recent efforts in multilingual NMT mainly focus on enlarging the model capacity, either by introducing multiple Encoders and Decoders to handle different languages (Firat et al., 2016;Zoph and Knight, 2016), or enhancing the attention mechanism with language-specific signals (Blackwood et al., 2018).


Because this decade often contributed strongly to the most frequent age category (25-54), predictions did not differ as much from gold in the previous test.

demographic factors (age, gender, etc.) all manifest in language, and therefore influence style: we do not expect a 6-year old to sound
like an adult, and would not translate a person to seem differently gendered.


This highlights a more general problem with template-based probing, namely, that the unnatural lack of sentence diversity imposed by the templates imposes unintended regularity for models to latch onto.

average surprisal was computed for each model for each test set, then each model was adapted to ("primed for") each sentence type.


The annotator judgements are relative (amount of decrease/increase in suspense from sentence to sentence), but the model predictions are absolute values.

these baselines also reflect how much change occurs from one sentence to the next in a story: WordOverlap is the Jaccard similarity between the two sentences, GloveSim is the cosine similarity between the averaged Glove (Pennington et al., 2014) word embeddings of the two sentences, and GPtSim is the cosine similarity between the GPt embeddings of the two sentences.


Given a probability distribution over possible next sentences P(e i t+1 ), we calculate the entropy of that distribution.

the Ely model is more in line with the second view that uncertainty matters less than consequentially different outcomes.


We categorize the factual errors made by the two models.

fACTE-DITOR can seldomly make such errors.


The summarizer refines the FINDINGS word representation based on salient ontology word representation encoded by a separate encoder.

we also evaluated the inter-rater agreement using Fleiss' Kappa (Fleiss, 1971) for our system's scores and obtained 52% for readability, 47% for accuracy, and 50% for completeness, all of which are characterized as moderate agreement rate.


Our work differs in that we hypothesize that all of the ontological terms in the FINDINGS are not equally important, but there is a notion of odds of saliency for each of these terms; thus, we focus on refining the FINDINGS representations.

73% (readability), and 71% (accuracy) of our system-generated Impressions ties with human-written Impressions, both achieving full-score of 3; nonetheless, this percentage is 62% for completeness metric.


ConceptFlow utilizes it to model the conversation and guide the response generation.

con-ceptFlow indicates the filtered two-hop graph.


Apart from standard vector embeddings, we also experiment with contextualized ELMo (Peters et al., 2018) embedding with the DA model using the version implemented in AllenNLP (Gardner et al., 2017).

A linear classifier using features inspired by Heilman and Smith (2010) and Wan et al. (2006), who have implemented similar linear models for other sentence pair classification tasks.


We perform a human evaluation similar to §3.2 on this sample.

next, we developed an annotation task on Amazon Mechanical Turk to select the best responses for the questions.


Softmax: We will discuss in §2.3 that annotators are expected to miss a few good responses since good and bad answers are often very similar (may only differ by a single preposition or pronoun).

furthermore, the D-GPT model with oracle answers is able to generate conversational responses on the CoQA dev set 77 % of the time showcasing the model's scalability.


Sample of responses from different models on SQuAD-dev-test set §3.2. ”Q:” is the Question, ”R:” is the Response, ”B.” stands for baseline and ”(o)” stands for oracle answer spans.

we observe that it does very well in option b (i.e. correct answer but not a complete-sentence).


The resulting SG data split is summarized in Table 1.

we explore a ranking objective that calculates errors based on the margin with which incorrect responses are ranked above correct ones (Collins and Koo, 2005).


To fulfill this goal, we come up a label guessing method to generate labels for the unlabeled data in the training process.

since the combination is continuous, TMix has the potential to create infinite mount of new augmented data samples, thus can drastically avoid overfitting.


Both measures are significantly lower for the depressed class for responses to starters, but not to backchannels.

accurately diagnosing depression is difficult-requiring time-intensive interviews, assessments, and analysis.


For example, exploring correlations between counselor-patient interaction dynamics and counseling outcomes (Althoff et al., 2016); studying linguistic development of mental healthcare counsellors (Zhang et al., 2019); identifying differences in how people disclose mental illnesses across gender and culture (De Choudhury et al., 2017); predicting a variety of mental health conditions from social media posts (Sekulic and Strube, 2019;De Choudhury et al., 2013a;Guntuku et al., 2019;Coppersmith et al., 2014); and analyzing well-being (Smith et al., 2016) and distress (Buechel et al., 2018).

we hope that this combination will encourage the research community to make more progress in this direction.


Entropy threshold S = 0 is the baseline, and all samples exit at the last layer; as S increases, gradually more samples exit earlier.

an inference sample can exit earlier at an off-ramp, without going through the rest of the transformer layers.


Responses are processed in batches of 8 and are padded/truncated to a length of 128.

difference in ASR quality has a bigger influence on the RMSE when using an LSTM encoder compared to a BERT encoder.


There have been other efforts in document representation learning such as extensions of word vectors to documents (Le and Mikolov, 2014; Ganesh et al., 2016; Liu et al., 2017; Wu et al., 2018; Gysel et al., 2017), convolution-based methods (Liu et al., 2018; Zamani et al., 2018), and variational autoencoders (Holmer and Marfurt, 2018; Wang et al., 2019).

note that SPECTER will not be further fine-tuned on any of the tasks; we simply plug in the embeddings as features for each task.


There were a total of 9 parameter combinations.

citeomatic also performs well on the citation tasks, as expected given that its primary design goal was citation prediction.


Note that methods like SGC cannot be used in real-world setting to embed new papers that are not cited yet.

While the fact that SPEcTER does not require finetuning makes its paper embeddings less costly to use, often the best performance from pretrained Transformers is obtained when the models are finetuned directly on each end task.


The latter needs thousands of times more computation to attain the same level of performance as the former.

Since no off-the-shelf C++ parser extracts the information we need from code pieces, we implement our own primary expression parser to extract high level control information.


This heavily depends on the underlying model to generate potentially correct code pieces.

this dataset consists of C++ solutions to problems from Codeforces, a competitive programming website, along with the input-output test cases used for each problem to evaluate correctness.


Finally, we build a non-auto model that performs SQG in an nonautoregressive way, i.e., each question is generated in parallel.

it is worth noticing that all models get rather poor performances under these two aspects, indicating that making a concise question meaningful (i.e., targeting on given answers) with more information from input passage (i.e., performing proper information elimination) is a major challenge in SQG.


Does training on premise encoded as structure help?

in the Dev set, the model performs poorly for the knowledge & common sense, multi-row, coreference, and temporal reasoning categories.


We see that when n is close to or exceeds the average phone duration (n ≥ 5), an evident reduction in L f after adding L r is observed, which validates the effectiveness of L r in assisting with the optimization of L f .

here  we will only focus on analyzing the phonetic content contained in a representation, and leave other properties such as speaker for future work.


For example, PLANENC misses "Buzz Aldrin" and also wrongly expresses the subject of "retirement" as "Frank Borman", indicating that LSTM is less powerful at capturing the semantic roles of entities.

to narrow this gap, we propose DUA-LENC, a dual encoding model that can not only incorporate the graph structure, but can also cater to the linear structure of the output text.


s and o are identified by their entity mentions, and p is identified by a unique ID.

we remove all one-triple instances for planner's evaluation since the planning for these instances is trivial.


For the SEEN set, the most competitive models are GRU and Transformer.

for the plan encoder, we use the final state H T of LSTM as the context representation.


We train the discriminator on the normalized hidden representations generated by each sentence Z = z/||z|| 2 .

In addition, Adel et al. (2018) used an adversarial loss for getting better sentence representation.


This can be replaced by context-aware embeddings.

when dealing with low resource datasets, transfer learning is a common solution.


To speed up the training process, we employ mix-precision training technique.

the basic idea behind the connection of two categories of models is similar to MADE (Germain et al., 2015).


For Wikitext103 dataset, the context length is set to 128, and each context containing multiple coherent sentences.

for GPT, the input text can only be placed in the beginning and the generation process become uncontrollable, resulting in generating sentences with topic drift.


Fed with X as input, the final output of the Transformer, denoted as H = {h 1 , h 2 , ..., h N }, captures the contextual representation of the tokens in the sequence.

for PPL on One-Billion Words, the performances of u-PMLM and BERT are not satisfactory in comparison with GPT.


B-TSort consistently performs better on all the metrics.

the classifier is trained to predict whether s 2 follows s 1 or not i.e the the classifier predicts the constraint s 1 < s 2 .


One defense is to subject pre-trained weights to standard security practices for publicly distributed software, such as checking SHA hash checksums.

training these large models is computationally prohibitive, and thus practitioners generally resort to downloading pre-trained weights from a public source.


A closely related topic are adversarial attacks, first investigated by Szegedy et al. (2013) and Goodfellow et al. (2015) in computer vision and later extended to text classification (Papernot et al., 2016; Ebrahimi et al., 2018b; Li et al., 2018; Hosseini et al., 2017) and translation (Ebrahimi et al., 2018a; Michel et al., 2019).

we conclude that the position of the trigger keyword has minimal effect on the success of the attack.


They are only understood by a limited set of people, because they require domain knowledge.

they provide an overview of what characterizes technical vocabulary, and observe two main categories.


A consolidated representation is finally achieved by a linear combination of the outputs from these N different GCNs.

mOH-X is more focused and VmWEs, for the most part, coincide with the target verb.


Analysis on the influence of the composition of S, i.e., instance numbers of two types, is presented in Section 4.5.

6 volunteers are asked to select user-posed questions and the corresponding review sentences that can serve as answers.


The preliminary neural QG models (Du et al., 2017; Zhou et al., 2017; Du and Cardie, 2017) outperform the rule-based methods relying on hand-craft features, and thereafter various models have been proposed to further improve the performance via incorporating question type (Dong et al., 2018), answer position (Sun et al., 2018), long passage modeling (Zhao et al., 2018b), question difficulty (Gao et al., 2019), and to the point context (Li et al., 2019).

we adapt the Seq2Seq model for the aspect-focused generation model, which is updated gradually via the transferred and augmented instances.


In each iteration of the algorithm, the generator is trained with current S, and then S is adapted accordingly.

one major challenge of this generation task is the lack of training data, i.e.


the most relevant review sentence for each question is retrieved via BM25 method, and such review-question pairs are added into the training set.


We also present a method of generating this adversarial training set in linear time by making use of the adversarial examples' inflectional distribution to perform weighted random sampling.

the difference becomes significant if we look only at the SQuAD 2.0-fine-tuned models' performance on answerable questions (7% difference).


This is reasonable as the "O" labels by far make up the majority of all labels in NER datasets.

vector representations are obtained for each unique word in the datasets.


The reward R((y), y) is the nondifferentiable evaluation metric, i.e., BLEU and ROUGE (details are in Section 7).

if the copying operation is selected, then a type-restricted copying mechanism is enabled to restrict the search space by masking down the illegal grammar types.


Our proposed Type-associated Encoder is designed as a variant N -ary Tree-LSTM.

our proposed framework also verifies the necessity of the type information in the code translation related tasks with a practical framework and good results.


Operation Selection Stage: Operation Selection Stage determines either using the copying operation or the generation operation to select the words based on the attention vector and hidden states from the encoder.

Xu et al. (2018b) considers a SQL query as a directed graph and adopts a graph-to-sequence model to encode the global structure information.


Although it is possible to train our framework by using the maximum likelihood estimation (MLE) method which constructs pseudo-labels or marginalize all the operations in the operation selection stage (Jia and Liang, 2016;Gu et al., 2016), the loss-evaluation mismatch between MLE loss for training and non-differentiable evaluation metrics for testing lead to inconsistent results (Keneshloo et al., 2019;Ranzato et al., 2015).

for the case in SQL, the keyword "Otkrytie Area" is missing in all the baselines but accurately generated by our framework.


The incorporation of pseudo-tags is a standard technique widely used in the NLP community, (Rico et al., 2016; Melvin et al., 2017).

these results support the claim that explicit operations for defining K virtual models have a significant effect for a single model and are complementary to normal dropout.


We followed the instructions of the task settings used in CoNLL-2000 andCoNLL-2003.

nORMALEnS produced the best results in this setting.


The two results are from the state-of-the-art system on the WMT2017 test set, which is selected based on METEOR.

finally, these representations provide an attention-based context vector for the decoder.


To deal with this issue, in this paper, we propose a novel graph-based multi-modal fusion encoder for NMT.

we then stack multiple graph-based multi-modal fusion layers that iteratively perform semantic interactions to learn node representations.


Since the size of training corpus is small and the trained model tends to be over-fitting, we first perform a small grid search to obtain a set of hyper-parameters on the En⇒De validation set.

our multi-modal NMT model significantly outperforms several competitive baselines.


Non-autoregressive neural machine translation (NAT) predicts the entire target sequence simultaneously and significantly accelerates inference process.

step" denotes the average number of decoding steps.


We thank the anonymous reviewers for their valuable feedback.

Besides, the correlations of labels (e.g. hierarchical label structures (Banerjee et al., 2019)) and the existence of tail labels make MLC a hard task (Bhatia et al., 2015).


Extensive experiments are conducted on four benchmark datasets.

adaptive routing is also applied with the maximum candidate label number set equally to 200.


Since NER is not the focus of this study, the readers can choose the public Chinese NER API 4 from Baidu for fast experiments.

interpretability is very important in the Ai-empowered healthcare studies.


Besides the patient's basic information like name, age and gender, an EMR document contains Chief Complaint (CC), History of Present Illness (HPI), Physical Examination (PE), Test Reports (TR, e.g.

upper Respiratory Infection (uRI) is easy to diagnose because it causes throat pain and rhinorrhea unlike the other respiratory diseases.


In the offline processing of the EMR corpus, we preserved the Top-K most frequent entities of all types as the entity vocabulary.

it is very hard for computers to automatically understand all the diverse sections and capture the key information before making an appropriate diagnosis.


Similar conclusions can be drawn when comparing ECPE-2D (Inter-CE) and ECPE-2D (Inter-CE+WC/CR) as well as ECPE-2D(Inter-EC) and ECPE-2D(Inter-EC+WC/CR).

the above results again demonstrate the effectiveness of the proposed 2D transformer.


Early stance detection methods were concentrated on debates (Thomas et al., 2006;Somasundaran and Wiebe, 2009;Walker et al., 2012).

the second error category is caused by special hashtags with implicit meanings.


For each word w t , we extract the corresponding entity from SE-graph by performing n-gram matching and acquire a subgraph representation M 0 t .

first, we utilize the whole words from the semantic lexicon SenticNet (Cambria et al., 2018) as the word-nodes and add edges between the semantic words that capture the wordword semantic connections.


In this paper, we proposed a Semantic-Emotion Knowledge Transferring (SEKT) model for cross-target stance detection, which uses the external knowledge (semantic and emotion lexicons) as a bridge to enable knowledge transfer across different targets.

For example, Augenstein et al. (2016) proposed a bidirectional conditional encoding method by incorporating the target to learn the target-specific words.


The data in different domains usually shares certain background knowledge that can possibly be transferred from the source domain to the target domain.

instead of leveraging only the input context in each BiLSTM unit, we take external commonsense knowledge into consideration by adding a novel knowledge-aware memory unit to the BiL-STM, which dynamically controls the amount of external knowledge at each encoding step and thus balances the contextual and knowledge information for stance detection.


The SE-graph can capture the semantic connections between words and emotion tags with multihop connections.

we randomly select 100 instances that are incorrectly predicted by SEKT from the expanded SemEval-2016 dataset.


For domain-exclusive graphs, approaches include SSL with GCNs (Shen and Chung, 2019) and domainadversarial learning (Dai et al., 2019).

rather than using explicit divergence measures or domain-adversarial losses for domain invariance, we uniquely adopt a shared-autoencoder strategy on GCNs.


In conclusion, if we would like to observe the desirable behavior as discussed for the attention mechanism, it is important for us to choose an appropriate λ value or we shall possibly find ways to control the value of VTW6 .

their corresponding attention scores may end up with large positive scores eventually.


We show the tracked gradient norms on all datasets in Figure 5.

the vocabulary size for PtB/Yelp/Yahoo is 10K/15K/20K.


it is a myth that the UNK of the war is not a war , but it is not possible to be able to see the war .

the (coupled) posterior network and the (coupled) decoder are additionally conditioned.


The upper bound follows the same derivation.

in this paper, we propose a structure-free certified defense method that applies to arbitrary models that can be queried in a black-box fashion, without any requirement on the model structures.


We hypothesise that exposure bias (Ranzato et al., 2016), a discrepancy between training and inference, makes this problem worse.

the model trained with teacher forcing may over-rely on previously predicted words, which would exacerbate error propagation.


Over time, the number of such initiatives grew substantially, e.g., at the time of writing, the Duke Reporters' Lab lists 237 active fact-checking organizations plus another 92 inactive.

we can see that, for PolitiFact, only 27% of the pairs have a similarity score that is above 0.25, while for Snopes, this percentage is at 50%, which suggests Snopes should be easier than PolitiFact.


We used information from it to create a second dataset, this time focusing on tweets.

We propose a learning-to-rank model that achieves sizable improvements over state-ofthe-art retrieval and textual similarity models.


We can see on line 1 of Table 1 a trivial case, where the verified claim is identical to the input claim; however, such cases are not very frequent, as the experiments with the BM25 baseline in Section 7 below will show.

we further try semantic matching using BERT.


In section 3, we discussed that matching some of the input claims with the corresponding verified claims can be a non-trivial task, and we gave examples of easy and hard cases.


Going one step further, to further validate the actions, we input the texts into our transition system to obtain the "pseudo-gold" emotion-cause pairs P based on the annotation, which can give us the correct action to take for a given parse state.

besides, action history is stored to a list A.


Different from the traditional emotion analysis, which aims to identify emotion categories in text.

the acyclicity and single-head constraints are not necessary for our model, as arbitrary graphs are allowed.


Firstly, in the same unimodal task, the results under unimodal labels are better than those under multimodal labels.

in this part, we only report the results in MLF-DNN.


"M, A" as an example, the sub-network of subtask "A" is supervised by multimodal loss and unimodal loss.

Zadeh et al. (2018a) designed a memory fusion network with a special attention mechanism for cross-view interactions.


Besides, we introduce three late-fusion models into this framework as strong baselines for SIMS.

sIMs has three modalities and unimodal annotations except for multimodal annotations for each clip.


This contextual contrast and semantic association information acquired, in turn, can provide salient evidence to interpret the detection of multimodal sarcasm.

the D-Net breaks down the raw visual or textual representation into a shared subspace and unique visual or textual subspace through three layers.


Our proposed ANP-aware cross-modality attention mechanism explicitly calculates the cross interactive attention between text words and image ANPs, providing the explainable reasoning evidence for sarcasm detection.

we encode the multimodal inputs into 200-dimensional hidden space, and set the dimension of invariant shared feature to 40, the dimension of unique variant contrast feature to 40, Finally, we optimize our model by Adam update rule with learning rate 0.01, mini-batch 128, and weight of orthogonal loss 0.5.


In Fig.4b, our model pays more attention to the textual phrase 'these lovely books' with stupid sign, strange sign, and bad sign ANPs which refer to the emoji in the attached image.

sarcasm is a sophisticated linguistic phenomenon, defined by Merriam-Webster Dictionary as 'The use of words that mean the opposite of what you really want to say, especially in order to insult someone, to show irritation, or to be funny'.


Our work focus on the multimodal sarcasm detection using image and text modalities.

To tackle the above challenges, in this paper, we propose a novel method to model both crossmodality contrast and semantic association by constructing the Decomposition and Relation Network (i.e. D&R Net) for multimodal sarcasm detection task.


To examine the generality of our evaluation method, we conduct experiments on two NMT systems, i.e. RNN-SEARCH (denoted by RNN) and TRANSFORMER (denoted by Trans.), both of which are implemented with fairseq (Ott et al., 2019).

It is worth noting that Arras et al. (2016); Denil et al. (2014) directly measure the performance of the target model P on the extracted words without constructing Q to evaluate explanation methods for classification tasks. H


Each human coder segmented talk turns into utterances (i.e., complete thoughts) and assigned one code per utterance for all utterances in a session.

here  we also compare our model to the multimodal approach proposed by (Singla et al., 2018; Chen et al., 2019) where they use word-level prosodic features along with lexical word embeddings.


Models trained on the current interval are used to predict users' opinions in the next interval.

inspired by the multivariate Hawkes process (Aalen et al., 2008; Du et al., 2016), we propose to model a user's posting behaviour by a temporal point process that when user u posts a tweet d at time t, they need to decide on whether they want to post a new topic/opinion, or post a topic/opinion influenced by past tweets either posted by other users or by themselves.


Text style transfer consists of rewriting a sentence from a given style s i (e.g., informal) into a different style s j (e.g., formal) while maintaining the content and keeping the sentence fluent.

the main difference between our approach and these previous work consists in the fact that we use the feature matching loss to perform distribution matching.


This loss consists of matching statistics of the features for each style separately.

gANs fail to train when a very deep network is used as the discriminator Moreover, SeqgFMN also outperforms gAN generators even when shallow word embeddings (glove / FastText) are used to perform feature matching.


We adopt the metrics from the referred papers.

oDQA: (1) For each question, DSQA (Lin et al., 2018) aggregates multiple relevant paragraphs from ClueWeb09, and then infers an answer from these paragraphs.


The third stage must be run for each variant separately.

furthermore, rare words are not well represented in commonly used downstream task datasets.


These approaches exploit either the contexts in which rare words occur (Lazaridou et al., 2017; Herbelot and Baroni, 2017; Khodak et al., 2018; Liu et al., 2019a), their surfaceform (Luong et al., 2013; Bojanowski et al., 2017; Pinter et al., 2017), or both (Schick and Schutze, 2019a,b; Hautte et al., 2019).

We evaluate BERTRAM on the WNLaMPro dataset (Schick and Schutze , 2020).


Nevertheless, Schick and Schutze (2020) recently showed that BERT's (Devlin et al., 2019) performance on a rare word probing task can be significantly improved by explicitly learning representations of rare words using Attentive Mimicking (AM) (Schick and Schutze, 2019a).

as it performs best for both the RaRE and MEDIUM subset, we always use the aDD configuration of BERTRaM in the following experiments.


Knowing the Most Frequent Sense (MFS) of a word has been proved to help Word Sense Disambiguation (WSD) models significantly.

our approach, in fact, achieves on average a significant improvement of approximately 9 F1 points over the existing state of the art.


We randomly split this data into train (80%), dev (10%) and test (10%) sets.

The heatmap in Figure 2 shows that emotions that are related in intensity (e.g. annoyance and anger, joy and excitement, nervousness and fear) have a strong positive correlation.


In the past decade, NLP researchers made available several datasets for language-based emotion classification for a variety of domains and applications, including for news headlines (Strapparava and Mihalcea, 2007), tweets (CrowdFlower, 2016;, and narrative sequences (Liu et al., 2019), to name just a few.

raters were free to select multiple emotions, but were asked to only select those ones for which they were reasonably confident that it is expressed in the text.


Given the simplicity of that model, it performs surprisingly well.

we want the parser to choose the transition sequence that would make the most out of this bad situation.


It can also recognise garden-paths: if we are in a state that cannot generate the following word that means we are in a garden-path and will punish all transitions from that state.

with MBR they all manage to outperform the non-incremental model.


For metrics, we use FPED and FNED in IPTTS to evaluate how discriminatory the models are, and lower scores indicate better equality.

although the method is reasonable, due to high cost, it is not always practical to add additional labeled data with specific identity-terms, as careful selection of the additional sentences w.r.t.


Based on the theorem, in this paper, we propose to evaluate models under a distribution where the demographic identity information is not predictive of labels to unify the three widely-used criteria.

aUC to all the other methods, indicating that our method does not hurt models' generalization ability very much.


As the discrimination distribution is directly observable, estimating P (y|z) is not hard.

recent research points out that there widely exist some unintended biases in text classification datasets.


In one of the their experiments Alishahi et al. (2017) use a linear classifier to predict phonemes from local activation patterns of a grounded language learning model, where images and their spoken descriptions are processed and mapped into a shared semantic space

we find that in our setting, RSA applied to local representations shows low correlations between phonemes and neural activation patterns for both trained and randomly initialized target models, and for one of the target models the local diagnostic classifier only shows a minor difference in the decodability of phonemes from randomly initialized versus trained network.


The user, convinced by the nicer-looking explanations, performs better using this system.

despite the difference between the two criteria, many authors do not clearly make the distinction, and sometimes conflate the two.


Vanilla LSTM: The model described in section 2.1 which uses the vanilla LSTM.

the output predictions will not change much even if the attention weights are permuted.


The idea of orthogonalizing representations in an LSTM have been used by (Nema et al., 2017) but they use a different diversity model in the context of improving performance of Natural Language Generation models In this work, we have analyzed why existing attention distributions can neither provide a faithful nor a plausible explanation for the model's predictions.

there is little change in the vanilla LSTM model's output for several datasets suggesting that the attention weights are not so meaningful.


Different from all these methods, in this work, we consider both the high-order connectivity information and latent preference factor underlying the user-news interactions.

after obtaining the latent variables {r d,k }, we can find an estimate of z u,k by aggregating information from the clicked news, which is computed as Eq.


We still adopt the Transformer (Vaswani et al., 2017) as our backbone, with a embedding size of 512 and FFN layer dimension of 1024.

among existing NMT models, most of them are generally equipped with 4-8 encoder and decoder layers (Wu et al., 2016;  Vaswani et al., 2017).


In this section, we introduce the details of the proposed approach, a MultiScale Collaborative (MSC) framework for constructing extremely deep NMT models.

the 72-layer MSC exhibits higher training BLEU than the 36-layer counterpart and is generalizable to the validation data.


Finally, as opposed to considering results across all possible E, dw, and wp, we first select those settings that maximize the Pearson correlation for each dimension.

this variance is on display in the right-hand plot in Figure 5, which gives results for the blackness dimension.


The measurements in our survey are compared to results from Smith-Lovin and Robinson (2015).

three sets are characterized by salient trait similarities and differences on gender, age or race/ethnicity.


And stronger correlations between the embedding and survey-based measures for Evaluation and Potency, relative to Activity, reflects the increased importance in affective perceptions of these two dimensions (Rogers et al., 2013).

finally, we include three random identities as a mechanism for comparison in other work not described here.


Here, we investigate the extent to which publicly-available word embeddings accurately reflect beliefs about certain kinds of people as measured via traditional survey methods.

here, Figure 4B) shows a significant positive correlation between variance in the survey data along a dimension (and hence measurement accuracy) and that dimensions' importance in explaining patterns of labeling in our identity labeling task.


(Rajani et al., 2019) We perform minimal processing, primarily deletion of any questions without a rationale or questions with rationales that were not possible to automatically map back to the underlying text.

we build upon these methods in Section 4.


However, for some datasets we have explicitly collected comprehensive rationales for at least a subset of the test set.

more recent work has aimed to exploit rationales in training neural text classifiers.


Towards this end, we first initialize the parameters of DeFormer with the parameters of a pretrained full Transformer, and fine-tune it on the downstream tasks.

we measure the FLOPs and memory consumption through the TensorFlow Profiler.


Previous QA neural models like BIDAF (Seo et al., 2016), QANet (Yu et al., 2018) and many others contain decomposition as part of their neural architecture design.

the effectiveness of decomposition generalizes further beyond QA tasks as long as the input sequences are paired.


For all experiments in this paper, we used RoBERTa (Liu et al., 2019), a state-of-the-art transformer-based method.

after retrieving N parallel evidence chains, we take the union of all the individual justification sentences to create the supporting evidence text for that candidate answer.


In QASC, which has a large KB4 of 17.4 million sentences), similar to Khot et al. (2019a) candidates are retrieved using the Heuristic+IR method which returns 80 candidate sentences for each candidate answer from the provided QASC KB.

to extract N parallel chains, we run AIR N times, ensuring that the first justification sentences in each chain are different (in practice, we start a new chain for each justification in the top N retrieved sentences in the first hop).


We detail these components in the next two sub-sections.

highlighting strengths of AIR over the standard IR baselines.


Access to such data can greatly facilitate investigation of phonetic typology at a large scale and across many languages.

notice that phonemes that typically had high accuracy in Figure 6, such as /p/ and /b/, now have far more variable accuracy in Figure 8, suggesting difficulty in aligning the Unitran pronunciations to the correct parts of the audio.


In fact, human perception of complexity is most affected by syntactic features related to sentence structure (Brunato et al., 2018).

we have introduced ASSET, a new dataset for tuning and evaluation of SS models.


We also included an example where all transformations were performed.


Sentence Simplification (SS) consists in modifying the content and structure of a sentence to make it easier to understand, while retaining its main idea and most of its original meaning (Alva-Manchego et al., 2020).

aSSET offers more variability, perhaps signalling that annotators consider deleting infor-mation as an important simplification operation.


Crucially, our method is fully general and assumes only that the contextualized model maps word sequences to vector sequences.

additionally, while most prior work has discussed the static embedding setting, recent work has considered sentence encoders and contextualized models.


comparing downstream performance for models initialized using the static embeddings and the original contextualized representations).

in this case, we back-offed to using the Decontextualized representation for w i .


This pattern suggests that using bidirectional versus unidirectional context has more impact on distribution of context information than does depth or architecture type.

we use several classifier architectures in our probing tasks, in order to explore the impact of classifier complexity on extraction of our target information types.


Using similar methods to test syntactic awareness in BERT, Goldberg (2019) finds the model to perform almost at ceiling on syntactic tests.

gLoVe embeddings are at chance on all words but the target word, while gPT embeddings are at chance before the target word, and pattern similarly to BERT afterwards in the object identity task.


"If we are unable to make payroll Oct. 19, we will definitely be able to make it next week Oct. 26 based on the nature of our sales taxes coming in at the end of the month.

training on this new data yielded the state of the art on existing NLI benchmarks.


CheckList provides a framework for such techniques to systematically evaluate these alongside a variety of other capabilities.

further, neither is robust to typos or simple paraphrases.


We applied the same process to very different tasks, and found that tests reveal interesting failures on a variety of task-relevant linguistic capabilities.

"behavioral testing" (also known as black-box testing) is concerned with testing different capabilities of a system by validating the input-output behavior, without any knowledge of the internal structure (Beizer, 1995).


We next conduct a thorough investigation of the similarities and differences between dialoguebased and traditional relation extraction tasks by comparing DialogRE and the Slot Filling dataset (McNamee and Dang, 2009; Ji et al., 2010, 2011; Surdeanu, 2013; Surdeanu and Ji, 2014), and we argue that a relation extraction system should be aware of speakers in dialogues

It has been shown that there is a longer distance between two arguments in the SF dataset (Surdeanu, 2013; Huang et al., 2017) compared to that in many widely used human-annotated relation extraction datasets such as ACE (Doddington et al., 2004) and SemEval (Hendrickx et al., 2010).


While it is tempting to equate such information with the meaning of an utterance, a large body of literature in linguistics and psycholinguistics argues that an utterance conveys much more than a simple set of facts: it carries with it a halo of intimations arising from the speaker's choices, including considerations of perspective, emphasis, and framing.

Systems following instructions also require a means of segmenting continuous sensorimotor data and linking it to discrete linguistic categories (Regneri et al., 2013; Yagcioglu et al., 2018) (cf. the symbol grounding problem (Harnad, 1990)).


For example, in "of course, [we] 1 still have the [storm surge] 2 coming," given the context, we can reasonably infer discontent towards the "storm surge" despite the absence of polarizing words.

intuitively, this makes sense as the decisions are only slightly better than being in complete disagreement (i.e., orthogonal).


we introduce binary classification tasks, one for each Plutchik-8 emotion.


Such a pattern has become prominent and is misleading the whole community.

hence, we propose to use RANDOM evaluation scheme for all model performance comparisons.


Several recently proposed methods report high performance gains on a particular dataset.

from these results, we can draw similar conclusions as in Section 5.


In Figure 3, we plot the ratio of neurons becoming zero after ReLU activation for the valid triplets vs. their normalized frequency on FB15k-237 dataset.

we notice that several recent papers report very high performance, which largely outperforms previous state-of-the-art methods.


Combined with inappropriate evaluation protocol, such methods reported inflated performance.

bOTTOM evaluation protocol can be unfair to the model during inference time because it penalizes the model for giving the same score to multiple triplets, i.e., if many triplets have the same score as the correct triple, the correct triplet gets the least rank possible.


Another line of work (Ning et al., 2019) explores the cost vs. benefit of collecting full vs. partial annotations for various structured predictions tasks.

by providing intermediate reasoning steps explicitly, the annotations we collect help the model overcome some of these biases in the training data.


We investigate the effects of spending a small additional budget, either by adding more QA pairs (from the biased data distribution) or by collecting intermediate annotations, on this bias.

they do not focus on intermediate reasoning required to learn the task.


Despite its simplicity, this yields competitive or even better performance than other sampling strategies.

our method significantly improves the performance of RoBERTa-based models by 1-5% across datasets.


Recent proposed approaches have made promising progress in dialogue state tracking (DST).

in this section, we will describe DST-SC model in detail.


Our model achieves significant improvements on two public datasets and shows effectiveness on relatedslot problem tests.

ellipsis and reference phenomena among multi-domain slots are still less explored in existing literature.


We use our partial classifiers to throw away a fraction α of candidates, to increase throughput.

depending on the values of k i , SR models with different trade-offs between accuracy and efficiency can be obtained.


Asadi and Lin (2013) studied effectiveness/efficiency trade-offs with three candidate generation approaches.

as we will show in Section 5, an opportune choice of a threshold and good accuracy of early classifiers ensure high probability of having at least one positive example in the candidate set for the last classifier of the cascade.


First, most of these models are pre-trained on formal writing, which is notably different from colloquial writing in dialogue; thus, fine-tuning for the end tasks is often not sufficient enough to build robust dialogue models.

for span-based QA where the evidence documents are in the form of multiparty dialogue, the performance is still poor even with the latest transformer models (Sun et al., 2019; Yang and Choi, 2019) due to the challenges in representing utterances composed by heterogeneous speakers.


Recently, pretrained language representation models (Kocijan et al., 2019;Radford et al., 2019;Liu et al., 2019) have demonstrated significant improvements in both unsupervised and supervised settings.

as the original purpose of proposing both WSC and WinoWhy is to evaluate how good current aI systems can understand commonsense knowledge rather than solve these questions by fitting the dataset, the unsupervised setting might be the more reasonable evaluation setting.


Moreover, experimental results on different knowledge types prove that such a conclusion is universal rather than for a specific kind of knowledge.

we divide all WSC questions based on how many knowledge types are required to solve these questions and show the result in Table 5.


We set the hidden dimension to 128 for both clas-sifier and discriminator.

they are only used during the test period.


This is a multilingual sentiment classification dataset (Duh et al., 2011) in four languages, in-cluding English (en), French (fr), German (de), and Japanese (ja), covering three products (book, DVD, and music).

we consider two views of input: (i) the encoded labeled documents from the source language; (ii) the encoded back-translations of the source documents from the target language.


Recruiting assessors on Amazon Mechanical Turk (MTurk), they use intra-class correlation to estimate inter-rater agreement, with an average value of 0.42 over all topics, thus also indicating a poor reliability (Portney et al., 2009).

the collected data and a reference implementation of our model are made available in form of the Webis-ArgQuality-20 corpus, one of the largest and most detailed corpora for pairwise argument quality.


By example, going from x = 1, k = 16 to x = 1, k = 4 ends up at the same number of comparisons as x = 2, k = 8, but has a slightly higher ranking accuracy.

we explore the lower bound of effort needed to infer labels of sufficient quality.


To ensure some topic diversity and relevance of the arguments to the topic, while keeping the amount of judgments within our budget limits, we (1) indexed the args.me corpus using three retrieval models from the Terrier information retrieval library (Ounis et al., 2006) (namely BM25, DPH, and Dirich-letLM), (2) retrieved texts for 20 topic queries at a depth of 50 texts per topic per model, (3) and pooled all 3000 retrieved texts to remove overlap between the different models.

we refrain from making a general suggestion for the choice of k. Thus, if the model is to be adapted to drastically different domains or item counts, exploratory studies are advised to estimate the quality tradeoff for a specific use case.


More specifically, our ConKADI outperforms baseline models in terms of all metrics except BLEU-3 on the Chinese Weibo, and our ConKADI outperforms baseline models in terms of almost all metrics on the English Reddit.

1) An entity word usually can refer to different concepts, i.e., an entity has multiple meanings, but only one specific concept is involved in a particular context.


In fact, this issue comes from the unusual score distribution, e.g., the score function for some invalid triples gets the same values as the valid triples.

it is useful to determine the importance of each neighbor for a specific query.


We will leave the comparison and deep analysis in the future work.

we thank anonymous reviewers for their thorough review comments on this paper.


Please note that lower MR, higher MRR and Hits@10 indicate better performance.

convE is limited by the number of interactions between the head and relation embeddings (Jiang et al., 2019; Vashishth et al., 2020).


Pronoun-Focused Automatic Evaluation For the models in Table 3, we employ three types of pronoun-focused automatic evaluation: 1.

when no context is used, there is a statistically significant drop in BLEU, while APT and F1-scores are equivalent to that of the baseline.


We implement the approach with the LSTM-based recurrent network and use the following set of hyper parameters to train models: hidden size is 256, learning rate is 0.001, learning rate decay is 0.5, dropout is 0.3, batch size is 150.

we evaluate our system on TABFACT (Chen et al., 2019), a large benchmark dataset for tablebased fact checking.


In terms of the model efficiency, neural topic models can be trained much faster than HDP on a large corpus by GPU acceleration.

for the batch size, the learning rate, and other model parameters, grid search is carried out on the training set to determine their optimal values and achieve the held-out performance.


Unfortunately, it needs to traverse every word of all topics (i.e., threads) in the whole corpus when updating the topic-word distribution, rendering a large time cost for thread communication.

gSM and gNB-NTM are slightly slower than others because the former introduces more parameters to model the topic-word distribution, while the latter introduces more sampling operations.


To evaluate the quality of topics generated by different models, we use perplexity and topic coherence as evaluation criteria.

nVDM, nVLDA, ProdLDA, and GSM are all neural topic models based on nVI.


Then we run our NB-NTM and GNB-NTM on the entire 20News testing set to get the corresponding values of r. After normalization, the proportion of different topics obtained by NB-NTM and GNB-NTM at the corpus level is presented in Figure 4 (b) and Figure 4 (c), respectively.

document modeling via mixed counting models is easy to interpret but difficult to infer.


We use the variational lower bound to calculate gradients and apply Adam to update parameters of GNB-NTM, which are the same as NB-NTM.

how to effectively integrate the distributed dependencies in mixed counting models into the framework of variational inference is still quite a challenging problem.


Here again the issue is representing multiple entities at the same time, but with the additional requirement of representing the structural relationships between these entities.

the rules were soon learned with statistical methods, followed by the use of neural networks to replace symbols with induced vectors, but the most effective models still kept structured representations, such as syntactic trees.


But they still required linguistically-motivated designs to work well.

boV representations are nonparametric representations, in that the number of vectors in the bag can grow arbitrarily large, and these vectors are exchangeable.


We think that it is far more likely that we will be able to design neural architectures which induce multiple levels of representation than it is that we can ignore this problem entirely.

lP/lR/F1: labelled constituent precision/recall/F-measure.


In a more linguisticallyinformed approach to the same problem, statistical models have been proposed for morphology induction (e.g. (Elsner et al., 2013)).

attention-based models have variable binding, which sequential LSTMs do not.


NMN estimates the similarities between entities to capture both the topological structure and the neighborhood difference.

it follows a four-stage processing pipeline: (1) KG structure embedding, (2) neighborhood sampling, (3) neighborhood matching, and (4) neighborhood aggregation for generating embeddings.


This is because there are many city entities for England which also have the three entities in their neighborhoods -e.g., the entity Birmingham.

nMn achieves the best Hits@1 score on DBP15KZH−EN , with a gain of 2.5% compared with RDGCn, and 5.4% over GMnn.


Therefore, we first pretrain the GCN-based KG embedding model to produce quality entity representations.

to evaluate different components of our model, we provide two implementation variants of NMN: (1) NMN (w/o nbr-m), where we replace the neighborhood matching part by taking the average of sampled neighbor representations as the neighborhood representation; and (2) NMN (w/o nbr-s), where we remove the sampling process and perform neighborhood matching on all one-hop neighbors.


"mIoU" is the average IoU over all testing samples.

after transforming moment annotations in NLVL dataset, we obtain a set of (V, Q, a) triples.


Third, there is no risk of compounding errors as compared to the transitionbased approach.

after the above procedures, there are still minor differences to PTB-Concat.


The WSJ test set has 2416 sentences with arbitrary lengths, while WSJ10 consists of 7422 sentences of the whole WSJ corpora that contain no more than 10 words.

unlike for transitionbased approaches (Kim et al., 2019b), for distancebased approaches there have been no studies examining the relationship between induced syntactic structure and human labeled syntactic structure, or whether human labeled syntactic trees can be used to improve language modeling (Dyer et al., 2016;Kim et al., 2019b).


The idea is that an EDIT TREE is only valid if it can be applied to multiple given lemmas.

we compare to one version of this baseline that has access to the ground-truth paradigm size (LB-Truth), and a second version which predicts the paradigm size as the average over the development languages (LB-Dev).


An intuitive method to obtain additional lemmas would be to train a lemmatizer and to generate new lemmas from words in our corpus.

note also that only the corpus and the lexicon can be accessed by our system, and no ground-truth morphological information (including paradigm size) is given.


We utilize the same dataset (CamRest676 or Mul-tiWOZ) to train all the models for fair comparison.

we do not substitute key words associated with slot values to maintain the dialog function of utterances.


During testing, only the ground truth user utterances are used as input.

for example, phrase dictionary (Cao et al., 2017) and semantic annotations (Wang et al., 2019) are used to assist the paraphrase model to improve the language quality.


Besides, they need to logically correlate with the answers by deducing over multiple entities and relations in several sentences and paragraphs of the given text.

with only limited labeled data, our model can effectively leverage unlabeled data to guide the generation.


These data sets were collected by crowd-sourcing, consisting of 97k, 35k, and 97k examples, respectively.

we first produce a segment based on the latent state z t , and then emits term with a length of l t on that segment.


In speech translation, Sperber et al.  (2019) used the longest distance to the start node to indicate lattice structure, and Zhang et al.

for lattice LSTM, our model has an average f1 improvement of 1.51 over it.


Table 3 shows two metrics of three models on the devlopment set of Ontonotes and MSRA.

due to NER's strong alignment of label and input, their model needs an RNN module for encoding.


In this case, Q gen is similar to Q in that the item is somewhat related to Q gen , and there's a chance that I may be matched to Q gen due to keyword stuffing by sellers, or poor semantic matching.

we also compare our model trained on the original training data and one augmented by naively adding the 3M matched pairs.


Holding the great promise of deep neural networks in language and images, Transformer capitalizes on the stacked multi-headed self-attention mechanism based on the conventional encoderdecoder architecture in a sequence-to-sequence (seq2seq) manner to learn the global soft signals without explicit recurrence mechanism.

To probe whether it is required to augment SDUs on each Transformer layer, we supplement gates on layer 1-3, layer 3-6, and layer 1-6 but removing gates on FFN components (denoted “\FFN”) as in Table 5 (see Fig. 8 in Appendix B for detailed convergence curvatures).


The de-identification performance of STACKED is comparable to the PIPELINE model, the de-identification performance of MULTITASK is slightly lower, however, we only found statistically significant differences for the MULTITASK model for English.

state-of-the-art methods for de-identification typically rely on recurrent neural networks (RNNs) (Dernoncourt et al., 2016;Lange et al., 2019b;Kajiyama et al., 2018).


Newman-Griffis and Zirikly (2018) study the performance of RNNs for medical named entity recognition in the context of patient mobility and find that they benefit from domain adaption.

Feutry et al. (2018) and Friedrich et al. (2019) create pseudo-de-identified text representations with adversarial training.


The original data contains 1.1M multi-turn conversations.

the second approach is to take a weighted average score of a seq2seq model trained on D and a language model trained on Dt when decoding responses.


The seq2seq model S2S is trained on D as an indicator of how relevant each response is with the context.

the model is still trained on the limited dialogue corpus which restricts its power at covering broad topics in opendomain chitchat.


The user requests to book one ticket in the second example, yet both HDSA and Human Response ask about the number once again.

a global vector is unable to capture the inter-relationships among acts, nor is it flexible for response generation especially when more than one act is mentioned.


Extensive experiments demonstrate the superiority of MMT over state-of-the-arts, and continuous improvements can be achieved on different backbone networks on both supervised and unsupervised domain adaptation settings.

only the "similar" source data will be beneficial for multi-source meta learning.


However, multi-source meta learning alone cannot guarantee the desirable performance due to the data distribution gap between multiple sources and target data.

while, the weaker backbone network is, the better improvement MMT can achieve.


For example, the MMT(BERT-Base) improves BERT-Base over 14% on MCTEST.

in this way, a more generalized representation could be obtained by considering multiple source datasets.


In this experiment, we evaluate the transferability between different datasets and further give the suggestion on the source selection for MMT.

multi-source meta learning alone cannot guarantee the desirable performance due to the data distribution gap between multiple sources and target data.


This often led to a mismatch between ASR trained on data from the spoken domain, and MT trained on data from the written domain.

models that do not suffer from erroneous early decisions will expectedly exhibit an advantage over other models especially for acoustically challenging inputs, and less so for inputs with clean acoustics.


Being a span enumeration type model, DYGIE++ only works on paragraph level texts and extracts relations between mentions in the same sentence only.

we also tried integrating it into our model, where we classify pairs of "span embeddings" (not the surface form) but found the separate model that uses surface forms to work much better.


The 0.5 threshold enjoys the property that, assuming all predicted clusters are disjoint from each other (which is the case by construction) and gold clusters are disjoint from each other (which is the case for 98.5% of them), a single predicted cluster can be assigned to atmost one gold cluster.

we predict clusters of mentions using our model (mention identification, pairwise coreference, and mention clustering).


For training all of the above models, we use the batch size of 6, a learning rate of 2e-5, and 3 epochs.

in this section, we compare the performance of the models as presented in Table 1, draw empirical observations, and attempt to provide a rationale for the performances observed.


For each point from both XR and Y, we compute a "personalized" translation vector, as the weighted average of the translation vectors of its closest dictionary entries.

iV is supported by the ERC Consolidator Grant LEXiCAL (no 648909).


Hellendoorn et al. (2020) develop a similar model concurrently with this work, where they use relation-aware self-attention to encode data flow structure in source code embeddings.

the alignment matrix successfully infers to use these two tables instead of car_makers using the evidence that they contain the three mentioned columns.


Brunner et al. (2020) suggest that representations produced by Transformers mix the information from different positions and cease to be directly interpretable after 2+ layers, which might explain our observations.

in many applications (including text-to-SQL parsing) we are aware of some preexisting relational features between the inputs, and would like to bias our encoder model toward them.


To the best of our knowledge, text generation based on the predefined rigid formats has not been well investigated.

in this work, we propose a simple and elegant framework named SongNet to address this challenging problem.


We remove the cross-modality Transformer and use single-modality representations as entity representations.

researchers also extended Transformers with both textual and visual modalities (Li et al., 2019b;Sun et al., 2019;Tan and Bansal, 2019;Su et al., 2020;Tsai et al., 2019).


There are several challenging research directions for addressing learning representations for crossmodality data and enabling reasoning for target tasks.

for NLVR 2 dataset, the task is binary classification that determines whether the statement is correct regarding the images.


Sophisticated pre-training strategies were introduced to boost the performance (Tan and Bansal, 2019).

nLVR 2 : The results of nLVR task are listed in Table 1.


The random caption speaker outputs one of the ground-truth captions for the target image at random.

in the absence of a human listener to provide rewards for learning, we use the oracle fixed listener, which was found in Section 5 to be predictive of human referential success.


Single-headed cross attention speeds up decoding: Despite removing learned self-attention from both the encoder and decoder, we did not observe huge efficiency or speed gains.

hC-SA even outperforms BASE for sentences with fewer off-diagonal tokens.


It is possible that, at some future point in time, spelling will be deemed to lie outside of the construct targeted by the consequential assessment of writing and spell-correction software will be made available to test-takers.

add to the mix the finding that automated spelling correction system is more accurate on essays that are of better quality to begin with (Flor, 2012), and it's likely that the automated assessment of an automatically spell-corrected version of an essay might show an unexpected relationship with original scores that would need to be closely examined for bias or for an increase in construct-irrelevant variance.


Our synthetic corpus study (Table 5) shows that BERT is able to memorize negative facts that occur in the corpus.

while the prediction probability generally is somewhat lower in the negated compared to the positive answer, there is no threshold across cloze questions that could be used to distinguish valid positive from invalid negative answers (cf.


In the preprocessing step, we removed stop words, converted all text into lowercase, and stemming.

more importantly, OSDm tried to incorporate semantic information in the proposed graphical representation model to remove the term ambiguity problem in short-text clustering.


The reason behind might be the multiple iterations of each batch in the stream.

(4) MStreamF (Yin et al., 2018) is the latest model to deal with infinite number of latent topics in short text while processing one batch at a time.


A commonly used technique is to define the "context" for each node, for example by random walks, and train a predictive model to perform context prediction.Similar network-based learning is employed in node2vec (Grover and Leskovec, 2016).

albeit more accurate thanks to capturing stylistic variation elements, statistical models are prone to stereotype propagation as well (Costa-jussa et al., 2019; Koolen and van Cranenburgh, 2017).


A pure language model performs best, suggesting that BART is less effective when the output is only loosely constrained by the input.

to better understand these effects, we also report an ablation analysis that replicates other recently proposed training objectives.


Nevertheless, BART outperforms all existing work.

this objective allows predictions to condition on both left and right context.


Because BART has an autoregressive decoder, it can be directly fine tuned for sequence generation tasks such as abstractive question answering and summarization.

xSum is highly abstractive, and extractive models perform poorly.


It is also evident that removing more components results in more performance drop, thus demonstrating the complementary nature of the three proposed components in this work.

this model seeks to show that the use of RDP for the similarity encouragement with RW is more effective than employing RDP directly in V .


In such models, the dependency trees are often used to directly structure the network architectures or to obtain the dependency relations between the word pairs to inject the syntactic information into the models via multi-task learning.

(v) SP-CEON-LSTM: This model removes the SDPS component and includes the representation vector of the dependency path DP (i.e., R DP ) in the final representation V for relation prediction.


In CE-LSTM, we replace the ON-LSTM network with the usual LSTM model in CEON-LSTM.

other countries including the Philippines, South Korea, Qatar and Australia agreed to send other help such as field hospitals, engineers, explosive ordnance disposal teams or nuclear, biological and chemical weapons experts.


The goal of this component is to obtain a score for each word w t that indicates the contextual importance of w t with respect to the relation prediction between w s and w o in W .

we replace the term L import in the overall loss function (i.e., Equation 6) with the dependency edge prediction loss (using the ON-LSTM hidden vectors) in DRPC for EP-ON-LSTM.


We train our LID models using this setting for 50 epochs.

to accomplish that, we introduce CS-ELMo, an extended version of ELMo that contains a position-aware hierarchical attention mechanism over ELMo's character n-gram representations.


We achieve the best results on Exp 3.3, which outperforms both the baselines and the previous state of the art on the full LID label scheme (see Table 2).

Figure 2B illustrates the position embeddings for bi-grams and tri-grams.


The feature vector f has an element corresponding to each named entity type in the KB (e.g., LOC, PER, and ORG).

the autoencoder takes the hidden states of the BiLStM as input to a fully connected layer with a sigmoid activation function and reconstructs the features.


In this section, we discuss our experiments on four low-resource languages and attempt to answer the following research questions: 1) "Although gazetteer-based features have been proven useful for neural NER on English, is the same true in the low-resource setting?"

this feature vector is applied to each word in the span, considering the position of the specific word in the span according to the BIO scheme; we use the "B-" vector elements for the first word in the span, "I-" otherwise.


that excludes the negative class name selection step and uses only the single positive class name in the entity selection module.

their method requires a user-provided class name and utilizes web tables as additional knowledge, while our method can automatically generate both positive and negative class names and utilize them to guide the set expansion process.


ince these synonyms consistently rank lower than the positive one for the initial seeds based on the given corpus, they are indeed not good class names for this specific corpus.

it only incorporates the average BERT representation to select entities.


Table A4 shows VQA accuracy for each answer type on VQACPv2's test set.

in the test set, answer 'yes' is more frequent.


Learning generic answers is a well-known problem for open-domain dialog systems, e.g. (Li et al., 2016).

Dense annotations, i.e.  crowd-sourced relevance weights, are provided for 0.16% of training set, which we use to fine-tune the model to select multiple semantically equivalent answers.


Multi-head attention provides multiple representation spaces to capture different linguistic/grounding phenomena, which are otherwise lost by averaging using a single head.

For example, in order to answer the question “About how many?” in Figure 1, we have to infer from what was previously said, that the conversation is about the skiers.


Of course, we expect pre-trained visual BERT models to show even more improvements on this task, e.g. Vilbert (Lu et al., 2019), LXMert (Tan and Bansal, 2019), UNITER (Chen et al., 2019) etc.

we then embed the question to pool the relevant grounded history using another MCA module.


Other studies consider additional linguistic features such as POS tags and word segmentation (i.e. tokens or morphemes) (Ananthakrishnan et al., 2005; Zitouni et al., 2006; Zitouni and Sarikaya, 2009; Shaalan et al., 2009).

to learn word based tasks with diacritic restoration, we pass WordtoChar representation to the diacritic restoration and/or ChartoWord representation for word-based tasks.


Identifying morpheme boundaries did not increase accuracy as we expected.

diacritic restoration is the task of restoring missing diacritics in the written text.


Inflectional diacritics are also affected by word's root (e.g. weak roots) and semantic or morphological properties (e.g. with the same grammatical case, masculine and feminine plurals take different diacritics).

we examined whether information learned from the BiLSTM layer would help us learn morpheme interactions by passing the output of last BiLSTM layer to the diacritic restoration model along with segmentation labels.


Zalmout and Habash (2017) develop a morphological disambiguation model to determine Arabic morphological features including diacritization.

inflectional diacritics are related to the syntactic positions of words in the sentence and are added to the last letter of the main morphemes of words (word finally), changing their pronunciations.


We use Modern Standard Arabic (MSA) and Egyptian Arabic (EGY) as test cases.

the context in the tagging network spans the entire input sentence.


Although increases exist across all domains, these are most prominent in domains like TC (+5.36) that have a low density of named entities and where indomain models have access to limited amounts of data.

we expect that the two tasks at different levels of granularity rely on shared structure in the original semantic space.


After that, the Similarity Module computes the similarity of these vectors via the same L2-distance-based operation as in CT.

in the second example in Table 4, it fails to understand that there is also a dictionary involved and ends up returning the wrong command.


Row 3 of Table 2 shows that the MP model on its own under-performs and actually has the worst results out of all the models we tested.

we compare four models: a baseline model (CT) that only considers text and source code, a (CAT) model that also includes embedding of Abstract Syntax Trees, a multi-perspective model (MP) that leverages multi-perspective matching operations as defined in a bilateral multi-perspective model (wang et al., 2017), and our MP-CAT model that combines both MP and CAT architectures.


This suggests that pre-training provides useful signal for mentions that consist of variations appearing in the ontology.

in the case of entities with CUis, we find that pre-training the model does provide a gain in ranking accuracy (MRR).


As mentioned in Section Â§1, however, such a brute-force method is computationally infeasible.

formally, assume that we have a set N , comprising experimental records (both features and scores) of n datasets for one task.


Next, we test whether models identify presuppositions of the premise as entailments, e.g., that Jo's cat went entails that Jo has a cat.

otherwise, a model might correctly predict entailment essentially by accident if, for instance, it systematically ignores negation.


We now list all the hyper-parameters used.

neutral: Not all rides at amusement parks are a movie.


For BERT models, we fine-tune the representations from the final layer for our parsing and tagging tasks.

a parsing approach taking into account the special status of headless structural representations can potentially benefit models for a large number of languages and treebanks.


To demonstrate the effectiveness of CRF, we compare results from models with or without CRF using each training technique mentioned above.

character-level information has proved to help improve the sequence labeling accuracy by capturing morphological features (Ma and Hovy, 2016).


To address this issue, we propose TLUA model, utilizing an attention scheme to automatically weight different LUs for the frame, according to target word T in the given sentence, shown in Figure 1.

f Eki denotes the i-th frame element of fk , and Pki denotes the ith span fulfilling f Eki .


For instance, in Figure 1 the entity types are movie and person, which are combined with the table cast.

for each node, the annotators can inspect the result of the execution.


we can limit the number of overall constraints and the number of maximum constraints for each entity type.


Furthermore, the table shows the average precision for each query complexity category.

this approach is very time-consuming and has a major issue.


Each question in the QuAC dataset is assigned one of three categories: should ask, could ask, or should not ask a followup question.

the similarity scores are computed based on vector cosine similarity.


The last follow-up question example is invalid since Verhoeven is a he, not she.

our proposed model relies on three attention matrices, where the two additional attention matrices make use of the associated passage.


Since MWPs are usually clearly specified (with a sure answer), there is no ambiguous interpretation once the answer is given.

furthermore, low-diversity corpora are typically characterized by highly similar problems, which usually yields over-optimistic results (Huang et al., 2016) (as the answer frequently can be simply obtained from the existing equation template associated with the most similar MWP in the training-set).


As can be seen, possession implicitly acts as an agreement feature (i.e., possession of the object and person of the verb must match).

typelevel probing tasks have the advantage of containing less bias (domain, annotator, and majority class); whereas token-level tests might be sensitive to the domain biases from the underlying full-text data.


For instance, while probing for the OddFeat between two forms, we assume that there exists a word pair differing only by one feature.

a word annotated with UniMorph features can be used in several probing tests.


In order to diagnose whether it does indeed extract morphologically relevant information during training, we save the model states for different epochs and generate the word representations via the aforementioned internal biLSTM layer and use our intrinsic evaluation suite from Section 4.3, to evaluate these representations.

dictionary-based tasks do not contain any frequency information of the surface forms, whereas token-level tasks do.


Although the performance boost for Case and POS are still visible in intermediate layer results, we notice that Person and Tense features may have been forgotten in the next layer.

the results for MUSE, word2vec, GloVe-BPE, and fastText are only provided for comparison among each other, and to gain insights on some of the aspects discussed in Section 3.5.


For instance, the German lemma Teilnehmerin would be inflected as Teilnehmerinnen as a plural noun marked either with accusative, dative, or a genitive case marker.

several recent studies analyze representations generated by neural models to get insights on what kind of linguistic information is learned by the models.


for instance, it can also be used for the synset: ((biology) a taxonomic group containing one or more genera) since all the chosen translations also fit in that sense.


But in general, there are too many constructions to keep track of, and special domains of language use come with their own constructions.

one possible solution is a layer of non-compositional post processing that finds paraphrases for inadequate trees (Section 3.5).


There, every synset corresponds to one abstract function and then the function's linearization in each language produces all words in the language as variants.

note that there is the Princeton Wordnet Gloss Corpus, which also contains sense annotations but only for some words-in order to construct complete trees we had to annotate all senses.


The common abstract syntax of the RGL can be seen as an intersection of the structures available in different languages.

this assumption has recently been challenged by an analysis of hyperbatic constructions in Latin, which make it possible to interleave any words contained in discontinuous constituents (Spevak 2010).


However, the recent advances of UD (Nivre et al. 2016) have not only shown that it is possible to assign the same syntactic structures to vastly different languages, but also ended up with structures very similar to the RGL (Kolachina and Ranta 2016).

aMR equipped with GF strengthens the status of aMR being an interlingua.


given Q + A, we use the procedure described in (Khot et al., 2020) to assemble candidate chains from T (below).


Similarly, another entailmentbased dataset is FEVER (Thorne et al., 2018), testing where a larger context entails a claim.

our work fills this gap by providing an explanation (e.g., "spoon is made of metal", "heat is energy" from a larger retrieved context).


More recently, multihop datasets where the decomposition is not evident have appeared, e.g., WikiHop (Welbl et al., 2018), OBQA (Mihaylov et al., 2018), and QASC (Khot et al., 2020), posing more a realistic explanation challenge.

this may be useful for helping a user understand the rationale behind a chain, and a repository of high-scoring patterns may be useful as a knowledge resource in its own right.


Multitask learning (Caruana, 1997; Collobert and Weston, 2008) seeks to learn a single model that can solve multiple tasks simultaneously, similar to our framework that seeks to learn a model that can solve many tasks.

our baselines build off the T5 11B model (Raffel et al., 2019): a text-to-text encoder-decoder structured transformer pretrained via masked language modeling and multi-tasking.


Attribution of natural disasters/collective misfortune is a widely-studied political science problem.

two different pre-trained LM weights are used to bootstrap our model.


Secondly, for their Distant Supervised Stance Detection (DSSD) method, Augenstein et al. train an LSTM on tweets where stance towards various entities or topics is labeled (cf. the SemEval 2016 Stance Detection shared task A dataset (Mohammad et al., 2017)).

we reimplement TD-LSTM using the code made available by the authors (c.f Table 2 and Appendix C).


First, in both domains, as more logical forms are replaced, the performance of all MRs declines gradually.

these values are not explicitly mentioned in the utterance and may even change over time.


Dataset Statistics Sentence-level annotations were aggregated by majority vote, yielding 252, 418, and 369 instances for the "Unrelated", "Some meaning difference", and "No meaning difference" classes, respectively.

we train the Divergent mBERT model by learning to rank synthetic divergences.


All models are trained on the OpenSubtitles 2018 corpus.

the training data for our models is a mixture of OpenSubtitles 2018 4 en-fr data and en-fr Gigaword 5 data.


We hypothesize that these datasets contain many examples where their gold scores are easy to predict by either having similar structure and word choice and a high score or dissimilar structure and word choice and a low score.

this trick allows for the expectation under q to be approximated through sampling in a way that preserves backpropagation.


The enhanced pre-trained model could be directly used in our approach.

we propose to recover the alignment information by masking signals of unimportant cells during self-attention.


To tackle these issues, some works adopt the distantly supervised methods (Mintz et al., 2009) to automatically annotate data with existing event facts in knowledge bases (Chen et al., 2017; Zeng et al., 2018; Araki and Mitamura, 2018) or use bootstrapping methods to generate new data (Ferguson et al., 2018; Wang et al., 2019b)

to verify the effectiveness of modeling multiple event correlations, the two models both adopt the conditional random field (CRF) (Lafferty et al., 2001) as their output layers, which can model structured output dependencies.


Nevertheless, the benchmark datasets for ED are upgraded slowly.

intermediate pre-training can improve ED on ACE 2005 with the general event knowledge learned on MAVEN, which indicates MAVEN's high coverage of event types can benefit other ED tasks.


We extract the sentence pair representation via the [CLS] token and treat it as a unary probing task.

we observe that grammatical tasks score high, while core role labeling lags behind -in line with the findings of Tenney et al.


In English, PropBank predicate token mixing weights emphasize the same layers as dependency parsing -in line with the previously published results.

2 We train the probes for 20 epochs using the Adam optimizer with default parameters and a batch size of 32.


In some of the embedding-based approaches, attribute types or values are utilized to generate attribute embeddings, which are integrated with structure embeddings of entities to get more accurate entity alignments.

if seeding alignments between two KGs are available, a set of aligned entities in an entity's neighborhood will be taken as the set-representation.


To let the entity-pair embeddings memorize the original attribute features, we add residual connections from the input features to the output layer of the GNN model.

in this work, our KG alignment model considers both relational and attributional triples in KGs.


Entities having the same or similar attribute values tend to be equivalent.

mRR is the average of the reciprocal ranks of the results.


For both models, we want the aligned entitypairs having higher scores than the non-aligned entity-pairs.

multiKE (Zhang et al., 2019) uses a framework unifying the views of entity names, relations and attributes to learn embeddings for aligning entities.


We form an action matrix A using hj , where the j-th row Aj corresponds to the j-th sentence (sj ) in D. A pseudo sentence indicating the STOP action, whose representation is randomly initialized, is also included in A, and sentence extraction is finalized when the STOP action is taken (Mao et al., 2018, 2019).

they lack holistic modeling of summary quality and the capability of end-to-end representation learning.


The KPE task is a foundational task, which plays a facilitating role in many Information Retrieval (IR) tasks, including classification, summarization, and document indexing (Hasan and Ng, 2014).

for each term, features describing itself and its parent block in the DOM Tree are included.


In this section, we will explain our model in detail.

multistage fusion consists of the cross fusion block and hierarchical fusion decoder, which aims to model the correlation and complementarity between modalities spontaneously.


We can observe that our model without the graph encoder has a 1.6% absolute value loss (over 25% in ratio) in BLEU score and a 1.1% absolute value loss (9.8% in ratio) in entity F1 on MultiWOZ 2.1, which suggests that the overall quality of the generated sentences are better improved by our graph encoder.

the model loops over K hops on an input graph.


It should be noted that even with a memory of 10k examples, the performance of continual learning algorithms are far from the i.i.d setting.

we construct our data streams using two popular vision-language datasets: COCO-captions (Chen et al., 2015) and Flickr30k Entities (Plummer et al., 2015) which provide multiple captions for each image in MSCOCO (Lin et al., 2014) and Flickr30k (Young et al., 2014) respectively.


Our proposed VisCOLL benchmark reveals that the gains observed in image classification tasks from state-of-art continual learning algorithms fail to transfer to VisCOLL even with increased memory.

(ii) effect of the large search space on memory-based continual learning algorithms.


Humans acquire language continually with much more limited access to data samples at a time, as compared to contemporary NLP systems.

visCOLL rules out many continual learning algorithms which require explicit task identity and boundary (Kirkpatrick et al., 2017;Rusu et al., 2016).


In VSM, we follow XML (Lei et al., 2020b) to
compute the matching scores between the query
and visual frames at both local and global levels.

in Figure 2a, we human could probably identify the moment by the speaker information and the visual clue of character's emotion.


It contains 200K unique video clip-caption pairs.

reordering happens after the multimodal fusion of subtitle and visual frames.


Premised on the assumption that the adversarial text generator is unknown beforehand, we propose to evaluate articles based on the semantic consistency between the linguistic and visual components.

we propose to leverage possible visual-semantic inconsistency between the article text, images, and captions, such as missing or inconsistent named entities (underlined in red).


The show had started more than an hour earlier.

mr. Germano was placed on leave after a New York Times investigation last month detailed the treatment of women at the company.


One plausible reason for this is that the main subject in the caption does not match the person who was mentioned in the article body and DIDAN is able to pick up on this relatively easily.

the human-generated articles are sourced from the GoodNews (Biten et al., 2019) dataset.


Masking implicitly performs gradient descent, analogy to the weights update achieved by finetuning; the observations complement our arguments in the main text.

(i) a large initial sparsity removing most pretrained parameters, e.g., 95%, leads to bad performance for the four tasks.


Figure 3 shows the accumulated number of parameters in million and memory in megabytes (MB) required when an increasing number of downstream tasks need to be solved using finetuning and masking.

we exclude 5 annotated words in dev and 87 annotated words in test.


TRANSFORMER (Vaswani et al., 2017) that is based solely on attention mechanisms.

it's worth noting that our approach achieves significant improvements without introducing any additional data and model modification.


Intuitively, if a training example has a low sentence-level probability, it is less likely to provide useful information for improving model performance, and thus is regarded as an inactive example.

the final NMt model is trained on the combination of the active examples and rejuvenated examples.


Increased training of the baseline also results in a drop in BLEU scores.

although backtranslation achieves highest BLEU score at 32.57, our fine-tuned CONCaT model achieves the highest F1 for pronoun translation at 72.39, without having been trained on any extra monolingual data.


This is similar to the features that Lin et al. (2019) consider, but without training a ranking model fed with scores from pairwise MT systems.

we need to obtain the NMT-learned embeddings first in order to fulfil those methods (from a 53-languages massive model).


For example, our QE model beat BLEU for zh-en, en-de and en-kk directions but got negative Pearson correlation with human evaluation for gu-en, ru-en, en-ru and fr-de directions.

we obtain new state-of-the-art results on the wMT 2019 QE as a Metric task and outperform sentBLEU on the wMT 2019 Metrics task (Ma et al., 2019).


For system-level evaluation, metrics which can use the reference translations for quality estimation, such as BLEU, generally achieved consistently high correlation with human evaluation for all language pairs.


We introduce 'Batch Filtering', a fast and effective method for filtering out incorrect alignments.

For each combination of the aligners (4 combinations in total; see Table 2), we took the union of sentence pairs extracted by each constituent aligner of the said combination for each document.


We also make sure to check that there are no morpho-syntactic dependencies, e.g., agreement, between these variables.

unlike all other previous challenge datasets focusing on gender bias, our examples quantify to what extent gender bias in models leads to prediction errors, rather than unwarranted disambiguation.


For all languages, we report the performance of Google Translate.

we see a similar tendency to Chinese, but since the overall performance is poor, and the model is in general rather insensitive to differences in pronouns, we do not include correlation results.


In this example, input #1, #2 and #3 have the same meaning.

the dominant methods for NMT only observe one of them from the parallel corpora for the model training but have to deal with adequate variations under the same meaning at inference.


Assuming that a token is selected to be replaced, and its candidate substitutes are retrieved by the mapping, we want the selected substitute can fit in well with the token's context and maintain both the semantic and syntactic coherence.

we found that adversarial examples also exist in the GEC models and up to 100% of input examples admit adversarial perturbations.


Prior hyperparameters controlling the degree of sparsity were set to α = 0.1, β = 0.01, γ = 1.

this expression is similar to the stick-breaking representation of a Dirichlet process DP(·, F)— however, it has different weights and does not include random atoms drawn from F as part of its definition-see Appendix B for more details.


We denote loss functions of these two parts as L new and L old respectively.

existing GE models are not practical in real-world applications since it overlooked the streaming nature of incoming data.


In WD-Match, a Wasserstein distance-based regularizer is defined to regularize the features vectors projected from different domains.

in this paper, we proposed a novel Wasserstein distance-based regularizer to improve the sequence representations, for text matching in asymmetrical domains.


The diagonal positions of the matrices are almost always important for the MNLI task, which exhibits similar trends with TinyBERT with the one-to-one "Skip" layer mapping strategy.

we summarize our main contributions as follows.


When the answer spans appear only once in the input, this is simple, since the ground-truth tagging is immediately available.


For both methods, we find that the pruned "good" subnetworks alone reach the performance comparable with the full model, while the "bad" ones do not.

ideally, the pre-trained weights would provide transferable linguistic knowledge, fine-tuned only to learn a given task.


Given that we did not find even the "good" subnetworks to be stable, or preferentially containing the heads that could have interpretable linguistic functions, the latter seems more likely.

the degree to which the "good" subnetworks overlap across tasks may be a useful way to characterize the tasks themselves.


Based on the defined roles, we perform a preliminary analysis across all layers.

we note that all 12 layers are majorly focused on entity based words (common nouns, proper nouns and numerical entities).


Due to the large number of parameters and nonlinearity of deep NN models, the answer to the question "how did the model arrive at the prediction?

the current works do not provide an insight into how BERT is able to achieve near human-level performance on the task of Reading Comprehension based Question Answering.


Nevertheless, the counterfactual samples are simply added to the training data for augmentation, ignoring that the relationship between original samples and counterfactual samples are vital for the reasoning of VQA models.

the dimension of the joint embedding is 2048.


By examining the distribution of the relevance scores, we find that only 31% of them are aligned well with the sparse annotations and 9% are totally misaligned.

their models neglect the important early interaction of the answer entity and cannot naturally leverage the pretrained language representations from BERT like ours.


The response of StepGAN is not infor-mative enough as well.

we have tested the learning rate from 1e-6 to 1e-3.


Aspect Based Sentiment Analysis (ABSA) (Pang and Lee, 2008; Liu, 2012) is an extensively studied sentiment analysis task on a fine-grained semantic level, i.e., opinion targets explicitly mentioned in sentences.

such behavior is caused by the different distribution of positive, neutral, and negative sentiment between training and test set, shown in Table 1.


In this group, the baselines use different approaches to deal with the multi-label issue without considering the modality dependence issue.

comparison with the multi-modal and multilabel classification approaches.


Kant et al. (2018) leverage the pre-trained BERT to perform multi-label emotion task and Kim et al. (2018) propose an attention-based classifier that predicts multiple emotions of a given sentence.

while the conditional generation framework naturally models the label dependence by predicting the next emotion label upon other potential labels, we propose Multi-head soft modality attention at each predicting step inside the emotion decoder to capture the modality dependence.


Although stateof-the-art models have shown high accuracy on existing test sets, we still question their robustness.

finally, we ask the annotators to fix the rejected samples by minimal edit which does not change the aspect term or the sentence meaning, but satisfies both criteria.


We first form a set of aspect expressions AspectSet 4 by extracting all aspect expressions from the entire dataset.

Moreover, when we test on the subset of the test set (59 instances in Laptop, and 122 instances in Restaurant) where the target aspect sentiment differs from all nontarget aspect sentiments (so that the confounding factor is disentangled), the best model (Xu et al., 2019a) drops from 78.53% to 59.32% on Laptop and from 86.70% to 63.93% on Restaurant.


ADDDIFF: Add aspects with the opposite sentiment from the target aspect Tasty burgers, crispy fries, but poorest service ever!

we propose a flexible method, adversarial training, for aspect robustness, which is applicable to any given dataset.


A sequence of tokens is encoded into a sequence of token embeddings H by the BERT encoder.

sun and Nenkova (2019) discussed both reference-based and reference-free settings for summarization evaluation.


That's because these approaches are difficult to measure the salience and redundancy simultaneously with error propagation.


We use P BLANK to denote the ratio of replaced entities and set P BLANK = 0.7 following Baldini Soares et al.

there is no clear understanding which type of information affects existing RE models to make decisions and how to further improve the performance of these models.


For the case from Mased Language Modeling, we also observe that the sentence that can be used to recover the masked words receives much higher attention weight compared to others, validating our motivation on retrieving the useful sentence embeddings from other sequences to enhance masked word recovery in the current sequence.

all the models are pre-trained on Wikipedia and finetuned on downstream tasks.


During training, we fix the position embeddings for the pre-appended special tokens, and randomly select 64 continuous positions from 0 to 564 for the other words.

instead of using the original signals of full sentences, we train a Transformer-based sequence encoder over a large set of short sequences, which allows the model to automatically select the most useful information for predicting masked words.


For the other two datasets -MAWPS and ALG514, -we report the average accuracy and standard error using 5-fold cross-validation.

in the rest of the paper, we introduce existing approaches to solve algebraic word problems in Section 2.


Early attempts for solving algebraic word problems noted the importance of Expressions in building models with hand-crafted features  (Kushman et al., 2014; Roy et al., 2015; Roy and Roth, 2015; Zhou et al., 2015; Upadhyay et al., 2016).

existing neural models solve algebraic word problems by using symbols to provide an abstraction of problem-dependent numbers or unknowns.