They actually equal or improve the current state of the art in tagging and parsing for all five languages.

they also improve over multilingual Wikipedia-based contextual embeddings (multilingual BERT), which almost always constitutes the previous state of the art, thereby showing that the benefit of a larger, more diverse corpus surpasses the crosslingual benefit of multilingual embedding architectures.


The domain proportions of the words can be naturally integrated into the Transformer model for capturing domain-shared/specific knowledge, as the multi-head dot-product attention mechanism is applied at the word-level.

we carefully design multi-head dot-product attention modules for different domains, and eventually mix these modules by taking weighted averages of their parameters by their layer-wise domain proportions.


Based on the above attention function in (1), Vaswani et al. (2017) further develop a multi-head attention module, which allows the NMT model to jointly attend to information from different representations at different positions.

we consider a multi-head attention module with m heads.


We follow the fairseq re-implementation of 12-layer Transformer designed for IWLST data.

the embedding dimension is 512 for both the encoder and decoder, the number of heads is 4, and the embedding dimension in the feed-forward layer is 1024.


The computation of the gradient, on the other hand, is the key to combining two methods.

we encourage the embedding to be domain aware via MTL, AdvL and PAdvL, where we use the domain proportion layers to guide the training of the embedding.


We use vertices to represent utterance content, and edges to represent dialog transitions between utterances.

there are two types of vertices: (1) a what-vertex that contains a keyword, and (2) a how-vertex that contains a responding mechanism (from a multi-mapping based generator in Section 3.1) to capture rich variability of expressions.


Using this information, the decoder performs the third subtask, which requires reasoning across language representations.

it must determine how the source code changes that are relevant to the current decoding step should manifest as natural language updates to the relevant portions of C old .


Although two of the baselines achieve slightly higher BLEU-4 scores than our best model, these differences are not statistically significant, and our model is better at editing comments, as shown by the results on exact match, SARI, and GLEU.

our edit models beat all other models with wide, statistically significant, margins on SARI, which explicitly measures performance on edit operations.


The return type substitution and return type substitution w/ null handling baselines produce predictions that are identical to C old for 74.73% and 65.76% of the test examples, respectively, while it is only 9.33% for the reranked edit model.

the baselines attain high scores on automatic metrics and even beat our model on BLEU-4, without actually performing edits on the majority of examples.


First, our proposed model, FACTEDITOR, achieves significantly better performances than the main baseline, ENCDECEDITOR, in terms of almost all measures.

fACTEDITOR  obtains significant gains in DELETE scores on both WEBEDIT and ROTOEDIT.


We observed that while there is still a gap between the systemgenerated and human-written Impressions, over 80% of our system-generated Impressions are as good 9 as the associated human-written Impressions.

73% (readability), and 71% (accuracy) of our system-generated Impressions ties with human-written Impressions, both achieving full-score of 3; nonetheless, this percentage is 62% for completeness metric.


In all the models, regardless of the training dataset used and the model architecture, adding a logistic regression on top of the LSTM output significantly improves average F1 scores.

the highest F1 scores are always achieved with logits scores as LogReg input features.


From a technical perspective, we use data augmentation to generate training data for an end-to-end system.

we develop Syntactic Transformations (STs) to produce question-specific candidate answer responses and rank them using a BERT-based classifier (Devlin et al., 2019).


To obtain data needed to train these models, rather than constructing yet-another crowdsourced QA dataset, we transform the answers from an existing QA dataset into fluent responses via data augmentation.

we synthetically generate supervised training data by converting questions and associated extractive answers from a SQuADlike QA dataset into fluent responses via Syntactic Transformations (STs).


Also, we only test sentences in the same domain such that the distribution in target corpus T is the same as well.

the marginal distribution of target corpus PT (x) is the same with that for each individual source dataset, i.e. PT (x) = Pk(x).


Recent work (Jawahar et al., 2019) has studied what BERT learned at different layers.

the authors found {3,4,5,6,7,9,12} layers have the most representation power in BERT and each layer captures different types of information ranging from surface, syntactic to semantic level representation of text.


UDA (Xie et al., 2019): Since we do not have access to TPU and need to use smaller amount of unlabeled data, we implemented Unsupervised Data Augmentation(UDA) using pytorch by ourselves.

we used the same BERT-based-uncased model, unlabeled augment data and batch size as our MixText, used original unlabeled data to predict the labels with the same softmax sharpen temperature as our MixText and computed consistency loss between augmented unlabeled data.


Our final layer-wise knowledge transfer loss L KT for the th layer is a linear combination of the two objectives stated below: Feature Map Transfer (FMT) Since each layer in BERT merely takes the output of the previous layer as input, the most important thing in layerwise knowledge transfer is that the feature maps of each layer should be as close as possible to those of the teacher.

the mean squared error between the feature maps of the MobileBERT student and the IB-BERT teacher is used as the knowledge transfer objective: where is the index of layers, T is the sequence length, and N is the feature map size.


This motivates us to use self-attention maps from the well-optimized teacher to help the training of MobileBERT in augmentation to the feature map transfer.

we minimize the KL-divergence between the per-head self-attention distributions of the MobileBERT student and the IB-BERT teacher: where A is the number of attention heads.


An alternative to importance sampling is to directly marginalize over a subset of z values where we expect p(x|z) is large.

we propose using the top-k most likely values of z identified by performing beam search using the proposal distribution q(z|x).


We note that the bias terms are wordspecific and can only adjust the stolen probability effect by a constant factor.

it does not change the fact that words in the interior set are probability-bounded.


We also propose a novel method of directed acyclic graph (DAG) generation for taxonomy construction.

our proposed Graph2Taxo uses a noisy graph constructed from automatically extracted noisy hyponym-hypernym candidate pairs, and a set of taxonomies for some known domains for training.


The problem addressed in this paper is, given a list of domain-specific terms from a target unseen (aka missing) domain as input, how to construct a taxonomy for that target unseen domain.

the problem addressed in this paper is how to organize these terms into a taxonomy.


Adding the DAG Constraint (Row 1) to this yields can get a better Fscore.

we observe a major increase of +5% F1 for the Science (Eurovoc) domain.


Finally, we study the effect of initializing our model using pre-trained embeddings, rather than initializing at random.

we initialize the input matrix H 0 of our Graph2Taxo model with pre-trained fastText 5 embeddings.


Backchannels are short utterances that punctuate longer turns by another conversational participant (Yngve, 1970;  Goodwin, 1986; Bavelas et al., 2000).

the model assigns the backchannels "mhm," "mm," "nice," and "awesome" each to separate categories.


For example, exploring correlations between counselor-patient interaction dynamics and counseling outcomes (Althoff et al., 2016); studying linguistic development of mental healthcare counsellors (Zhang et al., 2019); identifying differences in how people disclose mental illnesses across gender and culture (De Choudhury et al., 2017); predicting a variety of mental health conditions from social media posts (Sekulic and Strube, 2019;De Choudhury et al., 2013a;Guntuku et al., 2019;Coppersmith et al., 2014); and analyzing well-being (Smith et al., 2016) and distress (Buechel et al., 2018).

many researchers have used NLP methods for identifying depression (Morales et al., 2017).


Devlin et al. (2019) suggest trying a limited number of combinations of learning rate and training epochs to optimize the BERT classification model.

the paper recommends combinations of 2, 3, or 4 epochs and learning rates of 2E-5, 3E-5, and 5E-5.


To tackle the above issue, we propose a novel probabilistic deep generative model for QA pair generation.

our model is a hierarchical conditional variational autoencoder (HCVAE) with two separate latent spaces for question and answer conditioned on the context, where the answer latent space is additionally conditioned on the question latent space.


For the answer encoding, we use a binary token type id of BERT.

we encode all context tokens as 0s, except for the tokens which are part of answer span (highlighted words of context in Figure 1-(a) or -(c)), which we encode as 1s.


Human Evaluation As a qualitative analysis, we first conduct a pairwise human evaluation of the QA pairs generated by our Info-HCVAE and Maxout-QG on 100 randomly selected paragraphs.

20 human judges performed blind quality assessment of two sets of QA pairs that are presented in a random order, each of which contained two to five QA pairs.


We proposed a novel probabilistic generative framework for generating diverse and consistent questionanswer (QA) pairs from given texts.

our model learns the joint distribution of question and answer given context with a hierarchically conditional variational autoencoder, while enforcing consistency between generated QA pairs by maximizing their mutual information with a novel In-  foMax regularizer.


For the semisupervised learning experiment on SQuAD, we follow Zhang and Bansal (2019)'s split for a fair comparison.

we receive the unique IDs for QA pairs from the authors and use exactly the same validation and test set as theirs.


In this work we study knowledge distillation with a focus on multilingual Named Entity Recognition (NER).

we study several distillation strategies and propose a stage-wise optimization scheme leveraging teacher internal representations, that is agnostic of teacher architecture, and show that it outperforms strategies employed in prior works.


In order to imitate an adversary who may not have the significant resources needed to train such models, we use off-the-shelf pre-trained neural language models.

we choose well-known context-aware neural language models GPT-2 (Radford et al., 2019) and BERT (Devlin et al., 2018).


Inspired by them, we explore obfuscation detection via image classification.

we explore a transfer learning approach wherein we use the VGG-19 classifier 4 trained for image classification on ImageNet dataset 5 .


Inspired by reversing the order of words in the input sequence (Sutskever et al., 2014), we propose an auxiliary synthetic task that changes the order of the target class levels in the output sequence.

we go upward from the child nodes to the parent.


In this work, we address the task of automatically grading the language proficiency of spontaneous speech based on ASR transcriptions only, and seek to investigate the extent to which current state-ofthe-art neural approaches to language assessment are effective for the task at hand.


We now apply such approaches to the grading of spoken transcriptions in a scenario where the audio, or information derived from it, is not available.

the task is analogous to essay scoring except for the presence of characteristic speech features such as false starts, repetitions and filled pauses (Moore et al., 2015;Carter and McCarthy, 2017).


These auxiliary objectives are based on previous work indicating that learning to make such predictions aids in tasks such as essay scoring and grammatical error detection (Cheng et al., 2015;Rei and Yannakoudakis, 2017;Cummins and Rei, 2018;Johan Berggren et al., 2019;Bell et al., 2019).

for the last three tasks, we predict a label y per word x t (Figure 3; left).


Our baseline approach is a feature-based model of the type which has been used in previous research (Vajjala and Rama, 2018; Yannakoudakis et al., 2018)

we train a linear regression model and use as features tf– idf weighted word and POS n-grams (up to trigrams), grammatical constructions extracted from the phrase-structure trees, the length of the transcript, and the number of errors, estimated by counting the number of trigrams that are absent from a large background corpus of correct English (Ferraresi et al., 2008).


Recently, pretrained Transformer networks have demonstrated success on various NLP tasks (Radford et al., 2018;Devlin et al., 2019;Liu et al., 2019); we use these models as the foundation for SPECTER.

we use SciBERT (Beltagy et al., 2019) which is an adaptation of the original BERT (Devlin et al., 2019) architecture to the scientific domain.


We test how well different paper representations can reproduce this signal through citation prediction tasks.

we focus on two sub-tasks: predicting direct citations, and predicting co-citations.


We experiment with fine-tuning SciBERT on our tasks, and find this to be generally inferior to using our fixed representations from SPECTER.

we finetune SciBERT directly on task-specific signals instead of citations.


We observe that the classifier performance when trained on our representations is better than when trained on any other baseline.

on the MeSH (MAG) dataset, we obtain an 86.4 (82.0) F1 score which is about a ∆= + 2.3 (+1.5) point absolute increase over the best baseline on each dataset respectively.


We also study the effect of applying MAG at different encoder layers of the XLNet.

we first apply the MAG to the output of the embedding layer.


To evaluate the effect of SPOLIN on generating yes-and responses and thus improving generated dialogue quality, we train a common architecture with a variety of fine-tuning data configurations, both with and without SPOLIN.

for each data configuration we fine-tune a doublehead GPT-2 model (117M-parameter version based on the implementation by Wolf et al. (2019b)), which achieved state-of-the-art performance on Personachat for the ConvAI-2 dialogue competition (Zhang et al., 2018).


Next, we intrinsically evaluate the performance of our SOW model (Section 2.3).

given a budget of 10 reorderings, we want to understand how close our SOW model comes to covering the target ordering.


We train MAUDE with downsampling, as we observe poor results when we run the recurrent network on top of 768 dimensions.

we downsample to 300 dimensions, which is the same used by our baselines RUBER and InferSent in their respective encoder representations.


In the last few years, a number of successful approaches have emerged that are able to adequately model various aspects of natural language.

language models based on neural networks have improved the state of the art with regard to predictive language modeling, while topic models are successful at capturing clear-cut, semantic dimensions.


In recent years, a number of fruitful NLP approaches have emerged that are able to adequately model various aspects of natural language.

neural network language models have improved the state of the art in language modeling, while topic models are successful at capturing clearcut, semantic dimensions.


First of all, we would like to experiment with different neural network architectures.

we believe hierarchical approaches (Serban et al., 2017) as well as the Transformer network (Vaswani et al., 2017) would be particularly suitable to poetry generation.


s and o are identified by their entity mentions, and p is identified by a unique ID.

two entities from different triples that have the same mentions will be regarded as the same node.


Learning a plan can be naturally regarded as a sequential decision-making process.

given a set of triples, we first determine which triple to mention/visit first, and then select the second triple from the remaining triples that have not been visited so far.


However, these architectures broaden the structural gap between the encoder and decoder.

while the encoder receives the input data as a graph, the decoder has to create the output text as a linear chain structure.


For each condition, we use a condition-specific PLUGINVAE to derive the conditional space.

pLUGINVAE is proposed to learn the transformation between the conditional and global latent space for each condition.


We conduct human annotations as a complementary evaluation beyond automatic metrics.

eight individual judges are asked to rate over 200 conditional samples generated from each model and each condition.


Specifically, eight individual judges are asked to rate over 200 conditional samples generated from each model and each condition.

for each model, a total of 4, 800 text samples are annotated.


We show that the amount of downstream training data is a critical factor in determining the relative performance of contextual vs. non-contextual embeddings.

we show in representative tasks in Figure 1 that the performance of the non-contextual embedding models improves quickly as the amount of training data is increased (plots for all tasks in Appendix B).


To provide theoretical support for why, given sufficient training data, a model trained with random embeddings might match the performance of one trained with pretrained embeddings, we consider the simple setting of Gaussian process (GP) regression (Rasmussen and Williams, 2006).

we assume that the prior covariance function for the GP is determined by the pretrained embeddings, and show that as the number of observed samples from this GP grows, the posterior distribution gives diminishing weight to the prior covariance function, and eventually depends solely on the observed samples.


For sentiment analysis, we need a sentence-level proxy for structural complexity; toward this end, we leverage the dependency parse tree for each sentence in the dataset.

we characterize a sentence as more structurally complex if the average distance between dependent words is higher


With the growth in use of KGs, researchers have explored ways to learn better representations of KGs in order to improve generalization and robustness in downstream tasks.

there has been interest in learning embeddings of KGs in continuous vector spaces (Bordes et al., 2011(Bordes et al., , 2013Socher et al., 2013).


We investigate several proposed approaches introduced in recent works and suggest a new loss that relies on sentence reconstruction from normalized embeddings.

our method demonstrates how by adding a decoding layer for sentence reconstruction, we can improve the performance of various baselines.


This work analyzes the contribution of various techniques proposed for transfer learning between languages for the task of sequence tagging.

we evaluate joint training and adversarial learning.


Inspired by these strategies, we propose a method for transfer learning between different languages for sequence tagging.

we focus on sentence representation alignment.


To obtain complete representations of the tokens in a sequence, researchers resort to bidirectional attention as shown in Figure 2(b).

the training instances are created by replacing a subset of tokens in the input X with a special token [MASK], and the objective is to predict the masked tokens.


Models without pretraining (e.g., BoW, LSTM word2vec) are often unable to reliably detect OOD examples.

these models' FAR95 scores are sometimes worse than chance because the models often assign a higher probability to out-of-distribution examples than in-distribution examples.


To maintain the realism of our study, we apply kernel density estimation to actual results, using the resulting probability density (or discretized mass) function as the ground truth distribution.

we examine the experimental results of the following neural networks: Document classification.


To evaluate the validity of bootstrapping the expected maximum, we measure the coverage probability of CIs constructed using the percentile bootstrap method (Efron, 1982).

Specifically, we set B = 50 and iterate n = 1, . . . , 50.


This is clear from their equation (7) where the empirical distribution is defined over the first n samples, instead of the B samples that we use here

they claim, at least in the text, to use Fˆ n instead of FˆB for their estimator Vˆ n n .


We find significant and consistent improvements over both BERT BASE and BERT LARGE on multiple Machine Reading Comprehension (MRC) datasets.

our proposed model has strong empirical evidence as it obtains SOTA results on Natural Questions, a new benchmark MRC dataset, outperforming BERT LARGE by 3 F1 points on short answer prediction.


The constraint learnt by the classifier is the relative ordering between the two sentences.

the classifier is trained to predict whether s 2 follows s 1 or not i.e the the classifier predicts the constraint s 1 < s 2 .


We use the Bidirectional Encoder Representations from Transformers (BERT) pre-trained uncased language model (Devlin et al., 2019) and fine-tune it on each dataset using a fully connected perceptron layer.

we leverage the Next Sentence Prediction objective of BERT and get a single representation for both sentences s 1 and s 2 .


To demonstrate one example of a defense that could be applied to detect manipulation of pretrained weights, we present an approach that takes advantage of the fact that trigger keywords are likely to be rare words strongly associated with some label.

we compute the LFR for every word in the vocabulary over a sample dataset, and plot the LFR against the frequency of the word in a reference dataset (we use the Books Corpus here).


Of particular relevance to our work is the concept of universal adversarial perturbations (Moosavi-Dezfooli et al., 2017;Wallace et al., 2019;Neekhara et al., 2019), perturbations that are applicable to a wide range of examples.

the adversarial triggers from Wallace et al. (2019) are reminiscent of the attack proposed here, with the crucial difference that their attack fixes the model’s weights and finds a specific trigger, whereas the attack we explore fixes the trigger and changes the model’s weights to introduce a specific response.


We show that much efficient light BERT models can be obtained by reducing algorithmically chosen correct architecture design dimensions rather than reducing the number of Transformer encoder layers.

our schuBERT gives 6.6% higher average accuracy on GLUE and SQuAD datasets as compared to BERT with three encoder layers while having the same number of parameters.


We propose to train a non-autoregressive machine translation model to minimize the energy defined by a pretrained autoregressive model.

we view our non-autoregressive translation system as an inference network (Tu and Gimpel, 2018) trained to minimize the autoregressive teacher energy.


As an alternative to training non-autoregressive translation systems on distilled corpora, we instead propose to train them to minimize the energy defined by a pretrained autoregressive teacher model.

we view non-autoregressive machine translation systems as inference networks  (Tu and Gimpel, 2018, 2019; Tu et al., 2019) trained to minimize the teacher's energy.


In this section we provide a qualitative analysis of our approach.

we are interested in the capability of the model to predict unseen synsets, thanks to the prior knowledge that is encoded in both the output embeddings O and the adjancency matrix A.


We propose to utilize question generation (QG) (Du et al., 2017) as a new means to overcome this problem.

given a review sentence, the generated question is expected to ask about the concerned aspect of this product, from the perspective of the review writer.


The generator is initially trained with (question, answer) data, and is gradually updated with adapted and augmented training instances, so that the rewards from the generator can reflect the ability of review for generating the corresponding question.

we propose a reinforcement objective that makes use of the reward from the generator, denoted as reward G (r, q).


6 volunteers are asked to select user-posed questions and the corresponding review sentences that can serve as answers.

the volunteers are given pairs  of question and review, and only consider the relevance between question and review.


G P N ar : Review data is incorporated via a retrieval-based method.

the most relevant review sentence for each question is retrieved via BM25 method, and such review-question pairs are added into the training set.


We provide visual evidence of desirable behavior of our model on collections of character extractions from historical books with mixed fonts.

we discus the performance of our model on the mysterious edition of Thomas Hobbes' Leviathan known as "the 25 Ornaments" edition.


In order to reduce the size of the vocabulary, low-frequency words are not kept in both the vocabulary for the source codes and the vocabulary for target comments.

the minimum threshold frequency for WikiSQL and ATIS is set as 4 while for CoNaLa it is set as 2.


We illustrate the n-gram based BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) evaluations to evaluate the quality of our generated comments and also use them to set the reward in the HRL based training.

bLEU-4, ROUGE-2 and ROUGE-L are used to evaluate the performance of our model since they are the most representative evaluation metric for context-based text generation.


To deal with this issue, in this paper, we propose a novel graph-based multi-modal fusion encoder for NMT.

we first represent the input sentence and image using a unified multi-modal graph, which captures various semantic relationships between multi-modal semantic units (words and visual objects).


Based on the graph, we then stack multiple graph-based multi-modal fusion layers that iteratively perform semantic interactions among the nodes to conduct graph encoding.

during this process, we distinguish the parameters of two modalities, and sequentially conduct intraand inter-modal fusions to learn multi-modal node representations.


Experimental results and in-depth analysis indicate that our encoder is effective to fuse multi-modal information for NMT.

our multi-modal NMT model significantly outperforms several competitive baselines.


Finally, we use the metrics BLEU (Papineni et al., 2002) and METEOR (Denkowski and Lavie, 2014) to evaluate the quality of translations.

we run all models three times for each experiment and report the average results.


It should be noticed that we do not explicitly delete a segment when DEL is encountered but do it via post-processing.

the model is trained to ignore the segment to be deleted implicitly.


Note that, by fixing segment length (token number of each segment) instead, the segment number K can be changed dynamically according to the sentence length.

we can predict the target sentence length to determine the segment number during inference.


In addition, we found that miscalibrated predictions correlate well with the translation errors in inference.

the over-estimated predictions correlate more with over-translation and mis-translation errors, while the under-estimated predictions correlate more with under-translation errors.


Reliability diagrams are a visual representation of model calibration, which plot accuracy as a function of confidence (Niculescu-Mizil and Caruana, 2005).

it partitions the output tokens into several bins according to their prediction confidence, and calculate the average confidence and accuracy of each bin.


In this section, we present our novel UPSA framework that uses simulated annealing (SA) for unsupervised paraphrasing.

we first present the general SA algorithm and then design our searching objective and searching actions (i.e., candidate sentence generator) for paraphrasing.


Therefore, we propose a copy mechanism for SA sampling, inspired by that in Seq2Seq learning (Gu et al., 2016).

we allow the candidate sentence generator to copy the words from the original sentence x 0 for word replacement and insertion.


The main drawback of existing methods is that they cannot solve the confusing charges issue.

due to the high similarity of several law articles, their corresponding law cases can be easily misjudged.


In this paper, we propose a Hyperbolic and Co-graph Representation method (HyperCore) to address the above problem.

we propose a hyperbolic representation method to leverage the code hierarchy.


The results demonstrate that the proposed HYPERCAPS can achieve competitive performance compared with the baselines.

effectiveness of HYPERCAPS is shown on tail labels.


In this study, we bring forward a novel framework of automatic diagnosis with EMR documents for CDS.

we propose to predict the main diagnosis based on the patient's current illness.


Then, each EMR document is transformed to a Kdimensional one-hot feature f .

if the i-th entity in the entity vocabulary appears as a positive finding in the input EMR, then the i-th dimension of f is set to 1, and otherwise, it is set to 0.


We propose the Bayesian network ensembles for the diagnosis inference.

a group of PGMs with the extracted relations and weights are ensembled towards the final predictions.


To address this problem, we have proposed a new task named emotion-cause pair extraction (ECPE), aiming to extract the potential pairs of emotions and their corresponding causes together in our previous work .

eCPe is defined as a fine-grained emotion analysis task, where the goal is to extract a set of valid emotion-cause pairs, given a document consisting of multiple clauses as the input.


BiLSTM can capture the left and right context of each word in the input.

for the t-th word w t in the input sequence of the target,  for word w t at the t-th position of the input target.


In addition to the four targets in SemEval-2016, we introduce an additional Trade Policy (TP) target as the fifth target, which is an incredibly hot topic nowadays.

1245 tweets related to TP are collected and manually labeled as "favor", "against" and "none".


Here, we extend TextCNN to the cross-target setting, denoted as TextCNN-E.

each word is represented as a 3D tensor by concatenating the embeddings of k semantically/emotionally-related words.


To investigate the impact of each part on our SEKT model, we perform the ablation test by discarding SE graph knowledge (denoted as w/o SE) and knowledge-aware memory unit (denoted as w/o KAMU), respectively.

for the w/o SE model, the external knowledge is expressed by a weighted sum of the embeddings of four semantically/emotionally-related words.


To better understand the limitations of SEKT, we additionally carry out an analysis of the errors made by SEKT.

we randomly select 100 instances that are incorrectly predicted by SEKT from the expanded SemEval-2016 dataset.


In this paper, we proposed a semantic-emotion knowledge transferring (SEKT) model for crosstarget stance classification, which used the external knowledge from semantic and emotion lexicons as commonsense knowledge to bridge the gap across different targets.

we first built a SE-graph from semantic and emotion lexicons, which leveraged external knowledge from both word-level and concept-level.


In this paper, we proposed a Semantic-Emotion Knowledge Transferring (SEKT) model for cross-target stance detection, which uses the external knowledge (semantic and emotion lexicons) as a bridge to enable knowledge transfer across different targets.

a semantic-emotion heterogeneous graph is constructed from external semantic and emotion lexicons, which is then fed into a graph convolutional network to learn multi-hop semantic connections between words and emotion tags.


We further analyze our framework and challenge our design choices.

we consider three variants of our architecture based on alternative ways to condition DANN with the graph features.


They are both novel and effective in generating parallel data that introduces additional formality transfer knowledge that cannot be derived from the original training data.

f-Dis identifies useful pairs from the paraphrased pairs generated by cross-lingual MT; while M-task leverages the training data of Grammatical Error Correction (GEC) task to improve formality, as shown in figure 1.


Our framework can effectively learn contextualized information with various base encoders.

we try two different encoders (BiLSTM encoder and BERT encoder).


Opinion and sentiment analysis has a wide range of real-world applications like social media monitoring (Bollen et al., 2011), stock market prediction (Nguyen et al., 2015), box office prediction (Yu et al., 2010), and general e-commerce applications (Kim et al., 2013;Hu et al., 2017;Cui et al., 2017).

fine-grained opinion analysis aims to identify users' opinions in a text, including opinion expressions, holders of the opinions, targets of the opinions, target-dependent attitude, and intensity of opinions (MarasoviÄ‡ and Frank, 2018), which is very important for understanding political stance, customers’ reviews, marketing trends, and other subjective information (Ravi and Ravi, 2015).


In summary, based on previous studies in using syntax to improve various tasks, this work investigates whether syntax can enhance the neural ORL model.

we try to answer the following three questions.


In this work, we exploit rich syntactic information contained in the edgeweighted graphs to mitigate the effects of parsing errors.

we firstly employ graph convolutional networks (GCN) to encode the edgeweighted graphs, and then integrate them into different processing levels of ORL with implicit parser hidden states.


We use BERT (Bidirectional Encoder Representations from Transformers) (Devlin et al., 2019) to obtain deep contextualized word representations as our extra inputs.

we use BERT-base (uncased) model and extract representations from the top-1 hidden layer.


We follow the previous works of Zhang et al. (2019b) and Marasovic and Frank (2018) without much parameter tuning.

we use the pretrained 100-dimensional glove embeddings (Pennington et al., 2014).


In this section, we report the overall performance of our approaches compared with previous methods on the test data, as shown in Table 3.

we list our syntax-agnostic baseline (BASELINE in Table 1), others’ works (Zhang et al. (2019b) and Marasovic and Frank (2018), using SRL for ORL), best non-MTL approaches based on our results on the dev data (DEPGCN for explicit syntactic information and DEPHDN for implicit syntactic information), and finally the MTL-based models.


In this section, we conduct analysis to better understand the contributions from the syntactic information and BERT.

we compute the exact F1 score according to different lengths of opinion arguments, as well as different distances between the arguments and their corresponding expressions.


Following the work of Kuribayashi et al. (2019) and Potash et al. (2017), we propose incorporating multiple types of token representation to provide rich input features.

the proposed system combines surface, part-of-speech (POS) tags, GloVe (Pennington et al., 2014) embedding, and ELMo (Peters et al., 2018) as input features for each token.


In this paper, we propose a novel framework ConWea, providing contextualized weak supervision for text classification.

we leverage contextualized representations of word occurrences and seed word information to automatically differentiate multiple interpretations of the same word, and thus create a contextualized corpus.


During this process, contextualized seed words are introduced by expanding and disambiguating the initial seed words.

for each word, we develop an unsupervised method to adaptively decide its number of interpretations, and accordingly, group all its occurrences based on their contextualized representations.


Second, we design a principled comparative ranking method to select highly label-indicative keywords from the contextualized corpus, leading to contextualized seed words.

we start with all possible interpretations of seed words and train a neural classifier.


The key objective of this contextualization is to disambiguate different occurrences of the same word into several interpretations.

we treat every word separately, so in the rest of this section, we focus on a given word w. given a word w, we denote all its occurrences as w1 , . . . , wn, where n is its total number of occurrences in the corpus.


We present a case study to showcase the power of contextualized weak supervision.

we investigate the differences between the expanded seed words in the plain corpus and contextualized corpus over iterations.


In this work, we use a much stronger basic parser and observe more significant UAS/LAS improvement than theirs.

we present an in-depth analysis showing that explicitly highorder modeling certainly helps the parsing model and thus is complementary to the BiLSTM encoder.


Therefore, the processing speed is faster than the evaluation phase.

for LOC, CRF and CRF2O, the average one-iteration training time is about 1min, 2.5min and 3.5min on PTB.


To address these limitations, we resort to existing pre-trained contextualized word representations, and propose a unified multimodal architecture based on Transformer (Vaswani et al., 2017), which can effectively capture inter-modality interactions and alleviate the visual bias.

we first adopt a recently pre-trained contextualized representation model (Devlin et al., 2018) as our sentence encoder, whose multi-head self-attention mechanism can guide each word to capture the semantic and syntactic dependency upon its context.


On the one hand, removing the whole ESD module will significantly drop the performance, which shows the importance of alleviating the visual bias.

discarding the conversion matrix in the ESD module also leads to the performance drop, which indicates the usefulness of capturing the label correspondence between the auxiliary module and our main MNER task.


In this paper, to obtain a new insight into the syntactic abilities of neural LMs, in particular RNN-LMs, we perform a series of experiments under a different condition from the prior work.

we extensively analyze the performance of the models that are exposed to explicit negative examples.


Since our goal is to understand to what extent LMs can be sensitive to the target syntactic constructions by giving explicit supervision via negative examples, we only prepare negative examples on the constructions that are directly tested at evaluation.

we mark the following words in the training data, and create negative examples


Further, TextING ranks top on all tasks, suggesting that the individual graph exceeds the global one.

the result of TextING on MR is remarkably higher.


We conduct a thorough study to diagnose the behaviors of pre-trained language encoders (ELMo, BERT, and RoBERTa) when confronted with natural grammatical errors.

we collect real grammatical errors from non-native speakers and conduct adversarial attacks to simulate these errors on clean text data.


Considering different target models, we observe that the impact of grammatical errors varies among models.

roBErTa exhibits a strong robustness against the impact of grammatical errors, with consistently lower attack success rates (20.28% on average) and F1 score decreases (17.50% on average) across all tasks, especially on MrPC and MNLI.


Then, we introduce the target error type to half of these sentences using probabilistic transformation and keep the error rate over each dataset around 3% (resulting in one or two  Models We study individual layers of ELMo (2 layers), BERT-base-uncased (12 layers) and RoBERTa-base (12 layers).

we fix each layer and attach a trainable self-attention layer on top of it to obtain a sentence representation.


We also observe patterns related to a specific model.

middle layers (layers 7-9) of BERT are better at identifying errors than lower or higher layers, as shown in Fig 2.


But its best performing layer in locating errors depends on the error type and varies between the first and the second layer.

the second layer of ELMo exhibits strong ability in locating Nn and outperforms BERt in accuracy.


There are some future directions that are worth exploring.

we can further examine the influence of using pre-trained word embeddingswhether similar words can help each other boost their polarity and attention scores.


PLMs are first trained on a huge general domain data set and then fine-tuned on specific domain datasets of different downstream tasks.

given a pre-trained GPT2 model, to generate sentences of email domain, we always need to fine-tune the GPT2 on a small set of email domain corpus.


However, SMC suffers from serious degeneracy problem.

samples from SMC tend to share a very small number of the ancestors because most of the ancestors are killed during resampling.


Our method significantly outperforms the recent IBP-based methods (Jia et al., 2019;Huang et al., 2019) on both IMDB and Amazon text classification.

we achieve an 87.35% certified accuracy on IMDB by applying our method on the state-of-the-art BERT, on which previous certified robust methods are not applicable.


In this paper, we focus on adversarial word substitution in which an attacker arbitrarily replaces the words in the sentence by their synonyms according to a synonym table to alert the prediction of the model.

for any word x, we consider a pre-defined synonym set S x that contains the synonyms of x (including x itself).


We can see that our method clearly outperforms the baselines.

our approach significantly outperforms IBP on Amazon by improving the 14.00% baseline to 24.92%.


To solve those issues, we propose a novel graphbased coarse-to-fine paradigm to generate initial solutions for learning cross-lingual word embeddings, from which we induce bilingual lexicons.

given source and target languages, our method first uses pre-trained monolingual embeddings to construct a graph for each language, with the vertices representing different words, so that the mutual relationship between words is preserved.


Instead of direct optimization of the constrained adversarial loss in Eq.6, we model discriminator D's output as survival rewards similar to that in gaming (Mnih et al., 2015).

the agent must survive for its goal by also fooling D, which attempts to terminate ill-perturbed modifications.


However, batch size is also an important hyperparameter, and some batch sizes empirically lead to better performance than the others.

it has been shown that the performance of the Transformer model (Vaswani et al., 2017) for Neural Machine Translation (NMT) (Bahdanau et al., 2015;Gehring et al., 2017;Vaswani et al., 2017) relies heavily on the batch size (Popel and Bojar, 2018;Ott et al., 2018;Abdou et al., 2017;Zhang et al., 2019a).


However, it can only translate between a single language pair and cannot produce translation results for multiple language pairs at the same time.

research on multilingual UNMT has been limited.


The BiRRE vector models the possibility of a term being mapped to another in the embedding space by hypernymy relations, learned via existing neural language models and supervised signals of the training set.

we introduce the Latent Projection Model with Negative Regularization (LPMNR) to simulate how hypernyms and hyponyms are generated in the the embedding space.


On the basis, we propose a Cooperative Graph Attention Networks (Co-GAN) approach for cooperatively learning the aspect-related sentence representation.

two graph attention networks are leveraged to model above two kinds of documentlevel sentiment preference information respectively, followed by an interactive mechanism to integrate the two-fold preference.


To well accommodate the above two kinds of document-level sentiment preference information, we propose a Cooperative Graph Attention Networks (CoGAN) approach to ASC.

two graph attention networks are constructed to model the two-fold sentiment preference with the attention weight to measure the preference-degree.


To better illustrate the effectiveness of modeling the intra-aspect consistency and inter-aspect tendency information, we systematically investigate both sentiment preference phenomena in all the four evaluation datasets respectively.

we sample 200 sentence 4 pairs inside each dataset and calculate the ratio that the two sentences in the pair have 4 Sentences are repeated in a document according to the unrolled aspects.


As phrase-level sentiment labels are expensive to obtain, we further explore if the compositional sentiment semantics learned from one task can be transferred to others.

we find that SentiBERT trained on SST can be transferred well to other related tasks such as twitter sentiment analysis (Rosenthal et al., 2017) and emotion intensity classification (Mohammad et al., 2018) and contextual emotion detection (Chatterjee et al., 2019).


As shown in Table 1, SentiBERT and SentiBERT w/ RoBERTa substantially outperforms their corresponding variants and the networks merely built on the tree.

we first observe that though our attention network (SentiBERT w/o BERT) is simple, it is competitive with Recursive NN, GCN and Tree-LSTM.


In this paper, we propose SentiBERT to incorporate recently developed contextualized representation models (Devlin et al., 2019; Liu et al., 2019) with the recursive constituency tree structure to better capture compositional sentiment semantics.

we build a simple yet effective attention network for composing sentiment semantics on top of BERT (Devlin et al., 2019).


Similar as the traditional ECE task, the ECPE is also defined at the clause level, because it is difficult to describe emotion causes at the word or phrase level.

in this paper, the "emotion" and "cause" are refer to "emotion clause" and "cause clause", respectively.


The reason is that the existing multimodal sentiment datasets only contain a unified multimodal annotation for each multimodal segment, which is not always suitable for all modalities.

all modalities share a standard annotation during intra-modal representation learning.


Compared with single-task models, multi-task models have better performance in most of evaluation metrics.

all three improved models (MLF-DNN, MLFM, and MTFN) have promotion significantly compared to corresponding original models (LF-DNN, LFM, and TFN) in all evaluation metrics except for Acc-5.


Thus, we reinforce discrepancy between image and text, and on the contrary, weaken their commonality.

we combine the above unique variant contrast features as the cross-modality contrast representation.


Since it is intractable to exactly calculate the principled metric for a given explanation method, we propose an approximate metric to address the optimization problem.

inspired by statistical learning theory (Vapnik, 1999), we cast the optimization problem into a standard machine learning problem which is addressed in a two-step strat-egy: firstly we follow empirical risk minimization to optimize the empirical risk; then we validate the optimized parameters on a held-out test dataset.


For example, on classification tasks, Bach et al. (2015) propose layer-wise relevance propagation to visualize the relationship between a pair of neurons within networks, and Li et al. (2016) introduce a gradientbased approach to understanding the compositionality in neural networks for NLP

on structured prediction tasks, many research works design similar methods to understand NMT models (Ding et al., 2017; Alvarez-Melis and Jaakkola, 2017; Ding et al., 2019; He et al., 2019)


After training, the model makes evidence predictions on unlabeled instances (the labeling arrow), and then Selector chooses the most confident instances from U to provide noisy evidence labels.

the instances with newly generated evidence labels are moved from U to L (the moving arrow), which are used to supervise evidence extraction in the next iteration.


Furthermore, the simple gating mechanism results in only a shallow combination of form and context.

the model is not able to combine form and context until the very last step: While it can learn to weight form and context components, the two embeddings (form and context) do not share any information and thus do not influence each other.


In the second stage of our training process, we use the same parameters as Schick and Schutze (2020), as our form-only model is the very same as theirs.

we use a learning rate of 0.01, a batch size of 64 words and we apply n-gram dropout with a probability of 10%.


The advance is achieved by exploiting locality as terminal edge-adjacency in HRG rules.

we highlight the importance of 1) a terminal edge-first parsing strategy, 2) a categorization of a subclass of HRG, i.e. what we call Weakly Regular Graph Grammar, and 3) distributing argumentstructures to both lexical and phrasal rules


Considering the above problem in the framework of chart parsing, we would like to construct a data structure to efficiently access all chart items.

when partial information is provided, this data structure can quickly find all compatible chart items.


As mentioned in Dixon et al. (2018), a
model is defined as biased if it performs better for
sentences containing some specific identity-terms
than for ones containing others.

a non-discrimination model should perform similarly across sentences containing different demographic groups.


The choice of RSA versus diagnostic classifier interacts with scope, and thus these are better considered as a combination.

local RSA as implemented in this study shows only weak correlations between neural activations and phoneme labels.


Although the temporal scale of the extracted representations has not received much attention and scrutiny, our experimental findings suggest that it is an important choice.

global representations are more sensitive to learning, and more consistent across different analysis methods.


Since the interpretation serves as a proxy for the model's "reasoning", it should satisfy the same constraints.

interpretations of similar decisions should be similar, and interpretations of dissimilar decisions should be dissimilar.


The aforementioned assumptions are currently utilized to evaluate faithfulness in a binary manner: whether an interpretation is strictly faithful or not.

they are most often used to show that a method is not faithful, by constructing cases in which the assumptions do not hold for it.


Specifically, they are most often used to show that a method is not faithful, by constructing cases in which the assumptions do not hold for it.

there is a clear trend of proof via counter-example, for various interpretation methods, that they are not globally faithful.


Through a series of experiments using 12 datasets spanning 4 tasks, we show that our model is more transparent while achieving comparable performance to models containing vanilla LSTM based encoders.

we show that in our proposed models, attention weights (i) provide useful importance ranking of hidden states (ii) are better indicative of words that are important for the model's prediction (iii) correlate better with gradient-based feature importance methods and (iv) are sensitive to random permutations (as should indeed be the case).


However, before we go there we need to show that the predictive performance of our models is comparable to that of a vanilla LSTM model and significantly better than non-contextual models.

we show that we do not compromise on performance to gain transparency and explainability.


We use the intermediate representation erasure by Serrano and Smith (2019) to evaluate an importance ranking over hidden representations.

we erase the hidden representations in the descending order of the importance (highest to lowest) until the model's decision changes.


Previous work on active learning focused on optimizing the system objective (blue).

only the system provides feedback to the active learning component (e.g., how certain it is about the predicted label of an instance).


Motivated learners (MOT) continually improve their language proficiency throughout our experiments with a fixed step size of t 1 C-tests.

we simulate that their proficiency level φ(v) increases by one every t 1 iterations.


In this section, we describe our approaches for incorporating a pre-trained MLM into our GEC model.

we chose the following approaches: (1) initializing a GEC model using BERT; (2) using BERT output as additional features for a GEC model, and (3) using the output of BERT fine-tuned with the GEC corpora as additional features for a GEC model.


With the advent of deep learning, the trend shifted, with abundant work focusing on neural architectures for abuse detection.

the use of convolutional neural networks (CNNs) for detecting abuse has shown promising results (Park and Fung, 2017;Wang, 2018).


For pre-training to be efficient, we restrict our word piece sequence length to a certain budget (e.g., we use 128 in our final experiments).

the combined length of tokenized text and table cells has to fit into this budget.


We implement a fully differentiable layer that latently learns the weights for the aggregation prediction layer pa(·), without explicit supervision for the aggregation type.

we recognize that the result of executing each of the supported aggregation operators is a scalar.


Evaluation of conversational systems has been one major obstacle in dialogue research.

for open-domain dialogues, automated metrics have been shown to correlate poorly with human judgement (Liu et al., 2016).


Due to this error, some of our identity pairs were not seen exactly ten times.

3.4% were asked less than 6 times, 40% were asked less than ten times, and 40.4% were asked more than ten times.


Therefore, instead of directly searching for a possible previous statement, we search for related context, where the source are describing a statement related to the claim.

we rank sentences in the given corpus, by computing the cosine similarity to the given claim with their ELMo (Peters et al., 2018) representations.


We reflect this idea by including a margin ranking loss within our model.

we design our model on top of a pre-trained language model for general purpose (BERT) (Devlin et al., 2018), so that we can have a representation of sentences that can capture both semantic and syntactic information.


Since the sources can be a url or a mention of an entity, we do wikification (Ratinov et al., 2011;Cheng and Roth, 2013) for the extracted sources.

to wikify a source mention, we first adapt a redirectbased wikification method (RedW) (Shnayderman et al., 2019), which is efficient and context free.


Therefore, we derive a claim evidence graph from the claim provenance graph based on which we do claim verification.

we keep the nodes and edges in the claim provenance graph, and add another label on each edge with one of support, contradiction and neutral.


Luckily, with the claim evidence graph, we can collect the opinions in statement level, and vote the veracity by sources that are more independent with each other.

given an evidence graph of a claim, we start with the sink node and do breadth first search to find all source nodes whose indegree are 0, and leverage those sources to vote by their opinions to get an estimation of the claim veracity.


We find that this ratio equates 0.90, 0.50, and 0.11 for posdis, bosdis, and topsim respectively.

while compositionality is not a necessary condition for generalization, it appears that the strongest form of compositionality, namely posdis, is at least sufficient for generalization.


We obtain test accuracies of 92.7%, 66.7% and 22.8% for densities 1, 0.51 and 0.25 respectively.

the high generalization observed in the main paper is (also) a consequence of density, and hence combinatorial variety, of the inputs the agents are trained on, and not (only) of the number of training examples.


Aside from interpretability considerations, collecting rationales from annotators may afford greater efficiency in terms of model performance realized given a fixed amount of annotator effort (Zaidan and Eisner, 2008).

recent work by McDonnell et al. (2017, 2016) has observed that at least for some tasks, asking annotators to provide rationales justifying their categorizations does not impose much additional effort.


Short for Fact Extraction and VERification; entails verifying claims from textual sources.

each claim is to be classified as supported, refuted or not enough information with reference to a collection of source texts.


We also try to split the entire record into different sections (e.g., "medical history", "family history") based on some heuristic measures.

in order to split the clinical notes into sections, we notice that most sections begin with easily identifiable headers.


We want DeFormer to behave more like the original model.

the upper layers of DeFormer should produce representations that capture the same kinds of information as the corresponding layers in the original model.


To assess this, we measured how the text representation changes when paired with different questions.

we computed the average passage representation variance when paired with different questions.


Question answering (QA) is one of the challenging natural language processing (NLP) tasks that benefits from explainability.

multihop QA requires the aggregation of multiple evidence facts in order to answer complex natural language questions (Yang et al., 2018).


To utilize this type of redundancy in answer classification, we extend AIR to extract parallel evidence chains.

to extract N parallel chains, we run AIR N times, ensuring that the first justification sentences in each chain are different (in practice, we start a new chain for each justification in the top N retrieved sentences in the first hop).


We train G2P models on WikiPron annotations to provide pronunciations for these words.

we use the WFST-based tool Phonetisaurus (Novak et al., 2016).


In our first study, we adapt the approach Keyes (2018) took for analyzing the degree to which computer vision papers encoded trans-exclusive models of gender.

we began with a random sample of ∼150 papers from the ACL anthology that mention the word “gender” and coded them according to the following questions


Overall, the situation more broadly is equally troubling, and generally also fails to escape from the folk theory of gender.

none of the differences are significant at a p = 0.05 level except for the first two questions, due to the small sample size (according to an n−1 chi-squared test).


Next, we analyze the effect that our different ablation mechanisms have on existing coreference resolutions systems.

we run five coreference resolution systems on our ablated data: the AI2 system (AI2; Gardner et al., 2017), hugging face (HF; Wolf, 2017), which is a neural system based on spacy, and the Stanford deterministic (SfdD; Raghunathan et al., 2010), statistical (SfdS; Clark and Manning, 2015) and neural (SfdN; Clark and Manning, 2016) systems.


If we accept mutual information as a natural operationalization for how much representations encode a target linguistic task (§2.2), the best estimate of that mutual information is the one where the probe qθ(t | r) is best at the target task.

we want the best probe qθ(t | r) such that we get the tightest bound to the actual distribution p(t | r).


We call the resultant sense-informed model SenseBERT.

we add a maskedword sense prediction task as an auxiliary task in BERT's pre-training.


Moreover, SenseBERT exhibits an improvement in lexical semantics ability (reflected by the Word in Context task score) even when compared to models with WordNet infused linguistic knowledge.

we compare to Peters et al. (2019) who re-contextualize word embeddings via a wordto-entity attention mechanism (where entities are WordNet lemmas and synsets), and to Loureiro and Jorge (2019) which construct sense embeddings from BERT’s word embeddings and use the WordNet graph to enhance coverage (see quantitative comparison in table 3).


For each word w in our vocabulary, we employ the WordNet word-sense inventory for constructing A(w), the set of its "allowed" supersenses.

we apply a WordNet Lemmatizer on w, extract the different synsets that are mapped to the lemmatized word in WordNet, and define A(w) as the union of supersenses coupled to each of these synsets.


Here we measure the quality of the collected simplifications using human judges.

we study if the abstractive simplifications in ASSET (test set) are preferred over lexical-paraphrase-only or splitting-only simplifications in TurkCorpus (test set) and HSplit, respectively.


In this section we study the behaviour of evaluation metrics for SS when using ASSET's simplifications (test set) as references.

we measure the correlation of standard metrics with human judgements of fluency, adequacy and simplicity, on simplifications produced by automatic systems.


if an attention head has a high negation-scope prediction performance before fine-tuning, will it increase in performance more than other attention heads that had lower performance before fine-tuning?


In Table 1, we see that performance for both BERT-12 and BERT-24 steadily increases across all datasets with increasing N ; this trend holds for the other 7 pretrained models.

in the largest setting with N = 1M, the BERT-24 embeddings distilled from the best-performing layer for each dataset drastically outperform both Word2Vec and GloVe.


Inspired by the results of Nissim et al. (2020), in this work we transparently report social bias in existing static embeddings as well as the embeddings we produce.

we exhaustively report the measured bias for all 3542 valid (pretrained model, layer, social attribute, bias definition, target word list) 5-tuples -all possible combinations of static embeddings and bias measures considered.


Taken together, our analysis suggests a concerning state of affairs regarding bias quantification measures for (static) word embeddings.

while estimates are seemingly stable to some types of choices regarding word lists, bias scores for a particular word embedding are tightly related to the definition being used and existing bias measures are markedly inconsistent with each other.


Beyond this, SIMVERB3500 was introduced by Gerz et al. (2016) to further increase coverage over all predecessors.

it shifted the focus towards verbs which had been heavily neglected in the prior datasets which centered on nouns and adjectives.


In each bio, we attempt to explain the predictions of the model.

we employ a technique that highlights words that (per our explanation method) are thought to be responsible for a particular prediction (colloquially, what the model focuses on).


We find that these models encode each of our tested word features richly across sentence tokens, often with perfect or near-perfect recoverability, but the details of how the models distribute this information vary across encoders.

bidirectional models show more nuance in information selectivity, while the deeper transformer models show more robustness to distance.


In this paper, we propose a video question answering model which effectively integrates multi-modal input sources and finds the temporally relevant information to answer questions.

we first employ dense image captions to help identify objects and their detailed salient regions and actions, and hence give the model useful extra information (in explicit textual format to allow easier matching) for answering questions.


We see that clusters correspond to topical aspects of the input (either a document or a word).

in the sentence-level case, documents in the same cluster often have the same ground-truth label.


In contrast, our method instead introduces a smaller rectifier network with ≈ 1000 additional parameters while still producing similar improvements

using trained constraints is computationally more efficient.


In this paper, we take an initial step towards studying relation extraction in dialogues by constructing the first human-annotated dialogue-based relation extraction dataset, DialogRE.

we annotate all occurrences of 36 possible relation types that exist between pairs of arguments in the 1,788 dialogues originating from the complete transcripts of Friends, a corpus that has been widely employed in dialogue research in recent years (Catizone et al., 2010;Chen and Choi, 2016;Zhou and Choi, 2018;Rashid and Blanco, 2018;Yang and Choi, 2019).


We next conduct a thorough investigation of the similarities and differences between dialoguebased and traditional relation extraction tasks by comparing DialogRE and the Slot Filling dataset (McNamee and Dang, 2009; Ji et al., 2010, 2011; Surdeanu, 2013; Surdeanu and Ji, 2014), and we argue that a relation extraction system should be aware of speakers in dialogues.

most relational triples in DialogRE (89.9%) signify either an attribute of a speaker or a relation between two speakers.


We compare DialogRE with the official SF (2013-2014) dataset (Surdeanu, 2013;Surdeanu and Ji, 2014) as 47.2% of relation types in DialogRE originate from the SF relation types (Section 2.1), and 92.2% of the source documents in it that contain ground truth relational triples are formally written newswire reports (72.8%) or well-edited web documents (19.4%) compared to the remaining documents from discussion fora.

we show the relation distributions in DialogRE and SF in Figure 1    the subjects of 77.3% of relational triples are speaker names, and more than 90.0% of relational triples contain at least one speaker argument.


We append the ground truth triggers to the input sequence on the baseline, and the F1 of this model is 74.9%, a 16.4% absolute improvement compared to the BERT baseline.

through the introduction of triggers, we observe a 22.9% absolute improvement in F1 on relation types whose inverse relation types are themselves (e.g., PER:ROOMMATE and PER:SPOUSE).


In this paper, we present a facetaware evaluation setup for better assessment of the information coverage in extracted summaries.

we treat each sentence in the reference summary as a facet, identify the sentences in the document that express the semantics of each facet as support sentences of the facet, and automatically evaluate extractive summarization methods by comparing the indices of extracted sentences and support sentences of all the facets in the reference summary.


In this paper, we argue that the information coverage in summarization can be better evaluated by facet overlap, i.e., whether the system summary covers the facets in the reference summary.

we treat each reference sentence as a facet, identify document sentences that express the semantics of each facet as support sentences of the facet, and measure information coverage by Facet-Aware Recall (FAR), i.e., how many facets are covered.


To verify the effectiveness of facet-aware evaluation, we annotate the FAMs of 150 documentsummary pairs from the test set of CNN/Daily Mail.

we take the first 50 samples in the test set, the 20 samples used in the human evaluation of Narayan et al. (2018c), and randomly draw another 80 samples.


Recently, question answering (QA) based automatic metrics have been proposed for evaluating content selection in summarization (Eyal et al., 2019;Scialom et al., 2019;.

cloze-style QA is used to evaluate whether important information in the source is recovered from the summary.


In this section, we analyze generated summaries along two dimensions: abstractiveness and faithfulness.

we aim to answer the following questions: (1) How to quantify abstractiveness of a summary? (2) Is abstractiveness encouraged more by the data or the model? (3) How does being abstractive affect faithfulness?


Under the encoder-decoder framework, we enhance the regular document encoder with a separate graph-structured encoder to maintain the global context and local characteristics of entities by using the outputs from an open information extraction (OpenIE) system.

we experiment with two graph variants, one mainly capturing entities' document-level interactions and the other reflecting such interactions within each paragraph plus topic shifts across paragraphs.


Two partitions of the same vocabulary are said to be independent if they do not share any information with respect to language L.

if we translate a sequence of symbols from L into their categories from one partition, this sequence of categories will not provide any information on how the sequence translates into categories from the other partition:


While it is tempting to equate such information with the meaning of an utterance, a large body of literature in linguistics and psycholinguistics argues that an utterance conveys much more than a simple set of facts: it carries with it a halo of intimations arising from the speaker's choices, including considerations of perspective, emphasis, and framing.

linguistic choices subtly color meaning; far from merely conveying objective facts, they reflect how speakers conceptualize meaning and affect listeners' interpretations in predictable ways.


One vexing issue is the lack of evaluation metrics that correlate with human judgments for tasks like paraphrase, image captioning, and textual entailment (see, e.g., Bhagat and Hovy, 2013;Pavlick and Kwiatkowski, 2019;Wang et al., 2019b).

it is unclear how closely a good paraphrase should hew to all aspects of the source sentence.


And a system that has learned the meaning of a human language can do things like answer questions posed in the language about things in the world (or in this case, in pictures).

what's interesting here is not that the tasks are impossible, but rather what makes them impossible: what's missing from the training data.


However, this kind of variation is not just restricted to text domains: it is a fundamental property of human-generated language: we talk differently than our parents or people from a different part of our country, etc. (Pennebaker and Stone, 2003; Eisenstein et al., 2010; Kern et al., 2016).

language reflects the diverse demographics, backgrounds, and personalities of the people who use it.


Our definition of predictive bias in NLP builds on its definition within the literature on standardized testing (i.e., SAT, GRE, etc.)

swinton (1981) states: By "predictive bias," we refer to a situation in which a [predictive model] is used to predict a specific criterion for a particular population, and is found to give systematically different predictions for subgroups of this population who are in fact identical on that specific criterion.


However, standard models trained on such data end up predicting people depicted in kitchens as women 63% of the time (Zhao et al., 2017).

an error in generating a gender reference within the text (e.g., "A [woman man] standing next to a counter-top") males an incorrect female reference much more common.


It does not require bias on the part of the annotator, data collector, or even the programmer/data analyst (though it can escalate existing biases and the models' statistical discrimination along a demographic dimension).

it extends countermeasures beyond the point some authors have made, that they are merely cosmetic and do not address the underlying cause: biased language in society (Gonen and Goldberg, 2019).


In §2 and §3, we analyze the language people use when they make forecasts in geopolitical and financial domains.

these two sections reveal how language is associated with accuracy both within and across forecasters.


In contrast to descriptive or predictive tasks, causal inference aims to understand how intervening on one variable affects another variable (Holland, 1986; Pearl, 2000; Morgan and Winship, 2015; Imbens and Rubin, 2015; Hernan and Robins ´ , 2020).

many applied researchers aim to estimate the size of a specific causal effect, the effect of a single treatment variable on an outcome variable.


Unlike predictive applications, causal applications have no ground truth and so it is difficult distinguish modeling errors and forking paths from the true causal effects.

we caution against using all available text in causal adjustment methods without any human validation or supervision, since one cannot diagnose any potential errors.


Rather than using bill text in models to explain the roll-call behavior of individual legislators, (Yano et al., 2012) include the legislation's text in a model that predicts whether a bill emerges from a standing committee, a point in the legislative process that most bills do not pass.

they use features based on the urgency and importance of the issue being addressed by the bill as well as a set of features extracted from co-sponsors of the bill.


Equipped with co-attention learning, our model is capable of the explainability by looking into the attention weights between retweet users in the propagation and words in the source tweet.

by extending the co-attention formulation (Lu et al., 2016), the proposed dual co-attention mechanism aims to attend to the source-tweet words and graphaware interaction users simultaneously (sourceinteraction co-attention), and also attend to the source-tweet words and propagated users simultaneously (source-propagation co-attention).


Since most previous work on ZP resolution uses gold syntactic trees and/or ZP positions, we also investigate our performance under these settings.

we take the noun phrases and/or ZP positions from gold trees to serve as constraints.


In-group language aims to capture whether the author of a post may be a member of the same social/demographic group that is targeted, as speaker identity changes how a statement is perceived (O'Dea et al., 2015).

in-group language (words or phrases that (re)establish belonging to a social group; Eble, 1996) can change the perceived offensiveness of a statement, such as reclaimed slurs (Croom, 2011;Galinsky et al., 2013) or self-deprecating language (Greengross and Miller, 2008).


Various work has tackled the task of making inferences about power and social dynamics.

previous work has analyzed power dynamics about specific entities, either in conversation settings (Prabhakaran et al., 2014;Danescu-Niculescu-Mizil et al., 2012) or in narrative text (Sap et al., 2017;Field et al., 2019;Antoniak et al., 2019).


However, with Affected methods, there is a considerable variation in performance.

we can observe that these models perform best when evaluated using TOP and worst when evaluated using BOTTOM 3 .


Multilingual BERT is pretrained on corpora in 104 languages; however, we probe the performance of the model in 11 languages (Arabic, Chinese, Czech, English, Farsi, Finnish, French, German, Indonesian, Latvian, and Spanish).

we probe the model on trees encoded in the Universal Dependencies v2 formalism (Nivre et al., 2020).


MBERTRAND: A model with the same parametrization as mBERT but no training.

all of the contextual attention layers are reinitialized from a normal distribution with the same mean and variance as the original parameters.


We first investigate whether mBERT builds syntactic subspaces, potentially private to each language, for a subset of the languages it was trained on; this is a prerequisite for the existence of a shared, cross-lingual syntactic subspace.

we train the structural probe to recover tree distances in each of our eleven languages.


If a probe trained to predict syntax from representations in language i also predicts syntax in language j, this is evidence that mBERT's syntactic subspace for language i also encodes syntax in language j, and thus that syntax is encoded similarly between the two languages.

we evaluate the performance of the structural probe in the following contexts: â€¢ Direct transfer, where we train on language i and evaluate on language j.


We train syntactic probes to learn a subspace on languages that primarily only use one ordering (i.e. majority class is greater than 95% of all adjectives), then evaluate their UUAS score solely on adjectives of the other ordering

we evaluate on French, which has a mix (69.8% prenominal) of both orderings, in the hope that evaluating both orderings in the same language may help correct for biases in pairwise language similarity.


Second, we provide strategies for improving module-wise faithfulness in NMNs (§4).

(a) we demonstrate how module architecture affects faithfulness (§4.1), (b) propose supervising module outputs with either a proxy task or heuristically generated data (§4.2), and (c) show that providing modules with uncontexualized token representations improves faithfulness (§4.3).


In this section, we train multiple models for the DROP and Quoref datasets, and evaluate the benefits of intermediate annotations as compared to traditional QA pairs.

we will focus on the cost vs benefit tradeoff of intermediate annotations, along with evaluating their ability to mitigate bias in the training data.


Improving the global consistency of predictions, our approach achieves large improvements over previous methods in a variety of question answering (QA) tasks including multiple-choice qualitative reasoning, cause-effect reasoning, and extractive machine reading comprehension.

our method significantly improves the performance of RoBERTa-based models by 1-5% across datasets.


Our implementations are all based on PyTorch.

to implement our classification based and span-based model, we use pytorch-transformers (Wolf et al., 2019) 6 .


Therefore, we use the pre-trained BERT-Largeuncasedd with its NSP layer to predict the most appropriate candidate for each blank given its context.

bERT is employed to predict the probability of the context and the candidate being consecutive.


We defer automatically generating distractors to future work since non-trivial distractor generation is a hard problem in itself.

we extract all passages from RACE (Lai et al., 2017) (which is also from exams) and filter out passages which have less than 10 sentences or more than 30 sentences.


Overall, we observed that our cascade models are competitive with monolithic transformers on both ASNQ and GPD datasets.

when no selection is applied (α = 0.0), a 12 layer cascade model performs equal or better to TANDABASE: on ASNQ, we improve P@1 by 2.1% (53.2 vs 52.1), and MAP by 1.2% (66.3 vs 65.5); on GDP, we achieve the same P@1 (67.5), and a slightly lower MAP (57.8 vs 58.0).


A drop rate α > 0.0 produces a small degradation in accuracy, at most, while significantly reducing the number of operations per batch (-37%).

when α = 0.3, we achieve less than 2% drop in P@1 on GPD, when compared to TANDA BASE ; on ANSQ, we slightly improve over it (52.9 vs 52.1).


To answer these two questions, in this work, we present the first deep diagnosis of essential commonsense knowledge for answering WSC questions.

we invite annotators to first provide reasons for why they choose the answers when they answer the questions, and then group all the WSC questions by different types of used commonsense knowledge (e.g., the property of entities, temporal knowledge, or spatial knowledge).


Top-100 words identified by our method cover all the words attested as real semantic shift in Hamilton et al. (2016b)’s top-10 except the word ‘wanting’.

three attested words, ‘gay’, ‘major’ and ‘check’ are present in our top-10, which also has more interesting words not present in Hamilton et al. (2016b)’s top-10 (1900 vs. 1990): van (captain vs. vehicle), press (printing vs. places), oxford (location vs. university).


Unlike previous language model (LM) based fine-tuning approaches that adjust parameters solely based on the classification error on training data, we employ the encoder-decoder framework of a UMT as a regularization component on the shared network parameters.

the cross-lingual encoder of our model learns a shared representation, which is effective for both reconstructing input sentences of two languages and generating more representative views from the input for classification.


Different from previous language model (LM) based fine-tuning approaches (Devlin et al., 2019;Conneau and Lample, 2019) that adjust parameters solely based on the classification error of training data, we utilize the encoder-decoder network from unsupervised machine translation (UMT) (Lample et al., 2018a) to regularize and refine the shared latent space.

the transformer-based encoder regularized by a language discriminator learns shared but more refined language-invariant representations, which are effective for both reconstructing sentences from two languages by the decoder and generating multi-view feature representations for classification from input documents.


We can thereby benefit from multi-view learning's superior generalization capability over single-view learning (Zhao et al., 2017).

we consider two views of input: (i) the encoded labeled documents from the source language; (ii) the encoded back-translations of the source documents from the target language.


We leverage pretrained language models (Conneau and Lample, 2019) to initialize a transformerbased UMT (Lample et al., 2018a) and train it on Wikipedia text 3 .

we sample 10 million sentences from each language pairs and use the XLM library 4 to train a UMT (Lample et al., 2018a) for 200K steps.


Finally, the contextual state sequence of X/Y is denoted as Hx/y = (h x/y 1 , . . . , hx/y n/m)

h x is the prior context; h y is the posterior context that is only available in the training stage.


Enhanced with the proposed pruning algorithm, we can see great improvement for CMAML methods against all the competing methods on both quality and diversity measurements.

our full model CMAML-Seq2SPG shows clearly better performance and the reasons can be ascribed to two aspects: firstly, the proposed Seq2SPG has a better model structure for our task and secondly, the pruning algorithm makes the models more likely to generate a user-coherent response.


We then apply pre-trained models (Radford et al., 2019) to leverage the deep attention neural networks to capture text and video dependencies with fine granularity.

we propose to capture dependencies between each token in text data and each spatial feature along the temporal dimension of the input video.


Compared with the baseline with Transformerbased neural networks (Le et al., 2019), our model treats both visual and text features with equal importance at different levels of different dimensions.

we aligned the token level with spatial level and turn level with temporal level between visual and text features.


Due to the constraint of emotion factors, the scale of proper responses shrinks, and the model is more likely to map any query to a frequentlyoccurring response in that emotion category.

given "Disgust", the response would be "You are so bad" in general, while given "Happy", it would be "Haha, you too" (Example 2 to 4 in Table 1).


Rewards designed here aim to encourage both emotion expression and content consistency.

emotion expression can be either explicit (embodied in some obvious emotion words) or implicit (reflected by the organization of the entire sentence).


Emotional Chatting Machine (ECM) (Zhou et al., 2018a) addresses the emotion factor using three new mechanisms: Emotion Category Embedding, Internal Memory, and External Memory.

1) Emotion Category Embedding models the high-level abstraction of emotion expression by embedding emotion categories, and concatenates corresponding embedding to the input at each decoding step.


Emotion-acc and Emotion-word are utilized to test the emotion expression.

emo-acc is the agreement between the ground truth labels and the predicted labels through the TextCNN classifier trained before.


Although discontinuous mentions may overlap, we discriminate this overlapping from the one in nested NER.

if one mention is completely contained by the other, we call mentions involved nested entity mentions.


We further explore attention mechanisms that take into account the relationships between hidden states.

we apply the multi-head self-attention mechanism in Transformer (Vaswani et al., 2017), which has shown promising results in many applications (Radford et al., 2018;Devlin et al., 2019).


In this paper, we take the benefits of ConvE and KBGAT together and propose a Relation-aware Inception network with joint local-global structural information for knowledge graph Embedding (ReInceptionE).

we first explore the Inception network to learn query embedding, which aims to further increase the interactions between head and relation embeddings.


Furthermore, our proposed ReInceptionE significantly outperforms ConvE and InceptionE for all relation types.

reInceptionE obtains larger improvements for complex relations, such as one-to-many, many-to-one and many-to-many.


This task is to assess the veracity of a statement when a table is given as evidence.

we evaluate our system on TABFACT (Chen et al., 2019), a large benchmark dataset for tablebased fact checking.


To address this problem, we use the graph structure to re-define the related contexts of each token for learning a graph-enhanced representation.

we present a graph-based mask matrix for self-attention mechanism in Transformer.


The basic idea is that the generation of a program tree is equivalent to the generation of a sequence of action, which is a traversal of the program tree following a particular order, like depth-first, left-to-right order.

our semantic parser works in a top-down manner in a sequence-to-sequence paradigm.


In unsupervised learning tasks such as topic discovery, mixture modeling has gained increasing attention from researchers (Wang et al., 2011;Zhou andCarin, 2012, 2015;Zhou, 2018;Zhao et al., 2019).

mixture modeling over document words devotes to assign these words to different topics via random probability measures.


These models used the neural network to learn the distribution relationship between input documents and latent topics due to its excellent function fitting ability and scalability.

the neural network parameters can be trained by back-propagation through the reparameterization of a continuous distribution (Naesseth et al., 2017) or using variance reduction techniques for a discrete distribution (Mnih and Gregor, 2014).


In contrast to variational auto-encoders on handling the case of continuous latent variables (Kingma and Welling, 2014), NVI can deal with both discrete and continuous latent variables.

a neural network is used to infer the proposed distribution q(h|d).


Considering that word embeddings have shown to capture both the semantic and syntactic relatedness in words and demonstrated impressive performance in natural language processing tasks, we also present the result of a neural autoregressive topic model that leverages word embeddings (i.e., iDocNADE).

the publicly available codes of HDP 2 , N-VDM 3 , NVLDA and ProdLDA 4 , and iDocNADE 5 are directly used.


Furthermore, the latent semantics of these corpora may be hierarchically dependent.

the topics at the corpus level and those of each document are not independent but correlated with one another.


The adequacy of vector-space representations was also questioned based on the regularities found in natural language.

fodor and Pylyshyn (1988) argued that connectionist architectures were not adequate to account for regularities which they characterised as systematicity (see also (Smolensky, 1990;fodor and McLaughlin, 1990)).


But they still required linguistically-motivated designs to work well.

feature engineering is necessary to make sure that these statistical machine-learning method can search a space of rules which is sufficiently broad to include good models but sufficiently narrow to allow learning from limited data.


Parameterised rules which are learned when paying attention to one of these vectors (in the set or in the stack) automatically generalise to the other vectors.

attention-based models have variable binding, which sequential LSTMs do not.


Secondly, the vectors in the bag are exchangeable, in the sense of Jordan (2010).

renumbering the indices used to refer to the different vectors will not change the interpretation of the representation.


These properties mean that BoV representations are nonparametric representations.

the specification of a BoV representation cannot be done just by choosing values for a fixed set of parameters.


In changing from a vector space to a bag-of-vector space, the only change is this discrete segmentation into entities.

no discrete representation of structure is added to the representation.


Using entity names to initialize node features, the GNN-based models, GMNN and RDGCN, show a clear improvement over structure-based models, suggesting that entity names provide useful clues for entity alignment.

gMNN achieves the highest Hits@10 on the DWY100K datasets, which are the only monolingual datasets (in English) in our experiments.


When comparing NMN and NMN (w/o nbr-m), we can observe around a 2.5% drop in Hits@1 and a 0.6% drop in Hits@10 on average, after removing the neighborhood matching module.

the Hits@1 scores between NMN and NMN (w/o nbr-m) differ by 3.9% on DBP15K FR−EN.


Compared to word spans in text, human is insensitive to small shifting between video frames.

small offsets between video frames do not affect the understanding of video content, but the differences of a few words or even one word could change the meaning of a sentence.


We study the effectiveness of our feature encoder and context-query attention (CQA) by replacing them with other modules.

we use bidirectional LSTM (BiLSTM) as an alternative feature encoder.


To this end, we investigate distance-based language models with explicit supervision.

we inject syntactic tree supervision into distance-based neural language models by breaking a syntactic tree into a label sequence, and extending a distance-based language model to include a multi-task objective that also learns to predict goldstandard labels.


The overall structure of our model is shown in Figure 1.

the ON-LSTM is taken as the base language model, and syntactic trees are added by conversion to distance metrics.


The first group aims to attack the OOV issue by utilizing predefined dictionaries from the target domain to facilitate cross-domain CWS (Liu et al., 2014;Zhao et al., 2018;, which are apt to suffer from scalability since not all domains possess predefined dictionaries.

these methods are directly restricted by external resources that are available in a target domain.


Nevertheless, after constructing an annotated dataset on the target domain, the OOV rate (target) drops significantly.

the DA method yields 9.92%, 13.1%, 14.09% 20.51% and 14.94% absolute OOV rate drop on the five out-domain datasets.


We define each lexeme in a language as a triple.

the i th triple consists of an orthographic word form w i , a distributional semantic vector v i that encodes the lexeme's semantics, and a declension class c i .


Here, we fill the gap between unsupervised morphological analysis and morphological generation with limited training data by proposing the task of unsupervised morphological paradigm completion.

we aim to construct and fill inflection tables exclusively from raw text and a lemma list for a known part of speech (POS), in a situation similar to those encountered by field linguists.


Different combinations of components result in major performance differences.

each step of our system has the potential to introduce errors.


After applying our framework, the response generation models can generate more informative responses that significantly improves the task completion rate.

our framework is extremely useful under low-resource settings.


We utilize the same dataset (CamRest676 or Mul-tiWOZ) to train all the models for fair comparison.

we use all the user utterances in the training corpus of CamRest676 or MultiWOZ to train the LSTM language model of WordSub and the policy network of TextSub.


And in each turn, the paraphrase generated by PARG is given one-to-one comparisons with each baseline's paraphrase by five judges.

we ask the judges to choose whether the paraphrase generated by PARG is of better, equal or worse quality than the paraphrase generated by NAEPara or SRPara, given the original utterance.


Such questions have to be syntactically valid and need to logically correlate with the answers by deducing over multiple relations on several sentences in the text.

we first build a multi-hop generation model and guide it to satisfy the logical rationality by the reasoning chain extracted from a given text.


Motivated by the above observations, we propose a practical two-stage approach to learn a multihop QG model from both a small-scale labeled data and a large-size unlabeled corpus.

we first exploit the neural hidden semi-Markov model (Dai et al., 2016) to parameterize the sophisticated structural patterns on the questions by latent variables.


In the pre-processing phase, we first mask the answer from the input contents by a special token <UNK>, to avoid the answer inclusion problem (Sun et al., 2018).

the answer words may appear in the question that would reduce the rationality.


In order to understand the effect of unlabeled data, we examined two variants of the proposed model.

ours-Pattn which was trained without unlabeled data, and ours-50% that used 50% unlabeled data for training.


We investigated the value of unlabeled data for the overall performance, especially when the labeled data was inadequate.

we randomly sampled {10%, 40%, 70%, 100%} of the labeled data, and split the unlabeled data into ten subsets.


Motivated by this fact, we are concerned with whether it is feasible to automatically normalize the texts first.

our strategy is correcting the grammatical errors contained in the input sentences, and then parsing the revised texts into semantic struc-tures with standard models.


Here we aim to train a model that can generate hard-to-classify mismatched examples, which tend to occur due to the query and product title being lexically similar.

we want to generate mismatched query-item examples that have a realistic chance of appearing in the search results for said query.


We propose uncertainty-aware curriculum learning, which is motivated by the intuition that: 1) the higher the uncertainty in a translation pair, the more complex and rarer the information it contains; and 2) the end of the decline in model uncertainty indicates the completeness of current training stage.

we serve cross-entropy of an example as its data difficulty and exploit the variance of distributions over the weights of the network to present the model uncertainty.


We introduce a novel uncertainty-aware curriculum learning framework, which serves uncertainty as its principle to order the input examples and control the duration of each training stage.

we measure the data uncertainty of a sentence pair according to its joint distribution that is estimated by a language model pre-trained on the training corpus.


In this work, we adopt a widely used CL strategy called baby step (Cirik et al., 2016; Zhang et al., 2018) to arrange training data and organize the training process.

the whole training set D is divided into different buckets, i.e. steps {D1, · · · , DT }, in which those examples with similar data uncertainty scores u data are categorized into the same bucket.


This verifies our hypothesis that data uncertainty is of higher relevance in respect to the difficulty of an example for a NMT model than its sentence length and word rarity counterparts.

the results show the utility of estimating the uncertainty on either the source or target side of a translation pair.


Surprisingly, the tendency highly accords with the psychology of human students when they getting into a new area, i.e. Dunning Kruger Curve (Figure 1, Kruger and Dunning, 1999).

starting from scratch, peoples rapidly grow their knowledge, they therefore have a large amount of confidence.


In this paper, we close this gap by reporting concept extraction performance on automatically anonymized data and investigating joint models for de-identification and concept extraction.

we propose a stacked model with restricted access to privacy-sensitive information and a multitask model.


Feutry et al. (2018) and Friedrich et al. (2019) create pseudo-de-identified text representations with adversarial training.

they replace personal information, such as names, by other names.


In addition, the annotation projection can be combined with model transferring naturally.

the projected SRL tags in annotation projection could contain much noise because of the source-side automatic annotations.


The task can be treated as a standard sequence labeling problem, and a simple multi-layer BiLSTM-CRF model is exploited here, which has archived state-of-the-art performance with contextualized word representations (He et al., 2018b; Xia et al., 2019; He et al., 2019).

we adapt the model to better support multilingual inputs by using a PGN module on the BiLSTM (Hochreiter and Schmidhuber, 1997).


Therefore, we first retrieve top 200 matched text with elastic search based on the similarity of Bert embeddings (Devlin et al., 2019).

we pass sentences through Bert and derive a fixed-sized vector by averaging the outputs from the second-to-last layer (May et al., 2019).


Our dialogue act generator and response generator share one same encoder and input, but having different masking strategies for the input to focus on different information.

only the current utterance is kept for act generation, while the entire history utterances are used for response generation.


These transfer learning methods exhibit clear advantages over more traditional task-specific approaches.

they can be trained in an unsupervized manner, thereby taking advantage of the information contained in large amounts of raw text.