For (i), the target probability of each word is set proportional to the square root of its frequency in the visited stream.

highly frequent words would take a smaller portion compared to reservoir sampling where the word distribution in the memory is linear to its frequency in the visited stream, leaving space for storing more diverse examples.


We find that (i) diversifying storage improves performance at the early stage of the stream but not in the later stages; (ii) prioritizing words likely to be forgotten does not improve performance.

future works should find a balance between storing more diverse or important examples and respecting original data distribution.


Unlike Multistream, which leverages fine-grained region-level features, our results are reported on global framelevel features.

it may be difficult for HERO to capture the inconsistency between hypothesis and video content.


Adversaries could easily exploit this to disseminate realistic-looking neural fake news.

exploring the visual-semantic consistency between the article text and images could prove to be an important area for research in defending against generated disinformation.


In contrast, Type C articles have the potential to be exploited by adversaries to disseminate large amount of misleading disinformation due to its generated article contents.

our proposed approach is geared towards addressing this particular type of generated articles.


Furthermore, each OIE system extracts the interested facts in the desired form at the time of development and omits the uninterested facts.

they are not adaptable to new requirements.


It is also very abstract that sentences with the same meaning but in very different expressions will share the same AMR annotation.

aMR is difficult to label (cost about 10 min to label a sample 4 ) and is very difficult to learn.


Due to resource limitations and in the spirit of environmental responsibility (Strubell et al., 2019;Schwartz et al., 2019), we conduct our experiments on the base models: BERT-baseuncased, RoBERTa-base, and DistilBERT-baseuncased.

the BERT/RoBERTa models we use have 12 transformer blocks (0–11 indexed) producing 768-dimension vectors; the DistilBERT model we use has the same dimension but contains 6 transformer blocks (0–5 indexed).


Masking and finetuning achieve accuracy 84.79% and 85.25%, which are comparable and both outperform the baseline 50%, demonstrating successful knowledge transfer.

finetuning and masking yield models with similar generalization ability.


For POS, after wordpiece tokenization, we see 1 sentence in dev and 2 sentences in test have more than 126 (the [CLS] and [SEP] need to be considered) wordpieces.

we exclude 5 annotated words in dev and 87 annotated words in test.


Intuitively, if a training example has a low sentence-level probability, it is less likely to provide useful information for improving model performance, and thus is regarded as an inactive example.

we adopt sentence-level probability P (y|x) as the metric to measure the activeness level of each training example: where T is the number of target words in the training example.


To save the time cost, a promising strategy is to let the identification model take the responsibility of rejuvenation.

we used the TRANSFORMER-BIG model with the large batch configuration trained on the raw data to accomplish both identification and rejuvenation.


In the target vocabulary, words are sorted in the descending order of their frequencies in the whole training data, and the frequency rank of a word is its position in the dictionary.

the higher the frequency rank is, the more rare the word is in the training data.


The language coverage could be broadened with other knowledge, such as that encoded in WALS, to distinguish even more language properties.

to obtain the best of both views (KB and task-learned) with minimal information loss, we project a shared space of discrete and continuous features using a variant of canonical correlation analysis (Raghu et al., 2017).


We then can assess their applicability on multilingual NMT tasks that require guidance from language relationships.

language clustering and ranking related partner languages for (multilingual) transfer are our study cases (§6).


The list is a crafted set of concepts for comparative linguistics (e.g. I, eye, sleep), and it is usually processed by lexicostatistics methods to study language relationship through time.

we prefer to argue that corpus-based embeddings could partially encode lexical similarity of languages.


However, the lack of an unequivocal definition of hate speech, the use of slurs in friendly conversations as opposed to sarcasm and metaphors in elusive hate speech (Malmasi and Zampieri, 2018), and the data collection timeline (Liu et al., 2019) contribute to the complexity and imbalance of the available datasets.

training hate speech classifiers easily produces false positives when tested on posts that contain controversial or search-related identity words (Park et al., 2018;Sap et al., 2019;Davidson et al., 2019;Kim et al., 2020).


The initial list of predefined keywords such as the ones we have shown in Table 1 carries additional words in English and Arabic.

for these two datasets, we have measured bias using two predefined lists of keywords: the initial list and one that is specific to the dataset in question.


The datasets were too large to be fit into GPU as a whole.

we shifted the neighbor search to CPU, but that again took more than a day to complete.


The reason is that both of them synthesize sentences are either of less diverse or of less quality.

we propose a controllable sampling strategy to generate reasonable source sentences: at each decoding step, if the word distribution is sharp then we take the word with the maximum probability, otherwise the sampling method formulated in Eq.


However, observe that this new rule performs two tests that could be done independently: 1. the right span boundary of the first antecedent must match the left span boundary of the second one; 2. the right span boundary of the second antecedent must match the left span boundary of the third antecedent.

we can break the deduction into two sequential deductions, first testing the "k" boundary then the "l" boundary.


A popular solution tackles the grammatical error correction as a monolingual machine translation task where ungrammatical sentences are regarded as the source language and corrected sentences as the target language (Ji et al., 2017;Chollampatt and Ng, 2018a).

the GEC can be modeled using some relatively mature machine translation models, such as the sequence-to-sequence (seq2seq) paradigm (Sutskever et al., 2014).


Assuming that a token is selected to be replaced, and its candidate substitutes are retrieved by the mapping, we want the selected substitute can fit in well with the token's context and maintain both the semantic and syntactic coherence.

we define a function s based on the edit distance (Marzal and Vidal, 1993) to estimate the similarity scores between two sentences.


We would also like to know which type of words to modify is most likely to form a successful attack.

we calculate the correction rates of the newly added errors with different types of part-ofspeech.


As it is difficult to ensure high quality annotations for 21 languages using crowdsourcing, we relied on colleagues by reaching out on NLP and Linguistics mailing lists.

the number of evaluators per language varies (cf.


Specifically, when the LLL model is trained on a new task, we assign a teacher model to first learn the new task, and pass the knowledge to the LLL model via knowledge distillation.

the LLL model can better adapt to the new task while keeping the previously learned knowledge.


Inspired by knowledge distillation (Bucila et al., 2006;Hinton et al., 2015;Kim and Rush, 2016), in which a student (smaller) model is trained to imitate the behavior of a teacher (larger) model in order to reach the performance closer to the teacher model, the LLL model in L2KD can be seen as a weak learner that needs to compress knowledge from different tasks into a compact single model.

lll can benefit from the similar procedure of knowledge distillation, although the model size is equal to its teacher model.


By way of assuming Ψ ∼ GEM(γ), an HDP assumes an infinite number of topics are present a priori, with the number of tokens per topic decreasing rapidly with the topic's index in a manner controlled by γ.

under the model, a topic with a sufficiently large index should contain no tokens with high probability.


The subcluster split-merge algorithm is designed to converge with fewer iterations, but is more costly to run per iteration.

we used a fixed computational budget of 24 hours of wall-clock time for both algorithms.


However, several alignment-based STS methods employ Euclidean distance  or dot product  to compute the word dissimilarity.

the question arises as to which is the most suitable method for computing word dissimilarity.


The key difference lies in the fact that ADD treats a sentence as a single vector (the barycenter of direction vectors), whereas WRD treats a sentence as a set of direction vectors.

it is natural that WRD had a positive effect on STS tasks, given that STS tasks require the word alignment (i.e., they assume that words are treated disjointedly).


The phenomenon, however, is often overlooked in existing matching models.

the feature vectors are constructed without any regularization, which inevitably increases the difficulty of learning the downstream matching functions.


In WD-Match, a Wasserstein distance-based regularizer is defined to regularize the features vectors projected from different domains.

the method enforces the feature projection function to generate vectors such that those correspond to different domains cannot be easily discriminated.


G module is implemented as a two-layer MLP (the number of neurons in the second layer is set as one).

the additional computing cost comes from the training of the two-layer MLP, which is of O(T * N * K * 1), where T is the number of training iterations, N number of training examples, K number of neurons in the first layer of MLP (without considering the compute cost of the activation function).


We observe that the proposed two-stage methods, GW-GeoMM and SL-GeoMM, obtain scores on par with state-of-the-art methods, UMWE and UMH.

multilingual approaches can learn an effective multilingual space for closeby languages.


In addition, most previous works do not consider the importance of each teacher layer and use the same layer weights among various tasks, which create a substantial barrier for generalizing the compressed model to different NLP tasks.

an adaptive compression model should be designed to transfer knowledge from all teacher layers dynamically and effectively for different NLP tasks.


Since different attention and hidden layers of BERT can learn different levels of linguistic knowledge, these layers should have different weights for various NLP tasks.

we propose a cost attention mechanism to assign weights for each attention and hidden layers automatically.


Given an event like "Jerry repels Tom's attack", to approximate the phrases in Atomic, we firstly annotate the person roles, that is, replacing the subject person with "PersonX" and the other person with "PersonY".

we get "PersonX repels PersonY's attack".


However, the counterfactual samples generated by most previous methods are simply added to the training data for augmentation and are not fully utilized.

we introduce a novel selfsupervised contrastive learning mechanism to learn the relationship between original samples, factual samples and counterfactual samples.


Although the MAMS described in Section 5.3 provides a training set with diversity, it remains difficult to improve aspect robustness for other domains, or future new datasets.

we propose a flexible method, adversarial training, for aspect robustness, which is applicable to any given dataset.


A possible explanation is that the short reference summaries fail to capture all the important information of original documents.

directly comparing with document representations will suffer much less information loss.


We make use of the heuristics that nearby sequences in the document contain the most important information to recover the masked words.

the challenging retrieval part can be replaced by soft-attention mechanism, making our model much easier to train.


During training, we fix the position embeddings for the pre-appended special tokens, and randomly select 64 continuous positions from 0 to 564 for the other words.

the model can be used to encode longer sequences in downstream tasks.


Rather a pointer is made to the location where the number 8 occurred in an algebraic word problem.

using such an "operand-context pointer" enables a model to access contextual information about the number directly, as shown in Figure 1 (c); thus, the operand-context separation issue can be addressed.


We observed that the existing pure neural model's performance on low-complexity dataset of MAWPS was relatively high at 78.9%, compared to that of high-complexity dataset of ALG514 (44.5%).

using Expression tokens and operand-context pointers contributed to higher performance when applied to high-complexity datasets of ALG514 and DRAW-1K, as shown in Table 5.


As the expression fragmentation issue can arise for each token, probability of fragmentation issues' occurrence increases exponentially as the number of unknowns/Op tokens in a problem increases.

the vanilla Transformer model, which could not handle the fragmentation issue, yields low accuracy on high-complexity datasets.


When generating solution equations for the comparative phrases, the order of arguments is a matter for an equation that contains non-commutative operators, such as subtractions or divisions.

errors occurred when the order of arguments for comparative phrases with non-commutative operators was mixed up.


However, neural networks are less interpretable and need to be trained with a large amount of data to make it possible to learn such implicit logic.

we consider tackling the problems by exploiting logic knowledge.


Notice that the intent classifier is typically implemented using standard text classification algorithms (Weiss et al., 2012;Larson et al., 2019;Casanueva et al., 2020).

to perform OOS sample detection, methods often rely on one-class classification or threshold rejectionbased techniques using the probability outputs for each class (Larson et al., 2019) or reconstruction errors (Ryu et al., 2017, 2018).


However, in practice, collecting OOS data can be a burden for intent classifier creation, which is generally carried out by domain experts and not by machine learning experts.

in the ideal world, one should rely solely on in-scope data for this task because it is very difficult to collect a set of data that appropriately represents the space of the very unpredictable OOS inputs.


It is a language representation model pre-trained on unlabeled text and conditioned on both the left and right contexts.

a simple output layer can be fine-tuned to attain strong results in many different tasks.


But, although BERT+ has not been significantly better than BERT with PT-BR chatbots, the proposed word graph-based approach had a great impact in reducing that 5% difference, since both present similar EER values, and still had a huge impact in FRR rates since BERT+ presented significantly better values.

it is likely that by improving the mapping of sentence and graph embeddings for those datasets, and consequently reducing that 5% gap in ISER, BERT+ will stand out as a significantly better approach than BERT.


Unlike the constraintbased methods, which use translations of the focus word to post-process the output of a WSD system, t emb provides the translation information in the form of an embedding directly as input to the WSD system.

translation information is used as an additional feature to improve sense predictions of the base WSD system.


Although attention weights in some NMT systems may be used to derive word alignment, such an approach is not necessarily more accurate than off-the-shelf alignment tools (Li et al., 2019).

our approach is to instead identify the word-level translations by performing a bitext-based alignment between the source focus words and their translations.


To highlight the improvement in contextualization alone, since the Within Word tasks before lemmatization may contain different word forms of the same lemma as the target words in each pair, we lemmatize all the target words in the dataset.

each pair in the Within Word tasks now contains the identical target word.


Moreover, latent-variable models can represent multimodal distributions.

for these conditional tasks, the latent variable can be used as a source of stochasticity to ensure more diverse translations (Pagnoni et al., 2018) or answers in a dialogue (Serban et al., 2017).


However, this model would not be more useful than a standard, left-to-right autoregressive models.

it is necessary to check that such useless, purely local features are not learned.


Moreover, if the latent variable of the VAEs did encode the label perfectly and exclusively, they would reconstruct the first words or recover sentence length with much lower accuracy than what is observed.

we conclude that seq2seq VAEs are biased towards memorizing the first few words and the sentence length.


As Long et al. (2019) reported, the max-pooling operator is better than the average operator, both when the encoder is a LSTM and BoW (possibly because the maximum introduces a non-linearity).


The latent variable is more predictive of global features and memorisation of the first words and sentence length is decreased.

these models are more suitable for diverse and controllable generation.


However, information related to the second word in the latent variable can help the decoder predict the first word.

gains in position i can only be attributed to information pertaining to the words in positions >=i.


However, previous work neglects the fact that there is usually a limited time budget to interact with domain experts (e.g., medical experts, biologists)  and high-quality natural language explanations are expensive, by nature.

we focus on eliciting fewer but more informative explanations to reduce expert involvement.


Contrastive Natural Language Explanations Existing research in social science and cognitive science (Miller, 2019; Mittelstadt et al., 2019) suggests contrastive explanations are more effective in human learning than descriptive explanations.

we choose contrastive natural language explanations to benefit our learners.


Unlike previous active learning on data-points, our class-based active learning is empirically insensitive to the change of random seeds and hyper-parameter (e.g., batch size).

we could collect the explanations in an on-demand manner.


However, Table 10 shows that even when continuing to train this model for a long time no multilinguality arises.

in this configuration the model has enough capacity to model the languages independently of each other -and due to the modifications apparently no incentive to try to align the language representations.


Recent work (Yu et al., 2020) shows that gradient conflict between dissimilar tasks, defined as a negative cosine similarity between gradients, is predictive of negative interference in multi-task learning.

we study whether gradient conflicts exist between languages in multilingual models.


As shown by the expansions per step in Table 2, VAR-STREAM uses the batch capacity of 100 most efficiently.

vAR-STREAM is faster than both vAR-BATCH and FIXED, despite overhead which is exacerbated in a small model.


However, since some languages do not typically use whitespace between words (e.g., Thai), we used the heuristic of SentencePiece meta symbol U+2581 to designate the beginning of the word.

a word is defined as the token span between two successive U+2581 symbols.


Using three encoders did not yield clear improvements over two encoders.

we do not experiment with using more than three encoders.


Bansal et al. (2019) demonstrated that better feature learning from supervised tasks helps few-shot learning.

we also evaluate multi-task learning and multi-task meta-learning for few-shot generalization.


A few recent projects reveal that GLUE tasks may be not sophisticated enough and do not require much tasks-specific linguistic knowledge (Kovaleva et al., 2019;Warstadt et al., 2019).

superGLUE benchmark, being more challenging, becomes much more preferable for evaluation of language models.


Furthermore, the normalization involves the prediction of both operation types and token labels, enabling TNT to learn from more challenging tasks than the standard task of masked word recovery.

the experiments demonstrate that TNT outperforms strong baselines on the hate speech classification task.


However, we showed that using En dev accuracy for checkpoint selection leads to somewhat arbitrary zero-shot results.

we propose reporting oracle accuracies, where one still fine-tunes using English data, but selects a checkpoint using target dev.


They use different language models and word embeddings (e.g., BERT, RoBERTa, or BiRNN), and have been trained on different data (e.g., DPR (Rahman and Ng, 2012), WinoGrande, or no additional data).

it is unclear whether the choice of the objective function is essential for pronoun resolution tasks.


The relatively free word order this allows creates much less emphasis on the collocation of a semantic unit's tokens.

as conversational assistants progress toward multiple languages it's important to consider that constraints that are acceptable if only English is considered will not analogously scale to other languages.


The refinement approach delegates responsibility of sessionbased semantic parsing to a separate dialog component.

refinement approaches tend to have a very limited ontology due to the semantic parser operating over a fixed input (non-session utterances).


However, for leaves with larger number of examples statistical significance alone is insufficient, because there are a large number of cases where there are small but significant differences from the ratio of agreement expected by chance.

in addition to comparing the p-value we also compute the effect size which provides a quantitative measure on the magnitude of an effect (Sullivan and Feinn, 2012).


Whereas the Statistical Threshold uses effect size with the significance test which takes into account the sample size within a leaf leading to better leaves.

we choose to use Statistical-Threshold for all our simulation experiments.


As (1) illustrates, not all sentences in a response to an advice-seeking question constitute advice.

we want annotators to highlight which parts of the response to a question are advice, and which  are not.


For instance, the "activation"-anchored event (Figure 2) is both THEME and CAUSE of "induced"and "promote"-anchored event heads, respectively.

both r and h are multi-label, and the label for "activation" is encoded as +REGULATION, [THEME, CAUSE] where the order of r and h items is preserved.


In a crowdsourcing setting, we oftentimes hire a much larger set of annotators that are not professionally trained and may be only working on the task sporadically.

it is infeasible to ask them to follow detailed guidelines.


In our experiments, we aim at evaluating the multilingual and the cross-lingual question answering capabilities of different models.

we split the data in order to support both evaluation strategies: Multilingual and Cross-lingual.


Social biases appear to be a natural component of human cognition that allow people to make judgments efficiently (Kahneman et al., 1982).

they are often implicit-people are unaware of their own biases (Blair, 2002; Bargh, 1999)-and manifest subtly, e.g., as microaggressions or condescension (Huckin, 2002;Sue, 2010).


The main challenge is encouraging the model to focus on text features that are indicative of bias, rather than artifacts in data that correlate with the gender of the addressee but occur because of confounding variables (confounds).

the core of our methodology focuses on reducing the influence of confounds.


However, we only want the model to learn that references to appearance are indicative of gender if they occur in unsolicited contexts.

our model needs to account for the effects of O TXT: Because of correlations between W GEN and O TXT, COM TXT values may contain features that are predictive of W GEN, but are caused by O TXT, rather than by W GEN. We face a similar problem with W TRAITS.


While there may still be overlap in some latent W TRAITS, we expect there to be less overlap in W TRAITS between the train and test set than within the train set.

improved performance over the held-out test set would suggest that demotion effectively reduces the influence of the latent confounding variables-the model learns characteristics of comments addressed to women generally rather than characteristics specific to the individual people in the training set.


Because most gender-related microaggressions target women, if our model predicts that the reported microaggression was addressed to a woman (e.g. W GEN = F), we assume that the post is a gender-tagged microaggression.

our models are not trained at all for identifying gender-tagged microaggressions.


However, by controlling for O TXT, propensity matching discards many of these comments.

by demoting a confounding variable, we make the prediction task more difficult.


Specifically, for the 6K words, we report the mean precision and recall of every 10 consecutive words in the vocabulary.

we have 600 data points, each representing precision/recall of 10 words.


The two UI elements have very similar images (magnifiers) although they are for searching different objects.

context information is critical for models to decode the correct objects.


To avoid information leaks, the split was done app-wise so that all the screens from the same app will not be shared across different splits.

all the apps and screens in the test dataset are unseen during training, which allow us to examine how each model configuration generalizes to unseen conditions at test.


If only textual inputs are given, they cannot effectively incorporate visual knowledge in their representations.

their help for entailing the contradiction between a and b is limited.


Fourth, individuals will mirror the language as a way of decreasing social distance which can increase trust (Scissors et al., 2008); Wang et al. (2015) found that lexical alignment is associated with increased emotional support.

we include a feature for lexical alignment as the % of the condolence's words that were also used in the distress comment.


To get rid of this annotation burden, we formulate the problem from the perspective of Multiple Instance Learning (MIL; Keeler and Rumelhart, 1992).

our model learns to spot story attributes in reviews in a weakly supervised fashion and does not expect direct tag level supervision.


However, we expect a latent correlation between Y P and Y C that can be jointly modeled while modeling P (Y P |X), hence helping the extraction of Y C without any direct supervision.

we first supervise a model containing a synopsis encoder and a review encoder to learn P (Y P |X) (Section 4.1), and later we use the trained review encoder to generate complementary tagset Y C (Section 4.2).


For example, we observe that the model usually puts higher attention weights on opinion-heavy words in the reviews.

we use the attention weights on words and sentences in reviews to extract an additional open-vocabulary tagset Y C .


However, directly fine-tuning such models for long texts like synopses and reviews is extremely memory expensive.

we employ Sentence-BERT (SBERT; Reimers and Gurevych, 2019) in our work, which is a state-of-the-art universal sentence encoder built with pre-trained BERT (Devlin et al., 2019).


Figure 7(a) shows that, for the predefined tags, our tagsets were more relevant than the baseline ones for 57% movies, the baseline tags were better than HN(A)+MIL for 24% movies, and both systems were equally performing for 19% movies.


Results in Table 4 show that, our system can indeed predict tags that are very relevant to the new types of stories.

we conclude that our approach also shows great promise for other domains and can be extended with little effort.


Beyond its interpretation as shared information, mutual information gives little in terms of interpretability: It has no consistent reference points, beyond that the minimum possible MI is zero.

several variants of MI are preferred in community detection.


Combinatory categorial grammar (CCG) is a lexicalized grammatical formalism, where the lexical categories (also known as supertags) of the words in a sentence provide informative syntactic and semantic knowledge for text understanding.

ccG parse often provides useful information for many downstream natural language processing (NLP) tasks such as logical reasoning (Yoshikawa et al., 2018) and semantic parsing (Beschke, 2019).


As high-quality dependency parsers are not always available, we do not want our CCG supertaggers to rely on the existence of dependency parsers.

we need another way to extract useful word pairs to build GCN models.


It is not very convenient to utilize a large amount of unlabeled data in the baseline method wt, since this will directly increase the amount of augmented data.

some of augmented data may not be utilized before sequence tagging models converge.


Their loss is not applicable in our setting, because the embedding space of an autoencoder is not unit-normalized like word vectors typically are.

we employ cosine loss and leave the exploration of other regression losses to future work.


Our model’s performance is close to that of Wang et al. (2019), even without FGIM at inference time.

our model has a much lower computational overhead.


Our motivation comes from the fact that sentence generation along parse trees can intrinsically capture and maintain the syntactic information (Eriguchi et al., 2017;Aharoni and Goldberg, 2017;Iyyer et al., 2018), and show better performances than sequential recurrent models (Li et al., 2015;Iyyer et al., 2014).

we design a novel tree-based autoencoder to generate adversarial text that can simultaneously preserve both semantic meaning and syntactic structures of original sentences.


Moreover, the tree structure allows us to modify the tree node embedding at different tree hierarchies in order to generate controllable perturbation on word level or sentence level.

we explore the following two types of attacks at root level and leaf level T3(SENT) and T3(WORD), which are shown in Figure 3 and Figure 4.


From the table, we find using different initialization methods will greatly affect the attack success rates.

the initial sentence selection methods are indeed important to help reduce the number of iteration steps and fastly converge to the optimal z * that can attack the model.


Previous methods craft adversarial samples mainly based on specific rules (Li et al., 2018;Gao et al., 2018;Alzantot et al., 2018;Ren et al., 2019;Jin et al., 2019;Zang et al., 2020).

these methods are difficult to guarantee the fluency and semantically preservation in the generated adversarial samples at the same time.


On the other hand, BERT is a pre-trained masked language model on extremely large-scale unsupervised data and has learned general-purpose language knowledge.

bERT has the potential to generate more fluent and semantic-consistent substitutions for an input text.


While most words are still single words, rare words are tokenized into sub-words.

we treat single words and sub-words separately to generate the substitutes.


Since the average sequence length is relatively long, the target model tends to make judgments by only a few words in a sequence, which is not the natural way of human prediction.

the perturbation of these keywords would result in incorrect prediction from the target model, revealing the vulnerability of it.


Nevertheless, candidates generated from the masked language model can sometimes be antonyms or irrelevant to the original words, causing a semantic loss.

enhancing language models to generate more semantically related perturbations can be one possible solution to perfect BERT-Attack in the future.


Previous studies have created synthetic data from generic news summarization corpora which have a small set of aspects (e.g., "sports", "health" and other 4 aspects in (Frermann and Klementiev, 2019)).

models trained on these data tend to be restricted to the pre-defined set and fall short of summarizing on other diverse aspects.


This evaluation setup clearly does not account for the entity distributions in the real data.

the reported performance scores do not reflect the effectiveness of these models when adapting to a new domain.


Tag set extension Our first set of experiments are motivated by the fact that new types of entities often emerge in some domains such as medical and social media.

we evaluate the performance of our systems on recognizing new entity types as they emerge in the source domain.


Concretely, "Apple", "Microsoft", "Coca", and "Cola" all contain only alphabetical letters with the first one capitalized; Tokens "Inc.", "Corp.", and "Co." all are alphabetical letters with first letter capitalized, and they all end with a dot.

"Apple Inc." and "Microsoft Corp." have the same sequence of structure vectors.


Moreover, for consecutive tokens with identical structure vectors, we combine them into one and hence "Coca Cola" shares the same structure vectors with the other two.

if one of the three is labeled as <name><suffix> , we can apply the same sequence of labels to the other two examples as weak labels without actual human annotation.


Current knowledge bases (such as ATOMIC) contain social rather than physical effects.

generation models trained on these knowledge bases incorrectly force the effects to be social.


We also found that when manually evaluating on ∼200 dev datapoints, the score was systematically a few (∼10%) points higher than BLEU, while the trends and model rankings remained the same, indicating robustness of the automatic metric.

the proposed metric aligns with human evaluation, and is able to use existing generation metrics thereby simplifying evaluation, allowing easier reproducibility.


Additionally, user feedback from these experiments suggested that we generate shorter entries, as longer ones frequently devolved into unrelated and incoherent sentences.

for our final experiments detailed in the next section, we also truncate model outputs to a maximum of four sentences.


The amount of author effort involved in evaluation, when combined with the relatively small size of the STORIUM community, can cause evaluation to take a considerable amount of time (i.e., to collect hundreds of judgements) as evidenced in our analysis (Section 5).

our platform is not currently suitable for "instant" evaluation of generated stories.


Supporting multi-domain conversations spanning multiple APIs quickly grows out of hand, requiring expert linguists and rigorous testing to ensure the grammatical correctness and appropriateness of generated utterances.

data-driven generative approaches have gained prominence.


The main objective of this work is to improve attention supervisions for the purpose of better text classification.

we evaluate the three attention methods by their contribution to the classification performance.


On the other hand, for the head categories with many instances, general approaches such as the single CNN model (Kim, 2014;Liu et al., 2017) may be more effective in terms of performance and more efficient in terms of complexity.

our basic idea for tackling the problem of extremely imbalanced multi-label text classification is a hybrid solution that adapts a general approach (i.e., a Single network) for head categories and a few-shot approach for tail categories, so that we can take the advantages of both of them.


For example, if an instance has multiple categories and each category has a representation vector, it is difficult to learn a representation of this instance near the representations of all these categories through a single similarity output.

we propose a category-specific similarity in the Siamese structure to capture the rich information in the similarities.


Moreover, in some situations, parsers are supplied by third parties, making it impossible to alter them.

assuming parsers are a black box, it is indispensable to conduct research on an interactive approach for enhancing the text-to-SQL technique in complex scenarios.


Compared to aligning an NL question with a SQL query, the alignment of two NL questions is more reasonable because it utilizes a similar linguistic structure and can make better use of pre-trained models (e.g., BERT).

before the alignment, we restate the predicted SQL y into a natural language question x .


As users are non-expert and unfamiliar with database operations, simply picking an option is more natural and friendly.

the Question Generator is designed to generate a multi-choice question for each uncertain token.


As analyzed in Section 2, most of the uncertain tokens are related to database information.

for each uncertain token in an NL question, we find out the corresponding database and add all the column and table names into the candidate set.


A major challenge to the proposed method is the absence of manual annotation of mentions and relations.

we propose an automatic annotation method (Section 2.4) based on aligning tokens in a SQL with corresponding question.


The main reason is that the sentence embeddings in the pipeline approach are not shared.

although these two subtasks can be well learned separately, they are not trained to collaborate with each other.


Indeed, in the rebuttal phase, authors reply reviewer's suggestions and questions very carefully to make the points clear, sometimes by citing the review arguments.

the structure and the format of rebuttals are relatively fixed, while reviewers have more flexibility in the style and the structure when writing reviews.


As shown in Figure 1, graph-attention will degenerate into a vanilla selfattention layer when the nodes in the graph are fully connected.

the graph-attention can be considered as a special case of self-attention.


It is obvious that graph attention can not cover the last three attention patterns.

we draw a conclusion that self attention has advantages on generality and flexibility.


As regards WSD, instead, we are no longer bound by the long-standing limits of predefined sense inventories.

it is possible to give (i) a meaningful answer for words that are not in the inventory, and (ii) one that fits the meaning and the granularity required by a given context better than any sense in the inventory.


Other errors stem from the fact that the model can only rely on the knowledge about possible definienda that it is able to store in the parameters during the pre-training and training stages.

if the contextual knowledge is not sufficient to extrapolate a definition, the model which is required to always generate an output will hallucinate an answer on the basis of contextual clues, incurring the risk of introducing nonfactualities.


Reif et al. (2019) showed that senses are encoded with finer-grained precision in higher layers, to the extent that their representation of the same token tends not to be self-similar across different contexts (Ethayarajh, 2019;  Mickus et al., 2020).

we hypothesise that abstract, type-level information could be codified in lower layers instead.


Interestingly, we find that the non-spurious correlations are more located in entity representation rather than context representation.

our method eliminates part of the spurious correlations between context representation and output labels.


Furthermore, fully-annotated training data will be rare due to the expensive cost.

training set can only cover a minor part of test mentions and diverse context patterns must be learned from minimal instances.


In open NER, however, most entities (e.g., movie, song and book) do not have such strong name regularity, and some of mentions can even be random utterances.

it is critical to evaluate the impact of name regularity on generalization.


This ability, obviously, is not what we desire because 1) in real world applications, most entity mentions are new and unseen, which means out-of-dictionary mentions will dominate the test process; 2) because the training instances are very limited in open situations, it is too expensive to achieve high mention coverage; 3) many longtail mentions in the training set would be oneshot, i.e., the mention only appears once in the training data.

it is necessary to exploit whether NER models can still reach reasonable performance in low mention coverage situation.


We believe this is because, as some previous studies in other tasks (Zhang et al., 2016;Lu et al., 2019) have pointed out, neural networks have strong ability and tendency to memorize training instances.

the high mention coverage will mislead the models to mainly memorize and disambiguate frequent entity names even though they are irregular, but ignore informative context patterns which are useful for generalization over unseen mentions.


While most graph-structured data has a wide variety of inherent geometric structures, e.g. partially tree-like and partially cyclical, the above studies model the latent structures in a single geometry with a constant curvature, limiting the flexibility of the model to match the hypothetical intrinsic manifold.

using a product of different constant curvature spaces  might be helpful to match the underlying geometries of temporal knowledge graphs and provide high-quality representations.


Besides, there is no statistically significant difference in the model performance when using different optimizers, such as Riemannian Adam (RADAM) and Riemannian stochastic gradient descent (RSGD).

for the model's simplicity, we decide to use RSGD.


Given the known bias that female characters are portrayed with less agency (Sap et al., 2017), our goal is to re-balance their agency levels to be more on par with those of male characters.

we revise only the sentences describing female characters to have higher agency, using POWERTRANS-FORMER.


Meanwhile, in both EURLEX57K and AMAZON13K, the performance of ATTENTION-XML is competitive with both TF-IDF-based PLT-based methods and BIGRU-LWAN, suggesting that the bag-of-words assumption holds in these cases.

we can fairly assume that word order and global context (longterm dependencies) do not play a drastic role when predicting labels (concepts) on these datasets.


We simulate a setting where we have not enough information about the biases for training a debiased model, and thus biased examples should be identified automatically.

we only use the existing challenge test set for each examined task strictly for evaluation and do not use the information about their corresponding bias types during training.


After the embeddings learning we can get a distribution over all of the subframe labels for each paragraph which is based on the cosine similarity between the embeddings of the paragraph and subframe labels.

our model combined with the labeled n-grams have the ability to expand the subframe labels to unlabeled text from other domains of the same topics without any human evaluation.


For each object with M instances in the image, we randomly remove m instances from the image s.t. m ∈ {0, . . . , M} using polygon annotations from the COCO (Lin et al., 2014) dataset

for each image, we get multiple masked images, with pixels inside the instance bounding-box removed, as shown in Figure 3.


However, the use of AL with deep pre-trained models for text classification -and BERT in particular -has so far received surprisingly little consideration.

while recent papers have demonstrated the value of AL for various deep-learning text classification schemes (Shen et al., 2017; Zhang et al., 2017; Siddhant and Lipton, 2018; Prabhu et al., 2019), the potential of AL combined with BERT is yet to be explored.


In this setting we assume high-precision heuristics that enable generating a relatively unbiased sample; but in many real-world cases such heuristics may not exist, or are expected to have limited coverage and would not enable sampling at will.

such heuristics cannot be assumed to yield a large training set, but may nevertheless be used for obtaining a small initial seed in an active learning setting.


However, unlike the above models, VVMAs focus on the low levels of execution: the VVMA is an architecture that speeds up matrix multiplications.

it is an efficient model that relates to hardware accelerators directly and it is universal, as matrix multiplication is the dominant computational factor for neural network inference.


While every neural network requires a certain budget of floating point operations for a target computation, how fast such computations are in practice depends not on the size of this budget but rather on the number of wall clocks needed in order to cover all floating point operations.

it is important to combine the software and the hardware advances in a co-design manner to optimize an efficient model for the correct metric: wall clocks.


As the Transformer is already getting noticeable impact in industrial settings, e.g., for machine translation and Web search, there is active research in developing more efficient Transformer architectures (Sanh et al., 2019; Kitaev et al., 2020; Beltagy et al., 2020;Zaheer et al., 2020).

with each new version of a Transformer architecture, new VVMA experiments would be needed in order to measure the potential improvements in efficiency that VVMA would yield.


As discussed in Section 2, most existing datasets were crowdsourced with a fixed set of prompts and no taboo constraints, which leads to limited diversity in the data.

models trained on the data may be brittle, failing when tested on new data in the same domain.


Following Shah et al. (2018); Rastogi et al. (2019), every grammar production in the simulator is paired with a template whose slots are synchronously expanded.

each dialog state or system act is associated with a template utterance.


One observation about the standard decoder is that it has to predict long strings with closing brackets to represent a tree structure in the linearization.

the total number of decoding LSTM recursions is the number of tree nodes plus the number of non-terminals.


While TF-IDF based document linking provides a co-occurence-based similarity measure between documents and conversations, there is no guarantee such linking will improve dialog modeling performance.

we aim to train a linking model such that conditioning on linked documents has a positive effect on dialog modeling performance.


The reason is that we cannot always rely on expensive human resources to annotate large-scale task-specific labeled data, especially considering the inestimable number of tasks to be explored.

a reasonable attempt is to map diverse NLP tasks into a common learning problem-solving this common problem equals to solving any downstream NLP tasks, even some tasks that are new or have insufficient annotations.


Although large-scale pre-trained language models like BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019b) have achieved super-human performances on these datasets, there have been concerns raised about these models exploiting idiosyncrasies in the data using tricks like pattern matching (McCoy et al., 2019).

various stress-testing datasets have been proposed that probe NLI models for simple lexical inferences (Glockner et al., 2018), quantifiers (Geiger et al., 2018), numerical reasoning, antonymy and negation (Naik et al., 2018).


They show that this instability largely arises from high inter-example similarity, as these datasets typically focus on a particular linguistic phenomenon by leveraging only a handful of patterns.

following their suggestion, we conduct an instability analysis of CONJNLI by training RoBERTa on MNLI with 10 different seeds (1 to 10) and find that the results on CONJNLI are quite robust to such variations.


Since BERT converts each word into word pieces, we first propagate the gold SRL tags, which are for each word, to word pieces.

for a word with multiple word pieces, we assign the tag B-ARG to the first word piece, and the tag I-ARG to the subsequent word pieces.


It makes words a special case of motif instances, and one can easily construct a similar bipartite graph for words.

in the rest of this section, we use motif instances to explain our ranking score design.


Intuitively, a highly label-indicative motif instance would not belong to the seed sets of multiple labels.

when any motif instance is expanded to seed sets of multiple classes, we stop the expansion of motif instances of the corresponding motif pattern.


We make the simplifying assumption that all changes in wikiHow's revision history are made for the better and therefore represent needed revisions to the original version of an article.

we treat all sentences that went through revision in wikiHowToImprove as requiring revision and all unrevised sentences from our extension (see 2.1) as requiring no revision.


State of the art research for date-time 1 entity extraction from text is task agnostic.

while the methods proposed in literature perform well for generic date-time extraction from texts, they don't fare as well on task specific date-time entity extraction where only a subset of the date-time entities present in the text are pertinent to solving the task.


“Let’s schedule for tomorrow. Next month, I plan on taking up Mr Baskerville’s case” Here, the model without Lt generates high attention weights for embeddings associated with “tomorrow”, since the localization of the attention weights is much more spread out.

it also uses the embeddings associated with "tomorrow" for predicting the label of "next month", and hence, predicts it to be relevant to scheduling when it is not.


Each resume is manually annotated to its most appropriate CRC position by experts through several rounds of triple annotation to establish guidelines.

a high Kappa score of 61% is achieved for interannotator agreement.


At any time, there are various positions posted for the same level from different divisions, cardiology, renal, infectious disease, etc.

it is common to see resumes from the same applicant applying to several job postings within the same CRC level.


To the best of our knowledge, no previous study has explicitly focused on this question.

the goal of this paper is to provide an answer to this question.


The representations for all mentions are then compared (usually sequentially) and mention pairs judged to be most similar are considered coreferent.

the mention representation is a key component in modern coreference resolution models.


The first signal is that the same name can appear multiple times in a text, and these mentions very likely corefer.

we can train mention representations to be similar for these mentions.


We do not want to use all mentions in the text since, for most of these, we don't have ground-truth clusters.

we only consider mentions that contain proper names that appear one or more times.


The decision of when to stop querying more external articles needs to be made after successive evaluations of the candidate answers.

the decision making process is inherently sequential.


Moreover, the samples show that the generation process is more sophisticated than just a trivial path flattening (i.e., merging text from all edge parts followed by minimal edits).

the proposed approach can eventually become a part of a more sophisticated system converting graphs to a coherent textual story and vice versa.


Entities, relationship are all sequences of tokens that need to be generated properly, which does not fit well in the conventional KB completion evaluation framework.

we define a new meaningful commonsense KB completion task for generative models, and present the challenges that arise from it.


Such automatic alignment between knowledge graph and texts provides distant supervision (Mintz et al., 2009) for pre-training but it is bound to be noisy.

we design a selection strategy and only retain plausible alignments with high semantic overlap.


Apparently, these pairs cannot serve our goal to build a knowledge-grounded language model.

we propose a data selection step to suppress the noise and filter out the data pairs of our interests.


We verify that there is zero RDF triple seen during pre-training though 31% entities are seen.

we can confirm the comparison with other baselines is still fair given no information from test/dev is leaked.


Unlike the previous two human-annotated datasets from different domains, WikiBio is also scraped from Wikipedia.

we filtered out the instances of KGTEXT from the first paragraph of the biography domain to ensure no overlap or leakage about Wikibio's dev/test set.


However, supervising the copy attention does not have much influence on the performance.

in the following experiments, we will run experiments for both encoding schemes with a copy mechanism without copy loss.


In a human-written document, subsequent text often refers back to entities and tokens present earlier in the preceding text.

it would increase coherence of text generated in downstream to incorporate the copy mechanism into pre-training on an unlabeled corpus.


For example, in Figure 1, one can observe that randomly sampling actions from the game vocabulary leads to several inadmissible ones like 'north a' or 'eat troll with egg'.

narrowing down the action space to admissible actions requires both syntactic and semantic knowledge, making it challenging for current systems.


The Jericho framework implements an admissible action handicap by enumerating all combinations of game verbs and objects at each state, and testing each action's admissibility by accessing the underlying simulator states and load-and-save functions.

the handicap runs no faster than a GPT-2 inference pass, and could in fact be unavailable for games outside Jericho.


In the MSCOCO validation set, 'man', 'elephant', and 'river' have more exposure, while 'traffic' and 'highway' are less mentioned.

the first group of references has a much higher consensus CIDEr score than the second group.


QA systems accurately answer simpler, related questions such as “What profession does H. L. Mencken have?” and “Who was Albert Camus?” (Petrochuk and Zettlemoyer, 2018).

a promising strategy to answer hard questions is divide-and-conquer: decompose a hard question into simpler sub-questions, answer the sub-questions with a QA system, and recompose the resulting answers into a final answer, as shown in Figure 1.


Second, we train a decomposition model on the mined data with unsupervised sequence-to-sequence learning, allowing ONUS to improve over pseudo-decompositions.

we are able to train a large transformer model to generate decompositions, surpassing the fluency of heuristic/extractive decompositions.


This format makes it difficult to determine stance in the typical topic-phrase (pro/con/neutral) setting with respect to a single topic, as opposed to a position statement (see Topic and ARC Stance columns respectively, Table 2).

we collect annotations on both topic and stance, using the ARC data as a starting point.


However, prior work used static word embeddings and we want to take advantage of contextual emebddings.

we embed a document and topic jointly using BERT (Devlin et al., 2019).


For both stance labels and models, performance increases when the majority sentiment polarity agrees with the stance label (M + for pro, M for con).

we investigate how susceptible both models are to changes in sentiment.


Given the limited amount of labeled data, we first want to augment our dataset by labeling the unlabeled data as well.

we have a two step approach: 1. Build a model to resolve ambiguous time terms (AM versus PM) and label the unlabeled data. 2. Train a model for time of day prediction by hour using the augmented dataset


These features do not exist in the unlabeled training set, to ensure the models learn to identify the proper AM/PM label without cheating.

we replace all the time phrases with the same special token.


While the dataset does not have the publication date of the book in the metadata, we were able to access the authors and the years the author was alive.

we created groupings of our data by time period based on the year of the author's birth.


However, since the semantic representation used by SCAN only covers a small subset of English grammar, SCAN does not enable testing various systematic linguistic abstractions that humans are known to make (e.g., verb argument structure alternation).

it is unclear whether progress on SCAN would generalize to natural language.


Our grammar does not generate VP-modifying PPs (the only PP verbal dependents are recipient to-phrases, which are always arguments rather than modifiers).

all PP modifiers in our dataset should strictly have an NP-attachment reading, although for human readers VP-attachment readings could sometimes be more prominent based on the lexical content of the sentences.


Note that as demonstrated by Reimers and Gurevych (2019), averaging context embeddings consistently outperforms the [CLS] embedding.

unless mentioned otherwise, we use average of context embeddings as BERT sentence embeddings and do not distinguish them in the rest of the paper.


Note that co-occurrence statistics is a typical tool to deal with “semantics” in a computational way — specifically, PMI is a common mathematical surrogate to approximate word-level semantic similarity (Levy and Goldberg, 2014; Ethayarajh et al., 2019).

roughly speaking, it is semantically meaningful to compute the dot product between a context embedding and a word embedding.


This is a common problem in the context of representation learining (Rezende and Viola, 2018; Li et al., 2019;Ghosh et al., 2020).

the resulted sentence embeddings can locate in the poorly-defined areas, and the induced similarity can be problematic.


we argue that the lexical proximity of BERT sentence embeddings is excessive, and can spoil their induced semantic similarity.


Concretely, the likelihood training pays little attention to the top ranks in terms of the target token probabilities (Welleck et al., 2020), or maximizing likelihood itself does not adequately reflect human language processing (Holtzman et al., 2019).

with the maximum likelihoodbased training, models learn to produce tokens frequently appearing in the data more often.


The more uniform a label distribution is, the less likely decision boundaries are biased in favor of frequent classes.

we aim to maximize the degree of uniformity of frequency distributions for both (i) tokens within each class and (ii) classes themselves (i.e., the sum of token frequencies within each class), to avoid the class imbalance problem (Buda et al., 2018) over the course of training.


Tokens in lyrics show a distribution largely different from general articles; for instance, repeated phrases are abundant in lyrics.

it provides an additional unique angle for model evaluations and comparisons.


We attribute this observation to the distinctive characteristics of lyrics, in which the same phrases are rhythmically repeated throughout the songs in the form of chorus or hook.

for lyrics dataset, forcing models to discourage reusing previously used tokens may adversely affect the likelihood of the generated texts.


The potential reason may be the lack of topics (i.e., keywords) in the model response, as illustrated in the graph that only contains context-topic nodes.

the graph reasoning module in our GRADE fails to induce an appropriate graph representation, which harms the coherence scoring.


Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers.

the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications.


From the perspective of data collection for studying a particular phenomenon, TORQUE has done more on defining the task and developing a scalable crowdsourcing pipeline.

tORQUE is also much larger than QA-tempEval and the annotation pipeline of tORQUE can be easily adopted to collect even more data.


Despite the progress on the encoder side, the current stateof-the-art models use a rather standard decoder: it functions as a language model, where each word is generated given only the previous words.

one limitation of such decoders is that they tend to produce fluent sentences that may not retain the meaning of input AMRs.


However, this dataset does not explicitly state the topics of individual poems.

we automatically predict a topic for each poem with the help of our topic prediction model, which will be described in Subsection 3.3.


Suprisingly, however, 12 out of 20 and 9 out of 20 poems are recognized correctly for know and unknown words, respectively.

our model works well even for topics it has not seen during training.


It is noteworthy that the text encoder part actually dominates the overall cost.

the gap between our model and the RGCN are further narrowed if we consider the cost of the entire model.


We compute the sum of the vector representations weighted by the probabilities, and then input it into the downstream model.

opTok becomes to assign a high probability to the tokenization which improves the performance of the downstream task.


The goal of this study is to improve the performance of downstream tasks by optimizing the tokenization.

we evaluate OpTok on various text classification tasks to validate its effect.


Moreover, many studies have reported that training models with a stochastic tokenization lead to a better performance of the downstream tasks than training a model using deterministic tokenization (Kudo, 2018;Hiraoka et al., 2019;Provilkov et al., 2019).

we trained the encoder and downstream model using subword regularization provided by SentencePiece.


It is still unclear whether the optimized tokenization leads to the improvement described in Section 3.3 because we trained all components simultaneously.

we investigate whether the optimized tokenization contributes to the improvement of the performance on the downstream task.


Sarcasm is prevalent in today's social media platforms, and it can completely flip the polarity of sentiment or opinion.

an effective sarcasm detector is beneficial to applications like sentiment analysis, opinion mining (Pang and Lee, 2007), and other tasks that require people's real sentiment.


Consider the given examples in Figure 1; people can not recognize sarcasm merely from text unless they find the contradiction between text and images.

capturing the incongruity between modalities is significant for multi-modal sarcasm detection.


Tay et al. (2018) argues that the words that contribute to the incongruity (usually accompany with a high attention value) should be highlighted.

a more discriminative pooling operator like max-pooling is desirable in our case.


Obviously, the methods based on text modality achieve better performance than the method based on image modality.

text information is more useful than image information for sarcasm detection.


Our model is designed to capture the incongruity information.

incongruous regions on the images are more likely to be attended by our model.


In addition, we find that our model might struggle in those instances requiring external knowledge, such as a speaker's facial gesture or contextual information.

external information is also essential for sarcasm detection.


Different users usually prefer different news information.

personalized news recommendation, which aims to display news articles to users based on their personal interest, is a useful technique to improve user experience and has been widely used in many online news services (Wu et al., 2019b).


Besides, these methods represent items using their IDs, and are difficult to handle new items since many news articles are posted every day which are all new items.

these federated learning based recommendation methods have their inherent drawbacks, and are not suitable for news recommendation.


According to the data processing inequality (McMahan et al., 2017), these gradients never contain more private information than the raw user behaviors, and usually contain much less information (McMahan et al., 2017).

the user privacy can be better protected compared with the centralized storage of user behavior data as did in existing news recommendation methods.


Moreover, different from these existing news recommendation methods which are all trained on centralized storage of user behavior data, in our FedNewsRec the user behavior data is stored on local user devices and is never uploaded.

our method can train accurate news recommendation model and meanwhile better protect user privacy.


Luckily, the gap between the performance of FedNewsRec and CenNewsRec is not very big.

our FedNewsRec method can achieve much better privacy protection at the cost of acceptable performance decline.


However, human ground-truth construction for summarization is time-consuming and laborintensive.

a more flexible summary generation framework could minimize manual labor and generate useful summaries more efficiently.


Each year, new shared tasks and datasets are proposed, ranging from classics like sentiment analysis to irony detection or emoji prediction.

it is unclear what the current state of the art is, as there is no standardized evaluation protocol, neither a strong set of baselines trained on such domainspecific data.


However, we acknowledge that other important tasks may need to be evaluated differently.

for future work we would like to include more tasks in the context of social media NLP research.


Although there are traditional non-neural parsers using n-grams as features to improve parsing (Sagae and Lavie, 2005;Pitler et al., 2010), they are limited in treating them euqally without learning their weights.

unimportant n-grams may deliver misleading information and lead to wrong predictions.


However, there are cases that long n-grams can play an important role in parsing when they carry useful context and boundary information.

we extend the span attention with a category mechanism (namely, categorical span attention) by grouping n-grams based on their lengths and weighting them within each category.


It is possible for the programmer to output illegal actions that do not follow the predefined action template (e.g., actions with missing a position component), especially when the programmer is not fully trained.

the interpreter checks if an action is valid and skips invalid actions by returning the input sequence.


We first experiment with N = 10, L = 5, and D = 10K, but all methods can reach a nearperfect sequence accuracy (see Figure 3).

we adjust N from 10 to 100 to make the task more challenging.


Note that online training is not part of the standard training procedure for End2end and Tagging, however, we use online training with End2end and Tagging for the sake of a fair comparison.

for End2end and Tagging, the online training acts like a data augmentation technique, providing more data points for training.


To estimate human performance within each metric, we treat each reference sentence in dev/test data as a "system prediction" to be compared with all other references, which is equivalent to compute inter-annotator agreement within each metric.

systems that have better generative ability than average crowd-workers should exceed this.


Moreover, considering the fact that summarization is not an easy task even for people, reliable human-labeled data are also difficult to obtain.

several unsupervised summarization approaches have been proposed, which do not require reference summaries for the target domain.


Taking advantage of corpora with billions of tokens, the pretrained language models learn universal and robust representations for various semantic structures and linguistic relationships.

pretrained models have been widely used with considerable success in applications such as question answering (Zhu et al., 2018), sentiment analysis (Peters et al., 2018) and passage reranking (Nogueira and Cho, 2019).


Gestures can be triggered at the sub-word level; for example, by a change of intonation in acoustics.

it is important to have sub-word level alignment between language and acoustics to generate the freeform gestures.


In these existing methods, their user models are trained in an end-to-end way using the labeled data of target task, which can only capture task-specific information.

in this paper we propose to pre-train user models from unlabeled user behavior data via self-supervision, which can exploit universal user information encoded in user behaviors.


We simply use the element-wise average operation for f .

if the segmentation of the word changes, the corresponding embedding and gradient vector will change accordingly.


However, n-best decoding is n-times time consuming compared to the standard decoding method.

we only use 1-best decoding which is the standard decoding framework for evaluating the translation quality.


Proposition 1 shows that the objective is maximized when the mutual information is maximized to I max .

maximizing the mutual information by other means can help side-step this issue.


In the biomedical domain, there exist several entities, such as genes, chemicals, and diseases, that are closely related to each other.

extracting the relationships among these entities is critical for biomedical research, particularly in fields such as construction of a knowledge base or drug development.


In other words, the output probability of the over-confident model does not indicate how uncertain the input example is, even if its classification performance is high.

several approaches, called "calibration" techniques, have been applied to several domains that require high reliability, such as autonomous driving and medical diagnosis (Guo et al., 2017;Jiang et al., 2012).


When a certain level of performance is guaranteed, unlabeled data predicted with a high probability in the classifier is likely to have a corresponding label.

such data can be used as pseudolabeled data, even if it contains slight noise.


Moreover, we applied self-training on our model to augment the training data and boost the performance, as our model is well-calibrated; it returned reliable output probabilities.

our model outperformed the other chemical-protein relationship extraction models and achieved state-of-the-art performance regarding the Biocreative VI ChemProt task.


As a meeting consists of utterances from different participants, it forms a natural multi-turn hierarchy.

the hierarchical structure carries out both token-level understanding within each turn and turn-level understanding across the whole meeting.


As the canonical transformer has the attention mechanism, its computational complexity is quadratic in the input length.

it struggles to handle very long sequences, e.g.  5,000 tokens.


This reconstructor quantifies the clinical correctness of the reports.

we can estimate the correctness of reports without rule-based annotators.


To train the language model, RL with only CRS and ROUGE as a reward is insufficient.

we use the cross-entropy loss to generate fluent sentences.


This evaluation is intended to assess whether our proposed method is also applicable to the MIMIC-CXR dataset or not.

in this evaluation, we focus only on the data-to-text module.


To the best of our knowledge, there are no publicly available entity typing datasets in the medical domain.

three entity typing datasets are constructed from the corresponding medical named entity recognition datasets.


Long tokens are relatively common in the medical domain, and these tokens will be split into short pieces when a domain-independent vocabulary is used, which will cause an overgeneralization of lexical features.

a medical vocabulary generated by the PubMed corpus can be introduced into BERT-MK in the following work.


The appropriate responses to the user queries are highly dependent on the visual information pertaining to the different aspects of the various images in the conversation.

it is natural to conclude that a conversational agent would be more effective if the visual information were part of its underlying conversational model.


We come up with the following two top-level principles for domain selection after closer review and extensive discussions: (i). it encompasses a broad group of task-oriented frameworks used by industries/service providers and is likely to build user interfaces; (ii). for deeper comprehension and clarification of the services, the domains need visual details.

we choose to curate conversations belonging to three distinct domains in our newly established large-scale MDMMD dataset1, namely restaurants, electronics, and furniture.


For example, the cuisine is the aspect category but Chinese is the aspect term that according to the user could change into Mexican, Japanese in the remaining utterances of a particular dialogue.

the labeling of both the aspect category and aspect term is essential for the generation of aspect guided responses to learn the subtle differences between the different aspect terms within the same category.


Conversely, the larger the L, the closer we are to sampling strategies.

by tuning L, it is possible to combine the advantages of both sampling and likelihood-based strategies.


This is also the basis on which we can continuously expand BERT's ER-Length and continue to benefit.

for a particular dataset, when we set the ERLength of the BERT, letting it exceed more data's DLength can always bring more improvements.


In the training phase, the current triplet prediction relies on the gold-standard labels of the previous triplets, while in the testing phase, the current triplet prediction relies on the model prediction of the previous triplets, which can be different from the gold-standard labels.

in the test phase, a skewed prediction will further deviate the predictions of the follow-up triplets; if the decoding length is large, the discrepancy from the gold-standard labels would be further accumulated.


For the sequence labeling tasks, because the data is relatively small and confined to a very specific domain (chat log from online apparel shops), we set a small vocabulary size of 500 for all the methods except NumAsTok and set the vocabulary size of NumAsTok to 550 to ensure that different methods have similar numbers of parameters for word embedding training.

our methods have (500 + |P|) × D parameters for word embedding training and Nu-mAsTok has 550 × D parameters, where P is the prototype set, whose size is typically smaller than 50, and D is the embedding dimension.


However, running these models on edge-devices, faces memory and latency issues due to limitations of the hardware.

there has been considerable interest towards research in reducing the memory footprint and faster inference speed for these models (Sainath et al., 2013;Acharya et al., 2019;Shi and Yu, 2018;Jegou et al., 2010;Chen et al., 2018;Winata et al., 2019).


In practice, state-of-the-art NLP models (Vaswani et al., 2017;Lioutas and Guo, 2020) have shown better performance with parameter sharing between the two (Press and Wolf, 2017).

there is a need for an exhaustive analysis of various embedding compression techniques, with parameter sharing.


Lastly, embedding compression models not based on linear SVD (Khrulkov et al., 2019;Shi and Yu, 2018) require the reconstruction of the entire embedding matrix or additional computations, when used at the output-layer.

during runtime, the model either uses the same amount of memory as the uncompressed model or pays a higher computation cost.


The consequence would be loss of information on words not seen during training and loss of generalization performance.

adding a loss for embedding reconstruction helps in grounding the embedding and not lose a lot of information.


Based on on (Chen et al., 2015b; Shao et al., 2017; Zhang et al., 2018), n-gram features are of great benefit to Chinese word segmentation and POS tagging tasks.

we use unigram and bigram embeddings for our models.


Each character can directly attend the criterion-token to be aware of the target criterion.

we can use a single model to produce different segmented results for different criteria.


In this work, we only adopt the vanilla Transformer encoder since we just want to utilize its selfattention mechanism to model the criterion-aware context representation for each character neatly.

it is promising for future work to look for the more effective adapted Transformer encoder for CWS task or to utilize the pre-trained models (Qiu et al., 2020), such as BERT-based MCCWS (Ke et al., 2020)


However as we show here - existing works are not optimized for dealing with pairs (or tuples) of texts.

they are either not scalable or demonstrate subpar performance.


For identity-hate, overlap with toxic is 1302/1405.

in this paper, we use the term toxic more generally, subsuming threat and identity-hate as particular types of toxic speech.


The pre-trained models are in principle already complete table entailment predictors.

it is interesting to look at their accuracy on the TABFACT evaluation set before fine-tuning them.


But in this sentence, a human is able to reason its label to LOCATION easily by recognizing the contextual phrase "went to" or "by car".

contextual information is crucial to improve model robustness in context level.


Here, replacing with similar words is more likely to produce valid natural language sentences and thus they should have higher probability during replacement.

instead of getting a random replacement word from the full vocabulary, a word similarity based replacement is proposed and applied here.


In addition, frequent substitution with extremely different words is not ideal either since it is not likely to produce reasonable sentences.

we want to make the probability distribution of sampling substitutes to focus more on the words which share some similarities with the original words but not far-away.


Though we keep the semantic latent variable, z s , and switch the gender latent variable, z g , to generate the gender-counterfactual word embedding, their concatenation during decoding can be vulnerable to the semantic information changes because of variances in the individual latent variables.

we constrain that the reconstructed word embedding with the counterfactual gender latent, w cf , differs only in the gender information from w n , which is the reconstructed word embedding with the original gender latent.


Arguments written in the TL provide a more realistic evaluation set than translated texts, specifically for tasks where labels are not well-preserved across automatic translation.

we created a new multilingual evaluation set by collecting arguments in all 5 languages (ES, FR, IT, DE, and NL) for all the 15 topics of the ArgsEN test set, using the Appen 5 crowdsourcing platform.


The results showed that many of the annotators labeled a vast majority (>80%) of the arguments as high-quality, even though they were instructed to consider only half as such.

only those labeling ≤ 80% of arguments as highquality were allowed further work.


Previous approaches typically apply only a single round of attention focusing on simple semantic information In our ADE detection task, instead, key elements of the sentence can be linked to multiple categories of task-specific semantic information of the named entities (ADE, Drug, Indication, Severity, Dose etc.).

single attention is insufficient in exploring this multi-aspect information and consequently risks losing important cues.


To reduce the word annotation burden, we are interested in understanding whether a word classifier trained on one domain can be applied in another.

we measure crossdomain accuracy, e.g., by fitting the word classifier on IMDB dataset and evaluating on Kindle dataset.


However, parsing with HRGs is not practical due to its complexity and large number of possible derivations per graph (Groschwitz et al., 2015).

work has looked at ways of constraining the space of possible derivations, usually in the form of align-1 See Gilroy (2019) for an extensive review of the issue.


So far, the generalization ability of current summarization systems when transferring to new datasets still remains unclear, which poses a significant challenge to design a reliable system in realistic scenarios.

in this work, we take a closer look at the effect of model architectures on cross-dataset generalization setting.


Despite recent impressive results on diverse summarization datasets, modern summarization systems mainly focus on extensive in-dataset architecture engineering while ignore the generalization ability which is indispensable when systems are required to process samples from new datasets or domains.

instead of evaluating the quality of summarization system solely based on one dataset, we introduce cross-dataset evaluation (a summarizer (e.g., L2L) trained on one dataset (e.g., CNNDM) will be evaluated on a range of other datasets (e.g., XSUM)).


When served as test set, such dataset brings great challenge for BERT match to correctly rank the can-didate summaries while it provides more training signals when served as training set.

the in-dataset (Bigpatent b) trained model obtain much higher score compared with cross-dataset models which trained from other datasets and cause lower stableness.


While the authors' analysis suggested the domains were similar enough to justify transfer attempts, only limited post-hoc analysis of the data platform effect was carried out.

it remains unclear to what extent the annotation methodologies as opposed to platform effects (or other confounds) caused the degradation.


Traditionally, different feature vocabularies account for domain transfer loss (Serra et al., 2017;Chen and Gomes, 2019;Stojanov et al., 2019).

we hypothesize that limited feature overlap and poor vocabulary alignment across datasets could hinder cross-domain generalization.


On the contrary, the recently proposed VizWiz dataset (Gurari et al., 2018) was collected from blind people taking photos and asking questions about those photos.

the images in VizWiz are often of poor quality, and questions are more conversational with some questions might even be unanswerable due to the poor quality of the images.


A natural way of verifying the instruction sets from Stage 1 is to have new workers follow them (Chen et al., 2019).

during Stage 2 Verification, a new worker is placed in the environment encountered by the Stage 1 worker and is provided with the NL instructions that were written by that Stage 1 worker.


As shown in Figure 9, the score of rPOD is decreased according to the placement error (the Manhattan distance) exponentially.

to score high in the rPOD metric, agents should place the target objects as close to the target place as possible.


Moreover, scoring all code snippets can be computationally inefficient in practice.

we use the method of Yang et al. (2019) to first uniformly sample a subset of data, whose size is much smaller than the entire training set size, and then perform adversarial sampling on this subset.


KERMIT (Chan et al., 2019) further simplified the Insertion Transformer model by removing the encoder and only having a decoder stack (Vaswani et al., 2017), by concatenating the original input and output sequence as one single sequence and optimizing over all possible factorizations.

KERMIT is able to model the joint p(x, y), conditionals p(x | y), p(y | x), as well as the marginals p(x), p(y).


Although promising results are obtained, existing models are limited in regarding extra features as gold references and directly concatenate them with word embeddings.

such features are not distinguished and separately treated when they are used in those NER models, where the noise in the extra features (e.g., inaccurate POS tagging results) may hurt model performance.


Therefore, such features are not distinguished and separately treated when they are used in those NER models, where the noise in the extra features (e.g., inaccurate POS tagging results) may hurt model performance.

it is still a challenge to find an appropriate way to incorporate external information into neural models for NER.


For example, as illustrated in Figure 2(c), for "Salt", its context features are "Salt" and "City" (the governor of "Salt"), and their corresponding dependency information are "Salt compound" and "City root".

for each type of syntactic inforamtion, we obtain a list of context features and a list of syntactic information instances, which are modeled by a KVMN module to enhance input text representation and thus improve model performance.


Second, on the contrary, SA is able to improve NER with integrating multiple types of syntactic information, where consistent improvements are observed among all datasets when more types of syntactic information are incorporated.

the best results are achieved by the model using all types of syntactic information.


Later, the syntax attention ensures that the constituent information should be emphasized and the gate mechanism also tends to use syntax for this input with higher weights.

this case clearly illustrates the contribution of each component in our attentive ensemble of syntactic information.


However, to enhance NER, it is straightforward to incorporate more knowledge to it than only modeling from contexts.

additional resources such as knowledge base (Kazama and Torisawa, 2008; Tkachenko and Simanovsky, 2012; Seyler et al., 2018; Liu et al., 2019b,a; Gui et al., 2019b,a) and syntactic information (McCallum, 2003; Mohit and Hwa, 2005; Finkel and Manning, 2009; Li et al., 2017; Luo et al., 2018; Cetoli et al., 2018; Jie and Lu, 2019) are applied in previous studies


Processing of legal contracts requires significant human resources due to the complexity of documents, the expertise required and the consequences at stake.

a lot of effort has been made to automate such tasks in order to limit processing costs-notice that law was one of the first areas where electronic information retrieval systems were adopted (Maxwell and Schafer, 2008).


One important advantage over static embeddings is the fact that every occurrence of the same word is assigned a different embedding vector based on the context in which the word is used.

it is much easier to address issues arising from pre-trained static embeddings (e.g., taking into consideration polysemy of words).


Our aim is for explanations to better communicate the task model's reasoning process, without adopting the trivial solution, i.e., directly stating its output.

while we optimize explanations for simulatability, we also penalize label leakage, which we formalize below.


We also try to incorporate the head information in constituent syntactic training process, namely max-margin loss for both two scores, but it makes the training process become more complex and unstable.

we employ a parameter to balance two different scores in joint decoder which is easily implemented with better performance.


Overall, joint semantic and constituent syntactic parsing achieve relatively better SRL results than the other settings.

the rest of the experiments are done with multi-task learning of semantics and constituent syntactic parsing (wo/dep).


Besides, LIMIT-BERT takes a semisupervised learning strategy to offer the same large amount of linguistics task data as that for the language model training.

lIMIT-BERT not only improves linguistics tasks performance, but also benefits from a regularization effect and linguistics information that leads to more general representations to help adapt to new tasks and domains.


(2) Naturally empowered by linguistic clues from joint learning, pre-trained language models will be more powerful for enhancing downstream tasks.

we propose Linguistics Informed Multi-Task BERT (LIMIT-BERT), making an attempt to incorporate linguistic knowledge into pre-training language representation models.


BERT is typically trained on quite large unlabeled text datasets, BooksCorpus and English Wikipedia, which have 13GB plain text, while the datasets for specific linguistics tasks are less than 100MB.

we employ semi-supervised learning to alleviate such data unbalance on multi-task learning by using a pre-trained linguistics model to label BooksCorpus and English Wikipedia data.


To absorb both strengths of span and dependency structure, we apply both span (constituent) and dependency representations of semantic role labeling and syntactic parsing.

it is a natural idea to study the relationship between constituent and dependency structures, and the joint learning of constituent and dependency syntactic parsing (Klein and Manning, 2004;Charniak and Johnson, 2005;Farkas et al., 2011;Green andÃ…Â½abokrtskÃƒÂ½, 2012;Ren et al., 2013;Xu et al., 2014;Yoshikawa et al., 2017).


Ideally, we expect that the representation vectors in the deep learning models for ABSA should mainly involve the related information for the aspect terms, the most important words in the sentences.

in this work, we propose to regulate the hidden vectors of the graph-based models for ABSA using the information from the aspect terms, thereby filtering the irrelevant information for the terms and customizing the representation vectors for ABSA.


In this work, we hypothesize that these overall importance scores from the dependency trees might also provide useful knowledge to improve the representation vectors of the graph-based models for ABSA.

we propose to inject the knowledge from these syntaxbased importance scores into the graph-based models for ABSA via the consistency with the modelbased importance scores.


Solving VLQA examples requires linking information from image and text.

vLQA can be considered a novel kind of multi-hop task involving images and text, which we believe will drive future vision-language research.


Now we want to determine where is A' located in the image I.

we formulate a new question Q' as "Where is A'?


Encoding hierarchical information from large type inventories has been proven critical to improve performance  (Lopez et al., 2019).

we hypothesize that our proposed hyperbolic model will benefit from this representation.


Because the OneCommon game framework rewards players if they successfully create common ground with each other, players may think to mention to more salient dots to increase the success rate.

the variation of expressions could be restricted.


However, our architecture is difficult to directly apply to referring expression generation because it outputs modulated feature maps.

the future direction is to extend our architecture to language generation.


For clusters with only negative edges like the triad in Figure 1c, even though the relation is imbalanced according to the definition, we are unable to determine whether there should be a pair of paraphrases in the graph without knowing the actual semantic meaning of the sentences.

we use the weaker form of structural balance to represent graphs with all negative edges.


The NELL-995 does not come with a validation set, and therefore we selected 3000 edges randomly from the full NELL KB.

many of the query relations were different from what was present in the splits of NELL-995 and hence is not a good representative.


The other kind (Zheng et al., 2019) is to generate and classify candidate regions in a two-stage paradigm, often leading to cascaded errors.

region based methods face efficiency and effectiveness challenges.


Moreover, the Transformer model is inherently much slower than conventional machine translation approaches (e.g., statistical approaches) mainly due to the auto-regressive inference scheme (Graves, 2013) incrementally generating each token.

deploying the Transformer model to mobile devices with limited resources involves numerous practical implementation issues.


This passive and relatively simple dialogue mechanism gains less attention from humans and consumes the interests of human beings rapidly.

some recent researches attempt to endow the bots with proactivity through external knowledge to transform the role from a listener to a speaker with a hypothesis that the speaker expresses more just like a knowledge disseminator.


Two humans take the topic leading role like a speaker to introduce something new in turns.

in human-machine conversation, the dialogue agent side needs to act as a speaker timely and appropriately.


When one leads the dialog, the other takes a backseat and forgets more.

we predict the role with this forget gate and generate a response not only on the default decoder state but also on the predicted role simultaneously.


While the listener requires more forgetting to decrease the influence of the knowledge input.

it seems the forget gate here is just a hidden variable which represents the role.


The Initiative-Imitate recognizes the role of the human and controls the knowledge utilizing in the next sentence generation.


Differently from Chirps, this model makes its event clustering decision based on the predicate, arguments, and the context of the full tweet, as opposed to considering the arguments alone.

we expect it not to cluster predicates whose arguments match lexically, if their contexts or predicates don't match (first example in Table 3).


Following Shwartz et al. 2017, we annotated the templates while presenting 3 argument instantiations from their original tweets.

we only included in the final data predicate pairs with at least 3 supporting pairs.


However, the extent of annotation and the utility of domain adaptation for training are unknown.

our main question is how successfully can a semantic parser learn with alternative data resources to generalize to novel queries in a new language?


In initial experiments, we found negligible difference in MT-Paraphrase using random sampling or roundrobin selection of each paraphrase.

we assume that both methods use all available paraphrases over training.


The model trained on MT achieves nearly the same generalization error as the model trained on the gold standard.

we consider the feasibility of our approach justified by this result.


As previously discussed, we translate only the development and test set of Overnight (Wang et al., 2015) into Chinese and German for assessment of crosslingual semantic parsing in a multi-domain setting.

we translate all 5,473 utterances in ATIS and 4,311 utterances in Overnight.


It is impossible for a NER system to cover all entity types (Ling and Weld, 2012;Mai et al., 2018).

in the industrial area, it often happens that some entity types required to recognize by the clients are not defined in the previously designed NER system.


Note that, there is no labeled data for class K of the source task.

a fully supervised learning algorithm is not applicable to train the classifier.


The difference between the compared work and our work is that, in the compared work, the mention recognition for one entity type is performed independently to the other types through a binary classifier.

it has to resolve the conflict between the recognition results of different binary classifiers for different entity types using a heuristic method at the inference time.


This is because the occurring frequency of mentions of the location type is much lower than the occurring frequency of mentions of the GPE type.

it requires to annotate more data for the location type to cover enough mentions of the type.


For ranking, the setup is different, as it is not feasible to encode all the candidate documents (from firststage retrieval) into a single input template.

ranking necessitates multiple inference passes with the model and somehow aggregating the outputs.


We map a relevant document to "hot" and a non-relevant document to a completely unrelated word "orange".

we force the model to build an arbitrary semantic mapping.


Our propose model OTE-MTL consistently outperforms all state-of-the-art baselines on all datasets with and without OOTs.

we conclude OTE-MTL is effective in dealing with opinion triplet extraction task.


Previous work has found that if the ratio of biased examples is high, down-weighting, or disregarding all of them results in an insufficient training signal, which leads to performance decreases (Clark et al., 2019;Utama et al., 2020).

we propose a novel multi-bias weighting function that weights each example according to multiple biases and based on each bias' strength in the training domain.


To apply our framework to training sets that may contain multiple biases of different strengths, we automatically weight the output of the bias models according to the strength of each bias in each training dataset.

we propose a scaling factor F S (B k , D t j ) to automatically control the impact of bias B k in dataset D t j in our debiasing framework, i.e., to reduce the impact of bias on the loss function when the bias is commonly observed in the dataset.


We observe that (1) different datasets are more affected by certain biases, e.g., the ratio of examples that can be answered without the question (the empty question bias) is 8% in SQuAD while it is 38% in NQ, (2) NewsQA is least affected by biases overall while NQ and HotpotQA are most affected, (3) only few instances are affected by all four biases, and (4) except for NewsQA, the majority of training examples are affected by at least one bias.

methods that down-weight or ignore all biased examples will considerably weaken the overall training signal.


Mahabadi et al. (2020) propose two different methods among which the Debiased Focal Loss (DFL) approach has a better performance.


To collect a large corpus of parallel data, heuristic rules are often used but they inevitably let noise into the data, such as phrases in the output which cannot be explained by the input.

models pick up on the noise and may hallucinategenerate fluent but unsupported text.