Deep generative models (DGMs) are probabilistic latent variable models parameterised by neural networks (NNs).

dGMs optimised with amortised variational inference and reparameterised gradient estimates (Kingma and Welling, 2014; Rezende et al., 2014), better known as variational auto-encoders (VAEs), have spurred much interest in various domains, including computer vision and natural language processing (NLP).


The overarching objective of our work is to access the large body of domain-specific information available in Frequently Asked Question sites (FAQ for short) via conversational Question Answering (QA) systems.

we want to know whether current techniques are able to work with limited training data, and without needing to gather data for each target FAQ domain.


More specifically, if a user has an information need and asks a question to a conversational QA system on a FAQ, the system can search for similar questions which have already been answered, or the system can directly search in existing answer passages.

there are two ways to check automatically if the forum contains a relevant answer passage to a new question: (1) question retrieval, where relevant or similar questions are searched (and thus, the answer for this relevant question is taken as a relevant answer), and (2) answer retrieval, where relevant answers are searched directly among existing answers.


Conversational QA systems are more complex, as they need to deal with a sequence of possibly inter-dependent questions.

the meaning of the current question may depend on the dialogue history.


In contrast, in our studied problem MCQA, data properties such as answer, question type, and commonsense are greatly vary across the MCQA datasets.

the passages and questions come from different scenarios (such as exams, dialogues, and stories), and the answering choice contains more complex semantic information than the fixed categories in Mini-ImageNet.


Due to the data heterogeneity of different sources, the performance of meta learning may drop if we consider some undesirable data sources in training.

these undesirable or called "dis-similar" data sources will cause negative transfer when their distribution is far away from the target one.


First, while state-of-the-art models for structure prediction in NLP used to rely heavily on intricate formal structures and carefully designed features (or feature-templates) (Zhang and Nivre, 2011; Zhang and Clark, 2011a), current neural models provide a form of representation learning and may be viewed as automatic feature-extractors (Kiperwasser and Goldberg, 2016; Dozat and Manning, 2018).

as long as the input object can be represented as a vector, the neural model will learn how to map it to the appropriate set of structural decisions, without having to write features or feature-templates by hand.


Subsequent SPMRL events and shared tasks (Seddah et al., 2013b; Tsarfaty, 2013; Seddah et al., 2014b) illustrated how methodologies and modeling assumptions for English NLP often break down in the face of such typologically diversity.

while most NLP models can in principle be trained on data in any given language, 1 such models are often developed with English in mind, and the bias injected into such models is not optimal for languages that exhibit flexible word order, and rich word-internal structure, as is the case in MRLs.


It is an open question whether these word-pieces capture relevant aspects of morphology.

it is unclear that the strategy of relying on chars or char-strings is adequate for encoding nonconcatenative phenomena that go beyond simple character sequencing, such as templatic morphology, substraction, reduplication, and more (Ackerman and Malouf, 2006; Blevins, 2016).


As a complementary area of investigation, a plausible direction would be to shift the focus from the decomposition of words into morphemes, to the organization of words as complete paradigms.

instead of relying on sub-word units, identify sets of words organized into morphological paradigms (Blevins, 2016).


Hebrew is a Semitic language which lies high on both the synthesis and fusion typological indices, and thus provides an interesting case study.

we devised a multi-tagging task where each raw input token is tagged with the sequence of Part-of-Speech tags that represent the functions of its constituent morphemes.


The performance on end-to-end binary relations shows the utility of incorporating a document level model for crosssection relations, rather than predicting on individual sections.

we observe a large difference in recall, which agrees with the fact that 55% of binary relation occur across sentence level.


In this setting, we run our end-to-end model but with "gold cluster saliency" information.

we predict clusters of mentions using our model (mention identification, pairwise coreference, and mention clustering).


Similarly, with just "oracle columns", the accuracy is 69.8%, which means that 81.0% of the questions that RAT-SQL gets wrong have incorrect structure.

most questions have both column and structure wrong, so both problems require important future work.


To further bias the soft alignment towards the real discrete structures, we add an auxiliary loss to encourage sparsity of the alignment matrix.

for a column/table that is mentioned in the SQL query, we treat the model's current belief of the best alignment as the ground truth.


We tailor-design several sets of indicating symbols to enhance the performance in terms of accuracy on format, rhyme, and sentence integrity.

symbols C = {c i } are introduced for format and rhyming modeling; Intra-position symbols P = {p i } are designed to represent the local positions of the tokens within each sentence aiming to improve the rhyming performance and the sentence integrity.


CMR also achieves the best overall accuracy.

we can see a 2.3% improvement over Visual-BERT (Li et al., 2019b), as in the above mentioned NLVR 2 results.


In this paper, we propose a novel cross-modality relevance (CMR) for language and vision reasoning.

we argue for the significance of relevance between the components of the two modalities for reasoning, which includes entity relevance and relational relevance.


However, this would require a careful examination of the impact of correction on the distributions and interpretations of the scores.

choi and cho (2018) found that manually-vetted correction of spelling errors yielded a significant increase in scores assigned to the essays by trained raters, and that, even after controlling for the error quantity and quality predictors, the magnitude of the average gain in the score was smaller for responses with higher original scores.


The field of AWE has thus progressed according to the trajectory charted by Page to a large extent, though not completely.

while Page imagined the main use case of AWE to be in the service of a harried English teacher and his feedback-thirsty students, in reality, the most visible use case has arguably evolved to be automated scoring of essays for standardized testing, which, in turn, has led to new challenges, such as ensuring the validity and fairness of scores.


In this study, we investigate the importance of each micro-cluster to handle the cluster evolution problem.

the importance of each micro-cluster is decreased over time if it is not updated.


Our approach starts from the content of a passage, which we define as the information it expresses, implies, or relies on.

we propose that task designers lay out a minimal body of content that MRC systems should demonstrate they understand.


First, cognitive science suggests that humans make more inferences when reading narrative text than expository text (Graesser et al., 1994).

a story entails a highly structured network of relations (timelines, causality, etc.).


We do not have a separate category for "who did what to whom" information, but we expect strong performance on the ToU to hinge on such analysis.

much of this information is captured in the characterization of events for temporal questions.


Even with guidelines, different annotators may give substantively different answers.

they may drill down to different levels of detail in a causal chain before bottoming out in general knowledge-e.g., rather than stopping at dogs disliking rain, one annotator might explain that Rover disprefers rain because he dislikes getting wet, which in turn is because dogs often dislike getting wet.


In order to overcome these issues, we propose a novel deep learning model for RE that uses the dependency trees to extract the syntax-based importance scores for the words, serving as a tree representation to introduce syntactic information into the models with greater generalization.

we leverage Ordered-Neuron Long-Short Term Memory Networks (ON-LSTM) to infer the model-based importance scores for RE for every word in the sentences that are then regulated to be consistent with the syntax-based scores to enable syntactic information injection.


A notable insight in these recent studies is that the syntactic trees of the input sentences (i.e., the dependency trees) can provide effective information for the deep learning models, leading to the stateof-the-art performance for RE recently (Xu et al., 2015; Tran et al., 2019).

the previous deep learning models for RE has mostly exploited the syntactic trees to structure the network architectures according to the word connections presented in the trees (e.g., performing Graph Convolutional Neural Networks (GCN) over the dependency trees ).


Although the dependency connections might be less specific to the training data than the whole tree structures, the major limitation of the edge-based representation is that it only captures the pairwise (local) connections between the words and completely ignores the overall (global) importance of the words in the sentences for the RE problem.

some words in a given sentence might involve more useful information for relation prediction in RE than the other words, and the dependency tree for this sentence can help to better identify those important words and assign higher importance scores for them (e.g., choosing the words along the shortest dependency paths between the two entity mentions).


We fist compare the proposed model (called CEON-LSTM) with the baselines on the popular ACE 2005 dataset.

the four following groups of RE models in the prior work on RE with the ACE 2005 dataset is chosen for comparison: (i) Feature based models: These models handdesign linguistic features for RE, i.e., FCM, Hybrid FCM, LRFCM, and SVM (Yu et al., 2015;Hendrickx et al., 2010).


Among all the models, we see that the proposed model CEON-LSTM is significantly better than all the baseline models over different test domains/datasets.

cEON-LSTM is 1.38% and 3.1% better than DRPc and SAcNN (respectively) on the average F1 scores over different test datasets.


To illustrate the benefit of the importance score representation for SMC, EP-ON-LSTM replaces the importance score representation for the dependency trees in CEON-LSTM with the dependency edge representation in DRPC.

we replace the term L import in the overall loss function (i.e., Equation 6) with the dependency edge prediction loss (using the ON-LSTM hidden vectors) in DRPC for EP-ON-LSTM.


In this paper, we aim at adapting monolingual models to code-switched text in various tasks.

we transfer English knowledge from a pre-trained ELMo model to different code-switched language pairs (i.e., Nepali-English, Spanish-English, and Hindi-English) using the task of language identification.


In the second part, we demonstrate the effectiveness of our CS-ELMo models by further fine-tuning them on tasks such as NER and POS tagging.

we show that the resulting models significantly outperform multilingual BERT and their homologous ELMo models directly trained for NER and POS tagging.


For this feature, the element corresponding to the entity type of the highest scoring candidate c 1 is updated with the score of the candidate.


To obtain multi-gram class names, we design a modified beam search algorithm to iteratively query a pretrained LM.

after we query a LM for the first time and retrieve top K most likely words (for the masked token), we construct K new queries by adding each retrieved word after the masked token.


Recent studies (Peters et al., 2018;Devlin et al., 2019;Yang et al., 2019) found that language models, simply trained for next word or missing word prediction, can generate high quality contextualized word representations which benefit many downstream applications.

these language models will output an embedding vector for each word appearance in a specific context that is usually the entire sentence where the target word occurs, rather than just words appearing before the target word.


This is similar in spirit to maintaining a pool of visual attention maps (Seo et al., 2017), where we argue that different questions in the conversation attend to different parts of the image.

we pass the history to attend to object-level image features using the MCA module to get visually grounded contextual history.


This is a subset of the VisDial val-set consisting of 97 dialogs, where the crowd-workers identified single turns (with dense annotations) requiring historical information.

we asked crowd-workers whether they could provide an answer to a question given an image, without showing them the dialog history, and select one of the categories in Table 4 (see further details in Appendix B).


We also tried to analyze the top-20 answers (Figure 11) which had non-zero relevance in the dense annotations.

we took all 2k example turns of training set with dense annotations for each of 100 options.


 To do this, we use the concept of
must-link (ML; if two mentions are judged coreferent) and cannot-link (CL; if two mentions are
judged non-coreferent) relations between mentions
introduced by Sachan et al. (2015), and adapt it for
our purposes

in our discrete setting, we build the links as follows: if the user deems the pair coreferent, it is added to ML.


To do so, we first train the model for 10 epochs using regular MultiDDS, then switch to a different dev set aggregation method.

we compare MultiDDS with three different priorities: Regular: this is the standard MultiDDS that optimizes all languages throughout training using the average dev risk aggregation in Eq.


While InferSent rarely predicts that presuppositions project, we find strong evidence that the BERT and BOW models do.

they correctly identify that the premise entails the presupposition (acc. ≥ 80% for BERT, acc. ≥ 90% for BOW).


Similar to Camburu et al. (2018), for all our experiments, we filter out non-informative examples where the explanations contain the entire text of the premise or hypothesis.

we drop any training example where the uncased premise or hypothesis text appears entirely in the uncased explanation.


We use the GPT-2 architecture (Radford et al., 2019), which is trained using a causal language modeling loss (CLM), and includes a leftto-right decoder suitable for text generation.


RoBERTa: For classification modules, we leverage RoBERTa , which is trained using a masked language modeling loss (MLM).


While the gains of higher-order features are small in the presence of a powerful encoder, they are consistent for long-range dependencies and long sentences.

higher-order models are more accurate on full sentence parses and on the exact match of modifier lists, indicating that they deal better with larger, more complex structures.


In this paper, we test rigorously the hypothesis of the utility of second-order features.

we experiment with consecutive sibling and grandparent features in a non-projective, graphbased dependency parser.


In this paper, we propose a novel spelling check convolutional graph network (SpellGCN) that cap-tures the pronunciation/shape similarity and explore the prior dependencies between characters.

two similarity graphs are constructed for the pronunciation and shape relationship correspondingly.


To address this issue, we propose TLUA model, utilizing an attention scheme to automatically weight different LUs for the frame, according to target word T in the given sentence, shown in Figure 1.

more specifically, we compute the weighted sum of target word T's representation and other LUs' representations based on their importance wrt T. we emphasize T as it occurs in the given sentence, which can reduce the potential noise introduced by irrelevant LUs in the same frame.


Note attention schemes have been designed for both intra-frame and inter-frames.

intra-frame attention focuses on relevant LUs, while inter-frames attention emphasizes relevant frames, avoiding the influence from less relevant but linked frames.


Meanwhile, its score can be regarded as an indicator of coherence during inference, helping to select the most coherent one from multiple candidate responses.

the training of response selection is carried out together with the bi-directional encoding of latent act recognition.


In this paper, we extend the idea of attentive pooling network to the proposed three-way attentive pooling network for the follow-up question identification task, where the model needs to capture the suitability of a candidate follow-up question by comparing with the conversation history and the associated passage.

the proposed model aims to capture topic shift and topic continuation in the follow-up question.


The rise of deep learning based methods in NLP has stimulated research on the interpretability of the neural models.

several recent studies analyze representations generated by neural models to get insights on what kind of linguistic information is learned by the models.


Token-level tasks based on running text—especially given that treebanks are often based on a homogeneous document collection—are inevitably biased to the domain of this text, whereas type-level probing tasks are expected to be domain-neutral.

dictionary-based tasks do not contain any frequency information of the surface forms, whereas token-level tasks do.


For fusional languages the ambiguity ratios are higher, mostly due to syncretism-one word form might encode several morphological feature bundles.

for German, the average is around 26% (before removal of ambiguous entries), which means loss of considerable amount of data points.


This effect can be easily observed in the performance gaps between D-ELMo and C-ELMo.

when a certain feature gets a performance boost by C-ELMo, this may suggest that the feature is highly ambiguous and a model that can use contextual information to resolve ambiguity outperforms the one that can't by a large margin.


Similarly, a predicate tagged with future tense may be more likely to have a goal argument.

these features are not mutually exclusive, and cannot be mutually exclusive.


If linearization were the only operation needed, it could be performed easily by evaluating expressions by usual functional programming techniques.

tables would rarely need to be expanded to their full forms, because each use of them would need only one form, which could be found by lazy evaluation without expanding the entire table.


The best performing versions of BERT-CHAIN and BERT-GRC significantly outperforms the baselines.

the AUC-ROC is 11% higher (absolute), NDCG rises from 0.60 to 0.64, and P@1 rises from 0.47 to 0.54 for BERT-GRC, indicating substantial improvement.


This zero-shot dataset differs from typical reading comprehension datasets, however, in that each task description is paired with twenty different passages, and we evaluate a model's ability to solve the task, not just give the correct answer for a single (question, passage) pair.

given a question, a model produces some decision function f , and it is this function which we comprehensively evaluate on many different inputs.


We followed the original hyperparameters recommended by the authors for fine tuning for summarization 11 with the exception of tuning the batch size, learning rate, number of training epochs, and sequence lengths.

we used a batch size of 32, maximum source/target sequence lengths of 512/64, four beams for decoding, weight decay of 0.01 and 0.1 label smoothing.


To compute Similarity fi, we use an idea similar to attention mechanism by (Bahdanau et al., 2015).

we use cosine similarity between the representations of the attribution factor f and representations of wi from the LM.


Moreover, to address the problem caused by the long tail distribution of entities in logical forms, Jia and Liang (2016) proposed an attention-based copying mechanism.

at each time step, the decoder takes one of two types of actions, one to predict a word from the vocabulary of logical forms and the other to copy a word from the input utterance.


We compare the performance of models trained with different sets of rules.

wong and Mooney (2006) and wong and Mooney (2007) have induced a set of grammar rules for Prolog and FunQL in Geo.


Among all the MRs, the performance of Prolog declines more seriously than the others in both domains.

it suffers from the program alias problem more seriously.


Second, to reduce program alias' negative effect on neural semantic parsing, developers should define a concrete protocol for annotating logical forms to ensure their consistency.

given an MR, developers should identify as many as possible sources where program alias can occur.


We recruit participants from an educational institution and ensure they are proficient in both languages of interest.

participants are either bilingual speakers or graduate students pursuing a Translation Studies degree.


Dataset Statistics Sentence-level annotations were aggregated by majority vote, yielding 252, 418, and 369 instances for the "Unrelated", "Some meaning difference", and "No meaning difference" classes, respectively.

64% of samples are divergent and 40% of samples contain finegrained meaning divergences, confirming that divergences vary in granularity and are too frequent to be ignored even in a corpus viewed as parallel.


While previous work learning from bilingual data perhaps takes advantage of this fact implicitly, the focus of this paper is modelling this intuition explicitly, and to the best of our knowledge, this has not been explored in prior work.

we propose a deep generative model that is encouraged to perform source separation on parallel sentences, isolating what they have in common in a latent semantic embedding and explaining what is left over with language-specific latent vectors.


We focus on a style-transfer task where we have original seed sentences from which we calculate our semantic vector z sem and language specific vector z en .

we feed in a Source sentence into the semantic encoder to obtain z sem , and another Style sentence into the English language-specific encoder to obtain z en .


As illustrated in Figure 2, understanding the table requires both horizontal and vertical views.

if the table is flattened by a horizontal scan, the vertical alignment information will be lost, and vise versa.


We explore to enhance the performance of counting verification by converting the counting problem into a semantic matching problem.

for every column, the frequency of duplicate cell contents is counted as a summary cell, leading to a summary row which is then appended to the table.


To avoid pairing all the entities from two KGs and control the size of the PCG, our approach selects entity-pairs having high equivalent possibilities as nodes in the PCG.

locality-Sensitive Hashing (lSH) is employed in our approach to efficiently find similar entities between two KGs.


To obtain feature representations of entity-pairs containing their neighbors' information, our approach propagates attribute features of entity-pairs following these edges.

our approach uses a Graph Neural Network (GNN) to propagate the attribute features of entity-pairs over the PCG.


Extensive experiments demonstrate that RL-MMR achieves state-of-the-art performance on benchmark MDS datasets.

we show the benefits of incorporating MMR into end-to-end learning when adapting SDS to MDS in terms of both learning effectiveness and efficiency.


Soft attention restrains the search space while allowing more exploration of the limited labeled data, leading to better representation learning.

rL-MMr infuses the entire prediction of MMr into its neural module by attending (restraining) to important sentences and downplaying the rest instead of completely discarding them.


Unlike prior studies on adapting SDS to MDS (Lebanoff et al., 2018), which concatenates all the documents chronologically and encodes them sequentially, we adapt hierarchical encoding for better efficiency and scalability.

we first encode each sentence s j via a CNN (Kim, 2014) to obtain its sentence representation.


The probability of neural sentence extraction is measured through a two-hop attention mechanism.

we first obtain the neural summary representation z t by feeding previously extracted sentences (A e i ) to an LSTM encoder.


The guidance of MMR is incorporated into neural representation learning through end-to-end RL training.

we formulate extractive MDS as a Markov Decision Process, where the state is defined by (D \ E t , g t ).


We study the balance between salience and redundancy, and the performance of different similarity measures.

we use TF-IDF and BERT (Devlin et al., 2019) as the sentence (document) representation and measure cosine similarity in S(sj , D) and R(sj , e)


We note that there are plenty of inconsistencies in the previous work on MDS and some results cannot be directly compared with ours.

there are three major differences that may lead to incomparable results as follows.


The KPE task is a foundational task, which plays a facilitating role in many Information Retrieval (IR) tasks, including classification, summarization, and document indexing (Hasan and Ng, 2014).

the KPE task requires accurate selection of the phrases that best capture the web document's topic.


The overall architecture of our proposed model is shown in Figure 2, and the fusion forget gate inside is illustrated in Figure 3.

multistage fusion consists of the cross fusion block and hierarchical fusion decoder, which aims to model the correlation and complementarity between modalities spontaneously.


Although the CFG builds an unsupervised lowlevel signal alignment between original multisource features, noise modality information generated by CFG is hard to be suppressed.

when the whole modality cannot guide the task at all, the forced normalization of the softmax function in the attention structure makes the calculated fusion vector generated by the noise modality hard to be suppressed.


It can also be seen that the performances of the pure video modality models are modest because of the frozen video features extracted from a task-independent pretraining model.

table 3 shows that when the performances of all the prior models trained with ASR output transcripts drop sharply due to the higher error rate (W ER = 32.9%) of speech recognition, our model still has good performance close to the models trained with ground-truth transcripts.


The observation results are similar to those observed in Table 4.

we can see a greater increase in the performance of the FFG when using high noise ASR-output trancript compared to using the ground-truth transcript.


We also conduct comparisons with BERT to validate the effectiveness of our proposed model.

we use the bert-base-uncased model (due to GPU memory limit) from huggingface library 3 as our encoder to encode the dialogue history and the remaining parts are the same as our model.


Based on the question “Where is a nearby parking garage?”, the generated response of our model is “palo alto garage is 1 miles away”.

the attention results at each generation timestep for the knowledge graph information of this example are shown in (a), (b), (c) and (d) respectively.


Our results also highlight the importance of explicit controlled sparsity inducing terms as effective inductive biases for improved task performance and rationale agreement.

sparsity inducing methods consistently outperform the No Sparsity-baseline (row 3).


We benchmark these datasets using multi-modal language modeling architectures (Tan and Bansal, 2019;Su et al., 2020) which achieve state-of-art performance on multiple vision-language tasks.

we don't use any pre-training, instead train randomly initialized models on streaming data using continual learning algorithms (Robins, 1995;Rolnick et al., 2019;Aljundi et al., 2019a) and evaluate their resistance to forgetting and compositional generalization.


Figure 1 shows that masking achieves decent performance without hyperparameter search.

(i) a large initial sparsity removing most pretrained parameters, e.g., 95%, leads to bad performance for the four tasks.


We further propose data rejuvenation to rejuvenate the inactive examples to improve the performance of NMT models.

we train an NMT model on the active examples as the rejuvenation model to re-label the inactive examples, resulting in the rejuvenated examples (§3.2).


We follow Domhan (2018) to implement LSTM by replacing the self-attention (SAN) layers in TRANSFORMER-BASE with LSTM layers.

we use a bidirectional LSTM for each layer of the encoder, and a unidirectional LSTM for each layer of decoder.


Using these specialized losses, we aim to improve the learning power of the MT model.

in this work, we target the improvement of pronoun translation by focusing our finetuning efforts through our proposed objectives and also through the fine-tuning data.


In accordance with our settings to alternate training between the upsampled full dataset and the subset data (2D + D prn ), we also alternate the additional loss such that it is only applied to the targeted subset.

in every alternate epoch, the model is trained on the upsampled full dataset (2D) with the standard CLM translation loss L g (Eq.


We differ from previous studies and use a tree edit distance metric, which is defined as the minimum cost of transforming one tree into another by inserting, deleting or modifying (the label of) a node.

we used the All Path Tree Edit Distance algorithm (APTED; Augsten, 2015, 2016), a novel one for the task.


Throughout all the bins, we observe that both SVCCA-53 and SVCCA-23 accomplish a comparable accuracy with the best setting in each group.

their clusters provide stable performance for both low or high-resource languages.


No automated model is used in this section, but instead, we examine the applicability of the youshould theory in argumentation.

we analyze whether each imperative preserves the original intent when it is transformed to an assertive by adding "should", along with appropriate changes in the verb form, (implicit) subject, and object.


We further empirically show that the choice of sentence aligner plays a significant role in the quantity of parallel sentences extracted from document pairs.

we study three aligners and show that combining their results, which we name 'Aligner Ensembling', increases recall.


We make use of the 60 occupations listed in Caliskan et al. (2017) containing statistics about gender distributions across professions, taken from the U.S. Bureau of Labor Statistics.

we generate a base set of 4,560 sentences from 38 templates, two tenses (present and past), and 60 occupations.


The remaining 75 are masculine (0 feminine).

we see a similar tendency to Chinese, but since the overall performance is poor, and the model is in general rather insensitive to differences in pronouns, we do not include correlation results.


To address these problems, we present a novel uncertainty-aware semantic augmentation method, which takes account of the intrinsic uncertainty sourced from the one-to-many nature of machine translation (Ott et al., 2018).

we first synthesize multiple reasonable source sentences to play the role of inherent uncertainty 1 for each target sentence.


We present an uncertainty-aware semantic augmentation method to bridge the discrepancy of the data distribution between the training and the inference phases for dominant NMT models.

we first synthesize a proper number of source sentences to play the role of intrinsic uncertainties via the controllable sampling for each target sentence.


A different line of work proposed to explore constituency parsing as a dependency parsing problem.

even if it is straightforward to represent constituency trees as hierarchical phrase structures, the same syntactic content can be represented with different mathematical objects (Rambow, 2010), including directed graphs commonly used for dependency parsing.


KAMG outperforms the other models in all the metrics across almost all the settings on both datasets with a notable margin, due to our multi-graph knowledge aggregation model.

while classifying zero-shot labels, ACNN-KAMG outperforms ZAGCNN, which uses only the label hierarchy (i.e., H g ), by 8% in R@10 and 6.5% in nDCG@10 on MIMIC-II and 4.1% in R@10 and 10.5% in nDCG@10 on MIMIC-III.


We focus on unsupervised STS, following  and Ethayarajh (2018).

we utilize only pre-trained word vectors, and do not use any supervision including training data for related tasks (e.g., natural language inference) and external resources (e.g., paraphrase database).


However, several studies have confirmed that the norm of word vectors has a large dispersion (Schakel and Wilson, 2015; Arefyev et al., 2018).

a sentence vector would contain word vectors of various sizes.


In the experiments, RE2 (Yang et al., 2019), De-cATT (Parikh et al., 2016), CAFE (Tay et al., 2018a) and BERT (Devlin et al., 2018) were set as the underlying models, achieving new models respectively denoted as "WD-Match (RE2)", "WD-Match (DecAtt)", "WD-Match (CAFE)", and "WD-Match (BERT)".

in WD-Match(RE2), F is a stacked blocks which consist of multiple convolution layers and multiple attention layers, and M is an MLP; in WD-Match(DecAtt), F is an attention layer and a aggregation layer, M is an MLP.


We conducted experiments to analyze how the feature vectors (i.e., hX and hY ) generated by WD-Match distributed in the common semantic space, using WD-Match(RE2) as an example.

we trained a RE2 model and a WD-Match (RE2) model based on SciTail dataset.


We conducted experiments to test how the Wasserstein distance-based regularizer guides the training of matching models.

we tested the WD-Match (RE2) and RE2 models generated at each training epochs.


The main idea behind the cost attention is to make the teacher and student Transformer networks be as close as possible.

we could reduce the overall cost of EMD by increasing the weights of the layers with low flow cost, while the weights of the layers with high flow cost should be decreased adaptively.


Similar to TinyBERT, our BERT-EMD method also contains a general distillation and a task-specific distillation.

we initialize our student model with the general distillation model provided by TinyBERT 1 .


Table 1 shows that the proposed BERT-EMD 6 method can effectively compress BERT BASE 12 into a 6-layer BERT model without performance sacrifice.

bERT-EMD 6 performs better than the 12-layer bERT bERT bASE 12 model on 7 out of 9 tasks, with only about 50% parameters and inference time of the original bERT bASE 12 model.


To verify the effectiveness of EMD and the cost attention mechanism, we perform ablation test of BERT-EMD on two large datasets (MNLI and QQP) and two small datasets (MRPC and RTE) in terms of removing EMD (denoted as w/o EMD) and cost attention (w/o CA), respectively.

for the method of removing EMD, we retain the many-to-many layer mapping by simply replacing the EMD with the mean squared error when measuring the distance between the teacher and student layers.


We follow Fan et al. (2020) and consider the computation of one layer as a unit of computational cost.

we will assess how many layers, on average, each method builds up for each passage.


We note that it reaches the similar results of the static baselines with much fewer layers.

it yields substantially higher performance than static methods when the computational budget is smaller than ten layers.


In this work we show that adaptive computation can lead to substantial efficiency improvements for ODQA.

we find that it is important to allocate budget dynamically across a large number of passages and prioritise different passages according to various features such as the probability that the passage has an answer.


Secondly, we utilize a structured description to restore the scene.

we build a scene graph based on the original narrative and the knowledge from Atomic.


To summarise, inspired by human behaviors, we propose a novel method to restore the scene for narrative MRC.

we introduce event knowledge from Atomic , and build the scene graph to describe the scene.


Multiple studies of BERT concluded that it is considerably overparametrized.

it is possible to ablate elements of its architecture without loss in performance or even with slight gains (Kovaleva et al., 2019;Michel et al., 2019;Voita et al., 2019).


This shows that even the worst possible selection of pre-trained BERT components for a given task still contains a lot of useful information.

some lottery tickets are "winning" and yield the biggest gain, but all subnetworks have a non-trivial amount of useful information.


We observed that the initial layers focus on query-passage interaction, whereas later layers focus more on contextual understanding and enhancing the answer prediction.

for quantifier questions (how much/how many), we notice that BERT focuses on confusing words (i.e., on other numerical quantities in the passage) in the later layers, but still manages to predict the answer correctly.


Nevertheless, the counterfactual samples are simply added to the training data for augmentation, ignoring that the relationship between original samples and counterfactual samples are vital for the reasoning of VQA models.

the model should be able to learn why the correct answer cannot be inferred after changing the original sample to the counterfactual sample.


In this paper, we present VD-BERT, a novel unified vision-dialog Transformer framework for Vis-Dial tasks.

we first encode the image into a series of detected objects and feed them into a Transformer encoder together with the image caption and multi-turn dialog.


We use two visually grounded training objectivesmasked language modeling (MLM) and next sentence prediction (NSP) to train our VD-BERT.

we aim to capture dense interactions among both inter-modality (i.e., image-dialog) and intra-modality (i.e., image-image, dialog-dialog).


For training in the discriminative setting, we transform the task of selecting an answer into a point-wise binary classification problem.

we sample an answer At from the candidate pool and append it to the input sequence, and ask the NSP head to distinguish whether the sampled answer is correct or not.


We evaluate our model on the VisDial v0.9 and v1.0 datasets (Das et al., 2017).

v0.9 contains a training set of 82,783 images and a validation set of 40, 504 images.


TDAEME combines both general and domain word embeddings in a supervised manner, which is implemented by a supervised autoencoder.

tDAEME predicts the words' category distribution.


Previous work (Wang and Lu, 2018) engage structured attention networks (Kim et al., 2017), which extend the previous attention mechanism to incorporate structure dependencies, to model the interaction among context words, and perform softselections of word spans.

they introduce two hand-coded regularizers to constrain the soft-selection process to attend to few short opinion spans.


The model achieves the best performance when the number of CRFs equals to 4.

the performance becomes relatively plateau when a large number of CRFs is adopted.


We propose a simple and effective MCRF-SA model to extract aspect-specific opinion span features.

with the proposed multi-CRF structured attention layer and the effective position decay function, our model is capable of extracting various aspect-specific opinion span features from different representation sub-spaces.


In this scenario, we need to consider both the dependence among different labels (label dependence) and the dependence between each predicting label and different modalities (modality dependence).

we propose a multi-modal sequence-to-set approach to effectively model both kinds of dependence in multi-modal multi-label emotion detection.


In this group, the baselines use different approaches to deal with the multi-label issue without considering the modality dependence issue.

in these approaches, the multi-modal inputs are early fused (simply concatenated) as a new input.


To solve this problem, we develop a simple but effective approach to enrich ABSA test sets.

we generate new examples to disentangle the confounding sentiments of the non-target aspects from the target aspect's sentiment.


Is our method generalizable to different datasets?

how does it perform if we train the metric on one dataset and test on another one?


PEGASUS proposes Transformerbased models with extracted gap-sentences for abstractive summarization.

these Transformer-based approaches are divided into Base and Large versions, according to the layers of Transformer.


The output of ALBERT encoder contains word hidden states h word and sentence hidden states h sent .

aLBERT takes subword units as input, which means that one word may correspond to multiple hidden states.


The proposed Cross-Thought architecture is illustrated in Figure 2(c).

we preappend each sequence with multiple special tokens, the final hidden states of which are used as the final sentence embedding.


In this section, we introduce our proposed pretraining model Cross-Thought, and describe how to finetune the pre-trained model on downstream tasks.

most parameters in downstream tasks can be initialized by the pre-trained Cross-Thought, and for certain tasks (e.g., ranking) the attention weights across sequences can be directly used without additional parameters (Figure 3).


However, such two-step procedures in these early attempts can be performed via a single-step procedure with neural models.

recent attempts have utilized sequence-to-sequence (seq2seq) models as a single-step procedure to learn the implicit relationship between operators and operands (Amini et al., 2019; Chiang and Chen, 2019; Wang et al., 2019).


The fully automatic EPT model, which does not use handcrafted features, yields comparable performance to existing models using hand-crafted features.

on the ALG514 dataset, the EPT outperforms the best-performing pure neural model by about 40% and shows comparable performance accuracy to the SoTA model that uses hand-crafted features.


From the ablation study, our data showed that the two components of generating 'Expression' token and applying operand-context pointer, each improved the accuracy of the EPT model in different ways.

as seen in Table 5, adding Expression token to the vanilla Transformer improved the performance accuracy by about 15% in ALG514 and DRAW-1K and about 1% in MAWPS.


First, using Expression tokens in highcomplexity datasets address the expression fragmentation issue when generating solution equations, which is more complex in ALG514 and DRAW-1K than MAWPS.

table 3 shows that on average the number of unknowns in ALG514 and DRAW-1K is almost twice (1.82 and 1.75, respectively) than MAWPS (1.0).


Second, using operand-context pointers in highcomplexity datasets addresses the operand-context separation issue when selecting an operand, which is more complex in ALG514 and DRAW-1K than MAWPS.

table 3 shows that on average the amount of Expression tokens is also twice in ALG514 and DRAW-1K (7.45 and 7.95, respectively) than that of MAWPS (3.60).


In addition, even with 800 generic templates, the synthesized sentences still lack naturalness and variety.

people often compress multiple concepts into simpler constructions (sublexical compositionality (Wang et al., 2015)), e.g. “books with at least 1 award” can be simplified to “award-winning books”.


Based on that, we inject two types of knowledge as additional objectives to regularize the learning process.

we exploit logic knowledge by transforming the consistency between sentencelevel and token-level predictions with propositional Boolean expressions.


Results show that our knowledge-augmented method significantly improves a strong multi-task learning approach.

our model greatly improves precision, demonstrating leveraging declarative knowledge expressed in both first-order logic and natural language can help the model to make more accurate predictions.


Therefore, we consider tackling the problems by exploiting logic knowledge.

we propose to employ propositional Boolean expressions to explicitly regularize the model with a logic-driven objective, which improves the logical consistency between two different grained predictions, and makes our method more interpretable.


To exploit this kind of knowledge, we adopt an additional encoder to encode the literal definition of each propaganda technique.

for each definition, the input sequence is modified as "[CLS]definition[SEP ]" and fed into BERT.


Focusing solely on improving the representation of the class label set, we show in experiments conducted in both private and public intent classification datasets, that better detection of out-of-scope examples (OOS) is achieved and, as a consequence, that the overall accuracy of intent classification is also improved.

using the recently-released Larson dataset, an error of about 9.9% has been achieved for OOS detection, beating the previous state-of-the-art result by more than 31 percentage points.


But, in many cases, the classes are in fact associated with inter-connected higher-level concepts which could be formatted into more meaningful representations and better exploited in the classification process for an enhanced representation of the scope of the classifier.

we explore here the use of graphs which represent information by means of nodes connected to each other by arcs.


Average word embeddings is also used in this work.

glove (Pennington et al., 2014) is employed for embedding each word of the document.


Even though BERT+ has not significantly outperformed BERT in some scenarios, such as with PT-BR chatbots, we can observe a great improvement that the word graphs can bring to intent recognition if we take into account the ISER metric.

the difference in ISER of BERT+ against BERT is of 5%, where the former achieved 37% and the latter 32%.


That is, the difference in ISER of BERT+ against BERT is of 5%, where the former achieved 37% and the latter 32%.

bERT+ can be considered quite worse than bERT for in-scope only intent classification.


We start with a formalisation of lexical ambiguity.

we formalise the lexical ambiguity of an entire language as Interpreting entropy as uncertainty, this definition implies that the harder it is to predict the meaning of a word from its form alone, the more lexically ambiguous that word must be.


We aim to address the above issues by quantifying the primary sources of errors over representative models.

following MQM (Mariana, 2014), we design 8 metrics on the Accuracy and Fluency aspects.


Pre-training: Pre-training is highly effective for summarization, and even achieves a better content selection capability without copy and coverage mechanisms.

joint pre-training combining text understanding and generation gives the most salient advantage, with the BART model achieving by far the state-of-the-art results on both automatic and our human evaluations.


PolyTope is an error-oriented fine-grained human evaluation method based on Multidimensional Quality Metric (MQM) (Mariana, 2014).

it consists of 8 issue types (Section 4.1), 8 syntactic labels (Section 4.2) and a set of severity rules (Section 4.3) to locate errors and to automatically calculate an overall score for the tested document.


We develop an operating interface for annotation, which is shown in Appendix A.1.

a human annotator is presented the original text and an output summary in juxtaposition, and is asked to select segments that are deemed incorrect after reading.


It gives large improvements on Addition, Omission and Inacc errors, proving that unified pre-training for both understanding and generation is highly useful for content selection and combination.

bART shows superior performance in handling the leading bias of CNN/DM dataset.


It is a recall-based metric calculating lexical overlap between system output and human summaries.

rOUGE-1 is based on unigram overlaps, rOUGE-2 on bigrams and rOUGE-L on longest common subsequences.


The situation changed with the introduction of Babel-Net (Navigli and Ponzetto, 2012a), a massive multilingual semantic network, created by automatically integrating WordNet, Wikipedia, and other resources.

babelNet synsets contain translations in multiple languages for each individual word sense.


We find that applying transformation is generally able to improve each contextualized model, obtaining the best performance across all the tasks.

we observe substantial improvements in Usim (ca. 0.04 increase of ρ) and SCWS (ca. 0.03 increase of ρ).


We observe improvement from the transformation on the static anchors in Inter Word tasks.

aligning towards FastText brings the largest and the most consistent gains.


We explore an alternative method to allow mentions of the same PoC to be connected with each other.

we direct one attention head to focus on tokens belonging to the same PoC, allowing these tokens to share semantic representations, similar to Strubell et al. (2019).


A human evaluator from Amazon Mechanical Turk (mturk.com) is asked to assess if the fusion sentence has successfully retained the original meaning.

an evaluator is tasked with reading the two article sentences and fusion sentence and answering yes or no to the following question, "Is this summary sentence true to the original article sentences it's been sourced from, and it has not added any new meaning?"


Existing work in neural story generation has established the strength of adding a content planning stage to structure the generated content (Yao et al., 2019; Fan et al., 2019) (discussed in more detail in Section 7).

this line of work trains a pipeline with one model that generates from prompt→ plot and another that generates from prompt + plot → story.


Shi et al. (2019) propose a visually grounded neural syntax learner (VG-NSL) to tackle the task.

they learn a parser from aligned imagesentence pairs (e.g., image-caption data), where each sentence describes visual content of the corresponding image.


We also assume that there is a limited budget to ask domain experts to provide explanations during training.

we consider k rounds of interactions with domain experts and each round has a query budget b.


Despite these empirical observations, little prior work analyzed or showed how to mitigate negative interference in multilingual language models.

it is natural to ask: (1) Can negative interference occur for low-resource languages also? (2) What factors play an important role in causing it? (3) Can we mitigate negative interference to improve the model’s cross-lingual transferability?


We seek to individually characterize the un-derlying factors of negative interference through a set of ablation studies and glean insights on its causes.

we examine if training corpus size and language similarity affect negative interference, and also measure gradient and parameter similarities between languages.


Our results show that negative interference can occur in both high-resource and low-resource languages.

we observe that neither subsampling the training corpus nor adding typologically similar languages substantially impacts negative interference.


We then retrain the monolingual and bilingual models and compare with results of their high-source counterparts.

we test if reducing lg 1 's training size also reduces negative interference on lg 2 .


Therefore, we study whether gradient conflicts exist between languages in multilingual models.

we sample one batch for each language in the model and compute the corresponding gradients' cosine similarity for every 10 steps during pretraining.


We focus on monolingual and bilingual models for a more controllable comparison, which we refer to as Mono and JointPair respectively.

we always include English (En) in bilingual models to compare on zero-shot transfer settings with prior work.


Similar to NER, POS is also a sequence labelling task but with a focus on synthetic knowledge.

we use the Universal Dependencies treebanks (Nivre et al., 2018).


In Table 2 and 3, we report our results on NER, POS and QA together with XLM-100, which is trained on 100 languages and contains 827M parameters.

we observe that monolin-  gual models outperform bilingual models for all languages except Swahili on all three tasks.


We also plot the similarity within English, measured using two independently sampled batches.

gradients between two different languages are indeed less similar than those within the same language.


We thus train variants of bilingual models that contain language-specific components for each language.

we consider adding language-specific feedforward layers, attention layers, and residual adapter layers (Rebuffi et al., 2017;Houlsby et al., 2019), denoted as ffn, attn and adpt respectively.


At the same time, our method also mitigates negative interference and outperforms JointPair on withinlanguage performance, closing the gap with monolingual models.

it performs better than ordinary adapters in both settings.


Our work is also related to transfer learning (Pan and Yang, 2010) and multi-task learning (Ruder, 2017).

prior work have observed (Rosenstein et al., 2005) and studied (Wang et al., 2019) negative transfer, such that transferring knowledge from source tasks can degrade the performance in the target task.


With carefully controlled experiments, we are confident that any improvement in the results is solely attributable to vocabulary composition.

our vocabularies improve models without increasing the model size, compute or data.


We see that the models converge to different representations, where the lower layer representations are more similar and they diverge as we move towards the upper layers.

note that this indicates that multi-task learning helps in learning different representations than self-supervised pre-training, and meta-learning model representations are different from the other models.


TNT enhances the language learning by utilizing text normalization pre-training objective, inspired by misspelling correction.

tNt randomly manipulates tokens from the input text.


To put this variation in context, we note that the MLDoc and XNLI test sets are relatively large, so a 2.5% difference in accuracy would be statistically significant (at the 5% significance level using the usual test of proportions), which means that a paper claiming 'state-of-the-art' performance on these zero-shot tasks may be reporting strong results because of the large between-run variation, rather than a genuine improvement due to their proposed technique.

using the En dev accuracy for choosing the final model for zero-shot transfer leads to inconsistent results across different runs.


A similar effect occurs with XLM-R and the more distant languages (Ar, Vi, Zh).

Chinese (Zh) degrades by 5.9 points as En improves, which may be evidence of directional disagreement in MLQA between En dev and Zh test.


In this paper, we study existing and novel auxiliary tasks in a BERT paradigm to guide future research in an informed manner.

we test and provide insight on: 1) NSP's effect on BERT pretraining; 2) the result of 14 other auxiliary tasks on BERT pre-training; 3) how to combine multiple tasks in BERT pre-training; and 4) the advantages of multi-task learning in BERT pre-training.


Using this analyzed data, we then learn an agreement prediction model that contains the desired rules.

we devise a binary classification problem of identifying whether agreement will be observed between a head and its dependent token on a given morphological property.


In our work, we used articles from Wikipedia to build a background knowledge corpus for each language.

we parsed the text from the entire Wikipage, removing non-textual content, e.g., HTML tags, tables, etc.


For all the elements of each type given to workers, the percentage of elements being captioned ranges from 75% to 94% (see Figure 3).

the View type has the lowest labeling ratio of 75%, which we suspect that elements with the View type, a generic widget type, tend to be quite arbitrary and are difficult for the workers to understand.


Figure 2 shows that our corpus has reasonable word-level agreement among the captions of the same widget.

for the 6K words, we report the mean precision and recall of every 10 consecutive words in the vocabulary.


Pre-training allows us to leverage the images of all the elements instead of only those with caption labels.

to reconstruct an image, we used 5 layers of transposed convolution where each layer has a residual connection architecture that is similar to the encoder part (that is discussed in the main paper).


Within our framework, we also experiment with using BERT in both the energy function and inference network architecture.

the "input feature vector" in Equation 3 is replaced by the features from BERT.


KumaGCN outperforms all the baselines in terms of both averaged accuracy scores and averaged F1 scores.

it improves the performance by 2.77 F1 points compared with the depGCN method.


While linguistic research has historically sought to uncover the rules for associating a noun with gender in terms of surface features or semantics (see Corbett 1991 for an overview), we take an extensional approach.

we treat a gender category in a language solely as the set of words it covers.


We could ask: What fraction of A agrees in gender across languages?

for each noun in our multilingual vocabulary, do both languages lexicalize it with the same gender?


In this paper, we propose attentive graph convolutional networks to enhance neural CCG supertagging through a novel solution of leveraging contextual information.

we build the graph from chunks (n-grams) extracted from a lexicon and apply attention over the graph, so that different word pairs from the contexts within and across chunks are weighted in the model and facilitate the supertagging accordingly.


The results show that the model performance drops when either part is ablated (ID 1 vs. ID 2-6).

removing attention significantly hurts the performance, where all configurations without attention (ID 4-6) shows worse-than-baseline (ID 7) results; this observation confirms the importance of applying attention on GCN.


Especially for the smaller sampled sets, our method demonstrates more significant performance improvement.

the proposed method achieved average 1.93 and 1.38 point improvement compared with the baseline methods in the 1k and 2k settings, respectively.


Similar to NER, our method demon-  strates more significant performance improvement for smaller sampled sets on POS tagging.

the proposed method achieved average 0.56, 0.60 and 0.46 point improvement compared with the baseline methods in the 1k, 2k and 4k settings, respectively.


The results show that even a small fraction of real data can significantly improve zero-shot performance.

5% of the MNLI English training dataset is enough to lift the performance of extraction to that of XLMR-Base for all languages.


In order to relieve the under-contextualization issue of many synsets, we propose a gloss augmentation approach to draw in more contextual information.

we simply use the short-length glosses as queries (words or phrases) to retrieve sequences from the web and combine the sequences with the original gloss and example sentences to learn a contextual representation from BERT.


We start with a generic summarization corpus.

in this work we use the CNN/DailyMail (Hermann et al., 2015) which consists of a set of (document, summary) pairs.


The systems equipped with IO tagging scheme always outperform those with BIO scheme.

sTRUCTsHOT and NNshot benefit from switching from BIO scheme to IO scheme by an average of 3.2% and 3.8% F1 scores on the five-shot tag set extension and domain transfer tasks respectively.


We proposed a framework for learning structured representation of entity names under low-resource settings.

we focus on a challenging scenario, where entity names are given as textual strings without context.


As mentioned earlier, we used huggine-Face pytorch-transformer (Wolf et al., 2019) to implement our BERT-CRF model.

we used the pretrained BertModel, and we obtained the tokenizer and pretrained weights using the bert-base-cased configuration.


Our approach achieves the state-of-the-art among all methods.

the recall of our approach is much higher than other methods, which demonstrate the effectiveness of the trigger identification step.


We aim to leverage human and machine intelligence together for attention supervision.

we show that human annotation cost can be kept reasonably low, while its quality can be enhanced by machine selfsupervision.


Specifically, we show that human annotation cost can be kept reasonably low, while its quality can be enhanced by machine selfsupervision.

for this goal, we explore the advantage of counterfactual reasoning, over associative reasoning typically used in attention supervision.


Key distinctions from (Bao et al., 2018) are (a) humans annotate even less, and (b) without additional training resources.

we start by loosening the definition of human annotation (Camburu et al., 2018; Zhong et al., 2019) into the task-level annotation: it reduces annotation cost to the size of vocabulary, or often to zero, when public resources such as sentiment lexicon replace such annotation.


Our adaptation goal is an unsupervised adaptation of task-level human annotation to samplelevel supervision signals for attention/classification models.

we propose Sample-level AttentioN Adaptation (SANA).


Specifically, we propose Sample-level AttentioN Adaptation (SANA).

for self-supervising such adaptation, SANA conducts what-if tests per each sample, of whether the permutation on human annotation changes the machine prediction.


Specifically, for self-supervising such adaptation, SANA conducts what-if tests per each sample, of whether the permutation on human annotation changes the machine prediction.

we collect the counterfactual (machine) supervisions for free, by observing whether highly attended word by human leads to the same machine prediction, compared to when such attention is counterfactually lowered.


To validate this, we employ the attention-permutation experiments designed in (Jain and Wallace, 2019), i.e., what-if simulation.

when given an input sample in the test phase, we look into whether the randomly mutated attention (i.e., cause) from the original attention yields any changes in the corresponding prediction result (i.e., effect).


We studied the problem of attention supervision, and showed that requiring sample-level human supervision is often less effective than task-level alternative with lower (and often zero-) overhead.

we proposed a counterfactual signal for self-supervision, to augment task-level human annotation, into sample-level machine attention supervision, to increase both the accuracy and robustness of the model.


We reimplement the model of Tenney et al. (2019b),4 which gives a unified architecture that works for a wide range of probing tasks.

it classifies single spans or pairs of spans in the following way: 1) construct token representations by pooling across encoder layers with a learned scalar mix (Peters et al., 2018a), 2) construct span representations from these token representations using self-attentive pooling (Lee et al., 2017), and 3) concatenate those span representations and feed the result through a fully-connected layer to produce input features for the classification layer.


When we build an empirical distribution over templates on the training set of Spider, we observe a 85% coverage of dev set templates.

85% of dev set examples have a query whose template occurs in the training set.


Figure 1 depicts such alignments with arrows and colors.

the proposed method is a two-step procedure.


In addition, we assign a binary label for each pair of review sentence and rebuttal sentence to indicate if the two sentences in this pair are aligned.

the binary label is 1 only when: 1) both two sentences are from arguments; 2) the rebuttal sentence is replying the argument where the review sentence originates.


MT-LSTM-CRF is a baseline without the hierarchical LSTM design.


For training the sentence pairing classifier, we investigate the effect of data sampling on the matching performance.

we compare the  performance of using different numbers (i.e., k) of negative samples for each review argument sentence.


During the evaluation of the argument pair extraction performance, we determine the argument pairing relation on span level with the sentence pairing results.

we perform binary classification on all sentence pairs enumerated from a candidate argument pair.


Intuitively, a larger confidence threshold means a stricter criterion for sentencelevel alignment for argument pairs.

the confidence of the argument pair prediction is controlled by p%.


Given the status quo, our goal is to further the computational study of exaggeration.

our contributions in this work are three-fold.


We begin by collecting hyperbolic sentences from two sources: webpages in professional educational websites 1 and linguistics research papers on hyperbole in Chinese (Huang, 2010; Zhao and Lu, 2013; Liao and Ge, 2014; Zhou and Jiang, 2014; Zhang, 2016).

we select 700 sentences from these two sources that have been discussed and determined to be hyperbolic by experts on exaggeration.


Finally, we ask the two annotators involved in Step 3 to judge whether each hyperbolic sentence obtained in the previous step is indeed hyperbolic.

if at least one of them thinks a sentence is not hyperbolic or does not truly reflect the meaning of its non-hyperbolic counterpart, we will delete it.


To answer this question, we first compute the cosine similarity of each pair of hyperbolic sentences in the same set as well as the cosine similarity of each pair of non-hyperbolic sentences in the same set, where cosine similarity is computed based on their one-hot word vectors.

the more word overlaps there are between two sentences, the higher their similarity is.


The attention mechanism computes an output vector by accumulating relevant information from a sequence of input vectors.

it assigns attention weights (i.e., relevance) to each input, and sums up input vectors based on their weights.


However, our proposal differs from theirs in some aspects.

we aim to analyze the behavior of the whole attention mechanism more accurately, whereas they aim to make the attention weights more accurate.


We included /s as the alignment targets and we considered the alignments to /s as no alignment.

if the model aligns a certain word with /s , we assume that the model decides that the word is not aligned to any word.


To verify whether the results obtained in the Section 5 are reproducible in different settings, we conducted an additional experiment using the model with a different number of attention heads.

we used a model with eight attention heads in both the encoder and decoder.


Suppose we have two non-parallel corpora X and Y with style S x and S y , the goal is training two transferrers, each of which can (i) transfer a sen-  tence from one style (either S x and S y ) to another (i.e., transfer intensity); and (ii) preserve the styleindependent context during the transformation (i.e., preservation).

we denote the two transferrers f and g. f : X → Y transfers a sentence x ∈ X with style S x to y * with style S y .


Nevertheless, when pre-trained models are used as a fine-tuning approach, which is a common practice, graph structure does not contribute to the final results.

the graph structure may not be necessary for multi-hop question answering.


We devise a simple encoding scheme that allows us to make the model aware of the target boundaries, without architectural modifications to BART.

we encode <c, t> pairs as sequences of subword tokens in which the boundaries of the t span in c are marked by two special tokens, i.e. <define> and </define>.


In this work, we explore several unsupervised word-level representation extraction strategies and configurations for lexico-semantic tasks (i.e., probes), stemming from different combinations of the components detailed in Table 1 and illustrated in Figure 1.

we assess the impact of: 1) encoding tokens with monolingual LM-pretrained Transformers vs. with their massively multilingual counterparts; 2) providing context around the target word in input; 3) including special tokens like [CLS] and [SEP]; 4) averaging across several layers as opposed to a single layer.


Considering the sentence "John lives in New York", we observe that the location entity "New York" and the context "John lives in" are highly correlated but are not causal to each other.

we can intervene on the location entity to set it to another different location entity without destroying the sentence correctness at the grammatical level.


As there is no currently available dataset to investigate this problem, this paper proposes to conduct randomization test on standard benchmarks.

we erase name regularity, mention coverage and context diversity respectively from the benchmarks, in order to explore their impact on the generalization ability of models.


Table 1 overviews their discrepancies on name regularity, mention coverage and context pattern acquisition.

mentions of many entity types do not follow regular compositional structures.


In this paper, we want to shed some light on the impact of the discrepancies between regular and open NER, and provides some valuable insights into the construction of general NER models in a more effective and efficient way.

we want to answer the following question: Can pretrained supervised neural networks still generalize well on NER when either weaker name regularity, lower mention coverage or inadequate context diversity exists?


To this end, this paper exploits the efficacy of the above three kinds of information by conducting a series of experiments based on randomization test (Edgington and Onghena, 2007;Zhang et al., 2016).

we design several mention replacing mechanisms, which can erase specific kinds of information on-demand from current NER benchmarks.


The results of our randomization test can serve as a frame of reference for open NER, where the erased information is often truly absent.

three kinds of information are particularly considered, and four kinds of strategies are used in our randomization test.


Surprisingly, the model performs significantly better in the MP setting than in the NP setting in all entity types.

high mention coverage undermines the models' ability to generalize to unseen mentions.


The purpose of context reduction is to reduce context diversity in training data but still keeps all name regularity in the vanilla setting.

cR only keeps a subset of sentences in the vanilla training data, and then duplicates preserved sentences and randomly replace mentions in them with mentions of the same type in the vanilla training data.


Because currently no suitable dataset is available for verifying our conclusions, this paper constructs a new dataset from Wikipedia.

we consider four entity types in our experiments, including movie, song, book and tv series.


We use the BERT-CRF tagger as our baseline.

a Transformer (Vaswani et al., 2017) is used as the encoder and then two dense layers are used to map the hidden representation into the label space.


For each entity, we define an initial embedding (at t 0 ) on each manifold and a velocity vector residing in the tangent space of the initial embedding to generate a temporal representation at each timestamp.

the initial embeddings represent the stationary structural dependencies across facts, while the velocity vectors capture the time-varying properties of entities.


Thus, we treat the curvature of each component and the dimension as hyperparameters selected a priori.

we use the parallelogram law' deviation  to estimate both the graph curvature of a given temporal KG and the number of components.


The number of interactions between the selected entities are depicted in Figure 8 and 9, which evolves over time.

we highlight Nigerian citizens, the Nigerian government, head of the Nigerian government, other authorities in Nigeria, and Nigerian minister in the first row of subplots.


To assess the contribution of the dynamic part of entity embeddings, we remove the dynamic part and run the model variant on static knowledge graphs.

we compress ICEWS05-15 into a static, cumulative graph by ignoring the time information.


Obviously answers contain numerous cues for subsequent dialog flow, but in this first work we focus on meaning prediction.

given a question-answer pair, we classify it into one of the meaning categories in Tables 7 and 8.


Our experiments show that POWERTRANSFORMER performs better than the baselines overall.

while the BST revisions obtain slightly higher accuracy on the output agency levels, these revisions have the both the lowest diversity and meaning preservation, suggesting the model ignores the input (Table 4).


We train the top-performing discourse parser by Wang et al. (2017) on MEGA-DT and compare its performance with the same parser trained on previously proposed treebanks.

we compare our discourse-annotated dataset against a smaller "silver-standard" treebank (Huber and Carenini, 2019) containing around ~100,000 documents with <=20 EDUs and two standard human annotated corpora in the news domain (RST-DT) (Carlson et al., 2002) and in the instructional domain (Subba and Di Eugenio, 2009).


In this paper, we propose a new strategy to enable the existing debiasing methods to be applicable in settings where there is minimum prior information about the biases.

models should automatically identify potentially biased examples without being pinpointed at a specific bias in advance.


Our goal is to obtain a fair comparison without the confounds that may result in performance differences on these two sets.

the examples from the two sets should be similar except for the presence of a feature that is biased in one set and anti-biased in the other.


As expected, the models show the tendency of relying on biases after only seeing a small fraction of the dataset.

at an early point during training, models achieve 100% accuracy on the biased test and drop to almost 0% on the anti-biased test.


We question whether this behavior persists once we set a closer to 0.

do models fall back to the baseline performance when the loss gets more equivalent to the standard cross-entropy at the end of the training?


We are, therefore, interested in characterizing these changes by analyzing their training loss curve.

we examine the individual losses on each training batch and measure their variability using percentiles (i.e., 0th, 25th, 50th, 75th, and 100th percentile).


Instead, our analysis reveals that this behavior is caused by subtle artifacts arising from the translation process itself.

we show that translating different parts of each instance separately (e.g., the premise and the hypothesis in NLI) can alter superficial patterns in the data (e.g., the degree of lexical overlap between them), which severely affects the generalization ability of current models.


In both cases, we expand the initial set of 100 labeled examples with another set of 100 instances, selected at random from the remaining data, which are all added to L with a negative label.

the low prior of the positive class naturally implies high prior of the negative class, enabling to expand the fully labeled 100 instances with an additional set of 100 instances that are -weakly -labeled as negative examples (without the need for additional annotation budget).


Neural networks that are specifically designed to work in conjunction with certain AI accelerators will encounter a similar trade-off.

the more a neural network is tuned for efficiency, the less flexibility for change the model will have (Han et al., 2015).


For the Seq2seq-LSTM models (Cho et al., 2014; Sutskever et al., 2014), we slightly modified the code by Luong et al. (2017), and we ran it on the two benchmarks provided in the repository.

we used WMT datasets to train German-English and English-German models.


Note, however, that the OPU would run into some of the same problems that the TPU has.

there will be a large delay when loading the matrix weights from the memory onto the OPU.


In addition to the evaluations described above, where all utterances are randomly assigned to train, development, and test sets, we do speakerindependent evaluation of the speech+text model.

we hold out a single speaker for testing and use all the other speakers for training and development.


However, for tasks involving complex reasoning and induction it remains beneficial to provide models with externally linked knowledge (Mitra et al., 2019; Fan et al., 2019).

for dialog modeling, the Wizard of Wikipedia (Dinan et al., 2019) and Topical Chat (Gopalakrishnan et al., 2019) corpora consist of grounding documents linked with open-domain chit-chat.


The research on TE dates back more than two decades and has made significant progress.

with the advances of deep neural networks and the availability of large-scale human annotated datasets, fine-tuned systems often claim surpassing human performance on certain benchmarks.


As an example, we apply it on the boolean subset of the dataset by Richardson et al. (2020) containing samples with “boolean and” and find that our model achieves a near perfect accuracy on their test set.

roBErTa, trained on only MNLI, achieves a low accuracy of 41.5% on the test set, but on applying IAFT with an equal mix of MNLI and their training data in every epoch, the test accuracy improves to 99.8%, while also retaining MNLI matched/mismatched results at 86.45/86.46%.


Unsurprisingly, all models perform significantly better on the boolean subset compared to the non-boolean one.

the accuracies for RoBERTa, IAFT and PA on the boolean subset are 68%, 72% and 69% respectively, while on the non-boolean subset, these are 58%, 61% and 58% respectively.


While the OWA is beneficial because it helps us assess KGE calibration under more realistic conditions, it is also challenging because it significantly changes the requirements for evaluation.

now we need a label for every triple considered, whereas with the CWA we only needed labels for a small group of positives.


In this paper, we propose a novel framework, META, which goes beyond the existing paradigm and leverages metadata as an additional source of weak supervision.

we organize the text data and metadata together into a text-rich network and adopt network motifs to capture appropriate combinations of metadata.


We use the official TempEval-3 scoring script and report the standard metrics.

we report the detection precision, recall and F1 with the relaxed and strict metrics.


To achieve the set goals, we started with a review of previous arguments about the criteria for identifying appropriate utterances in dialogues and designed our scoring function that is consistent with reflects as much of the community's consensus as possible.

the proposed scoring method estimates the quality of utterance pairs based on the following two aspects: (i) the connectivity between source and target utterances and (ii) their content relatedness (Section 4).


We automatically obtained dialogue paired-data from the corpus which does not contain speaker annotations on the dialogue turns following the previous methods (Vinyals and Le, 2015; Li et al., 2016).

we extracted the consecutive two lines as an utterance pair based on the assumption that each line corresponds to a full-speaker's turn.


In the RL setting, we use DQN to train the policy.

the DQN contains three layers of multilayer perceptron (MLP).


The remaining computation cost from RL framework is comparably small during both the training and the prediction process.

during our experiments, we trained a three-layer deep neural network model, which has much fewer parameters compared to the information extraction system.


Method Since the state defined in our framework is from a continuous space, we adopt a deep Q-network (DQN) to approximate Q(s, a) with a deep neural network denoted by Q(s, a; θ).

we parameterize an approximate value function Q(s, a; θ) using a three-layer deep neural network.


In contrast to the single-instance evaluation, examining each generated path independently from others, we propose now to look at the generated graph as a whole and compare it to the ground truth one (based on our weakly-supervised dataset).

given input sentences x A , we generate the corresponding paths x AB and compare them to the ground truth x B .


In the preliminary study, we evaluate our pretrained model's performance on the held-out set of KGTEXT to conduct ablation study over KGPT.

we investigate 1) which encoding mechanism is better, 2) whether we need copy mechanism or copy supervision.


We conduct human evaluation to assess the factual accuracy of the generated sentences.

we sample 100 test samples from WebNLG and observe the model's factual consistency with given fact triples.


Quantitative Study We further investigate how much sample complexity KGPT can reduce.

we specify a BLEU-4 score and vary the training data size to observe how much training samples are required to attain the performance.


To address comprehension-based generation like generative QA, PALM uses the pre-training objectives that are closely related to the downstream tasks.

it differs from existing generative pre-training methods in that PALM goes beyond the solely autoencoding/autoregressive methods and combines the merits of autoencoding and autoregression in a single framework.


On the other hand, an autoregressive model, such as GPT (Radford, 2018;Radford et al., 2019), is only trained to encode unidirectional context (either forward or backward).

at each output timestep, a token is sampled from the model's predicted distribution and the sample is fed back into the model to produce a prediction for the next output timestep, and so on.


To close the gap, PALM is carefully designed to autoregressively generate a text sequence by comprehending the given context in a bidirectional autoencoding manner.

pALM delegates autoencoding-based comprehension to the encoder in Transformer, and autoregressive generation to the Transformer decoder.


To understand the performance of PALM pretraining, we compare generation quality of the pretrained models of PALM and MASS 3 .

we feed a few sentences from a news article to both pre-trained models, and the models generate a continuation of the input sentences by beam search with a beam of size 5.


In this work, we propose the Contextual Action Language Model (CALM) to alleviate this challenge.

at each game step we use CALM to generate action candidates, which are fed into a Deep Reinforcement Relevance Network (DRRN) (He et al., 2015) that uses game rewards to learn a value function over these actions.


To mitigate this, we combine two recent effective approaches, adversarial human-and-model-in-theloop data collection (Nie et al., 2020) and adversarial matching (Zellers et al., 2019a), to build a larger, more-challenging, and less-biased dataset.

50% of the examples in VLEP are directly annotated by humans over two rounds: round one of standard data collection, i.e., crowd-workers perform the annotations with no model feedback, and round two of adversarial data collection, i.e., crowd-workers perform the annotations with the goal of fooling our basic models trained on round one data (thus avoiding obvious biases).


Hence, in order to collect harder and less-biased negatives, we make use of an adversarial collection procedure (see Figure 2), in a human-and-model-inthe-loop process (Nie et al., 2020), where models are used to provide real-time feedback to crowdworkers during data collection.

each submitted result is sent to the model for evaluation and writers are prompted to rewrite their negative event if our model predicts a much higher probability for the more-likely event (p m ) than the less likely event (p l ), i.e., p m - p l >  ∆, where ∆ is a hyperparameter that controls how difficult we want the collected examples to be and is set to 0.1 empirically.


Since labeling questions with decompositions is cumbersome, we take an unsupervised approach to produce sub-questions, also enabling us to leverage millions of questions from the internet.

we propose an algorithm for One-toN Unsupervised Sequence transduction (ONUS) that learns to map one hard, multi-hop question to many simpler, singlehop sub-questions.


As shown in Table 7, we decompose queries from several other datasets, using our decomposition model trained on only questions in HOTPOTQAand Common Crawl.

we generate subquestions for (1) questions in ComplexWebQuestions (Talmor and Berant, 2018), which are multihop questions about knowledge-bases, (2) questions in CLEVR (Johnson et al., 2017b), which are multi-hop questions about images, and (3) claims (statements) in fact-verification challenges, FEVER 1.0 (Thorne et al., 2018) and 2.0 (Thorne et al., 2019).


We split VAST such that all examples xi = (di, ti, yi) where di = d, for a particular document d, are in exactly one partition.

we randomly assign each unique d to one partition of the data.


Due to the linguistic variation in the topic expressions (§3.2), we examine the prevalence of lexically similar topics, LexSimTopics, (e.g., 'taxation policy' vs. 'tax policy') between the training and zero-shot test sets.

we represent each topic in the zero-shot test set and the training set using pre-trained GloVe (Pennington et al., 2014) word embeddings.


Finally, we investigate the connection between stance and sentiment vocabulary.

we use the MPQA sentiment lexicon (Wilson et al., 2017) to identify positive and negative sentiment words in texts.


We observe that in the test set, the majority (80%) of pro examples have more positive than negative sentiment words, while only 41% of con examples have more negative than positive sentiment words.

con stance is often expressed using positive sentiment words but pro stance is rarely expressed using negative sentiment words and therefore there is not a direct mapping between sentiment and stance.


This was done using SUTime library (Chang and Manning, 2012) to tag time terms, which uses a regular expression based approach.

we focused on times that pinpointed hours within a day such as "two o'clock" or "noon".


We consider both BERT base and BERT large in our experiments.

we use an average pooling over BERT context embeddings in the last one or two layers as the sentence embedding which is found to outperform the [CLS] vector.


In addition to semantic similarity, we further study lexical similarity induced by different sentence embeddings.

we use edit distance as the metric for lexical similarity between a pair of sentences, and focus on the correlations between the sentence similarity and edit distance.


However, it is not evident that gold standard semantic similarity correlates with edit distance.

it is often the case where the semantics of a sentence can be dramatically changed by modifying a single word.


Training a classifier with the inherently imbalanced data on the maximum likelihood estimation (MLE) leads to biased classification boundaries in favor of majority classes (Khan et al., 2019).

models play a difficult role in learning with the imbalanced label (i.e., token) distribution (He et al., 2008b).


We generate texts for the evaluation by completing sequences from prefixes.

we batchify a test set, select the first 50 tokens from each batch as prefixes, and guide models to generate a continuation of 100 tokens from the prefixes.


Capitalized on the topic-level dialogue graph, we propose a new evaluation metric GRADE, which stands for Graph-enhanced Representations for Automatic Dialogue Evaluation.

gRADE incorporates both coarsegrained utterance-level contextualized representations and fine-grained topic-level graph representations to evaluate dialogue coherence.


As a result, our method can capture more accurate semantic transition information, thus measuring dialogue coherence in a more human-like manner.

our GRADE consists of two semantic extraction branches.


Since our goal is to predict a coherence score of a response based on a context, we only consider the edges between the context nodes V c and the response nodes V r .

the edges only exist between each context-topic node V i c and each response-topic node V j r .


We consider both retrievalbased and generation-based dialogue models to obtain diverse responses for metric evaluation so that the performance of the metrics can be assessed comprehensively.

we first deploy Transformer-Ranker and Transformer-Generator from the ParlAI platform (Miller et al., 2017), where the former is retrieval-based and the latter is generation-based.


It is noteworthy that all Pearson and Spearman correlations of GRADE are statistically significant with p-value < 0.05, and most of them are with p-value < 0.01.

gRADE achieves a significant Pearson correlation of 0.606 and Spearman correlation of 0.617 for evaluating Transformer-generator on the ConvAI2 dataset, bringing an improvement of 0.411 (Pearson) and 0.417 (Spearman) compared with BLEURT.


A limitation of GRADE is the inconsistency between the training objective (relative ranking) and the expected behavior (absolute scoring).

the ranking loss we adopted only requires good responses to be ranked higher than bad responses, which is a relatively loose constraint compared with the absolute scoring that humans do.


In this paper, we empirically study the efficiency issue for NMT models.

we first investigate the effects of weight pruning on advanced Transformer models, showing that 20% parameters can be directly pruned, and by continuously training the sparse networks, we can prune 50% with no performance loss.


Pruning The redundant parameters in neural networks can be pruned according to a certain criterion while the left ones are significant to preserve the accuracy of the model.

we mask weight connections with low magnitudes in the forward pass and these weights are not updated during optimization.


To solve TORQUE in an end-to-end fashion, the model here takes as input a passage and a question, then looks at every token in the passage and makes a binary classification of whether this token is an answer to the question or not.

the model has a one-layered perceptron on top of BERT (Devlin et al., 2019) or RoBERTa , and the input to the perceptron layer is the transformers' output corresponding to the token we're looking at.


We modify the deep biaffine attention classifier (Dozat and Manning, 2016) to model these edges.

we factorize the probability for each labeled edge into the "arc" and "label" parts, computing both based on the current decoder hidden state and the states of all previous words.


Apart from that, we study discourse (Prasad et al., 2008) relations, which are essential for generating a good sentence with correct meaning.

we consider 4 common discourse relations ("Cause", "Contrast", "Condition", "Coordinating").


We conduct ablation tests to study the contribution of each component to the proposed model.

we evaluate models with only the node prediction loss (Node Prediction, Section 3.1) and the edge prediction loss (Edge Prediction, Section 3.2), respectively, and further investigate the effect of integrating node and edge information into the next state computation (Section 3.3) by comparing models without and with (Int.)


We further investigate the influence of AMR-structure preservation on the performance of the main text generation task.

we first force our model to generate a gold sentence in order to calculate the accuracies for node prediction and edge prediction.


An alternative, also considered here, is to share output parameters across words as well as with the input embeddings.

this involves making the output embedding a function of the input embedding using a shared parameterization across words, E out = g(E in ), as displayed in Figure 1(b).


A key innovation of this proposed model is a novel parameter-gate (p-gate) mechanism that regulates the flow or transfer of the previously learned knowledge to the new task.

it can selectively use the network parameters (which represent the retained knowledge gained from the previous tasks) to assist the learning of the new task t. Knowledge distillation is also employed in the process to preserve the past knowledge by approximating the network output at the state when task t Ã¢Ë†â€™ 1 was learned.


However, the main objective of the current CL techniques is to solve the catastrophic forgetting (CF) problem (McCloskey and Cohen, 1989).

in learning each new task, the network parameters need to be modified in order to learn the new task.


Unlike traditional gates that regulate the feature information flow through the sequence chain, the goal of the proposed p-gates is to select useful parameters (which represent the learned knowledge from previous tasks) to be transferred to the new task to make it learn better.

p-gates regulate the knowledge transfer from the past to the present.


OpTok explores an appropriate tokenization for a downstream task.

opTok explores a tokenization that yields a better score for a downstream task.


As of the summer of 2018, 352,554 tweets were available, and we extracted only tweets with a single sentiment label of positive, negative, or neutral.

we removed both positive and negative and unrelated to prevent confusion.


Concretely, we obtained a tokenized sentence based on SentencePiece, and then treated the tokenized sentence as an input to the encoder.

we replaced the unigram language model in OpTok with the Sentence-Piece tokenizer and used one tokenized sentence as an input to the same architecture.


To validate the effect of only tokenization, we trained only the neural unigram language model in OpTok.

we fixed the neural encoder in OpTok and the downstream model with random initialization.


Inter-modality incongruity information can be represented as a kind of interaction between the features of multi modalities.

the input tokens will give high attention values to the image regions contradicting them as incongruity is a key character of sarcasm.


The experimental results illustrate that our model achieves the best performance across the baseline models.

our model obtains a 2.74% improvement in terms of F1 score compared with the state-of-the-art Hierarchical Fusion Model (HFM) proposed by Cai et al.


In the first experiment we compared data augmentation (via generation) to naive data balancing.

we compared baseline results to: (1) balancing w/o augmentation; (2) augmentation w/o balancing; and (3) balancing-via-augmentation.


For balancing experiments (no. 1 and 3), We used the simplest balancing scheme depicted by naive-OS balancing policy C (B low = B high = |c n |, as defined in Section 3).

for balancing w/o augmentation (1) we used basic sample-copy over-sampling, and for balancing-viaaugmentation (3) we applied BalaGen (using GPT-2 as generator) to generate additional samples according to policy C. For augmentation w/o balancing (2) we applied BalaGen using Augment-only data policy B -adding a fixed number of generated samples to all classes.


Partial-OS balancing policy (βlow  < 100) appears to be superior for all datasets.

for CQAβlow = 90, and for SEAQ and ATIS βlow = 80.


We explore the possibility of disentangling the three sub-aspects that are commonly used to characterize summarization: POSITION for choosing sentences by their position, IMPORTANCE for choosing relevant and repeating content across the document, and DIVERSITY for ensuring minimal redundancy between summary sentences (Jung et al., 2019) during the summary generation process.

we use these three sub-aspects as control codes for conditional training.


We proposed a neural framework for conditional extractive news summarization.

subaspect functions of importance, diversity and position are used to condition summary generation.


In this study, we propose a novel Temporally Expressive Networks (TEN) to jointly model the temporal feature dependencies and temporal state dependencies (Figure 1 (c)).

to improve the turn-level state prediction, we exploits hierarchical recurrent networks to capture temporal feature dependencies across dialogue turns.


Factor graphs are powered by a highly efficient algorithm, called the belief propagation or the sum-product algorithm, for computing the marginal distribution.

the algorithm executes by passing "messages" along the edges of the factor graph and the sent message is computed from all incoming messages on its "upstream".


In Table 3, we observe that both domain-specific BERT models can reduce greatly the number of false positives made by the baseline BERT.

159 false positives made by the baseline BERT are fixed by the domain-specific BERT models.


Recently, neural parsers (Vinyals et al., 2015; Dyer et al., 2016; Stern et al., 2017; Kitaev et al., 2019) without using any grammar rules significantly outperform conventional statistical grammar-based ones (Collins, 1997; Sagae and Lavie, 2005; Glaysher and Moldovan, 2006; Song and Kit, 2009), because neural networks, especially recurrent models (e.g, Bi-LSTM), are adept in capturing long range contextual information, which is essential to modeling the entire sentence.

a significant boost on the performance of chart-based parsers is observed from some recent studies (Kitaev and Klein, 2018; Kitaev et al., 2019; Zhou and Zhao, 2019) that employ advanced text encoders (i.e., Transformer, BERT, and XLNet), which further demonstrates the usefulness of contexts for parsing.