{"doc_id":"1602.03483","title":"Learning Distributed Representations of Sentences from Unlabelled Data","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Unsupervised methods for learning distributed representations of words are ubiquitous in today's NLP research, but far less is known about the best ways to learn distributed phrase or sentence representations from unlabelled data. This paper is a systematic comparison of models that learn such representations. We find that the optimal approach depends critically on the intended application. Deeper, more complex models are preferable for representations to be used in supervised systems, but shallow log-linear models work best for building representation spaces that can be decoded with simple spatial distance metrics. We also propose two new unsupervised representation-learning objectives designed to optimise the trade-off between training time, domain portability and performance."]},{"idx":1,"heading":"Introduction","paragraphs":["Distributed representations - dense real-valued vectors that encode the semantics of linguistic units - are ubiquitous in today's NLP research. For single-words or word-like entities, there are established ways to acquire such representations from naturally occurring (unlabelled) training data based on comparatively task-agnostic objectives (such as predicting adjacent words). These methods are well understood empirically BIBREF0 and theoretically BIBREF1 . The best word representation spaces reflect consistently-observed aspects of human conceptual organisation BIBREF2 , and can be added as features to improve the performance of numerous language processing systems BIBREF3 .","By contrast, there is comparatively little consensus on the best ways to learn distributed representations of phrases or sentences. With the advent of deeper language processing techniques, it is relatively common for models to represent phrases or sentences as continuous-valued vectors. Examples include machine translation BIBREF8 , image captioning BIBREF9 and dialogue systems BIBREF10 . While it has been observed informally that the internal sentence representations of such models can reflect semantic intuitions BIBREF11 , it is not known which architectures or objectives yield the `best' or most useful representations. Resolving this question could ultimately have a significant impact on language processing systems. Indeed, it is phrases and sentences, rather than individual words, that encode the human-like general world knowledge (or `common sense') BIBREF12 that is a critical missing part of most current language understanding systems.","We address this issue with a systematic comparison of cutting-edge methods for learning distributed representations of sentences. We constrain our comparison to methods that do not require labelled data gathered for the purpose of training models, since such methods are more cost-effective and applicable across languages and domains. We also propose two new phrase or sentence representation learning objectives - Sequential Denoising Autoencoders (SDAEs) and FastSent, a sentence-level log-linear bag-of-words model. We compare all methods on two types of task - supervised and unsupervised evaluations - reflecting different ways in which representations are ultimately to be used. In the former setting, a classifier or regression model is applied to representations and trained with task-specific labelled data, while in the latter, representation spaces are directly queried using cosine distance.","We observe notable differences in approaches depending on the nature of the evaluation metric. In particular, deeper or more complex models (which require greater time and resources to train) generally perform best in the supervised setting, whereas shallow log-linear models work best on unsupervised benchmarks. Specifically, SkipThought Vectors BIBREF13 perform best on the majority of supervised evaluations, but SDAEs are the top performer on paraphrase identification. In contrast, on the (unsupervised) SICK sentence relatedness benchmark, FastSent, a simple, log-linear variant of the SkipThought objective, performs better than all other models. Interestingly, the method that exhibits strongest performance across both supervised and unsupervised benchmarks is a bag-of-words model trained to compose word embeddings using dictionary definitions BIBREF14 . Taken together, these findings constitute valuable guidelines for the application of phrasal or sentential representation-learning to language understanding systems."]},{"idx":2,"heading":"Distributed Sentence Representations","paragraphs":["To constrain the analysis, we compare neural language models that compute sentence representations from unlabelled, naturally-ocurring data, as with the predominant methods for word representations. Likewise, we do not focus on `bottom up' models where phrase or sentence representations are built from fixed mathematical operations on word vectors (although we do consider a canonical case - see CBOW below); these were already compared by milajevs2014evaluating. Most space is devoted to our novel approaches, and we refer the reader to the original papers for more details of existing models."]},{"idx":3,"heading":"Existing Models Trained on Text","paragraphs":["SkipThought Vectors For consecutive sentences INLINEFORM0 in some document, the SkipThought model BIBREF13 is trained to predict target sentences INLINEFORM1 and INLINEFORM2 given source sentence INLINEFORM3 . As with all sequence-to-sequence models, in training the source sentence is `encoded' by a Recurrent Neural Network (RNN) (with Gated Recurrent uUnits BIBREF11 ) and then `decoded' into the two target sentences in turn. Importantly, because RNNs employ a single set of update weights at each time-step, both the encoder and decoder are sensitive to the order of words in the source sentence.","For each position in a target sentence INLINEFORM0 , the decoder computes a softmax distribution over the model's vocabulary. The cost of a training example is the sum of the negative log-likelihood of each correct word in the target sentences INLINEFORM1 and INLINEFORM2 . This cost is backpropagated to train the encoder (and decoder), which, when trained, can map sequences of words to a single vector.","ParagraphVector le2014distributed proposed two log-linear models of sentence representation. The DBOW model learns a vector INLINEFORM0 for every sentence INLINEFORM1 in the training corpus which, together with word embeddings INLINEFORM2 , define a softmax distribution optimised to predict words INLINEFORM3 given INLINEFORM4 . The INLINEFORM5 are shared across all sentences in the corpus. In the DM model, INLINEFORM6 -grams of consecutive words INLINEFORM7 are selected and INLINEFORM8 is combined with INLINEFORM9 to make a softmax prediction (parameterised by additional weights) of INLINEFORM10 .","We used the Gensim implementation, treating each sentence in the training data as a `paragraph' as suggested by the authors. During training, both DM and DBOW models store representations for every sentence (as well as word) in the training corpus. Even on large servers it was therefore only possible to train models with representation size 200, and DM models whose combination operation was averaging (rather than concatenation).","Bottom-Up Methods We train CBOW and SkipGram word embeddings BIBREF17 on the Books corpus, and compose by elementwise addition as proposed by mitchell2010composition.","We also compare to C-PHRASE BIBREF18 , an approach that exploits a (supervised) parser to infer distributed semantic representations based on a syntactic parse of sentences. C-PHRASE achieves state-of-the-art results for distributed representations on several evaluations used in this study.","Non-Distributed Baseline We implement a TFIDF BOW model in which the representation of sentence INLINEFORM0 encodes the count in INLINEFORM1 of a set of feature-words weighted by their tfidf in INLINEFORM2 , the corpus. The feature-words are the 200,000 most common words in INLINEFORM3 ."]},{"idx":4,"heading":"Models Trained on Structured Resources","paragraphs":["The following models rely on (freely-available) data that has more structure than raw text.","DictRep hill2015learning trained neural language models to map dictionary definitions to pre-trained word embeddings of the words defined by those definitions. They experimented with BOW and RNN (with LSTM) encoding architectures and variants in which the input word embeddings were either learned or pre-trained (+embs.) to match the target word embeddings. We implement their models using the available code and training data.","CaptionRep Using the same overall architecture, we trained (BOW and RNN) models to map captions in the COCO dataset BIBREF19 to pre-trained vector representations of images. The image representations were encoded by a deep convolutional network BIBREF20 trained on the ILSVRC 2014 object recognition task BIBREF21 . Multi-modal distributed representations can be encoded by feeding test sentences forward through the trained model.","NMT We consider the sentence representations learned by neural MT models. These models have identical architecture to SkipThought, but are trained on sentence-aligned translated texts. We used a standard architecture BIBREF11 on all available En-Fr and En-De data from the 2015 Workshop on Statistical MT (WMT)."]},{"idx":5,"heading":"Novel Text-Based Models","paragraphs":["We introduce two new approaches designed to address certain limitations with the existing models.","Sequential (Denoising) Autoencoders The SkipThought objective requires training text with a coherent inter-sentence narrative, making it problematic to port to domains such as social media or artificial language generated from symbolic knowledge. To avoid this restriction, we experiment with a representation-learning objective based on denoising autoencoders (DAEs). In a DAE, high-dimensional input data is corrupted according to some noise function, and the model is trained to recover the original data from the corrupted version. As a result of this process, DAEs learn to represent the data in terms of features that explain its important factors of variation BIBREF22 . Transforming data into DAE representations (as a `pre-training' or initialisation step) gives more robust (supervised) classification performance in deep feedforward networks BIBREF23 .","The original DAEs were feedforward nets applied to (image) data of fixed size. Here, we adapt the approach to variable-length sentences by means of a noise function INLINEFORM0 , determined by free parameters INLINEFORM1 . First, for each word INLINEFORM2 in INLINEFORM3 , INLINEFORM4 deletes INLINEFORM5 with (independent) probability INLINEFORM6 . Then, for each non-overlapping bigram INLINEFORM7 in INLINEFORM8 , INLINEFORM9 swaps INLINEFORM10 and INLINEFORM11 with probability INLINEFORM12 . We then train the same LSTM-based encoder-decoder architecture as NMT, but with the denoising objective to predict (as target) the original source sentence INLINEFORM13 given a corrupted version INLINEFORM14 (as source). The trained model can then encode novel word sequences into distributed representations. We call this model the Sequential Denoising Autoencoder (SDAE). Note that, unlike SkipThought, SDAEs can be trained on sets of sentences in arbitrary order.","We label the case with no noise (i.e. INLINEFORM0 and INLINEFORM1 ) SAE. This setting matches the method applied to text classification tasks by dai2015semi. The `word dropout' effect when INLINEFORM2 has also been used as a regulariser for deep nets in supervised language tasks BIBREF24 , and for large INLINEFORM3 the objective is similar to word-level `debagging' BIBREF25 . For the SDAE, we tuned INLINEFORM4 , INLINEFORM5 on the validation set (see Section SECREF23 ). We also tried a variant (+embs) in which words are represented by (fixed) pre-trained embeddings.","FastSent The performance of SkipThought vectors shows that rich sentence semantics can be inferred from the content of adjacent sentences. The model could be said to exploit a type of sentence-level Distributional Hypothesis BIBREF26 , BIBREF27 . Nevertheless, like many deep neural language models, SkipThought is very slow to train (see Table TABREF14 ). FastSent is a simple additive (log-linear) sentence model designed to exploit the same signal, but at much lower computational expense. Given a BOW representation of some sentence in context, the model simply predicts adjacent sentences (also represented as BOW) .","More formally, FastSent learns a source INLINEFORM0 and target INLINEFORM1 embedding for each word in the model vocabulary. For a training example INLINEFORM2 of consecutive sentences, INLINEFORM3 is represented as the sum of its source embeddings INLINEFORM4 . The cost of the example is then simply: DISPLAYFORM0","where INLINEFORM0 is the softmax function.","We also experiment with a variant (+AE) in which the encoded (source) representation must predict its own words as target in addition to those of adjacent sentences. Thus in FastSent+AE, ( EQREF12 ) becomes DISPLAYFORM0","At test time the trained model (very quickly) encodes unseen word sequences into distributed representations with INLINEFORM0 ."]},{"idx":6,"heading":"Training and Model Selection","paragraphs":["Unless stated above, all models were trained on the Toronto Books Corpus, which has the inter-sentential coherence required for SkipThought and FastSent. The corpus consists of 70m ordered sentences from over 7,000 books.","Specifications of the models are shown in Table TABREF14 . The log-linear models (SkipGram, CBOW, ParagraphVec and FastSent) were trained for one epoch on one CPU core. The representation dimension INLINEFORM0 for these models was found after tuning INLINEFORM1 on the validation set. All other models were trained on one GPU. The S(D)AE models were trained for one epoch ( INLINEFORM3 days). The SkipThought model was trained for two weeks, covering just under one epoch. For CaptionRep and DictRep, performance was monitored on held-out training data and training was stopped after 24 hours after a plateau in cost. The NMT models were trained for 72 hours."]},{"idx":7,"heading":"Evaluating Sentence Representations","paragraphs":["In previous work, distributed representations of language were evaluated either by measuring the effect of adding representations as features in some classification task - supervised evaluation BIBREF3 , BIBREF28 , BIBREF13 - or by comparing with human relatedness judgements - unspervised evaluation BIBREF14 , BIBREF0 , BIBREF29 . The former setting reflects a scenario in which representations are used to inject general knowledge (sometimes considered as pre-training) into a supervised model. The latter pertains to applications in which the sentence representation space is used for direct comparisons, lookup or retrieval. Here, we apply and compare both evaluation paradigms."]},{"idx":8,"heading":"Supervised Evaluations","paragraphs":["Representations are applied to 6 sentence classification tasks: paraphrase identification (MSRP) BIBREF30 , movie review sentiment (MR) BIBREF31 , product reviews (CR) BIBREF32 , subjectivity classification (SUBJ) BIBREF33 , opinion polarity (MPQA) BIBREF34 and question type classification (TREC) BIBREF35 . We follow the procedure (and code) of kiros2015skip: a logistic regression classifier is trained on top of sentence representations, with 10-fold cross-validation used when a train-test split is not pre-defined."]},{"idx":9,"heading":"Unsupervised Evaluations","paragraphs":["We also measure how well representation spaces reflect human intuitions of the semantic sentence relatedness, by computing the cosine distance between vectors for the two sentences in each test pair, and correlating these distances with gold-standard human judgements. The SICK dataset BIBREF36 consists of 10,000 pairs of sentences and relatedness judgements. The STS 2014 dataset BIBREF37 consists of 3,750 pairs and ratings from six linguistic domains. Example ratings are shown in Table TABREF15 . All available pairs are used for testing apart from the 500 SICK `trial' pairs, which are held-out for tuning hyperparameters (representation size of log-linear models, and noise parameters in SDAE). The optimal settings on this task are then applied to both supervised and unsupervised evaluations."]},{"idx":10,"heading":"Results","paragraphs":["Performance of the models on the supervised evaluations (grouped according to the data required by their objective) is shown in Table TABREF20 . Overall, SkipThought vectors perform best on three of the six evaluations, the BOW DictRep model with pre-trained word embeddings performs best on two, and the SDAE on one. SDAEs perform notably well on the paraphrasing task, going beyond SkipThought by three percentage points and approaching state-of-the-art performance of models designed specifically for the task BIBREF38 . SDAE is also consistently better than SAE, which aligns with other findings that adding noise to AEs produces richer representations BIBREF22 .","Results on the unsupervised evaluations are shown in Table TABREF21 . The same DictRep model performs best on four of the six STS categories (and overall) and is joint-top performer on SICK. Of the models trained on raw text, simply adding CBOW word vectors works best on STS. The best performing raw text model on SICK is FastSent, which achieves almost identical performance to C-PHRASE's state-of-the-art performance for a distributed model BIBREF18 . Further, it uses less than a third of the training text and does not require access to (supervised) syntactic representations for training. Together, the results of FastSent on the unsupervised evaluations and SkipThought on the supervised benchmarks provide strong support for the sentence-level distributional hypothesis: the context in which a sentence occurs provides valuable information about its semantics.","Across both unsupervised and supervised evaluations, the BOW DictRep with pre-trained word embeddings exhibits by some margin the most consistent performance. Ths robust performance suggests that DictRep representations may be particularly valuable when the ultimate application is non-specific or unknown, and confirms that dictionary definitions (where available) can be a powerful resource for representation learning."]},{"idx":11,"heading":"Discussion","paragraphs":["Many additional conclusions can be drawn from the results in Tables TABREF20 and TABREF21 .","Different objectives yield different representations It may seem obvious, but the results confirm that different learning methods are preferable for different intended applications (and this variation appears greater than for word representations). For instance, it is perhaps unsurprising that SkipThought performs best on TREC because the labels in this dataset are determined by the language immediately following the represented question (i.e. the answer) BIBREF35 . Paraphrase detection, on the other hand, may be better served by a model that focused entirely on the content within a sentence, such as SDAEs. Similar variation can be observed in the unsupervised evaluations. For instance, the (multimodal) representations produced by the CaptionRep model do not perform particularly well apart from on the Image category of STS where they beat all other models, demonstrating a clear effect of the well-studied modality differences in representation learning BIBREF39 .","The nearest neighbours in Table TABREF24 give a more concrete sense of the representation spaces. One notable difference is between (AE-style) models whose semantics come from within-sentence relationships (CBOW, SDAE, DictRep, ParagraphVec) and SkipThought/FastSent, which exploit the context around sentences. In the former case, nearby sentences generally have a high proportion of words in common, whereas for the latter it is the general concepts and/or function of the sentence that is similar, and word overlap is often minimal. Indeed, this may be a more important trait of FastSent than the marginal improvement on the SICK task. Readers can compare the CBOW and FastSent spaces at http://45.55.60.98/.","Differences between supervised and unsupervised performance Many of the best performing models on the supervised evaluations do not perform well in the unsupervised setting. In the SkipThought, S(D)AE and NMT models, the cost is computed based on a non-linear decoding of the internal sentence representations, so, as also observed by BIBREF40 , the informative geometry of the representation space may not be reflected in a simple cosine distance. The log-linear models generally perform better in this unsupervised setting.","Differences in resource requirements As shown in Table TABREF14 , different models require different resources to train and use. This can limit their possible applications. For instance, while it was easy to make an online demo for fast querying of near neighbours in the CBOW and FastSent spaces, it was not practical for other models owing to memory footprint, encoding time and representation dimension.","The role of word order is unclear The average scores of models that are sensitive to word order (76.3) and of those that are not (76.6) are approximately the same across supervised evaluations. Across the unsupervised evaluations, however, BOW models score 0.55 on average compared with 0.42 for RNN-based (order sensitive) models. This seems at odds with the widely held view that word order plays an important role in determining the meaning of English sentences. One possibility is that order-critical sentences that cannot be disambiguated by a robust conceptual semantics (that could be encoded in distributed lexical representations) are in fact relatively rare. However, it is also plausible that current available evaluations do not adequately reflect order-dependent aspects of meaning (see below). This latter conjecture is supported by the comparatively strong performance of TFIDF BOW vectors, in which the effective lexical semantics are limited to simple relative frequencies.","The evaluations have limitations The internal consistency (Chronbach's INLINEFORM0 ) of all evaluations considered together is INLINEFORM1 (just above `acceptable'). Table TABREF25 shows that consistency is far higher (`excellent') when considering the supervised or unsupervised tasks as independent cohorts. This indicates that, with respect to common characteristics of sentence representations, the supervised and unsupervised benchmarks do indeed prioritise different properties. It is also interesting that, by this metric, the properties measured by MSRP and image-caption relatedness are the furthest removed from other evaluations in their respective cohorts.","While these consistency scores are a promising sign, they could also be symptomatic of a set of evaluations that are all limited in the same way. The inter-rater agreement is only reported for one of the 8 evaluations considered (MPQA, INLINEFORM0 BIBREF34 ), and for MR, SUBJ and TREC, each item is only rated by one or two annotators to maximise coverage. Table TABREF15 illustrates why this may be an issue for the unsupervised evaluations; the notion of sentential 'relatedness' seems very subjective. It should be emphasised, however, that the tasks considered in this study are all frequently used for evaluation, and, to our knowledge, there are no existing benchmarks that overcome these limitations."]},{"idx":12,"heading":"Conclusion","paragraphs":["Advances in deep learning algorithms, software and hardware mean that many architectures and objectives for learning distributed sentence representations from unlabelled data are now available to NLP researchers. We have presented the first (to our knowledge) systematic comparison of these methods. We showed notable variation in the performance of approaches across a range of evaluations. Among other conclusions, we found that the optimal approach depends critically on whether representations will be applied in supervised or unsupervised settings - in the latter case, fast, shallow BOW models can still achieve the best performance. Further, we proposed two new objectives, FastSent and Sequential Denoising Autoencoders, which perform particularly well on specific tasks (MSRP and SICK sentence relatedness respectively). If the application is unknown, however, the best all round choice may be DictRep: learning a mapping of pre-trained word embeddings from the word-phrase signal in dictionary definitions. While we have focused on models using naturally-occurring training data, in future work we will also consider supervised architectures (including convolutional, recursive and character-level models), potentially training them on multiple supervised tasks as an alternative way to induce the 'general knowledge' needed to give language technology the elusive human touch."]},{"idx":13,"heading":"Acknowledgments","paragraphs":["This work was supported by a Google Faculty Award to AK and FH and a Google European Doctoral Fellowship to FH. Thanks also to Marek Rei, Tamara Polajnar, Laural Rimell, Jamie Ryan Kiros and Piotr Bojanowski for helpful discussion and comments."]},{"idx":14,"heading":"Figures and Tables","paragraphs":["Table 1: Properties of models compared in this study OS: requires training corpus of sentences in order. R: requires structured resource for training. WO: encoder sensitive to word order. SD: dimension of sentence representation. WD: dimension of word representation. TR: approximate training time (hours) on the dataset in this paper. * indicates trained on GPU. TE: approximate time (s) taken to encode 0.5m sentences.","Table 2: Example sentence pairs and ‘similarity’ ratings from the unsupervised evaluations used in this study.","Table 3: Performance of sentence representation models on supervised evaluations (Section 3.1). Bold numbers indicate best performance in class. Underlined indicates best overall.","Table 4: Performance of sentence representation models (Spearman/Pearson correlations) on unsupervised (relatedness) evaluations (Section 3.2). Models are grouped according to training data as indicated in Table 3.","Table 6: Internal consistency (Chronbach’s α) among evaluations when individual benchmarks are left out of the (supervised or unsupervised) cohorts. Consistency rank within cohort is in parentheses (1 = most consistent with other evaluations)."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1602.07563","title":"Multilingual Twitter Sentiment Classification: The Role of Human Annotators","sections":[{"idx":0,"heading":"Abstract","paragraphs":["What are the limits of automated Twitter sentiment classification? We analyze a large set of manually labeled tweets in different languages, use them as training data, and construct automated classification models. It turns out that the quality of classification models depends much more on the quality and size of training data than on the type of the model trained. Experimental results indicate that there is no statistically significant difference between the performance of the top classification models. We quantify the quality of training data by applying various annotator agreement measures, and identify the weakest points of different datasets. We show that the model performance approaches the inter-annotator agreement when the size of the training set is sufficiently large. However, it is crucial to regularly monitor the self- and inter-annotator agreements since this improves the training datasets and consequently the model performance. Finally, we show that there is strong evidence that humans perceive the sentiment classes (negative, neutral, and positive) as ordered."]},{"idx":1,"heading":"Abstract","paragraphs":["What are the limits of automated Twitter sentiment classification? We analyze a large set of manually labeled tweets in different languages, use them as training data, and construct automated classification models. It turns out that the quality of classification models depends much more on the quality and size of training data than on the type of the model trained. Experimental results indicate that there is no statistically significant difference between the performance of the top classification models. We quantify the quality of training data by applying various annotator agreement measures, and identify the weakest points of different datasets. We show that the model performance approaches the inter-annotator agreement when the size of the training set is sufficiently large. However, it is crucial to regularly monitor the self- and inter-annotator agreements since this improves the training datasets and consequently the model performance. Finally, we show that there is strong evidence that humans perceive the sentiment classes (negative, neutral, and positive) as ordered."]},{"idx":2,"heading":"Introduction","paragraphs":["Sentiment analysis is a form of shallow semantic analysis of texts. Its goal is to extract opinions, emotions or attitudes towards different objects of interest BIBREF0 , BIBREF1 . For example, one might be interested in consumers opinion about products, voters attitude towards political parties, or investors expectations about stocks. From the first approaches in 2000s, sentiment analysis gained considerable attention with massive growth of the web and social media. Different forms of textual information are becoming easily accessible (e.g., news, blogs, reviews, Facebook comments, Twitter posts, etc.), and different approaches to sentiment analysis were developed.","There are two prevailing approaches to large-scale sentiment analysis: (i) lexicon-based and (ii) machine learning. In the first case, the sentiment in the text is computed from the set of sentiment-bearing words identified in the text. In the second case, a sentiment classification model is constructed first, from a large set of sentiment labeled texts, and then applied to the stream of unlabelled texts. The model has the form of a function that maps features extracted from the text into sentiment labels (which typically have discrete values: negative, neutral, or positive). In both approaches, one needs a considerable involvement of humans, at least initially. Humans have to label their perception of the sentiment expressed either in individual words or in short texts. This sentiment labeling is language-, domain- and often even topic-specific.","An example of a lexicon-based approach that involves a massive human sentiment labeling of words is described by Dodds et al. BIBREF2 . They collected around 5 million human sentiment assessments of 10,000 common words, each in 10 languages and labeled 50 times. Another well-known sentiment lexicon is SentiWordNet BIBREF3 , constructed semi-automatically for over 100,000 words, but limited to English only.","In this paper we analyze a set of over 1.6 million Twitter posts, in 13 European languages, labeled for sentiment by human annotators. The labeled tweets are used as training data to train sentiment classifiers for different languages. An overview of the state-of-the-art of Twitter sentiment analysis is given in BIBREF4 . A more recent overview of the lexicon-based and machine learning methods, and their combination, is in BIBREF5 .","We focus on the quantity and quality of the labeled tweets, and their impact on the performance of sentiment classifiers. The quality of the labeled tweets is estimated from the agreement between human annotators. The main hypothesis of the paper is that the annotators agreement provides an upper bound for the classifier performance.","There are several more specific research questions we address:","(1) Are the sentiment classes ordered?","(2) Which evaluation measures are appropriate to quantify and compare the labeled data quality and classifiers performance?","(3) How to estimate the quality of the training data?","(4) How to compare and select appropriate classifiers?","(5) What are acceptable levels of the annotators agreement?","(6) How many labeled Twitter posts are needed for training a sentiment classifier?","In the paper we present three lines of experiments and results. One is related to manual annotation of Twitter posts and estimations of their quality and dataset properties. Another is about training sentiment classifiers, their performance and comparisons. The third line compares the labeled data quality with the classifier performance and provides support for our main hypothesis.","The paper is organized as follows. In the Results and Discussion section we provide the main results on the comparison of the annotators agreement and classifiers performance. We briefly outline the main evaluation measure used and the datasets analyzed. The evaluation procedures and methods are just sketched, to facilitate the discussion of the results—all the details are in the Methods section. The main emphasis is on an in-depth analysis of the datasets. We consider their evolution through time, as new tweets get annotated, and how the performance of the classifiers varies with time. We also discuss the effects of different distributions of the training and application datasets.","Conclusions provide answers to the research questions addressed, and give short- and long-term directions of future research.","The Methods section provides all the details about the first two lines of experiments and results, specifically about the data, annotations, and sentiment classifiers. We define four evaluation measures, common in the fields of inter-rater agreement and machine learning. The measures are used to compute the self- and inter-annotator agreements for all the datasets. From these results we derive evidence that human annotators perceive the sentiment classes as ordered. We present the related work on methods used for the Twitter sentiment classification, and publicly available labeled datasets. We compare the performance of six selected classifiers by applying a standard statistical test. We give the necessary details of the evaluation procedure and the standard Twitter pre-processing steps.","In the following subsection we give an overview of the related work on automated sentiment classification of Twitter posts. We summarize the published labeled sets used for training the classification models, and the machine learning methods applied for training. Most of the related work is limited to English texts only."]},{"idx":3,"heading":"Results and Discussion","paragraphs":["In this paper we analyze a large set of sentiment labeled tweets. We assume a sentiment label takes one of three possible values: negative, neutral, or positive. The analysis sheds light on two aspects of the data: the quality of human labeling of the tweets, and the performance of the sentiment classification models constructed from the same data. The main idea behind this analysis is to use the same evaluation measures to estimate both, the quality of human annotations and the quality of classification models. We argue that the performance of a classification model is primarily limited by the quality of the labeled data. This, in turn, can be estimated by the agreement between the human annotators."]},{"idx":4,"heading":"The limits of performance","paragraphs":["Determining sentiment expressed in a tweet is not an easy task, and depends on subjective judgment of human annotators. Annotators often disagree between themselves, and even an individual is not always consistent with her/himself. There are several reasons for disagreements, such as: inherent difficulty of the task (e.g., estimating the “sentiment” about the future stock movement), different vocabularies used in different domains (e.g., financial markets vs. environmental issues), topic drift in time (e.g., events which abruptly shift the topic of discussions on Twitter), or simply a poor quality of the annotator's work. In the data we analyze, we observe all the above issues, try to identify them by computational means, and draw lessons how the annotation process should be conducted in the future.","During the manual sentiment labeling of tweets, a fraction of tweets (about 15%) was intentionally duplicated to be annotated twice, either by the same annotator or by two different annotators (see details in the Datasets subsection in Methods). From multiple annotations of the same annotator we compute the self-agreement, and from multiple annotations by different annotators we compute the inter-annotator agreement (abbreviated as inter-agreement). The confidence intervals for the agreements are estimated by bootstrapping BIBREF11 . The detailed results are in the Annotator agreements subsection in Methods. It turns out that the self-agreement is a good measure to identify low quality annotators, and that the inter-annotator agreement provides a good estimate of the objective difficulty of the task, unless it is too low.","To manually label over 1.6 million tweets requires a considerable effort. The purpose of this effort is to use the labeled data to built sentiment classification models for each of the 13 languages. A classification model can then be applied to unlabeled data in various application scenarios, as was the case with our four application datasets.","A classification model can be build by any suitable supervised machine learning method. To evaluate the model, a standard approach in machine learning is to use 10-fold cross-validation. The whole labeled set is partitioned into 10 folds, one is set apart for testing, and the remaining nine are used to train the model and evaluate it on the test fold. The process is repeated 10 times until each fold is used for testing exactly once. The reported evaluation results are the average of 10 tests, and the confidence intervals are estimated from standard deviations.","We constructed and evaluated six different classification models for each labeled language dataset. The results for the application datasets are extracted from the original papers. Our classifiers are all based on Support Vector Machines (SVM) BIBREF12 , and for reference we also constructed a Naive Bayes classifier BIBREF13 . Detailed results are in the Classification models performance subsection in Methods. When comparing the classifiers' performance with the Friedman-Nemenyi test BIBREF14 , BIBREF15 , it turns out that there is no statistically significant difference between most of them (see the Friedman-Nemenyi test subsection in Methods). For subsequent analyses and comparisons, we selected the TwoPlaneSVMbin classifier that is always in the group of top classifiers according to two most relevant evaluation measures.","The main results of this paper are summarized in Fig FIGREF8 . It shows a comparison of the self-agreement, the inter-annotator agreement, and the TwoPlaneSVMbin classifier performance, for the 13 language datasets and the four application datasets.","The self-agreement for most of the datasets is above 0.6. The exceptions, Albanian and Spanish, indicate low quality annotators which should be eliminated from further considerations. In the applications corpus, the Emojis dataset is the only one with the self-agreement lower than the inter-annotator agreement, due to a high number of low quality Spanish annotations included. The other three application datasets have relatively high self-agreement (0.7–0.9, due to more carefully selected annotators), and higher variability (due to a lower number of tweets annotated twice, 2–4% only).","The inter-annotator agreement varies a lot, and is always lower than the self-agreement, except for Emojis. The high inter-annotator agreement for Facebook(it) is consistent with the high self-agreement. Values below 0.2 (Albanian and Spanish) indicate low quality annotators, consistent with the low self-agreement. Values in the range between 0.3–0.4 (Ser/Cro/Bos, Bulgarian, and German) indicate a problem with the annotation process, and are discussed in more detail in the next subsection.","The classifier performance is typically in the range between 0.4–0.6. Notable exceptions are Albanian and Spanish, with the performance barely above random, but very close to the inter-annotator agreement. More interesting are the datasets with a relatively low performance, around 0.4, that cannot be explained by low quality annotations alone: Ser/Cro/Bos, Bulgarian, German, Portuguese, and Environment. They are analyzed in the next subsections.","The main hypothesis of this paper is that the inter-annotator agreement approximates an upper bound for a classifier performance. In Fig FIGREF8 we observe three such cases where the classifier performance, in the range 0.4–0.6, approaches its limit: Polish, Slovenian, and DJIA30. There are also three cases where there still appears a gap between the classifier performance and the inter-annotator agreement: English, Facebook(it), and Environment. In order to confirm the hypothesis, we analyze the evolution of the classifiers performance through time and check if the performance is still improving or was the plateau already reached. This is not always possible: There are datasets where only one annotator was engaged and for which there is no inter-annotator agreement (Russian, Swedish, Hungarian, Slovak, and Portuguese). For them we can only draw analogies with the multiply annotated datasets and speculate about the conclusions.","In the next two subsection we first analyze the language datasets, and then the four application datasets."]},{"idx":5,"heading":"Language datasets analyses","paragraphs":["To label the 1.6 million tweets in the 13 languages, 83 native speakers were engaged, and an estimated effort of 38 person-months was spent. Can one reduce the efforts and focus them on more problematic datasets instead? It seems, for example, that the annotation of over 200,000 Polish tweets was an overkill. Worse, the annotation of over 250,000 Spanish tweets was largely a waste of efforts, due to the poor annotation quality.","We perform a post-hoc analysis of the 13 language datasets by measuring the performance of the sentiment classifiers through time. We emulate the evolution of the performance by feeding increasingly large labeled sets into the classifier training process. The labeled sets are ordered by the post time of the tweets, so one can detect potential topic shifts during the Twitter discussions. At each stage, the labeled set is increased by 10,000 tweets, and the set accumulated so far is used for training and testing the classifier. After each stage, the evaluation by 10-fold cross-validation is performed and the results are reported in the following charts. The final stage, when all the labeled sets are exhausted, corresponds to the results reported in Fig FIGREF8 . In subsequent figures, the x-axis denotes labeled sets increases by 10,000 tweets, the y-axis denotes the TwoPlaneSVMbin classifier performance measured by INLINEFORM0 , and the error bars are the 95% confidence intervals estimated from 10-fold cross-validations. The inter-annotator agreement is represented by a blue line—it is constant and is computed from all the available data.","We identify five cases, characterized by different relations between the classifier performance and the inter-annotator agreement: (i) a performance gap still exists, (ii) a performance limit is approached, (iii) low inter-annotator agreement, (iv) topic shift, and (v) very low annotation quality.","Fig FIGREF11 (chart on the left) shows the evolution of the English classifier performance, as it is fed increasingly large training sets. On top (in blue) is the inter-annotator agreement line ( INLINEFORM0 = 0.613). The classifier's INLINEFORM1 is increasing from the initial 0.422 to 0.516, but is still considerably below the inter-annotator agreement. Despite the relatively large training set (around 90,000 labeled tweets) there is still a performance gap and even more annotations are needed to approach the inter-annotator agreement.","We observe a similar pattern with the Russian (Fig FIGREF11 , chart on the right) and Slovak datasets (not shown). The inter-annotator agreement is unknown, but the classifier's performance is still increasing from the initial INLINEFORM0 of 0.403 to 0.490 for Russian, and from the initial 0.408 to 0.460 for Slovak. The size of the labeled sets for Russian is around 90,000, for Slovak around 60,000, and we argue that more training data is needed to further improve the performance.","A different pattern from the above can be observed in Fig FIGREF13 for the Polish dataset. After a slow improvement of the classifier's performance, the peak is reached at around 150,000 labeled tweets, and afterwards the performance remains stable and is even slightly decreasing. The maximum INLINEFORM0 is 0.536, close to the inter-annotator agreement of 0.571. At the same point, at 150,000 tweets, another performance measure, INLINEFORM1 , also peaks at its maximum value, even above the corresponding inter-annotator agreement. These results suggest that beyond a certain point, when the classifier's performance is “close enough” to the inter-annotator agreement, it does not pay off to further label tweets by sentiment. This is valid, however, only until a considerably new topic occurs.","Similar conclusions can be drawn for the Slovenian dataset (Fig FIGREF14 , chart on the left). The classifier's performance reaches its peak earlier, at 70,000 tweets, with the maximum INLINEFORM0 of 0.459, as well as the maximum INLINEFORM1 . INLINEFORM2 is close to the inter-annotator agreement of 0.485, and INLINEFORM3 even exceeds the corresponding agreement. However, notice that the inter-annotator agreement for Slovenian is almost 10% points lower than for Polish.","We observe a similar pattern for the Bulgarian dataset (Fig FIGREF14 , chart on the right). The classifier's peak performance is reached even earlier, at 40,000 tweets ( INLINEFORM0 is 0.378), but the inter-annotator agreement is also considerably lower, more than 10% points below the Slovenian ( INLINEFORM1 is 0.367). In such cases, when the inter-annotator agreement is “too low” (our estimate is when INLINEFORM2 INLINEFORM3 ), the inter-annotator agreement is a poor estimator of the difficulty of the task, and should not be used as a performance approximation. Instead, one could analyze the reasons for the disagreements, as we do with cases in the following paragraphs.","The inter-annotator agreement for the German dataset is low, INLINEFORM0 is 0.344. The classifier's performance is higher already with the initial small datasets, and soon starts dropping (Fig FIGREF16 , chart on the left). It turns out that over 90% of the German tweets were labeled by two annotators only, dubbed annotator A and B. The annotation quality of the two annotators is very different, the self-agreement INLINEFORM1 for the annotator A is 0.590, and for the annotator B is 0.760. We consider the German tweets labeled by A and B separately (Fig FIGREF16 , charts in the middle and on the right). The lower quality A dataset reaches its maximum at 30,000 tweets, while the performance of the higher quality B dataset is still increasing. There was also a relatively high disagreement between the two annotators which resulted in a low classifier's performance. A conclusions drawn from this dataset, as well as from the Bulgarian, is that one should constantly monitor the self- and inter-annotator agreements, and promptly notify the annotators as soon as the agreements drop too low.","Fig FIGREF17 gives the results on the joint Ser/Cro/Bos dataset. We observe a low inter-annotator agreement ( INLINEFORM0 is 0.329) and a high variability of the classifier's performance.","The three languages, Serbian, Croatian, and Bosnian, are very similar and difficult to distinguish in short Twitter posts. However, we argue that the reason for poor performance is not in mixing the three languages, but in different annotation quality. Serbian (73,783 tweets) was annotated by 11 annotators, where two of them account for over 40% of the annotations. All the inter-annotator agreement measures come from the Serbian only (1,880 tweets annotated twice by different annotators, INLINEFORM0 is 0.329), and there are very few tweets annotated twice by the same annotator (182 tweets only, INLINEFORM1 for the self-agreement is 0.205). In contrast, all the Croatian and Bosnian tweets were annotated by a single annotator, and we have reliable self-agreement estimates. There are 97,291 Croatian tweets, 13,290 annotated twice, and the self-agreement INLINEFORM2 is 0.781. There are 44,583 Bosnian tweets, 6,519 annotated twice, and the self-agreement INLINEFORM3 is 0.722. We can conclude that the annotation quality of the Croatian and Bosnian tweets is considerably higher than of the Serbian. If we construct separate sentiment classifiers for each language we observe very different performance (see Fig FIGREF18 ). The Serbian classifier reaches the inter-annotator agreement (albeit low) at 70,000 tweets. The Croatian classifier has much higher performance, and reaches it maximum at 50,000 tweets ( INLINEFORM4 is 0.590). The performance of the Bosnian classifier is also higher, and is still increasing at 40,000 tweets ( INLINEFORM5 is 0.494). The individual classifiers are “well-behaved” in contrast to the joint Ser/Cro/Bos model in Fig FIGREF17 . In retrospect, we can conclude that datasets with no overlapping annotations and different annotation quality are better not merged.","There is no inter-annotator agreement for the Portuguese dataset because only one annotator was engaged. However, the classifier shows interesting performance variability (Fig FIGREF20 ). After an initial peak is reached at 50,000 tweets ( INLINEFORM0 is 0.394), there is a considerable drop and a very high variability of performance. Inspection of the tweets (the set of 10,000 tweets added to the first 50,000 tweets at stage 6) revealed that at the beginning of November 2013, the Portuguese government approved additional austerity measures, affecting mainly public sector, to avoid the second international bailout. This provoked a flood of negative reactions on social media, in particular on Twitter, and a considerable shift of focus and sentiment of Twitter discussions. The classification model could not react immediately to the topic shift, and it took additional 100,000 tweets to accommodate the new topics, and the model to approach the peak performance ( INLINEFORM1 is 0.391 for the complete dataset).","What happens with the classifier's performance when the annotation quality is low? Fig FIGREF22 shows the evolution of performance for the Spanish dataset. We observe high variability and consistent drop in performance. Most (over 95%) of the Spanish tweets were annotated by one annotator, and out of them, 40,116 tweets were annotated twice. Therefore we have a reliable estimate of the low quality of her/his annotations since the self-agreement INLINEFORM0 is only 0.244. 2,194 tweets were annotated twice by two annotators and, not surprisingly, the inter-annotator agreement is ever lower, INLINEFORM1 is 0.120.","We observe a similar performance drop for the Albanian dataset (not shown). The main annotator (who annotated over 22% of the Albanian tweets) has self-agreement INLINEFORM0 only 0.269 (computed from 1,963 tweets annotated twice). The inter-annotator agreement INLINEFORM1 is only 0.126.","Such poorly labeled data is useless for training sentiment classifiers. However, the lesson learned is that the annotators should be monitored throughout the annotation process, that the low quality annotators (identified by a low self-agreement) should be excluded, and that the low inter-annotator agreements should be promptly investigated."]},{"idx":6,"heading":"Application datasets analyses","paragraphs":["The purpose of building sentiment classification models is to apply them in particular domains, e.g., to monitor elections or to predict stock prices. The models are build from labeled data (where the sentiment is given) and applied to unlabeled data (where the sentiment is to be predicted). The models are also evaluated on the labeled data (typically by 10-fold cross-validation) and the estimated performance can be extended to the application if the labeled data is representative, i.e., drawn from the same distribution as the application data. In the context of Twitter sentiment classification this means that the labeled tweets have to be not only language-, but also domain-specific.","In the previous subsection we analyzed the classifiers performance on the labeled datasets and in relation to the annotator agreements. The potential improvements can be achieved by providing additional training data, by improving the inter-annotator agreements, and by excluding low quality annotators. In this subsection we also consider the relation between the training and application dataset distributions.","There are four applications where we already applied and published Twitter sentiment classification to different domains. Details about the sizes and distributions of the labeled and application datasets are in the Datasets subsection in Methods. Sentiment distribution is captured by the sentiment score which is computed as the mean of a discrete probability distribution—details are in BIBREF10 . Here we briefly analyze and suggest possible improvements with reference to the results in Fig FIGREF8 .","This is the only domain that is not limited to Twitter, but where the same sentiment classification methodology was applied to Facebook comments, in Italian. There was over 1 million Facebook comments collected, and a sample of about 20,000 was labeled for sentiment. The sentiment distribution in both sets is similar. The self-agreement and inter-annotator agreement are both high, however, there is a gap between the inter-annotator agreement ( INLINEFORM0 is 0.673) and the classifier's performance ( INLINEFORM1 is 0.562). Based on the lessons from the language datasets, we speculate that 20,000 training examples is not enough, and that additional Facebook comments have to be labeled to approach the inter-annotator agreement.","This domain deals with English tweets, but very specific for financial markets. The sentiment labeling requires considerable domain knowledge about specific financial terminology. There were over 1.5 million tweets about the Dow Jones stocks collected, and a sample of about 100,000 was annotated for sentiment. The sentiment distribution in both sets is very similar. The annotators self-agreement is high, but the inter-annotator agreement is relatively low ( INLINEFORM0 is 0.438), and the classifier even slightly exceeds it. Also, in the period from June 2013 to September 2014, a relatively small fraction of tweets was annotated twice (5,934), so the agreement estimates are less reliable. These considerations were taken into account in the subsequent period: from June 2014 to May 2015 altogether 19,720 tweets were annotated twice, and the inter-annotator agreement improved for 10% points (new INLINEFORM1 is 0.482).","This domain deals with sentiment leaning towards various environmental issues (like climate change, fossil fuels, fracking, etc.)—not so well-defined problem. Consequently, the self-agreement and inter-annotator agreement are relatively low in comparison to the Facebook(it) dataset. Still, there is a gap between the inter-annotator agreement ( INLINEFORM0 is 0.510) and the classifier's performance ( INLINEFORM1 is 0.397). The training set consists of only about 20,000 labeled tweets, and in analogy to the language datasets and Facebook(it) we conclude that additional tweets have to be labeled to improve the classifier performance.","However, there is another issue. There were altogether over 3 million tweets collected, and sentiment distribution in the training set is considerably different from the application set (sentiment scores are INLINEFORM0 and INLINEFORM1 , respectively; see Fig FIGREF27 ). The sampling was done just in the initial phases of the Twitter acquisition and is not representative of the whole application dataset.","We conducted an additional experiment to demonstrate the effects of different training and application sets. We applied the general English language sentiment classification model from the previous subsection, trained on all 90,000 English tweets, to the labeled environmental tweets. The classifier's performance ( INLINEFORM0 is 0.243) is considerably lower in comparison to the environment-specific model ( INLINEFORM1 is 0.397) which was trained on only 20,000 domain-specific tweets. The same holds for the INLINEFORM2 measure. Detailed evaluation results are in the Classification models performance subsection in Methods. This result confirms our thesis that Twitter sentiment classification is sensitive to domain of application and that sentiment labeling has to be domain-specific.","Note also that the general English classifier has higher accuracy ( INLINEFORM0 is 0.604) than the environment-specific model ( INLINEFORM1 is 0.556). Our conclusion is that this is a clear indication that accuracy is a misleading evaluation measure for the ordered three-class sentiment classification problem.","There is no automated sentiment classification with the Emojis dataset. From the 13 language datasets which consist in total of over 1.6 labeled tweets, we selected only the tweets that contain emojis, about 70,000 in total. The goal was to attribute the sentiment to emojis, based on the sentiment of all the tweets in which they occur. Fig FIGREF8 shows that Emojis is the only dataset where the self-agreement ( INLINEFORM0 is 0.544) is lower than the inter-annotator agreement ( INLINEFORM1 is 0.597). The reason for this anomaly is a large share of Spanish tweets with emojis (about 20,000) that have very low self-agreement ( INLINEFORM2 is 0.245). If we remove them from the Emojis set, the self-agreement increases considerably (new INLINEFORM3 is 0.720), while the inter-annotators agreement remains almost unchanged (new INLINEFORM4 is 0.598). This reconfirms our conclusion that low quality annotators have to be excluded and their annotations removed from the datasets."]},{"idx":7,"heading":"Conclusions","paragraphs":["We present an analysis of over 1.6 million sentiment annotated Twitter posts, by far the largest set made publicly available until now. The labeled datasets are used to train sentiment classification models, and our analysis focuses on four main aspects: quality, quantity and sampling of the training data, and performance of the classifiers. Our main conclusion is that the choice of a particular classifier type is not so important, but that the training data has a major impact on the results.","There are several specific research questions we address:","(1) What is the nature and proper formalization of the sentiment classification problem, in particular, are the sentiment values ordered or not? We show that there is strong evidence that the sentiment values, negative, neutral, and positive, are perceived as ordered by human annotators (see subsection on Ordering of sentiment values in Methods).","(2) Which evaluation measures should be used to properly quantify the data quality and classifiers performance? In all the experiment, we compute values for four evaluation measures ( INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , and INLINEFORM3 ). Since there is evidence that sentiment values are ordered, INLINEFORM4 and INLINEFORM5 are the most appropriate as they take the ordering into account.","(3) How to estimate the quality of the training data? We propose to invest an extra effort to label a portion of tweets twice, and then to compute the annotator self-agreement and the inter-annotator agreement. The self-agreement yields a useful indication when to exclude low quality annotators, and the inter-annotator agreement approximates an upper bound on the performance of sentiment classifiers.","(4) How to select the most appropriate classifier? Our results show that there are no statistically significant differences between the top classifiers. As a consequence, one should better direct the efforts into higher training data quality.","(5) What are acceptable levels of annotators agreement? On the basis of the 17 datasets analyzed, we propose the following rule-of-thumb: for self-agreement, INLINEFORM0 INLINEFORM1 , and for the inter-annotator agreement, INLINEFORM2 INLINEFORM3 .","(6) How many posts should be labeled with sentiment for training? We cannot provide conclusive answers here. It seems that 20,000 high-quality annotations already provide reasonable performance. The peak performance depends on the inter-annotator agreement and we estimate that around 100,000 annotations are needed. However, more important than sheer quantity is the quality, and domain- and topic-specific coverage of the posts, as demonstrated on several use-cases.","This gives the following directions for the short-term future work. The annotation process has to be redesigned to allow for systematic monitoring of the annotation quality. In particular, more than one annotator per language/domain has to be engaged. We propose an increased overhead of posts to be labeled twice, from 15% to 20%, both by individuals as well as by two different annotators. The posts to be labeled multiple times could be based on their “importance” as measured by their retweet count BIBREF9 , for example. The self- and the inter-annotator agreements have to be continuously monitored and warnings issued when they drop below the selected thresholds. Extreme disagreements (as measured by INLINEFORM0 ) should be promptly directed to a “master” annotator who has to resolve the disagreement and issue a proper annotation together with a brief guideline. After each batch of a few thousand annotations, a classification model should be trained and its performance evaluated. This would help in monitoring the progress towards the inter-annotator agreement as well as in detecting possible abrupt topic shifts.","There is a number of open research questions to be addressed. One is how to combine the lexicon-based and machine learning approaches to sentiment classification. In BIBREF5 , authors already showed that the combination of both outperforms the individual approaches. However, sentiment lexicons are rarely available for languages other than English and require considerable efforts to construct. For several languages, one could use the data published by Dodds et al. BIBREF2 . For the languages covered in this study, one can construct a basic sentiment lexicon from the annotated tweets, in the analogy to derivation of the emoji sentiment lexicon BIBREF10 .","Another research direction, with the potential of considerable performance improvements, is the construction and selection of informative features from short Twitter posts. In this study we apply a number of standard text pre-processing steps to extract just the textual features and eliminate noise in tweets. However, there is a lot of additional information on Twitter to be exploited. For example, the importance of tweets (estimated by the retweet count, for example), the influence and reliability of Twitter users (estimated by their followers, retweets, and correlations to the real-world events), and the network features (e.g., neighbourhood and centrality) that can be attributed to the users, and indirectly to their tweets. We expect that proper considerations of the broader context in which the tweets are posted can provide for a major leap in quality and predictive potential of the Twitter sentiment classifiers.","Finally, since the analysis of opinions expressed in social media is an active and evolving research area, we plan to keep up with the newest trends, such as performing entity-based sentiment analysis BIBREF16 , applying deep learning techniques BIBREF17 , BIBREF18 , BIBREF19 , analyzing figurative language (e.g., irony or sarcasm) BIBREF20 , and detecting different types of emotions (e.g., joy, sadness or anger) BIBREF21 . The most interesting direction seems to be a shift from the basic sentiment categories (negative, neutral, and positive) of the whole tweet, to the finer-grained emotions about a discussed entity or topic."]},{"idx":8,"heading":"Ethics statement","paragraphs":["The tweets were collected through the public Twitter API and are subject to the Twitter terms and conditions. The human annotators were engaged for the purpose of sentiment labeling, and were aware that their annotations will be used to construct the sentiment classification models, and to estimate the annotator self-agreement and the inter-annotator agreement."]},{"idx":9,"heading":"Datasets","paragraphs":["In this study we analyze two corpora of data (see Table TABREF30 ). The first corpus is a collection of tweets, in 13 European languages, posted between April 2013 and February 2015. The tweets, except English, were collected during a joint project with Gama System (http://www.gama-system.si), using their PerceptionAnalytics platform (http://www.perceptionanalytics.net). The tweets were acquired through Twitter Search API, by specifying the geolocations of the largest cities. For English tweets, we used Twitter Streaming API (a random sample of 1% of all the public tweets), and filtered out the English posts.","83 native speakers (except for English) were engaged to manually label with sentiment over 1.6 million of the collected tweets. The annotation process was supported by the Goldfinch platform (provided by Sowa Labs, http://www.sowalabs.com), designed specifically for sentiment annotation of short texts (such as Twitter posts, Facebook comments, etc.). The annotators were instructed to label each tweet as either negative, neutral, or positive, by estimating the emotional attitude of the user who posted the tweet. Tweets that were skipped or excluded are not considered in this study.","The second corpus of data are four application datasets, used in different application scenarios and already published BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 .","The datasets in Table TABREF30 are used to analyze the annotator agreements, and to build the sentiment classification models. The classification models build from three out of four application datasets were actually applied to much larger sets of unlabeled data, to predict the sentiment. Details are in Table TABREF31 . For each of the three application domains we also show the difference between the application and training phase in terms of the sentiment score (the mean of a discrete probability distribution, see BIBREF10 for details). For the Emojis dataset, no sentiment classification model was trained—the tweets with emojis were just extracted from the above corpus of 13 language datasets.","Table TABREF32 gives the details of the number of posts annotated twice, by the same annotator or by two different annotators.","The 13 language datasets are publicly available for further analyses. Actually, our analysis reveales that it is better to partition the Ser/Cro/Bos dataset into the three constituent languages, therefore we provide the sentiment annotation data for the 15 languages. The data is available as 15 language files, in the csv format, in a public language resource repository clarin.si at http://hdl.handle.net/11356/1054. For each language and for each labeled tweet, there is the tweet ID (as provided and required by Twitter), the sentiment label (negative, neutral, or positive), and the annotator ID (anonymized). From this data, one can compute the annotator agreement measures, construct the “gold standard” training data, and train the classifiers for different languages."]},{"idx":10,"heading":"Evaluation measures","paragraphs":["In general, the agreement can be estimated between any two methods of generating data. One of the main ideas of this work is to use the same measures to estimate the agreement between the human annotators as well as the agreement between the results of automated classification and the “gold standard”. There are different measures of agreement, and to get robust estimates we apply four well-known measures from the fields of inter-rater agreement and machine learning.","Krippendorff's Alpha-reliability ( INLINEFORM0 ) BIBREF6 is a generalization of several specialized agreement measures. It works for any number of annotators, and is applicable to different variable types and metrics (e.g., nominal, ordered, interval, etc.). INLINEFORM1 is defined as follows: INLINEFORM2","where INLINEFORM0 is the observed disagreement between annotators, and INLINEFORM1 is a disagreement, expected by chance. When annotators agree perfectly, INLINEFORM2 INLINEFORM3 , and when the level of agreement equals the agreement by chance, INLINEFORM4 INLINEFORM5 . The two disagreement measures are defined as follows: INLINEFORM6","INLINEFORM0","The arguments, INLINEFORM0 , and INLINEFORM1 , refer to the frequencies in a coincidence matrix, defined below. INLINEFORM2 is a difference function between the values of INLINEFORM3 and INLINEFORM4 , and depends on the metric properties of the variable. INLINEFORM5 (and INLINEFORM6 ) is a discrete sentiment variable with three possible values: negative ( INLINEFORM7 ), neutral (0), or positive (+). We consider two options: either the sentiment variable INLINEFORM8 is nominal or ordered. This gives rise to two instance of INLINEFORM9 , INLINEFORM10 (nominal, when INLINEFORM11 is unordered) and INLINEFORM12 (interval, when INLINEFORM13 is ordered), corresponding to two difference functions INLINEFORM14 : INLINEFORM15","INLINEFORM0","Note that in the case of the interval difference function, INLINEFORM0 assigns a disagreement of 1 between the neutral and the negative or positive sentiment, and a disagreement of 2 between the extremes, i.e., the negative and positive sentiment. The corresponding disagreements INLINEFORM1 and INLINEFORM2 between the extreme classes are then four times larger than between the neighbouring classes.","A coincidence matrix tabulates all pairable values of INLINEFORM0 from two annotators into a INLINEFORM1 -by- INLINEFORM2 square matrix, where INLINEFORM3 is the number of possible values of INLINEFORM4 . In the case of sentiment annotations, we have a 3-by-3 coincidence matrix. The diagonal contains all the perfect matches, and the matrix is symmetrical around the diagonal. A coincidence matrix has the following general form: INLINEFORM5","In our case, INLINEFORM0 and INLINEFORM1 range over the three possible sentiment values. In a coincidence matrix, each labeled unit is entered twice, once as a INLINEFORM2 pair, and once as a INLINEFORM3 pair. INLINEFORM4 is the number of units labeled by the values INLINEFORM5 and INLINEFORM6 by different annotators, INLINEFORM7 and INLINEFORM8 are the totals for each value, and INLINEFORM9 is the grand total.","The computed values of INLINEFORM0 are subject to sampling variability, determined by an unknown sampling distribution. The sampling distribution can be approximated by bootstrapping BIBREF11 . In our case, we set the number of bootstrap samples to 1,000, and estimate the 95% confidence interval of true INLINEFORM1 .","F score ( INLINEFORM0 ) is an instance of a well-known effectiveness measure in information retrieval BIBREF22 . We use an instance specifically designed to evaluate the 3-class sentiment classifiers BIBREF23 . INLINEFORM1 is defined as follows: INLINEFORM2","INLINEFORM0 implicitly takes into account the ordering of sentiment values, by considering only the negative INLINEFORM1 and positive INLINEFORM2 labels. The middle, neutral, label is taken into account only indirectly. In general, INLINEFORM3 is a harmonic mean of precision and recall for class INLINEFORM4 . In the case of a coincidence matrix, which is symmetric, the `precision' and `recall' are equal, and thus INLINEFORM5 degenerates into: INLINEFORM6","In terms of the annotator agreement, INLINEFORM0 is the fraction of equally labeled tweets out of all the tweets with label INLINEFORM1 .","Accuracy ( INLINEFORM0 ) is a common, and the simplest, measure of performance of the model which measures the agreement between the model and the “gold standard”. INLINEFORM1 is defined in terms of the observed disagreement INLINEFORM2 : INLINEFORM3","INLINEFORM0 is simply the fraction of the diagonal elements of the coincidence matrix. Note that it does not account for the (dis)agreement by chance, nor for the ordering of the sentiment values.","Accuracy within 1 ( INLINEFORM0 ) is a special case of accuracy within n BIBREF24 . It assumes ordered classes and extends the range of predictions considered correct to the INLINEFORM1 neighbouring class values. In our case, INLINEFORM2 considers as incorrect only misclassifications from negative to positive and vice-versa: INLINEFORM3","Note that it is easy to maximize INLINEFORM0 by simply classifying all the examples as neutral; then INLINEFORM1 INLINEFORM2 .","The four agreement measures are always computed from the same coincidence matrix. In the case of the annotator agreements, the coincidence matrix is formed from the pairs of sentiment labels assigned to a tweet by different annotators (or the same when she/he annotated the tweet several times). In the case of a classification model, an entry in the coincidence matrix is a pair of labels, one from the model prediction, and the other from the “gold standard”. Experiments show that a typical ordering of the agreement results is: INLINEFORM0 INLINEFORM1 INLINEFORM2 INLINEFORM3 INLINEFORM4 INLINEFORM5 INLINEFORM6 ."]},{"idx":11,"heading":"The annotator agreements","paragraphs":["Table TABREF35 gives the results of the annotator agreements in terms of the four evaluation measures. The self-agreement is computed from the tweets annotated twice by the same annotator, and the inter-annotator agreement from the tweets annotated twice by two different annotators, where possible. The 95% confidence intervals for INLINEFORM0 are computed from 1,000 bootstrap samples.","Note that the Albanian and Spanish datasets have very low INLINEFORM0 agreement values. All the results for INLINEFORM1 , reported here and throughout the paper, refer to the INLINEFORM2 instance, for the reasons outlined in the next subsection."]},{"idx":12,"heading":"Ordering of sentiment values","paragraphs":["Should the sentiment classes negative ( INLINEFORM0 ), neutral (0), and positive (+) be treated as nominal (categorical, unordered) or ordered? One can use the agreement measures to estimate how are the three classes perceived by the human annotators.","First, lets compare the agreements in terms of two variants of INLINEFORM0 : INLINEFORM1 (interval) and INLINEFORM2 (nominal). The difference between the two measures is that INLINEFORM3 assigns four times higher cost to extreme disagreements (between the negative and positive classes) than INLINEFORM4 . A measure which yields higher agreements hints at the nature of sentiment class ordering as perceived by humans. The results in Table TABREF37 , column two, show that INLINEFORM5 always yields higher agreement than INLINEFORM6 , except for Spanish. We compute the average relative agreement gains by ignoring the Albanian and Spanish datasets (which have poor annotation quality), and Emojis (which are already subsumed by the 13 language datasets). We observe that the average agreement is 18% higher with INLINEFORM7 than with INLINEFORM8 . This gives a strong indication that the sentiment classes are perceived as ordered by the annotators.","Second, we can use the agreement as a proxy to measure the “distance” between the sentiment classes. Lets assume that the difficulty of distinguishing between the extreme classes ( INLINEFORM0 , +), as measured by INLINEFORM1 , is normalized to 1. If it is more difficult to distinguish between the neutral (0) and each extreme ( INLINEFORM2 or +) then the normalized agreement will be lower than 1, otherwise it will be greater than 1. The results in Table TABREF37 , columns three and four, indicate that for almost all the datasets the normalized agreement is lower than 1. The only exceptions are Slovak and Spanish. If we ignore the Albanian, Spanish, and Emojis we observe the following average differences: (i) it is 27% ( INLINEFORM3 ) more difficult to distinguish between the negative ( INLINEFORM4 ) and neutral (0) than between the negative ( INLINEFORM5 ) and positive (+); and (ii) it is 35% ( INLINEFORM6 ) more difficult to distinguish between the positive (+) and neutral (0) than between the positive (+) and negative ( INLINEFORM7 ).","The above results support our hypothesis that the sentiment values are ordered: negative INLINEFORM0 neutral INLINEFORM1 positive. This has an implication on the selection of an appropriate performance measure and a classification model. The performance measure should take the class ordering into account, therefore our selection of INLINEFORM2 over INLINEFORM3 is justified. In this respect, INLINEFORM4 would also be appropriate, and it actually shows high correlation to INLINEFORM5 . The choice of an appropriate classification model is discussed in the next two subsections."]},{"idx":13,"heading":"Related sentiment classification approaches","paragraphs":["In this subsection we give an overview of the related work on automated sentiment classification of Twitter posts. We summarize the published labeled sets used for training the classification models, and the machine learning methods applied for training. Most of the related work is limited to English texts only.","To train a sentiment classifier, one needs a fairly large training dataset of tweets already labeled with sentiment. One can rely on a proxy, e.g., emoticons used in the tweets to determine the intended sentiment BIBREF25 , however, high quality labeling requires engagement of human annotators.","There exist several publicly available and manually labeled Twitter datasets. They vary in the number of examples from several hundreds to several thousands, but to the best of our knowledge, none exceeds 20,000 entries. Saif et al. BIBREF16 describe eight Twitter sentiment datasets and also introduce a new one which contains separate sentiment labels for tweets and entities. Rosenthal et al. BIBREF26 provide statistics for several of the 2013–2015 SemEval datasets. Haldenwang and Vornberger BIBREF27 present a publicly available collection of Twitter posts, which were labeled not only with the positive or negative sentiment, but also as uncertain or spam. Finally, several Twitter sentiment datasets are publicly available in CrowdFlower's “Data for Everyone” collection.","There are several supervised machine learning algorithms suitable to train sentiment classifiers from sentiment labeled tweets. For example, in the SemEval-2015 competition, for the task on Sentiment Analysis on Twitter BIBREF26 , the most often used algorithms are Support Vector Machines (SVM), Maximum Entropy, Conditional Random Fields, and linear regression. In other cases, frequently used are also Naive Bayes, k-Nearest-Neighbor, and even Decision Trees. In the following we cite several relevant papers, and report, where available, the comparison in performance between the algorithms used.","Go et al. BIBREF25 employ the keyword-based approach, Naive Bayes, Maximum Entropy, and SVM, and show that the best performing algorithm is Maximum Entropy. The authors in BIBREF28 show that Maximum Entropy outperforms Naive Bayes. In contrast, the authors in BIBREF29 report that Naive Bayes performs considerably better than Maximum Entropy. Pak and Paroubek BIBREF30 show that Naive Bayes outperforms the SVM and Conditional Random Fields algorithms. Asiaee et al. BIBREF31 employ a dictionary learning approach, weighted SVM, k-Nearest-Neighbor, and Naive Bayes—Naive Bayes and its weighted variant are among the best performing algorithms. Saif et al. BIBREF32 employ Naive Bayes for predicting sentiment in tweets.","Often, SVM is shown as the best performing classifier for Twitter sentiment. For example, BIBREF33 test several algorithms implemented in Weka, and SVM performed best. The authors in BIBREF5 test the Naive Bayes, Decision Trees, and SVM algorithms, and find that the best performing algorithm is SVM. Preliminary results reported in BIBREF23 show that linear SVM yields better performance than the Maximum Entropy classifier. Jiang et al. BIBREF34 employ SVM models for subjectivity and polarity classification of Twitter posts. Davidov et al. BIBREF35 employ k-Nearest-Neighbor. Kouloumpis et al. BIBREF36 employ AdaBoost.MH, and also test SVMs, but the performance results of SVMs are lower. Recently, researchers also applied deep learning for Twitter sentiment classification BIBREF17 , BIBREF18 , BIBREF19 .","A wide range of machine learning algorithms is used, and apparently there is no consensus on which one to choose for the best performance. Different studies use different datasets, focus on different use cases, and use incompatible evaluation measures. There are additional factors with considerable impact on the performance, such as the natural language pre-processing of tweets, and formation of appropriate features. Typically, features are based on the bag-of-words presentation of tweets, but there are many subtle choices to be made."]},{"idx":14,"heading":"Classification models performance","paragraphs":["As discussed in the previous subsection, there are many supervised machine learning algorithms suitable for training sentiment classification models. Variants of Support Vector Machine (SVM) BIBREF12 are often used, because they are well suited for large-scale text categorization tasks, are robust on large feature spaces, and perform well. The basic SVM is a two-class, binary classifier. In the training phase, SVM constructs a hyperplane in a high-dimensional vector space that separates one class from the other. During the classification, the side of the hyperplane then determines the class. A binary SVM can be extended into multi-class and regression classifiers BIBREF37 . For this study we implemented five extensions of the basic SVM; some of them take the sentiment class ordering explicitly into account. All the SVM algorithms, and several others, including Naive Bayes BIBREF13 , are implemented in the open-source LATINO library BIBREF38 (a light-weight set of software components for building text mining applications, available at https://github.com/latinolib).","NeutralZoneSVM is an extension of the basic two-class SVM and assumes that neutral tweets are “between” the negative and positive tweets. The classifier is trained just on the negative and positive tweets. During the classification, the side of the hyperplane determines the sentiment class (negative or positive). However, tweets which are “too close” to the hyperplane are considered neutral. Various realizations of “too close” are described in BIBREF39 , BIBREF40 .","TwoPlaneSVM assumes the ordering of sentiment classes and implements ordinal classification BIBREF24 . It consists of two SVM classifiers: One classifier is trained to separate the negative tweets from the neutral-or-positives; the other separates the negative-or-neutrals from the positives. The result is a classifier with two hyperplanes (nearly parallel for all practical cases) which separates the vector space into three subspaces: negative, neutral, and positive. During classification, the distances from both hyperplanes determine the predicted class.","TwoPlaneSVMbin is a refinement of the TwoPlaneSVM classifier. It partitions the space around both hyperplanes into bins, and computes the distribution of the training examples in individual bins. During classification, the distances from both hyperplanes determine the appropriate bin, but the class is determined as the majority class in the bin. Additionally, the classifier can also provide the confidence of the predicted class.","CascadingSVM also consists of two SVM classifiers, but does not assume that the classes are ordered. Instead, the first classifier separates the neutral tweets (“objective”) from the union of negatives and positives (“subjective”). The second classifier in the cascade then considers only the “subjective” tweets and separates the negatives from positives.","ThreePlaneSVM treats the three sentiment classes as nominal, unordered. It consists of three binary classifiers in the one-vs-one setting: the first separates negatives from neutrals, the second neutrals from positives, and the third negatives from positives. The three independent classifiers partition the vector space into eight subspaces. In analogy to the TwoPlaneSVMbin, the distribution of the training examples in each subspace determines the majority class to be predicted during classification.","NaiveBayes is a well-know supervised machine learning algorithm, and is included here for reference. It is a probabilistic classifier based on the Bayes theorem, and does not assume ordering of the sentiment classes.","All the above algorithms were applied to the 13 language datasets and evaluated by 10-fold cross-validation. Standard 10-fold cross-validation randomly partitions the whole labeled set into 10 equal folds. One is set apart for testing, the remaining nine are used to train the model, and the train-test procedure is run over all 10 folds. Cross-validation is stratified when the partitioning is not completely random, but each fold has roughly the same class distribution. With time-ordered data, as is the Twitter stream, one should also consider blocked form of cross-validation BIBREF41 , where there is no randomization, and each fold is a block of consecutive tweets. There are also other evaluation procedures suitable for time-ordered data, different than cross-validation, like ordered sub-sampling, but this is beyond the scope of the paper. In this study we applied blocked, stratified, 10-fold cross-validation in all the experiments.","The Twitter data is first pre-processed by standard text processing methods, i.e., tokenization, stemming/lemmatization (if available for a specific language), unigram and bigram construction, and elimination of terms that do not appear at least 5 times in a dataset. The Twitter specific pre-processing is then applied, i.e, replacing URLs, Twitter usernames and hashtags with common tokens, adding emoticon features for different types of emoticons in tweets, handling of repetitive letters, etc. The feature vectors are constructed by the Delta TF-IDF weighting scheme BIBREF42 .","Evaluation results, in terms of INLINEFORM0 , are summarized in Fig FIGREF40 . The classifiers are ordered by their average performance rank across the 13 datasets. More detailed results, in terms of all four evaluation measures, and also including the application datasets, are in Table TABREF41 . Note that the sizes of the training datasets are lower than the numbers of annotated tweets in Table TABREF30 . Namely, tweets annotated several times are first merged into single training examples, thus forming the “gold standard” for training and testing. If all the annotations are the same, the assigned label is obvious. If the annotations differ, the following merging rules are applied: neutral and negative INLINEFORM1 negative; neutral and positive INLINEFORM2 positive; and negative and positive INLINEFORM3 neutral."]},{"idx":15,"heading":"The Friedman-Nemenyi test","paragraphs":["Are there significant differences between the six classifiers, in terms of their performance? The results depend on the evaluation measure used, but generally the top classifiers are not distinguishable.","A standard statistical method for testing the significant differences between multiple classifiers BIBREF43 is the well-known ANOVA and its non-parametric counterpart, the Friedman test BIBREF14 . The Friedman test ranks the classifiers for each dataset separately. The best performing classifier is assigned rank 1, the second best rank 2, etc. When there are ties, average ranks are assigned. The Friedman test then compares the average ranks of the classifiers. The null hypothesis is that all the classifiers are equivalent and so their ranks should be equal. If the null hypothesis is rejected, one proceeds with a post-hoc test.","If one wants to compare a control classifier to other classifiers, the Bonferroni-Dunn post-hoc test is used. In our case, however, all the classifiers are compared to each other, and the weaker Nemenyi test BIBREF15 is used. The Nemenyi test computes the critical distance between any pair of classifiers. The performance of the two classifiers is significantly different if the corresponding average ranks differ by at least the critical distance.","Fig FIGREF43 gives the results of the Friedman-Nemenyi test for the six classifiers trained in this study. We focus on two evaluation measures that take the ordering of sentiment classes into account: INLINEFORM0 and INLINEFORM1 . There are two classifiers which are in the group of top indistinguishable classifiers in both cases: ThreePlaneSVM (ranked 3rd) and TwoPlaneSVMbin (ranked 4th and 1st). We decided to interpret and discuss all the results in this paper using the TwoPlaneSVMbin classifier, since it is explicitly designed for ordered classes."]},{"idx":16,"heading":"Acknowledgments","paragraphs":["This work was supported in part by the European Union projects SIMPOL (no. 610704), MULTIPLEX (no. 317532) and DOLFINS (no. 640772), and by the Slovenian ARRS programme Knowledge Technologies (no. P2-103).","We acknowledge Gama System (http://www.gama-system.si) who collected most of the tweets (except English), and Sowa Labs (http://www.sowalabs.com) for providing the Goldfinch platform for sentiment annotations. Special thanks go to Sašo Rutar who implemented several classification algorithms and evaluation procedures in the LATINO library for text mining (https://github.com/latinolib). We thank Mojca Mikac for computing the Krippendorff's INLINEFORM0 confidence intervals, and Dragi Kocev for help with the Friedman-Nemenyi test."]},{"idx":17,"heading":"Figures and Tables","paragraphs":["Figure 1. Comparison of annotators self-agreement (green), the inter-annotator agreement (blue), and an automated sentiment classifier (TwoPlaneSVMbin, red) in terms of Krippendorff’s Alpha. On the left-hand side are the 13 language datasets, and on the right-hand side the four application datasets. The datasets are ordered by decreasing self-agreement. The error bars indicate estimated 95% confidence intervals.","Figure 2. The English (left) and Russian (right) datasets. For English, there is still a gap (Alpha = 0.097) between the classifier (in red) and the inter-annotator agreement (in blue).","Figure 3. The Polish dataset. The classifier’s peak performance (in red, Alpha = 0.536) is at 150,000 labeled tweets.","Figure 4. The Slovenian (left) and Bulgarian (right) datasets. The Slovenian classifier peak is at 70,000 tweets (Alpha = 0.459). The Bulgarian classifier peak is at 40,000 tweets (Alpha = 0.378).","Figure 5. The German datasets. The complete dataset (left), and separate datasets labeled by the two main annotators (middle and right).","Figure 6. Joint Serbian/Croatian/Bosnian dataset. There is an oscillation in performance and high variability.","Figure 7. Separate Serbian (left), Croatian (middle), and Bosnian (right) datasets. Here, the lower quality Serbian set has no adverse effects on the higher quality Croatian and Bosnian sets.","Figure 8. The Portuguese dataset. There are two peaks (at 50,000 tweets, Alpha = 0.394, and at 160,000 tweets, Alpha = 0.391), and a large drop in between, due to a topic shift.","Figure 9. The Spanish dataset. There is a consistent drop of performance and high variability.","Figure 10. The sentiment distribution of the environmental tweets in the training and application sets. Negative tweets are denoted by red, neutral by yellow, and positive by green color. The grey bar denotes the sentiment score (the mean) of each dataset.","Table 1. The number and distribution of sentiment annotated posts, and the time period of the posts. The top part of the table refers to the 13 language datasets, and the bottom to the four application datasets.","Table 2. Sentiment distributions of the application datasets as predicted by the sentiment classifiers. The rightmost column shows the sentiment score (the mean) of the application and training datasets (the later from Table 1), respectively.","Table 3. The number of annotators, and the number and fraction of posts annotated twice. The self-agreement column gives the number of posts annotated twice by the same annotator, and the inter-agreement column the posts annotated twice by two different annotators.","Table 4. The self- and inter-annotator agreement measures. The 95% confidence intervals for Alpha are computed by bootstrapping. Albanian and Spanish (in bold) have very low agreement values.","Table 5. Differences between the three sentiment classes (−, 0,+). The differences are measured in terms of Alpha , for the union of self- and inter-annotator agreements. The second column shows the relative difference between the Alpha int (interval) and Alphanom (nominal) agreement measures. The third and fourth columns show the distances of the negative (−) and positive (+) class to the neutral class (0), respectively, normalized with the distance between them. The last row is the average difference, but without the low quality Albanian and Spanish, and the subsumed Emojis datasets (in bold). Only the numbers in bold do not support the thesis that sentiment classes are ordered.","Figure 11. Comparison of six classification models in terms of Alpha. Results are the average of 10-fold cross-validations.","Table 6. Evaluation results of the sentiment classifiers. The dataset sizes in the second column, used for training the classifiers, are the result of merging multiple annotated tweets (there was no merging for the Facebook(it) and DJIA30 datasets). The 95% confidence intervals for Alpha are estimated from 10-fold cross-validations. The last row is an evaluation of the general English language model (trained from the English dataset in row 3) on the Environment dataset.","Figure 12. Results of the Friedman-Nemenyi test of classifiers ranking. The six classifiers are compared in terms of their ranking using two evaluation measures, Alpha (left) and F1 (right). The ranks of classifiers within the critical distance (2.09) are not statistically significantly different."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1602.07776","title":"Recurrent Neural Network Grammars","sections":[{"idx":0,"heading":"Abstract","paragraphs":["We introduce recurrent neural network grammars, probabilistic models of sentences with explicit phrase structure. We explain efficient inference procedures that allow application to both parsing and language modeling. Experiments show that they provide better parsing in English than any single previously published supervised generative model and better language modeling than state-of-the-art sequential RNNs in English and Chinese."]},{"idx":1,"heading":"Introduction","paragraphs":["Sequential recurrent neural networks (RNNs) are remarkably effective models of natural language. In the last few years, language model results that substantially improve over long-established state-of-the-art baselines have been obtained using RNNs BIBREF0 , BIBREF1 as well as in various conditional language modeling tasks such as machine translation BIBREF2 , image caption generation BIBREF3 , and dialogue generation BIBREF4 . Despite these impressive results, sequential models are a priori inappropriate models of natural language, since relationships among words are largely organized in terms of latent nested structures rather than sequential surface order BIBREF5 .","In this paper, we introduce recurrent neural network grammars (RNNGs; § SECREF2 ), a new generative probabilistic model of sentences that explicitly models nested, hierarchical relationships among words and phrases. RNNGs operate via a recursive syntactic process reminiscent of probabilistic context-free grammar generation, but decisions are parameterized using RNNs that condition on the entire syntactic derivation history, greatly relaxing context-free independence assumptions.","The foundation of this work is a top-down variant of transition-based parsing (§ SECREF3 ). We give two variants of the algorithm, one for parsing (given an observed sentence, transform it into a tree), and one for generation. While several transition-based neural models of syntactic generation exist BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , these have relied on structure building operations based on parsing actions in shift-reduce and left-corner parsers which operate in a largely bottom-up fashion. While this construction is appealing because inference is relatively straightforward, it limits the use of top-down grammar information, which is helpful for generation BIBREF11 . RNNGs maintain the algorithmic convenience of transition-based parsing but incorporate top-down (i.e., root-to-terminal) syntactic information (§ SECREF4 ).","The top-down transition set that RNNGs are based on lends itself to discriminative modeling as well, where sequences of transitions are modeled conditional on the full input sentence along with the incrementally constructed syntactic structures. Similar to previously published discriminative bottom-up transition-based parsers BIBREF7 , BIBREF12 , BIBREF13 , greedy prediction with our model yields a linear-time deterministic parser (provided an upper bound on the number of actions taken between processing subsequent terminal symbols is imposed); however, our algorithm generates arbitrary tree structures directly, without the binarization required by shift-reduce parsers. The discriminative model also lets us use ancestor sampling to obtain samples of parse trees for sentences, and this is used to solve a second practical challenge with RNNGs: approximating the marginal likelihood and MAP tree of a sentence under the generative model. We present a simple importance sampling algorithm which uses samples from the discriminative parser to solve inference problems in the generative model (§ SECREF5 ).","Experiments show that RNNGs are effective for both language modeling and parsing (§ SECREF6 ). Our generative model obtains (i) the best-known parsing results using a single supervised generative model and (ii) better perplexities in language modeling than state-of-the-art sequential LSTM language models. Surprisingly—although in line with previous parsing results showing the effectiveness of generative models BIBREF7 , BIBREF14 —parsing with the generative model obtains significantly better results than parsing with the discriminative model."]},{"idx":2,"heading":"RNN Grammars","paragraphs":["Formally, an RNNG is a triple INLINEFORM0 consisting of a finite set of nonterminal symbols ( INLINEFORM1 ), a finite set of terminal symbols ( INLINEFORM2 ) such that INLINEFORM3 , and a collection of neural network parameters INLINEFORM4 . It does not explicitly define rules since these are implicitly characterized by INLINEFORM5 . The algorithm that the grammar uses to generate trees and strings in the language is characterized in terms of a transition-based algorithm, which is outlined in the next section. In the section after that, the semantics of the parameters that are used to turn this into a stochastic algorithm that generates pairs of trees and strings are discussed."]},{"idx":3,"heading":"Top-down Parsing and Generation","paragraphs":["RNNGs are based on a top-down generation algorithm that relies on a stack data structure of partially completed syntactic constituents. To emphasize the similarity of our algorithm to more familiar bottom-up shift-reduce recognition algorithms, we first present the parsing (rather than generation) version of our algorithm (§ SECREF4 ) and then present modifications to turn it into a generator (§ SECREF19 )."]},{"idx":4,"heading":"Parser Transitions","paragraphs":["The parsing algorithm transforms a sequence of words INLINEFORM0 into a parse tree INLINEFORM1 using two data structures (a stack and an input buffer). As with the bottom-up algorithm of BIBREF12 , our algorithm begins with the stack ( INLINEFORM2 ) empty and the complete sequence of words in the input buffer ( INLINEFORM3 ). The buffer contains unprocessed terminal symbols, and the stack contains terminal symbols, “open” nonterminal symbols, and completed constituents. At each timestep, one of the following three classes of operations (Fig. FIGREF9 ) is selected by a classifier, based on the current contents on the stack and buffer:","INLINEFORM0 introduces an “open nonterminal” X onto the top of the stack. Open nonterminals are written as a nonterminal symbol preceded by an open parenthesis, e.g., “(VP”, and they represent a nonterminal whose child nodes have not yet been fully constructed. Open nonterminals are “closed” to form complete constituents by subsequent reduce operations.","shift removes the terminal symbol INLINEFORM0 from the front of the input buffer, and pushes it onto the top of the stack.","reduce repeatedly pops completed subtrees or terminal symbols from the stack until an open nonterminal is encountered, and then this open NT is popped and used as the label of a new constituent that has the popped subtrees as its children. This new completed constituent is pushed onto the stack as a single composite item. A single reduce operation can thus create constituents with an unbounded number of children.","The parsing algorithm terminates when there is a single completed constituent on the stack and the buffer is empty. Fig. FIGREF10 shows an example parse using our transition set. Note that in this paper we do not model preterminal symbols (i.e., part-of-speech tags) and our examples therefore do not include them.","Our transition set is closely related to the operations used in Earley's algorithm which likewise introduces nonterminals symbols with its predict operation and later completes them after consuming terminal symbols one at a time using scan BIBREF15 . It is likewise closely related to the “linearized” parse trees proposed by BIBREF16 and to the top-down, left-to-right decompositions of trees used in previous generative parsing and language modeling work BIBREF11 , BIBREF17 , BIBREF18 .","A further connection is to INLINEFORM0 parsing which uses an unbounded lookahead (compactly represented by a DFA) to distinguish between parse alternatives in a top-down parser BIBREF19 ; however, our parser uses an RNN encoding of the lookahead rather than a DFA.","To guarantee that only well-formed phrase-structure trees are produced by the parser, we impose the following constraints on the transitions that can be applied at each step which are a function of the parser state INLINEFORM0 where INLINEFORM1 is the number of open nonterminals on the stack:","The INLINEFORM0 operation can only be applied if INLINEFORM1 is not empty and INLINEFORM2 .","The shift operation can only be applied if INLINEFORM0 is not empty and INLINEFORM1 .","The reduce operation can only be applied if the top of the stack is not an open nonterminal symbol.","The reduce operation can only be applied if INLINEFORM0 or if the buffer is empty.","To designate the set of valid parser transitions, we write INLINEFORM0 ."]},{"idx":5,"heading":"Generator Transitions","paragraphs":["The parsing algorithm that maps from sequences of words to parse trees can be adapted with minor changes to produce an algorithm that stochastically generates trees and terminal symbols. Two changes are required: (i) there is no input buffer of unprocessed words, rather there is an output buffer ( INLINEFORM0 ), and (ii) instead of a shift operation there are INLINEFORM1 operations which generate terminal symbol INLINEFORM2 and add it to the top of the stack and the output buffer. At each timestep an action is stochastically selected according to a conditional distribution that depends on the current contents of INLINEFORM3 and INLINEFORM4 . The algorithm terminates when a single completed constituent remains on the stack. Fig. FIGREF12 shows an example generation sequence.","The generation algorithm also requires slightly modified constraints. These are:","The INLINEFORM0 operation can only be applied if INLINEFORM1 .","The reduce operation can only be applied if the top of the stack is not an open nonterminal symbol and INLINEFORM0 .","To designate the set of valid generator transitions, we write INLINEFORM0 .","This transition set generates trees using nearly the same structure building actions and stack configurations as the “top-down PDA” construction proposed by BIBREF20 , albeit without the restriction that the trees be in Chomsky normal form."]},{"idx":6,"heading":"Transition Sequences from Trees","paragraphs":["Any parse tree can be converted to a sequence of transitions via a depth-first, left-to-right traversal of a parse tree. Since there is a unique depth-first, left-ro-right traversal of a tree, there is exactly one transition sequence of each tree. For a tree INLINEFORM0 and a sequence of symbols INLINEFORM1 , we write INLINEFORM2 to indicate the corresponding sequence of generation transitions, and INLINEFORM3 to indicate the parser transitions."]},{"idx":7,"heading":"Runtime Analysis","paragraphs":["A detailed analysis of the algorithmic properties of our top-down parser is beyond the scope of this paper; however, we briefly state several facts. Assuming the availability of constant time push and pop operations, the runtime is linear in the number of the nodes in the parse tree that is generated by the parser/generator (intuitively, this is true since although an individual reduce operation may require applying a number of pops that is linear in the number of input symbols, the total number of pop operations across an entire parse/generation run will also be linear). Since there is no way to bound the number of output nodes in a parse tree as a function of the number of input words, stating the runtime complexity of the parsing algorithm as a function of the input size requires further assumptions. Assuming our fixed constraint on maximum depth, it is linear."]},{"idx":8,"heading":"Comparison to Other Models","paragraphs":["Our generation algorithm algorithm differs from previous stack-based parsing/generation algorithms in two ways. First, it constructs rooted tree structures top down (rather than bottom up), and second, the transition operators are capable of directly generating arbitrary tree structures rather than, e.g., assuming binarized trees, as is the case in much prior work that has used transition-based algorithms to produce phrase-structure trees BIBREF12 , BIBREF13 , BIBREF21 ."]},{"idx":9,"heading":"Generative Model","paragraphs":["RNNGs use the generator transition set just presented to define a joint distribution on syntax trees ( INLINEFORM0 ) and words ( INLINEFORM1 ). This distribution is defined as a sequence model over generator transitions that is parameterized using a continuous space embedding of the algorithm state at each time step ( INLINEFORM2 ); i.e., INLINEFORM3","and where action-specific embeddings INLINEFORM0 and bias vector INLINEFORM1 are parameters in INLINEFORM2 .","The representation of the algorithm state at time INLINEFORM0 , INLINEFORM1 , is computed by combining the representation of the generator's three data structures: the output buffer ( INLINEFORM2 ), represented by an embedding INLINEFORM3 , the stack ( INLINEFORM4 ), represented by an embedding INLINEFORM5 , and the history of actions ( INLINEFORM6 ) taken by the generator, represented by an embedding INLINEFORM7 , INLINEFORM8","where INLINEFORM0 and INLINEFORM1 are parameters. Refer to Figure FIGREF26 for an illustration of the architecture.","The output buffer, stack, and history are sequences that grow unboundedly, and to obtain representations of them we use recurrent neural networks to “encode” their contents BIBREF22 . Since the output buffer and history of actions are only appended to and only contain symbols from a finite alphabet, it is straightforward to apply a standard RNN encoding architecture. The stack ( INLINEFORM0 ) is more complicated for two reasons. First, the elements of the stack are more complicated objects than symbols from a discrete alphabet: open nonterminals, terminals, and full trees, are all present on the stack. Second, it is manipulated using both push and pop operations. To efficiently obtain representations of INLINEFORM1 under push and pop operations, we use stack LSTMs BIBREF23 . To represent complex parse trees, we define a new syntactic composition function that recursively defines representations of trees."]},{"idx":10,"heading":"Syntactic Composition Function","paragraphs":["When a reduce operation is executed, the parser pops a sequence of completed subtrees and/or tokens (together with their vector embeddings) from the stack and makes them children of the most recent open nonterminal on the stack, “completing” the constituent. To compute an embedding of this new subtree, we use a composition function based on bidirectional LSTMs, which is illustrated in Fig. FIGREF28 .","The first vector read by the LSTM in both the forward and reverse directions is an embedding of the label on the constituent being constructed (in the figure, NP). This is followed by the embeddings of the child subtrees (or tokens) in forward or reverse order. Intuitively, this order serves to “notify” each LSTM what sort of head it should be looking for as it processes the child node embeddings. The final state of the forward and reverse LSTMs are concatenated, passed through an affine transformation and a INLINEFORM0 nonlinearity to become the subtree embedding. Because each of the child node embeddings ( INLINEFORM2 , INLINEFORM3 , INLINEFORM4 in Fig. FIGREF28 ) is computed similarly (if it corresponds to an internal node), this composition function is a kind of recursive neural network."]},{"idx":11,"heading":"Word Generation","paragraphs":["To reduce the size of INLINEFORM0 , word generation is broken into two parts. First, the decision to generate is made (by predicting INLINEFORM1 as an action), and then choosing the word, conditional on the current parser state. To further reduce the computational complexity of modeling the generation of a word, we use a class-factored softmax BIBREF26 , BIBREF27 . By using INLINEFORM2 classes for a vocabulary of size INLINEFORM3 , this prediction step runs in time INLINEFORM4 rather than the INLINEFORM5 of the full-vocabulary softmax. To obtain clusters, we use the greedy agglomerative clustering algorithm of BIBREF28 ."]},{"idx":12,"heading":"Training","paragraphs":["The parameters in the model are learned to maximize the likelihood of a corpus of trees."]},{"idx":13,"heading":"Discriminative Parsing Model","paragraphs":["A discriminative parsing model can be obtained by replacing the embedding of INLINEFORM0 at each time step with an embedding of the input buffer INLINEFORM1 . To train this model, the conditional likelihood of each sequence of actions given the input string is maximized."]},{"idx":14,"heading":"Inference via Importance Sampling","paragraphs":["Our generative model INLINEFORM0 defines a joint distribution on trees ( INLINEFORM1 ) and sequences of words ( INLINEFORM2 ). To evaluate this as a language model, it is necessary to compute the marginal probability INLINEFORM3 . And, to evaluate the model as a parser, we need to be able to find the MAP parse tree, i.e., the tree INLINEFORM4 that maximizes INLINEFORM5 . However, because of the unbounded dependencies across the sequence of parsing actions in our model, exactly solving either of these inference problems is intractable. To obtain estimates of these, we use a variant of importance sampling BIBREF31 .","Our importance sampling algorithm uses a conditional proposal distribution INLINEFORM0 with the following properties: (i) INLINEFORM1 ; (ii) samples INLINEFORM2 can be obtained efficiently; and (iii) the conditional probabilities INLINEFORM3 of these samples are known. While many such distributions are available, the discriminatively trained variant of our parser (§ SECREF32 ) fulfills these requirements: sequences of actions can be sampled using a simple ancestral sampling approach, and, since parse trees and action sequences exist in a one-to-one relationship, the product of the action probabilities is the conditional probability of the parse tree under INLINEFORM4 . We therefore use our discriminative parser as our proposal distribution.","Importance sampling uses importance weights, which we define as INLINEFORM0 , to compute this estimate. Under this definition, we can derive the estimator as follows: INLINEFORM1","We now replace this expectation with its Monte Carlo estimate as follows, using INLINEFORM0 samples from INLINEFORM1 : INLINEFORM2","To obtain an estimate of the MAP tree INLINEFORM0 , we choose the sampled tree with the highest probability under the joint model INLINEFORM1 ."]},{"idx":15,"heading":"Experiments","paragraphs":["We present results of our two models both on parsing (discriminative and generative) and as a language model (generative only) in English and Chinese."]},{"idx":16,"heading":"Discussion","paragraphs":["It is clear from our experiments that the proposed generative model is quite effective both as a parser and as a language model. This is the result of (i) relaxing conventional independence assumptions (e.g., context-freeness) and (ii) inferring continuous representations of symbols alongside non-linear models of their syntactic relationships. The most significant question that remains is why the discriminative model—which has more information available to it than the generative model—performs worse than the generative model. This pattern has been observed before in neural parsing by BIBREF7 , who hypothesized that larger, unstructured conditioning contexts are harder to learn from, and provide opportunities to overfit. Our discriminative model conditions on the entire history, stack, and buffer, while our generative model only accesses the history and stack. The fully discriminative model of BIBREF16 was able to obtain results similar to those of our generative model (albeit using much larger training sets obtained through semisupervision) but similar results to those of our discriminative parser using the same data. In light of their results, we believe Henderson's hypothesis is correct, and that generative models should be considered as a more statistically efficient method for learning neural networks from small data."]},{"idx":17,"heading":"Related Work","paragraphs":["Our language model combines work from two modeling traditions: (i) recurrent neural network language models and (ii) syntactic language modeling. Recurrent neural network language models use RNNs to compute representations of an unbounded history of words in a left-to-right language model BIBREF0 , BIBREF1 , BIBREF44 . Syntactic language models jointly generate a syntactic structure and a sequence of words BIBREF45 , BIBREF46 . There is an extensive literature here, but one strand of work has emphasized a bottom-up generation of the tree, using variants of shift-reduce parser actions to define the probability space BIBREF47 , BIBREF8 . The neural-network–based model of BIBREF7 is particularly similar to ours in using an unbounded history in a neural network architecture to parameterize generative parsing based on a left-corner model. Dependency-only language models have also been explored BIBREF9 , BIBREF48 , BIBREF10 . Modeling generation top-down as a rooted branching process that recursively rewrites nonterminals has been explored by BIBREF41 and BIBREF11 . Of particular note is the work of BIBREF18 , which uses random forests and hand-engineered features over the entire syntactic derivation history to make decisions over the next action to take.","The neural networks we use to model sentences are structured according to the syntax of the sentence being generated. Syntactically structured neural architectures have been explored in a number of applications, including discriminative parsing BIBREF34 , BIBREF49 , sentiment analysis BIBREF25 , BIBREF24 , and sentence representation BIBREF50 , BIBREF51 . However, these models have been, without exception, discriminative; this is the first work to use syntactically structured neural models to generate language. Earlier work has demonstrated that sequential RNNs have the capacity to recognize context-free (and beyond) languages BIBREF52 , BIBREF53 . In contrast, our work may be understood as a way of incorporating a context-free inductive bias into the model structure."]},{"idx":18,"heading":"Outlook","paragraphs":["RNNGs can be combined with a particle filter inference scheme (rather than the importance sampling method based on a discriminative parser, § SECREF5 ) to produce a left-to-right marginalization algorithm that runs in expected linear time. Thus, they could be used in applications that require language models.","A second possibility is to replace the sequential generation architectures found in many neural network transduction problems that produce sentences conditioned on some input. Previous work in machine translation has showed that conditional syntactic models can function quite well without the computationally expensive marginalization process at decoding time BIBREF54 , BIBREF55 .","A third consideration regarding how RNNGs, human sentence processing takes place in a left-to-right, incremental order. While an RNNG is not a processing model (it is a grammar), the fact that it is left-to-right opens up several possibilities for developing new sentence processing models based on an explicit grammars, similar to the processing model of BIBREF18 .","Finally, although we considered only the supervised learning scenario, RNNGs are joint models that could be trained without trees, for example, using expectation maximization."]},{"idx":19,"heading":"Conclusion","paragraphs":["We introduced recurrent neural network grammars, a probabilistic model of phrase-structure trees that can be trained generatively and used as a language model or a parser, and a corresponding discriminative model that can be used as a parser. Apart from out-of-vocabulary preprocessing, the approach requires no feature design or transformations to treebank data. The generative model outperforms every previously published parser built on a single supervised generative model in English, and a bit behind the best-reported generative model in Chinese. As language models, RNNGs outperform the best single-sentence language models."]},{"idx":20,"heading":"Acknowledgments","paragraphs":["We thank Brendan O'Connor, Swabha Swayamdipta, and Brian Roark for feedback on drafts of this paper, and Jan Buys, Phil Blunsom, and Yue Zhang for help with data preparation. This work was sponsored in part by the Defense Advanced Research Projects Agency (DARPA) Information Innovation Office (I2O) under the Low Resource Languages for Emergent Incidents (LORELEI) program issued by DARPA/I2O under Contract No. HR0011-15-C-0114; it was also supported in part by Contract No. W911NF-15-1-0543 with the DARPA and the Army Research Office (ARO). Approved for public release, distribution unlimited. The views expressed are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government. Miguel Ballesteros was supported by the European Commission under the contract numbers FP7-ICT-610411 (project MULTISENSOR) and H2020-RIA-645012 (project KRISTINA).","2pt Chris Dyer INLINEFORM0 Adhiguna Kuncoro INLINEFORM1 Miguel Ballesteros INLINEFORM2 Noah A. Smith INLINEFORM3 INLINEFORM4 School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA INLINEFORM5 NLP Group, Pompeu Fabra University, Barcelona, Spain INLINEFORM6 Google DeepMind, London, UK INLINEFORM7 Computer Science & Engineering, University of Washington, Seattle, WA, USA {cdyer,akuncoro}@cs.cmu.edu miguel.ballesteros@upf.edu, nasmith@cs.washington.edu [ Corrigendum to Recurrent Neural Network Grammars ] Due to an implentation bug in the RNNG's recursive composition function, the results reported in Dyer et al. (2016) did not correspond to the model as it was presented. This corrigendum describes the buggy implementation and reports results with a corrected implementation. After correction, on the PTB §23 and CTB 5.1 test sets, respectively, the generative model achieves language modeling perplexities of 105.2 and 148.5, and phrase-structure parsing F1 of 93.3 and 86.9, a new state of the art in phrase-structure parsing for both languages. RNNG Composition Function and Implementation Error The composition function reduces a completed constituent into a single vector representation using a bidirectional LSTM (Figure FIGREF47 ) over embeddings of the constituent's children as well as an embedding of the resulting nonterminal symbol type. The implementation error (Figure FIGREF47 ) composed the constituent (NP the hungry cat) by reading the sequence “NP the hungry NP”, that is, it discarded the rightmost child of every constituent and replaced it with a second copy of the constituent's nonterminal symbol. This error occurs for every constituent and means crucial information is not properly propagated upwards in the tree. Results after Correction The implementation error affected both the generative and discriminative RNNGs. We summarize corrected English phrase-structure PTB §23 parsing result in Table TABREF49 , Chinese (CTB 5.1 §271–300) in Table TABREF50 (achieving the the best reported result on both datasets), and English and Chinese language modeling perplexities in Table TABREF51 . The considerable improvement in parsing accuracy indicates that properly composing the constituent and propagating information upwards is crucial. Despite slightly higher language modeling perplexity on PTB §23, the fixed RNNG still outperforms a highly optimized sequential LSTM baseline."]},{"idx":21,"heading":"Figures and Tables","paragraphs":["Figure 2: Top-down parsing example.","Figure 4: Joint generation of a parse tree and sentence.","Figure 1: Parser transitions showing the stack, buffer, and open nonterminal count before and after each action type. S represents the stack, which contains open nonterminals and completed subtrees; B represents the buffer of unprocessed terminal symbols; x is a terminal symbol, X is a nonterminal symbol, and each τ is a completed subtree. The top of the stack is to the right, and the buffer is consumed from left to right. Elements on the stack and buffer are delimited by a vertical bar ( | ).","Figure 6: Syntactic composition function based on bidirectional LSTMs that is executed during a REDUCE operation; the network on the right models the structure on the left.","Figure 5: Neural architecture for defining a distribution over at given representations of the stack (St), output buffer (Tt) and history of actions (a<t). Details of the composition architecture of the NP, the action history LSTM, and the other elements of the stack are not shown. This architecture corresponds to the generator state at line 7 of Figure 4.","Table 1: Corpus statistics.","Table 2: Parsing results on PTB §23 (D=discriminative, G=generative, S=semisupervised). ? indicates the (Vinyals et al., 2015) result with trained only on the WSJ corpus without ensembling.","Table 3: Parsing results on CTB 5.1.","Table 4: Language model perplexity results.","Figure 7: Correct RNNG composition function for the constituent (NP the hungry cat).","Figure 8: Buggy implementation of the RNNG composition function for the constituent (NP the hungry cat). Note that the right-most child, cat, has been replaced by a second NP.","Table 6: Parsing results on CTB 5.1 including results with the buggy composition function implementation (indicated by †) and with the correct implementation.","Table 7: PTB and CTB language modeling results including results with the buggy composition function implementation (indicated by †) and with the correct implementation."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1603.01514","title":"A Bayesian Model of Multilingual Unsupervised Semantic Role Induction","sections":[{"idx":0,"heading":"Abstract","paragraphs":["We propose a Bayesian model of unsupervised semantic role induction in multiple languages, and use it to explore the usefulness of parallel corpora for this task. Our joint Bayesian model consists of individual models for each language plus additional latent variables that capture alignments between roles across languages. Because it is a generative Bayesian model, we can do evaluations in a variety of scenarios just by varying the inference procedure, without changing the model, thereby comparing the scenarios directly. We compare using only monolingual data, using a parallel corpus, using a parallel corpus with annotations in the other language, and using small amounts of annotation in the target language. We find that the biggest impact of adding a parallel corpus to training is actually the increase in mono-lingual data, with the alignments to another language resulting in small improvements, even with labeled data for the other language."]},{"idx":1,"heading":"Introduction","paragraphs":["Semantic Role Labeling (SRL) has emerged as an important task in Natural Language Processing (NLP) due to its applicability in information extraction, question answering, and other NLP tasks. SRL is the problem of finding predicate-argument structure in a sentence, as illustrated below:","INLINEFORM0","Here, the predicate WRITE has two arguments: `Mike' as A0 or the writer, and `a book' as A1 or the thing written. The labels A0 and A1 correspond to the PropBank annotations BIBREF0 .","As the need for SRL arises in different domains and languages, the existing manually annotated corpora become insufficient to build supervised systems. This has motivated work on unsupervised SRL BIBREF1 , BIBREF2 , BIBREF3 . Previous work has indicated that unsupervised systems could benefit from the word alignment information in parallel text in two or more languages BIBREF4 , BIBREF5 , BIBREF6 . For example, consider the German translation of sentence INLINEFORM0 :","INLINEFORM0","If sentences INLINEFORM0 and INLINEFORM1 have the word alignments: Mike-Mike, written-geschrieben, and book-Buch, the system might be able to predict A1 for Buch, even if there is insufficient information in the monolingual German data to learn this assignment. Thus, in languages where the resources are sparse or not good enough, or the distributions are not informative, SRL systems could be made more accurate by using parallel data with resource rich or more amenable languages.","In this paper, we propose a joint Bayesian model for unsupervised semantic role induction in multiple languages. The model consists of individual Bayesian models for each language BIBREF3 , and crosslingual latent variables to incorporate soft role agreement between aligned constituents. This latent variable approach has been demonstrated to increase the performance in a multilingual unsupervised part-of-speech tagging model based on HMMs BIBREF4 . We investigate the application of this approach to unsupervised SRL, presenting the performance improvements obtained in different settings involving labeled and unlabeled data, and analyzing the annotation effort required to obtain similar gains using labeled data.","We begin by briefly describing the unsupervised SRL pipeline and the monolingual semantic role induction model we use, and then describe our multilingual model."]},{"idx":2,"heading":"Unsupervised SRL Pipeline","paragraphs":["As established in previous work BIBREF7 , BIBREF8 , we use a standard unsupervised SRL setup, consisting of the following steps:","The task we model, unsupervised semantic role induction, is the step 4 of this pipeline."]},{"idx":3,"heading":"Monolingual Model","paragraphs":["We use the Bayesian model of garg2012unsupervised as our base monolingual model. The semantic roles are predicate-specific. To model the role ordering and repetition preferences, the role inventory for each predicate is divided into Primary and Secondary roles as follows:","For example, the complete role sequence in a frame could be: INLINEFORM0 INLINEFORM1 , INLINEFORM2 , INLINEFORM3 , INLINEFORM4 , INLINEFORM5 , INLINEFORM6 , INLINEFORM7 , INLINEFORM8 INLINEFORM9 . The ordering is defined as the sequence of PRs, INLINEFORM10 INLINEFORM11 , INLINEFORM12 , INLINEFORM13 , INLINEFORM14 , INLINEFORM15 INLINEFORM16 . Each pair of consecutive PRs in an ordering is called an interval. Thus, INLINEFORM17 is an interval that contains two SRs, INLINEFORM18 and INLINEFORM19 . An interval could also be empty, for instance INLINEFORM20 contains no SRs. When we evaluate, these roles get mapped to gold roles. For instance, the PR INLINEFORM21 could get mapped to a core role like INLINEFORM22 , INLINEFORM23 , etc. or to a modifier role like INLINEFORM24 , INLINEFORM25 , etc. garg2012unsupervised reported that, in practice, PRs mostly get mapped to core roles and SRs to modifier roles, which conforms to the linguistic motivations for this distinction.","Figure FIGREF16 illustrates two copies of the monolingual model, on either side of the crosslingual latent variables. The generative process is as follows:","All the multinomial and binomial distributions have symmetric Dirichlet and beta priors respectively. Figure FIGREF7 gives the probability equations for the monolingual model. This formulation models the global role ordering and repetition preferences using PRs, and limited context for SRs using intervals. Ordering and repetition information was found to be helpful in supervised SRL as well BIBREF9 , BIBREF8 , BIBREF10 . More details, including the motivations behind this model, are in BIBREF3 ."]},{"idx":4,"heading":"Multilingual Model","paragraphs":["The multilingual model uses word alignments between sentences in a parallel corpus to exploit role correspondences across languages. We make copies of the monolingual model for each language and add additional crosslingual latent variables (CLVs) to couple the monolingual models, capturing crosslingual semantic role patterns. Concretely, when training on parallel sentences, whenever the head words of the arguments are aligned, we add a CLV as a parent of the two corresponding role variables. Figure FIGREF16 illustrates this model. The generative process, as explained below, remains the same as the monolingual model for the most part, with the exception of aligned roles which are now generated by both the monolingual process as well as the CLV.","Every predicate-tuple has its own inventory of CLVs specific to that tuple. Each CLV INLINEFORM0 is a multi-valued variable where each value defines a distribution over role labels for each language (denoted by INLINEFORM1 above). These distributions over labels are trained to be peaky, so that each value INLINEFORM2 for a CLV represents a correlation between the labels that INLINEFORM3 predicts in the two languages. For example, a value INLINEFORM4 for the CLV INLINEFORM5 might give high probabilities to INLINEFORM6 and INLINEFORM7 in language 1, and to INLINEFORM8 in language 2. If INLINEFORM9 is the only value for INLINEFORM10 that gives high probability to INLINEFORM11 in language 1, and the monolingual model in language 1 decides to assign INLINEFORM12 to the role for INLINEFORM13 , then INLINEFORM14 will predict INLINEFORM15 in language 2, with high probability. We generate the CLVs via a Chinese Restaurant Process BIBREF11 , a non-parametric Bayesian model, which allows us to induce the number of CLVs for every predicate-tuple from the data. We continue to train on the non-parallel sentences using the respective monolingual models.","The multilingual model is deficient, since the aligned roles are being generated twice. Ideally, we would like to add the CLV as additional conditioning variables in the monolingual models. The new joint probability can be written as equation UID11 (Figure FIGREF7 ), which can be further decomposed following the decomposition of the monolingual model in Figure FIGREF7 . However, having this additional conditioning variable breaks the Dirichlet-multinomial conjugacy, which makes it intractable to marginalize out the parameters during inference. Hence, we use an approximation where we treat each of the aligned roles as being generated twice, once by the monolingual model and once by the corresponding CLV (equation ).","This is the first work to incorporate the coupling of aligned arguments directly in a Bayesian SRL model. This makes it easier to see how to extend this model in a principled way to incorporate additional sources of information. First, the model scales gracefully to more than two languages. If there are a total of INLINEFORM0 languages, and there is an aligned argument in INLINEFORM1 of them, the multilingual latent variable is connected to only those INLINEFORM2 aligned arguments.","Second, having one joint Bayesian model allows us to use the same model in various semi-supervised learning settings, just by fixing the annotated variables during training. Section SECREF29 evaluates a setting where we have some labeled data in one language (called source), while no labeled data in the second language (called target). Note that this is different from a classic annotation projection setting (e.g. BIBREF12 ), where the role labels are mapped from source constituents to aligned target constituents."]},{"idx":5,"heading":"Inference and Training","paragraphs":["The inference problem consists of predicting the role labels and CLVs (the hidden variables) given the predicate, its voice, and syntactic features of all the identified arguments (the visible variables). We use a collapsed Gibbs-sampling based approach to generate samples for the hidden variables (model parameters are integrated out). The sample counts and the priors are then used to calculate the MAP estimate of the model parameters.","For the monolingual model, the role at a given position is sampled as:","DISPLAYFORM0","where the subscript INLINEFORM0 refers to all the variables except at position INLINEFORM1 , INLINEFORM2 refers to the variables in all the training instances except the current one, and INLINEFORM3 refers to all the model parameters. The above integral has a closed form solution due to Dirichlet-multinomial conjugacy.","For sampling roles in the multilingual model, we also need to consider the probabilities of roles being generated by the CLVs:","DISPLAYFORM0","For sampling CLVs, we need to consider three factors: two corresponding to probabilities of generating the aligned roles, and the third one corresponding to selecting the CLV according to CRP.","DISPLAYFORM0","where the aligned roles INLINEFORM0 and INLINEFORM1 are connected to INLINEFORM2 , and INLINEFORM3 refers to all the variables except INLINEFORM4 , INLINEFORM5 , and INLINEFORM6 .","We use the trained parameters to parse the monolingual data using the monolingual model. The crosslingual parameters are ignored even if they were used during training. Thus, the information coming from the CLVs acts as a regularizer for the monolingual models."]},{"idx":6,"heading":"Evaluation","paragraphs":["Following the setting of titovcrosslingual, we evaluate only on the arguments that were correctly identified, as the incorrectly identified arguments do not have any gold semantic labels. Evaluation is done using the metric proposed by lang2011unsupervised, which has 3 components: (i) Purity (PU) measures how well an induced cluster corresponds to a single gold role, (ii) Collocation (CO) measures how well a gold role corresponds to a single induced cluster, and (iii) F1 is the harmonic mean of PU and CO. For each predicate, let INLINEFORM0 denote the total number of argument instances, INLINEFORM1 the instances in the induced cluster INLINEFORM2 , and INLINEFORM3 the instances having label INLINEFORM4 in gold annotations. INLINEFORM5 , INLINEFORM6 , and INLINEFORM7 . The score for each predicate is weighted by the number of its argument instances, and a weighted average is computed over all the predicates."]},{"idx":7,"heading":"Baseline","paragraphs":["We use the same baseline as used by lang2011unsupervised which has been shown to be difficult to outperform. This baseline assigns a semantic role to a constituent based on its syntactic function, i.e. the dependency relation to its head. If there is a total of INLINEFORM0 clusters, INLINEFORM1 most frequent syntactic functions get a cluster each, and the rest are assigned to the INLINEFORM2 th cluster."]},{"idx":8,"heading":"Closest Previous Work","paragraphs":["This work is closely related to the cross-lingual unsupervised SRL work of titovcrosslingual. Their model has separate monolingual models for each language and an extra penalty term which tries to maximize INLINEFORM0 and INLINEFORM1 i.e. for all the aligned arguments with role label INLINEFORM2 in language 1, it tries to find a role label INLINEFORM3 in language 2 such that the given proportion is maximized and vice verse. However, there is no efficient way to optimize the objective with this penalty term and the authors used an inference method similar to annotation projection. Further, the method does not scale naturally to more than two languages. Their algorithm first does monolingual inference in one language ignoring the penalty and then does the inference in the second language taking into account the penalty term. In contrast, our model adds the latent variables as a part of the model itself, and not an external penalty, which enables us to use the standard Bayesian learning methods such as sampling.","The monolingual model we use BIBREF3 also has two main advantages over titovcrosslingual. First, the former incorporates a global role ordering probability that is missing in the latter. Secondly, the latter defines argument-keys as a tuple of four syntactic features and all the arguments having the same argument-keys are assigned the same role. This kind of hard clustering is avoided in the former model where two constituents having the same set of features might get assigned different roles if they appear in different contexts."]},{"idx":9,"heading":"Data","paragraphs":["Following titovcrosslingual, we run our experiments on the English (EN) and German (DE) sections of the CoNLL 2009 corpus BIBREF13 , and EN-DE section of the Europarl corpus BIBREF14 . We get about 40k EN and 36k DE sentences from the CoNLL 2009 training set, and about 1.5M parallel EN-DE sentences from Europarl. For appropriate comparison, we keep the same setting as in BIBREF6 for automatic parses and argument identification, which we briefly describe here. The EN sentences are parsed syntactically using MaltParser BIBREF15 and DE using LTH parser BIBREF16 . All the non-auxiliary verbs are selected as predicates. In CoNLL data, this gives us about 3k EN and 500 DE predicates. The total number of predicate instances are 3.4M in EN (89k CoNLL + 3.3M Europarl) and 2.62M in DE (17k CoNLL + 2.6M Europarl). The arguments for EN are identified using the heuristics proposed by lang2011unsupervised. However, we get an F1 score of 85.1% for argument identification on CoNLL 2009 EN data as opposed to 80.7% reported by titovcrosslingual. This could be due to implementation differences, which unfortunately makes our EN results incomparable. For DE, the arguments are identified using the LTH system BIBREF16 , which gives an F1 score of 86.5% on the CoNLL 2009 DE data. The word alignments for the EN-DE parallel Europarl corpus are computed using GIZA++ BIBREF17 . For high-precision, only the intersecting alignments in the two directions are kept. We define two semantic arguments as aligned if their head-words are aligned. In total we get 9.3M arguments for EN (240k CoNLL + 9.1M Europarl) and 4.43M for DE (32k CoNLL + 4.4M Europarl). Out of these, 0.76M arguments are aligned."]},{"idx":10,"heading":"Main Results","paragraphs":["Since the CoNLL annotations have 21 semantic roles in total, we use 21 roles in our model as well as the baseline. Following garg2012unsupervised, we set the number of PRs to 2 (excluding INLINEFORM0 , INLINEFORM1 and INLINEFORM2 ), and SRs to 21-2=19. Table TABREF27 shows the results.","In the first setting (Line 1), we train and test the monolingual model on the CoNLL data. We observe significant improvements in F1 score over the Baseline (Line 0) in both languages. Using the CoNLL 2009 dataset alone, titovcrosslingual report an F1 score of 80.9% (PU=86.8%, CO=75.7%) for German. Thus, our monolingual model outperforms their monolingual model in German. For English, they report an F1 score of 83.6% (PU=87.5%, CO=80.1%), but note that our English results are not directly comparable to theirs due to differences argument identification, as discussed in section SECREF25 . As their argument identification score is lower, perhaps their system is discarding “difficult” arguments which leads to a higher clustering score.","In the second setting (Line 2), we use the additional monolingual Europarl (EP) data for training. We get equivalent results in English and a significant improvement in German compared to our previous setting (Line 1). The German dataset in CoNLL is quite small and benefits from the additional EP training data. In contrast, the English model is already quite good due to a relatively big dataset from CoNLL, and good accuracy syntactic parsers. Unfortunately, titovcrosslingual do not report results with this setting.","The third setting (Line 3) gives the results of our multilingual model, which adds the word alignments in the EP data. Comparing with Line 2, we get non-significant improvements in both languages. titovcrosslingual obtain an F1 score of 82.7% (PU=85.0%, CO=80.6%) for German, and 83.7% (PU=86.8%, CO=80.7%) for English. Thus, for German, our multilingual Bayesian model is able to capture the cross-lingual patterns at least as well as the external penalty term in BIBREF6 . We cannot compare the English results unfortunately due to differences in argument identification.","We also compared monolingual and bilingual training data using a setting that emulates the standard supervised setup of separate training and test data sets. We train only on the EP dataset and test on the CoNLL dataset. Lines 4 and 5 of Table TABREF27 give the results. The multilingual model obtains small improvements in both languages, which confirms the results from the standard unsupervised setup, comparing lines 2 to 3.","These results indicate that little information can be learned about semantic roles from this parallel data setup. One possible explanation for this result is that the setup itself is inadequate. Given the definition of aligned arguments, only 8% of English arguments and 17% of German arguments are aligned. This plus our experiments suggest that improving the alignment model is a necessary step to making effective use of parallel data in multilingual SRI, for example by joint modeling with SRI. We leave this exploration to future work."]},{"idx":11,"heading":"Multilingual Training with Labeled Data for One Language","paragraphs":["Another motivation for jointly modeling SRL in multiple languages is the transfer of information from a resource rich language to a resource poor language. We evaluated our model in a very general annotation transfer scenario, where we have a small labeled dataset for one language (source), and a large parallel unlabeled dataset for the source and another (target) language. We investigate whether this setting improves the parameter estimates for the target language. To this end, we clamp the role annotations of the source language in the CoNLL dataset using a predefined mapping, and do not sample them during training. This data gives us good parameters for the source language, which are used to sample the roles of the source language in the unlabeled Europarl data. The CLVs aim to capture this improvement and thereby improve sampling and parameter estimates for the target language. Table TABREF28 shows the results of this experiment. We obtain small improvements in the target languages. As in the unsupervised setting, the small percentage of aligned roles probably limits the impact of the cross-lingual information."]},{"idx":12,"heading":"Labeled Data in Monolingual Model","paragraphs":["We explored the improvement in the monolingual model in a semi-supervised setting. To this end, we randomly selected INLINEFORM0 of the sentences in the CoNLL dataset as “supervised sentences” and the rest INLINEFORM1 were kept unsupervised. Next, we clamped the role labels of the supervised sentences using the predefined mapping from Section SECREF29 . Sampling was done on the unsupervised sentences as usual. We then measured the clustering performance using the trained parameters.","To access the contribution of partial supervision better, we constructed a “supervised baseline” as follows. For predicates seen in the supervised sentences, a MAP estimate of the parameters was calculated using the predefined mapping. For the unseen predicates, the standard baseline was used.","Figures FIGREF33 and FIGREF33 show the performance variation with INLINEFORM0 . We make the following observations:","[leftmargin=*]","In both languages, at around INLINEFORM0 , the supervised baseline starts outperforming the semi-supervised model, which suggests that manually labeling about 10% of the sentences is a good enough alternative to our training procedure. Note that 10% amounts to about 3.6k sentences in German and 4k in English. We noticed that the proportion of seen predicates increases dramatically as we increase the proportion of supervised sentences. At 10% supervised sentences, the model has already seen 63% of predicates in German and 44% in English. This explains to some extent why only 10% labeled sentences are enough.","For German, it takes about 3.5% or 1260 supervised sentences to have the same performance increase as 1.5M unlabeled sentences (Line 1 to Line 2 in Table TABREF27 ). Adding about 180 more supervised sentences also covers the benefit obtained by alignments in the multilingual model (Line 2 to Line 3 in Table TABREF27 ). There is no noticeable performance difference in English.","We also evaluated the performance variation on a completely unseen CoNLL test set. Since the test set is very small compared to the training set, the clustering evaluation is not as reliable. Nonetheless, we broadly obtained the same pattern."]},{"idx":13,"heading":"Related Work","paragraphs":["As discussed in section SECREF24 , our work is closely related to the crosslingual unsupervised SRL work of titovcrosslingual. The idea of using superlingual latent variables to capture cross-lingual information was proposed for POS tagging by naseem2009multilingual, which we use here for SRL. In a semi-supervised setting, pado2009cross used a graph based approach to transfer semantic role annotations from English to German. furstenau2009graph used a graph alignment method to measure the semantic and syntactic similarity between dependency tree arguments of known and unknown verbs.","For monolingual unsupervised SRL, swier2004unsupervised presented the first work on a domain-general corpus, the British National Corpus, using 54 verbs taken from VerbNet. garg2012unsupervised proposed a Bayesian model for this problem that we use here. titov2012bayesian also proposed a closely related Bayesian model. grenager2006unsupervised proposed a generative model but their parameter space consisted of all possible linkings of syntactic constituents and semantic roles, which made unsupervised learning difficult and a separate language-specific rule based method had to be used to constrain this space. Other proposed models include an iterative split-merge algorithm BIBREF18 and a graph-partitioning based approach BIBREF1 . marquez2008semantic provide a good overview of the supervised SRL systems."]},{"idx":14,"heading":"Conclusions","paragraphs":["We propose a Bayesian model of semantic role induction (SRI) that uses crosslingual latent variables to capture role alignments in parallel corpora. The crosslingual latent variables capture correlations between roles in different languages, and regularize the parameter estimates of the monolingual models. Because this is a joint Bayesian model of multilingual SRI, we can apply the same model to a variety of training scenarios just by changing the inference procedure appropriately. We evaluate monolingual SRI with a large unlabeled dataset, bilingual SRI with a parallel corpus, bilingual SRI with annotations available for the source language, and monolingual SRI with a small labeled dataset. Increasing the amount of monolingual unlabeled data significantly improves SRI in German but not in English. Adding word alignments in parallel sentences results in small, non significant improvements, even if there is some labeled data available in the source language. This difficulty in showing the usefulness of parallel corpora for SRI may be due to the current assumptions about role alignments, which mean that only a small percentage of roles are aligned. Further analyses reveals that annotating small amounts of data can easily outperform the performance gains obtained by adding large unlabeled dataset as well as adding parallel corpora.","Future work includes training on different language pairs, on more than two languages, and with more inclusive models of role alignment."]},{"idx":15,"heading":"Acknowledgments","paragraphs":["This work was funded by the Swiss NSF grant 200021_125137 and EC FP7 grant PARLANCE."]},{"idx":16,"heading":"Figures and Tables","paragraphs":["Figure 4: Multilingual model. The CLVs and their associated parameters are drawn in bold. PR3 in language 1 is aligned to PR3 in language 2 with the corresponding CLV taking the value c2, and SRM is aligned to PR2 with the CLV taking the value c7.","Table 1: Main Results. A * denotes a significant improvement in the F1 score over the the previous line. We compute the significance using stratified shuffling and consider it significant if the p-value < 0.05.","Table 2: Results for the Multilingual Model with using labeled data for the source language.","Figure 5: F1 with a portion of the data labeled."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1604.02201","title":"Transfer Learning for Low-Resource Neural Machine Translation","sections":[{"idx":0,"heading":"Abstract","paragraphs":["The encoder-decoder framework for neural machine translation (NMT) has been shown effective in large data scenarios, but is much less effective for low-resource languages. We present a transfer learning method that significantly improves Bleu scores across a range of low-resource languages. Our key idea is to first train a high-resource language pair (the parent model), then transfer some of the learned parameters to the low-resource pair (the child model) to initialize and constrain training. Using our transfer learning method we improve baseline NMT models by an average of 5.6 Bleu on four low-resource language pairs. Ensembling and unknown word replacement add another 2 Bleu which brings the NMT performance on low-resource machine translation close to a strong syntax based machine translation (SBMT) system, exceeding its performance on one language pair. Additionally, using the transfer learning model for re-scoring, we can improve the SBMT system by an average of 1.3 Bleu, improving the state-of-the-art on low-resource machine translation."]},{"idx":1,"heading":"Introduction","paragraphs":["Neural machine translation (NMT) BIBREF0 is a promising paradigm for extracting translation knowledge from parallel text. NMT systems have achieved competitive accuracy rates under large-data training conditions for language pairs such as English-French. However, neural methods are data-hungry and learn poorly from low-count events. This behavior makes vanilla NMT a poor choice for low-resource languages, where parallel data is scarce. Table 1 shows that for 4 low-resource languages, a standard string-to-tree statistical MT system (SBMT) BIBREF1 , BIBREF2 strongly outperforms NMT, even when NMT uses the state-of-the-art local attention plus feed-input techniques from luong2015effective.","In this paper, we give a method for substantially improving NMT results on these languages. Neural models learn representations of their input that are often useful across tasks. Our key idea is to first train a high-resource language pair, then use the resulting trained network (the parent model) to initialize and constrain training for our low-resource language pair (the child model). We find that we can optimize our results by fixing certain parameters of the parent model, letting the rest be fine-tuned by the child model. We report NMT improvements from transfer learning of 5.6 Bleu on average, and we provide an analysis of why the method works. The final NMT system approaches strong SBMT baselines in in all four language pairs, and exceeds SBMT performance in one of them. Furthermore, we show that NMT is an exceptional re-scorer of `traditional' MT output; even NMT that on its own is worse than SBMT is consistently able to improve upon SBMT system output when incorporated as a re-scoring model.","We start by a brief description of our NMT model in Section \"NMT Background\" . Section \"Transfer Results\" gives some background on transfer learning and explains how we use it to improve machine translation performance. Our main experiments translating Hausa, Turkish, Uzbek and Urdu into English with the help of a French-English parent model are presented in Section \"Experiments\" . Section \"Analysis\" explores alternatives to our model to enhance understanding. We find that the choice of parent language pair affects performance, and provide an empirical upper bound on transfer performance using an artificial language. We experiment with English-only language models, copy models, and word-sorting models to show that what we transfer goes beyond monolingual information and using a translation model trained on bilingual corpora as a parent is essential. We show the effects of freezing, fine-tuning, and smarter initialization of different components of the attention-based NMT system during transfer. We compare the learning curves of transfer and no-transfer models showing that transfer solves an overfitting problem, not a search problem. We summarize our contributions in Section \"Conclusion\" ."]},{"idx":2,"heading":"NMT Background","paragraphs":["In the neural encoder-decoder framework for MT BIBREF3 , BIBREF4 , BIBREF0 , BIBREF5 , BIBREF6 , we use a recurrent neural network (encoder) to convert a source sentence into a dense, fixed-length vector. We then use another recurrent network (decoder) to convert that vector to a target sentence. In this paper, we use a two-layer encoder-decoder system (Figure 1 ) with long short-term memory (LSTM) units BIBREF7 trained to optimize maximum likelihood (via a softmax layer) with back-propagation through time BIBREF8 . Additionally, we use an attention mechanism that allows the target decoder to look back at the source encoder, specifically the local attention plus feed-input model from luong2015effective."]},{"idx":3,"heading":"Transfer Learning","paragraphs":["Transfer learning uses knowledge from a learned task to improve the performance on a related task, typically reducing the amount of required training data BIBREF9 , BIBREF10 . In natural language processing, transfer learning methods have been successfully applied to speech recognition, document classification and sentiment analysis BIBREF11 . Deep learning models discover multiple levels of representation, some of which may be useful across tasks, which makes them particularly suited to transfer learning BIBREF12 . For example, cirecsan2012transfer use a convolutional neural network to recognize handwritten characters and show positive effects of transfer between models for Latin and Chinese characters. Ours is the first study to apply transfer learning to neural machine translation.","The transfer learning approach we use is simple and effective. We first train an NMT model on a dataset where there is a large amount of bilingual data (e.g., French-English), which we call the parent model. Next, we initialize an NMT model with the already-trained parent model. This new model is then trained on a dataset with very little bilingual data (e.g., Uzbek-English), which we call the child model. This means that the low-data NMT model will not start with random weights, but with the weights from the parent model.","A justification for this approach is that in scenarios where we have limited training data, we need a strong prior distribution over models. The parent model trained on a large amount of bilingual data can be considered an anchor point, the peak of our prior distribution in model space. When we train the child model initialized with the parent model, we fix parameters likely to be useful across tasks so that they will not be changed during child-model training. In the French-English to Uzbek-English example, as a result of the initialization, the English word embeddings from the parent model are copied, but the Uzbek words are initially mapped to random French embeddings. The English embeddings should be kept but the Uzbek embeddings should be modified during training of the child model. Freezing certain portions of the parent model and fine tuning others can be considered a hard approximation to a tight prior or strong regularization applied to some of the parameters. We also experiment with ordinary L2 regularization, but find it does not significantly improve over the parameter freezing described above.","Our method results in large Bleu increases for a variety of low resource languages. In one of the four language pairs our NMT system using transfer beats a strong SBMT baseline. Not only do these transfer models do well on their own, they also give large gains when used for rescoring $n$ -best lists ( $n = 1000$ ) from the SBMT system. Section \"Experiments\" details these results."]},{"idx":4,"heading":"Experiments","paragraphs":["To evaluate how well our transfer method works we apply it to a variety of low-resource languages, both stand-alone and for re-scoring a strong SBMT baseline. We report large Bleu increases across the board with our transfer method.","For all of our experiments with low-resource languages we use French as the parent source language and for child source languages we use Hausa, Turkish, Uzbek, and Urdu. The target language is always English. Table 1 shows parallel training data set sizes for the child languages, where the language with the most data has only 1.8m English tokens. For comparison, our parent French-English model uses a training set with 300 million English tokens and achieves 26 Bleu on the development set. Table 1 also shows the SBMT system scores along with the NMT baselines that do not use transfer. There is a large gap between the SBMT and NMT systems without using our transfer method.","The SBMT system used in this paper is a string-to-tree statistical machine translation system BIBREF2 , BIBREF1 . In this system there are two count-based 5-gram language models. One is trained on the English side of the WMT 2015 English-French dataset and the other is trained on the English side of the low-resource bitext. Additionally, the SBMT models use thousands of sparsely-occurring, lexicalized syntactic features BIBREF13 .","For our NMT system, we use development sets for Hausa, Turkish, Uzbek, and Urdu to tune the learning rate, parameter initialization range, dropout rate and hidden state size for all the experiments. For training we use a minibatch size of 128, hidden state size of 1000, a target vocabulary size of 15K and a source vocabulary size of 30K. The child models are trained with a dropout probability of 0.5 as in dropout. The common parent model is trained with a dropout probability of 0.2. The learning rate used for both child and parents is 0.5 with a decay rate of 0.9 when the development perplexity does not improve. The child models are all trained for 100 epochs. We re-scale the gradient when the gradient norm is greater than 5. The initial parameter range is $[$ -0.08, +0.08 $]$ ."]},{"idx":5,"heading":"Transfer Results","paragraphs":["The results for our transfer learning method applied to the four languages above are in Table 2 . The parent models were trained on the WMT 2015 BIBREF14 French-English corpus for 5 epochs. Our baseline NMT systems (`NMT' column) all receive a large Bleu improvement when using the transfer method (the `Xfer' column) with an average Bleu improvement of 5.6. Additionally, when we use unknown word replacement from luong-EtAl:2015:ACL-IJCNLP and ensemble together 8 models (the `Final' column) we further improve upon our Bleu scores, bringing the average Bleu improvement to 7.5. Overall our method allows the NMT system to reach competitive scores and beat the SBMT system in one of the four language pairs."]},{"idx":6,"heading":"Re-scoring Results","paragraphs":["We also use the NMT model with transfer learning to re-score output $n$ -best lists ( $n = 1000$ ) from the SBMT system. Table 3 shows the results of re-scoring. Transfer NMT models give the highest gains over using re-scoring with a neural language model or an NMT system that does not use transfer. The neural language model is an LSTM RNN with 2 layers and 1000 hidden states. It has a target vocabulary of 100K and is trained using noise-contrastive estimation BIBREF15 , BIBREF16 , BIBREF17 , BIBREF18 . Additionally, it is trained using dropout with a dropout probability of 0.2 as in BIBREF19 . From re-scoring with the transfer model, we get an improvement of 1.1–1.6 Bleu points above the strong SBMT system.","We ran a number of additional experiments to understand what components of our final transfer model significantly contribute to the overall result. Section \"Analysis\" details these experiments."]},{"idx":7,"heading":"Analysis","paragraphs":["We analyze the effects of using different parent models, regularizing different parts of the child model and trying different regularization techniques."]},{"idx":8,"heading":"Different Parent Languages","paragraphs":["In the above experiments we use French-English as the parent language pair. Here, we experiment with different parent languages. In this set of experiments we use Spanish-English as the child language pair, rather then Hausa, Turkish, Uzbek, or Urdu. A description of the data used in this section is presented in Table 4 .","Our experimental results are shown in Table 5 , where we use French and German as parent languages. If we just train a model with no transfer on a small Spanish-English training set we get a Bleu score of 16.4. When using our transfer method using French and German as parent languages, we get Bleu scores of 31.0 and 29.8 respectively. As expected, French is a better parent than German for Spanish, which could be the result of the parent language being more similar to the child language.","Overall, we can see that the choice of parent language can make a difference in the Bleu score, so in our Hausa, Turkish, Uzbek, and Urdu experiments, a parent language more optimal than French might improve results."]},{"idx":9,"heading":"Effects of having Similar Parent Language","paragraphs":["Next, we look at a best-case scenario in which the parent language is as similar as possible to the child language. Here we devise a synthetic child language (called French') which is exactly like French, except its vocabulary is shuffled randomly. (e.g., “internationale” is now “pomme”, etc). This language, which looks unintelligible to human eyes, nevertheless has the same distributional and relational properties as actual French, i.e. the word that, prior to vocabulary reassignment, was `roi' (king) is likely to share distributional characteristics, and hence embedding similarity, to the word that, prior to reassignment, was `reine' (queen). Such a language should be the ideal parent model.","The results of this experiment are shown in Table 6 . We get a 4.3 Bleu improvement with an unrelated parent (i.e. French-parent and Uzbek-child), but we get a 6.7 Bleu improvement with a `closely related' parent (i.e. French-parent and French'-child). We conclude that the choice of parent model can have a strong impact on transfer models, and choosing better parents for our low-resource languages (if data for such parents can be obtained) could improve the final results."]},{"idx":10,"heading":"Ablation Analysis","paragraphs":["In all the above experiments, only the target input and output embeddings are fixed during training. In this section we analyze what happens when different parts of the model are fixed, in order to see what yields optimal performance. Figure 2 shows a diagram of the components of a sequence-to-sequence model. Table 7 shows how we begin to allow more components of the child NMT model to be trained and see the effect on performance in the model. We see that the optimal setting for transferring from French-English to Uzbek-English in terms of Bleu performance is to allow all of the components of the child model to be trained except for the input and output target embeddings.","Even though we use this setting for our main experiments, the optimum setting is likely to be language- and corpus-dependent. For Turkish, experiments show that freezing target attention parameters as well gives slightly better results. For parent-child models with closely related languages we expect freezing, or strongly regularizing, more components of the model to give better results."]},{"idx":11,"heading":"Learning Curve","paragraphs":["In Figure 3 we plot learning curves for both a transfer and a non-transfer model on training and development sets. We see that the final training set perplexities for both the transfer and non-transfer model are very similar, but the development set perplexity for the transfer model is much better.","The fact that the two models start from and converge to very different points, yet have similar training set performances, indicates that our architecture and training algorithm are able to reach a good minimum of the training objective regardless of the initialization. However, the training objective seems to have a large basin of models with similar performance and not all of them generalize well to the development set. The transfer model, starting with and staying close to a point known to perform well on a related task, is guided to a final point in the weight space that generalizes to the development set much better."]},{"idx":12,"heading":"Dictionary Initialization","paragraphs":["Using the transfer method, we always initialize input language embeddings for the child model with randomly-assigned embeddings from the parent (which has a different input language). A smarter method might be to initialize child embeddings with similar parent embeddings, where similarity is measured by word-to-word t-table probabilities. To get these probabilities we compose Uzbek-English and English-French t-tables obtained from the Berkeley Aligner BIBREF20 . We see from Figure 4 that this dictionary-based assignment results in faster improvement in the early part of the training. However the final performance is similar to our standard model, indicating that the training is able to untangle the dictionary permutation introduced by randomly-assigned embeddings."]},{"idx":13,"heading":"Different Parent Models","paragraphs":["In the above experiments, we use a parent model trained on a large French/English bilingual corpus. One might hypothesize that our gains come from exploiting the English half of the corpus as an additional language model resource. Therefore, we explore transfer learning for the child model with parent models that only use the English side of the bilingual corpus. Table 8 shows the results for these experiments where we train one parent model to copy English sentences (English-English) and another parent model to un-permute scrambled English sentences (EngPerm-English). Additionally, we train a parent model that is just an RNN language model. These results show that our transfer learning is not simply importing an English language model, but making use of translation parameters learned from the parent's large bilingual text."]},{"idx":14,"heading":"Conclusion","paragraphs":["Overall our transfer method improves NMT scores on low-resource languages by a large margin and allows our transfer NMT system to come close to the performance of a very strong SBMT system, even exceeding its performance on Hausa-English. In addition, we consistently and significantly improve state-of-the-art SBMT systems on low-resource languages when the transfer NMT system is used for re-scoring. Our experiments suggest that there is still room for improvement in selecting parent languages that are more similar to child languages, provided data for such parents can be found."]},{"idx":15,"heading":"Acknowledgments","paragraphs":["This work was carried out with funding from DARPA (HR0011-15-C-0115) and ARL/ARO (W911NF-10-1-0533)."]},{"idx":16,"heading":"Figures and Tables","paragraphs":["Table 1: NMT models with attention are outperformed by standard string-to-tree statistical MT (SBMT) when translating lowresource languages into English. Train/test bitext corpus sizes are English token counts. Single-reference, case-insensitive BLEU scores are given for held-out test corpora.","Figure 1: The encoder-decoder framework for neural machine translation (NMT) (Sutskever et al., 2014). Here, a source sentence C B A (presented in reverse order as A B C) is translated into a target sentence W X Y Z. At each step, an evolving real-valued vector summarizes the state of the encoder (blue, checkerboard) and decoder (red, lattice). Not shown here are the attention connections present in our model used by the decoder to access encoder states.","Table 2: Our method significantly improves NMT results for the translation of low-resource languages into English. Results show test-set BLEU scores. The ‘NMT’ row shows results without transfer, and the ‘Xfer’ row shows results with transfer. The ‘Final’ row shows BLEU after we ensemble 8 models and use unknown word replacement.","Table 3: Our transfer method applied to re-scoring output nbest lists from the SBMT system. The first row shows the SBMT performance with no re-scoring and the other 3 rows show the performance after re-scoring with the selected model. Note: the ‘LM’ row shows the results when an RNN LM trained on the large English corpus was used to re-score.","Table 4: Data used for a low-resource Spanish–English task. Sizes are English-side token counts.","Table 5: For a low-resource Spanish–English task, we experiment with several choices of parent model: none, French– English, and German–English. We hypothesize that French– English is best because French and Spanish are similar.","Figure 2: Our NMT model architecture, showing six blocks of parameters, in addition to source/target words and predictions. During transfer learning, we expect the source-language related blocks to change more than the target-language related blocks.","Table 6: A better match between parent and child languages should improve transfer results. We devised a child language called French′, identical to French except for word spellings. We observe that French transfer learning helps French′ (13.3→20.0) more than it helps Uzbek (10.7→15.0).","Figure 3: Uzbek–English learning curves for the NMT attention model with and without transfer learning. The training perplexity converges to a similar value in both cases. However, the development perplexity for the transfer model is significantly better.","Figure 4: Uzbek–English learning curves for the transfer model with and without dictionary-based assignment of Uzbek word types to French word embeddings (from the parent model). Dictionary-based assignment enables faster improvement in early epochs. The model variants converge, showing that the unaided model is able to untangle the initial random Uzbek/French word-type mapping without help.","Table 7: Starting with the parent French–English model (BLEU =24.4, PPL=6.2), we randomly assign Uzbek word types to French word embeddings, freeze various parameters of the neural network model ( ), and allow Uzbek–English (child model) training to modify other parts (1). The table shows how Uzbek–English BLEU and perplexity vary as we allow more parameters to be re-trained.","Table 8: Transfer for Uzbek–English NMT with parent models trained only on English data. The English–English parent learns to copy English sentences, and the EngPerm–English learns to un-permute scrambled English sentences. The LM is a 2-layer LSTM RNN language model."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1605.04278","title":"Universal Dependencies for Learner English","sections":[{"idx":0,"heading":"Abstract","paragraphs":["We introduce the Treebank of Learner English (TLE), the first publicly available syntactic treebank for English as a Second Language (ESL). The TLE provides manually annotated POS tags and Universal Dependency (UD) trees for 5,124 sentences from the Cambridge First Certificate in English (FCE) corpus. The UD annotations are tied to a pre-existing error annotation of the FCE, whereby full syntactic analyses are provided for both the original and error corrected versions of each sentence. Further on, we delineate ESL annotation guidelines that allow for consistent syntactic treatment of ungrammatical English. Finally, we benchmark POS tagging and dependency parsing performance on the TLE dataset and measure the effect of grammatical errors on parsing accuracy. We envision the treebank to support a wide range of linguistic and computational research on second language acquisition as well as automatic processing of ungrammatical language. The treebank is available at universaldependencies.org. The annotation manual used in this project and a graphical query engine are available at esltreebank.org."]},{"idx":1,"heading":"Introduction","paragraphs":["The majority of the English text available worldwide is generated by non-native speakers BIBREF0 . Such texts introduce a variety of challenges, most notably grammatical errors, and are of paramount importance for the scientific study of language acquisition as well as for NLP. Despite the ubiquity of non-native English, there is currently no publicly available syntactic treebank for English as a Second Language (ESL).","To address this shortcoming, we present the Treebank of Learner English (TLE), a first of its kind resource for non-native English, containing 5,124 sentences manually annotated with POS tags and dependency trees. The TLE sentences are drawn from the FCE dataset BIBREF1 , and authored by English learners from 10 different native language backgrounds. The treebank uses the Universal Dependencies (UD) formalism BIBREF2 , BIBREF3 , which provides a unified annotation framework across different languages and is geared towards multilingual NLP BIBREF4 . This characteristic allows our treebank to support computational analysis of ESL using not only English based but also multilingual approaches which seek to relate ESL phenomena to native language syntax.","While the annotation inventory and guidelines are defined by the English UD formalism, we build on previous work in learner language analysis BIBREF5 , BIBREF6 to formulate an additional set of annotation conventions aiming at a uniform treatment of ungrammatical learner language. Our annotation scheme uses a two-layer analysis, whereby a distinct syntactic annotation is provided for the original and the corrected version of each sentence. This approach is enabled by a pre-existing error annotation of the FCE BIBREF7 which is used to generate an error corrected variant of the dataset. Our inter-annotator agreement results provide evidence for the ability of the annotation scheme to support consistent annotation of ungrammatical structures.","Finally, a corpus that is annotated with both grammatical errors and syntactic dependencies paves the way for empirical investigation of the relation between grammaticality and syntax. Understanding this relation is vital for improving tagging and parsing performance on learner language BIBREF8 , syntax based grammatical error correction BIBREF9 , BIBREF10 , and many other fundamental challenges in NLP. In this work, we take the first step in this direction by benchmarking tagging and parsing accuracy on our dataset under different training regimes, and obtaining several estimates for the impact of grammatical errors on these tasks.","To summarize, this paper presents three contributions. First, we introduce the first large scale syntactic treebank for ESL, manually annotated with POS tags and universal dependencies. Second, we describe a linguistically motivated annotation scheme for ungrammatical learner English and provide empirical support for its consistency via inter-annotator agreement analysis. Third, we benchmark a state of the art parser on our dataset and estimate the influence of grammatical errors on the accuracy of automatic POS tagging and dependency parsing.","The remainder of this paper is structured as follows. We start by presenting an overview of the treebank in section SECREF2 . In sections SECREF3 and SECREF4 we provide background information on the annotation project, and review the main annotation stages leading to the current form of the dataset. The ESL annotation guidelines are summarized in section SECREF5 . Inter-annotator agreement analysis is presented in section SECREF6 , followed by parsing experiments in section SECREF7 . Finally, we review related work in section SECREF8 and present the conclusion in section SECREF9 ."]},{"idx":2,"heading":"Treebank Overview","paragraphs":["The TLE currently contains 5,124 sentences (97,681 tokens) with POS tag and dependency annotations in the English Universal Dependencies (UD) formalism BIBREF2 , BIBREF3 . The sentences were obtained from the FCE corpus BIBREF1 , a collection of upper intermediate English learner essays, containing error annotations with 75 error categories BIBREF7 . Sentence level segmentation was performed using an adaptation of the NLTK sentence tokenizer. Under-segmented sentences were split further manually. Word level tokenization was generated using the Stanford PTB word tokenizer.","The treebank represents learners with 10 different native language backgrounds: Chinese, French, German, Italian, Japanese, Korean, Portuguese, Spanish, Russian and Turkish. For every native language, we randomly sampled 500 automatically segmented sentences, under the constraint that selected sentences have to contain at least one grammatical error that is not punctuation or spelling.","The TLE annotations are provided in two versions. The first version is the original sentence authored by the learner, containing grammatical errors. The second, corrected sentence version, is a grammatical variant of the original sentence, generated by correcting all the grammatical errors in the sentence according to the manual error annotation provided in the FCE dataset. The resulting corrected sentences constitute a parallel corpus of standard English. Table TABREF4 presents basic statistics of both versions of the annotated sentences.","To avoid potential annotation biases, the annotations of the treebank were created manually from scratch, without utilizing any automatic annotation tools. To further assure annotation quality, each annotated sentence was reviewed by two additional annotators. To the best of our knowledge, TLE is the first large scale English treebank constructed in a completely manual fashion."]},{"idx":3,"heading":"Annotator Training","paragraphs":["The treebank was annotated by six students, five undergraduates and one graduate. Among the undergraduates, three are linguistics majors and two are engineering majors with a linguistic minor. The graduate student is a linguist specializing in syntax. An additional graduate student in NLP participated in the final debugging of the dataset.","Prior to annotating the treebank sentences, the annotators were trained for about 8 weeks. During the training, the annotators attended tutorials on dependency grammars, and learned the English UD guidelines, the Penn Treebank POS guidelines BIBREF11 , the grammatical error annotation scheme of the FCE BIBREF7 , as well as the ESL guidelines described in section SECREF5 and in the annotation manual.","Furthermore, the annotators completed six annotation exercises, in which they were required to annotate POS tags and dependencies for practice sentences from scratch. The exercises were done individually, and were followed by group meetings in which annotation disagreements were discussed and resolved. Each of the first three exercises consisted of 20 sentences from the UD gold standard for English, the English Web Treebank (EWT) BIBREF12 . The remaining three exercises contained 20-30 ESL sentences from the FCE. Many of the ESL guidelines were introduced or refined based on the disagreements in the ESL practice exercises and the subsequent group discussions. Several additional guidelines were introduced in the course of the annotation process.","During the training period, the annotators also learned to use a search tool that enables formulating queries over word and POS tag sequences as regular expressions and obtaining their annotation statistics in the EWT. After experimenting with both textual and graphical interfaces for performing the annotations, we converged on a simple text based format described in section SECREF6 , where the annotations were filled in using a spreadsheet or a text editor, and tested with a script for detecting annotation typos. The annotators continued to meet and discuss annotation issues on a weekly basis throughout the entire duration of the project."]},{"idx":4,"heading":"Annotation Procedure","paragraphs":["The formation of the treebank was carried out in four steps: annotation, review, disagreement resolution and targeted debugging."]},{"idx":5,"heading":"Annotation","paragraphs":["In the first stage, the annotators were given sentences for annotation from scratch. We use a CoNLL based textual template in which each word is annotated in a separate line. Each line contains 6 columns, the first of which has the word index (IND) and the second the word itself (WORD). The remaining four columns had to be filled in with a Universal POS tag (UPOS), a Penn Treebank POS tag (POS), a head word index (HIND) and a dependency relation (REL) according to version 1 of the English UD guidelines.","The annotation section of the sentence is preceded by a metadata header. The first field in this header, denoted with SENT, contains the FCE error coded version of the sentence. The annotators were instructed to verify the error annotation, and add new error annotations if needed. Corrections to the sentence segmentation are specified in the SEGMENT field. Further down, the field TYPO is designated for literal annotation of spelling errors and ill formed words that happen to form valid words (see section SECREF13 ).","The example below presents a pre-annotated original sentence given to an annotator.","#SENT=That time I had to sleep in <ns type= \"MD\"><c>a</c></ns> tent.","#SEGMENT=","#TYPO= *1.1cm*1.1cm*1.1cm*1.1cm#IND WORD UPOS POS HIND REL","1 That","2 time","3 I","4 had","5 to","6 sleep","7 in","8 tent","9 .","Upon completion of the original sentence, the annotators proceeded to annotate the corrected sentence version. To reduce annotation time, annotators used a script that copies over annotations from the original sentence and updates head indices of tokens that appear in both sentence versions. Head indices and relation labels were filled in only if the head word of the token appeared in both the original and corrected sentence versions. Tokens with automatically filled annotations included an additional # sign in a seventh column of each word's annotation. The # signs had to be removed, and the corresponding annotations either approved or changed as appropriate. Tokens that did not appear in the original sentence version were annotated from scratch."]},{"idx":6,"heading":"Review","paragraphs":["All annotated sentences were randomly assigned to a second annotator (henceforth reviewer), in a double blind manner. The reviewer's task was to mark all the annotations that they would have annotated differently. To assist the review process, we compiled a list of common annotation errors, available in the released annotation manual.","The annotations were reviewed using an active editing scheme in which an explicit action was required for all the existing annotations. The scheme was introduced to prevent reviewers from overlooking annotation issues due to passive approval. Specifically, an additional # sign was added at the seventh column of each token's annotation. The reviewer then had to either “sign off” on the existing annotation by erasing the # sign, or provide an alternative annotation following the # sign."]},{"idx":7,"heading":"Disagreement Resolution","paragraphs":["In the final stage of the annotation process all annotator-reviewer disagreements were resolved by a third annotator (henceforth judge), whose main task was to decide in favor of the annotator or the reviewer. Similarly to the review process, the judging task was carried out in a double blind manner. Judges were allowed to resolve annotator-reviewer disagreements with a third alternative, as well as introduce new corrections for annotation issues overlooked by the reviewers.","Another task performed by the judges was to mark acceptable alternative annotations for ambiguous structures determined through review disagreements or otherwise present in the sentence. These annotations were specified in an additional metadata field called AMBIGUITY. The ambiguity markings are provided along with the resolved version of the annotations."]},{"idx":8,"heading":"Final Debugging","paragraphs":["After applying the resolutions produced by the judges, we queried the corpus with debugging tests for specific linguistics constructions. This additional testing phase further reduced the number of annotation errors and inconsistencies in the treebank. Including the training period, the treebank creation lasted over a year, with an aggregate of more than 2,000 annotation hours."]},{"idx":9,"heading":"Annotation Scheme for ESL","paragraphs":["Our annotations use the existing inventory of English UD POS tags and dependency relations, and follow the standard UD annotation guidelines for English. However, these guidelines were formulated with grammatical usage of English in mind and do not cover non canonical syntactic structures arising due to grammatical errors. To encourage consistent and linguistically motivated annotation of such structures, we formulated a complementary set of ESL annotation guidelines.","Our ESL annotation guidelines follow the general principle of literal reading, which emphasizes syntactic analysis according to the observed language usage. This strategy continues a line of work in SLA which advocates for centering analysis of learner language around morpho-syntactic surface evidence BIBREF13 , BIBREF6 . Similarly to our framework, which includes a parallel annotation of corrected sentences, such strategies are often presented in the context of multi-layer annotation schemes that also account for error corrected sentence forms BIBREF14 , BIBREF5 , BIBREF15 .","Deploying a strategy of literal annotation within UD, a formalism which enforces cross-linguistic consistency of annotations, will enable meaningful comparisons between non-canonical structures in English and canonical structures in the author's native language. As a result, a key novel characteristic of our treebank is its ability to support cross-lingual studies of learner language."]},{"idx":10,"heading":"Literal Annotation","paragraphs":["With respect to POS tagging, literal annotation implies adhering as much as possible to the observed morphological forms of the words. Syntactically, argument structure is annotated according to the usage of the word rather than its typical distribution in the relevant context. The following list of conventions defines the notion of literal reading for some of the common non canonical structures associated with grammatical errors.","Extraneous prepositions We annotate all nominal dependents introduced by extraneous prepositions as nominal modifiers. In the following sentence, “him” is marked as a nominal modifier (nmod) instead of an indirect object (iobj) of “give”.","#SENT=...I had to give <ns type=\"UT\"><i>to</i> </ns> him water... *1.5cm*1.3cm*1.1cm*1.1cm...","21 I PRON PRP 22 nsubj","22 had VERB VBD 5 parataxis","23 to PART TO 24 mark","24 give VERB VB 22 xcomp","25 to ADP IN 26 case","26 himPRON PRP 24 nmod","27 water NOUN NN 24 dobj","...","Omitted prepositions We treat nominal dependents of a predicate that are lacking a preposition as arguments rather than nominal modifiers. In the example below, “money” is marked as a direct object (dobj) instead of a nominal modifier (nmod) of “ask”. As “you” functions in this context as a second argument of “ask”, it is annotated as an indirect object (iobj) instead of a direct object (dobj).","#SENT=...I have to ask you <ns type=\"MT\"> <c>for</c></ns> the money <ns type= \"RT\"> <i>of</i><c>for</c></ns> the tickets back. *1.5cm*1.3cm*1.1cm*1.1cm...","12 I PRON PRP 13 nsubj","13 have VERB VBP 2 conj","14 to PART TO 15 mark","15 ask VERB VB 13 xcomp","16 you PRON PRP 15 iobj","17 the DET DT 18 det","18 money NOUN NN 15 dobj","19 of ADP IN 21 case","20 the DET DT 21 det","21 ticketsNOUN NNS 18 nmod","22 back ADV RB 15 advmod","23 . PUNCT . 2 punct","Cases of erroneous tense usage are annotated according to the morphological tense of the verb. For example, below we annotate “shopping” with present participle VBG, while the correction “shop” is annotated in the corrected version of the sentence as VBP.","#SENT=...when you <ns type=\"TV\"><i>shopping</i> <c>shop</c></ns>... *1.5cm*1.3cm*1.1cm*1.1cm...","4 when ADV WRB 6 advmod","5 you PRON PRP 6 nsubj","6 shopping VERB VBG 12 advcl","...","Erroneous word formations that are contextually plausible and can be assigned with a PTB tag are annotated literally. In the following example, “stuffs” is handled as a plural count noun.","#SENT=...into fashionable <ns type=\"CN\"> <i>stuffs</i><c>stuff</c></ns>... *1.8cm*1.3cm*1.1cm*1.1cm...","7 into ADP IN 9 case","8 fashionable ADJ JJ 9 amod","9 stuffs NOUN NNS 2 ccomp","...","Similarly, in the example below we annotate “necessaryiest” as a superlative.","#SENT=The necessaryiest things... *2.1cm*1.3cm*1.1cm*1.1cm1 The DET DT 3 det","2 necessaryiest ADJ JJS 3 amod","3 things NOUN NNS 0 root","..."]},{"idx":11,"heading":"Exceptions to Literal Annotation","paragraphs":["Although our general annotation strategy for ESL follows literal sentence readings, several types of word formation errors make such readings uninformative or impossible, essentially forcing certain words to be annotated using some degree of interpretation BIBREF16 . We hence annotate the following cases in the original sentence according to an interpretation of an intended word meaning, obtained from the FCE error correction.","Spelling errors are annotated according to the correctly spelled version of the word. To support error analysis of automatic annotation tools, misspelled words that happen to form valid words are annotated in the metadata field TYPO for POS tags with respect to the most common usage of the misspelled word form. In the example below, the TYPO field contains the typical POS annotation of “where”, which is clearly unintended in the context of the sentence.","#SENT=...we <ns type=\"SX\"><i>where</i> <c>were</c></ns> invited to visit...","#TYPO=5 ADV WRB *1.5cm*1.3cm*1.1cm*1.1cm...","4 we PRON PRP 6 nsubjpass","5 where AUX VBD 6 auxpass","6 invited VERB VBN 0 root","7 to PART TO 8 mark","8 visit VERB VB 6 xcomp","...","Erroneous word formations that cannot be assigned with an existing PTB tag are annotated with respect to the correct word form.","#SENT=I am <ns type=\"IV\"><i>writting</i> <c>writing</c></ns>... *1.5cm*1.3cm*1.1cm*1.1cm1 I PRON PRP 3 nsubj","2 am AUX VBP 3 aux","3 writting VERB VBG 0 root","...","In particular, ill formed adjectives that have a plural suffix receive a standard adjectival POS tag. When applicable, such cases also receive an additional marking for unnecessary agreement in the error annotation using the attribute “ua”.","#SENT=...<ns type=\"IJ\" ua=true> <i>interestings</i><c>interesting</c></ns> things... *1.5cm*1.3cm*1.1cm*1.1cm...","6 interestings ADJ JJ 7 amod","7 things NOUN NNS 3 dobj","...","Wrong word formations that result in a valid, but contextually implausible word form are also annotated according to the word correction. In the example below, the nominal form “sale” is likely to be an unintended result of an ill formed verb. Similarly to spelling errors that result in valid words, we mark the typical literal POS annotation in the TYPO metadata field.","#SENT=...they do not <ns type=\"DV\"><i>sale</i> <c>sell</c></ns> them...","#TYPO=15 NOUN NN *1.5cm*1.3cm*1.1cm*1.1cm...","12 they PRON PRP 15 nsubj","13 do AUX VBP 15 aux","14 not PART RB 15 neg","15 sale VERB VB 0 root","16 them PRON PRP 15 dobj","...","Taken together, our ESL conventions cover many of the annotation challenges related to grammatical errors present in the TLE. In addition to the presented overview, the complete manual of ESL guidelines used by the annotators is publicly available. The manual contains further details on our annotation scheme, additional annotation guidelines and a list of common annotation errors. We plan to extend and refine these guidelines in future releases of the treebank."]},{"idx":12,"heading":"Editing Agreement","paragraphs":["We utilize our two step review process to estimate agreement rates between annotators. We measure agreement as the fraction of annotation tokens approved by the editor. Table TABREF15 presents the agreement between annotators and reviewers, as well as the agreement between reviewers and the judges. Agreement measurements are provided for both the original the corrected versions of the dataset.","Overall, the results indicate a high agreement rate in the two editing tasks. Importantly, the gap between the agreement on the original and corrected sentences is small. Note that this result is obtained despite the introduction of several ESL annotation guidelines in the course of the annotation process, which inevitably increased the number of edits related to grammatical errors. We interpret this outcome as evidence for the effectiveness of the ESL annotation scheme in supporting consistent annotations of learner language."]},{"idx":13,"heading":"Parsing Experiments","paragraphs":["The TLE enables studying parsing for learner language and exploring relationships between grammatical errors and parsing performance. Here, we present parsing benchmarks on our dataset, and provide several estimates for the extent to which grammatical errors degrade the quality of automatic POS tagging and dependency parsing.","Our first experiment measures tagging and parsing accuracy on the TLE and approximates the global impact of grammatical errors on automatic annotation via performance comparison between the original and error corrected sentence versions. In this, and subsequent experiments, we utilize version 2.2 of the Turbo tagger and Turbo parser BIBREF18 , state of the art tools for statistical POS tagging and dependency parsing.","Table TABREF16 presents tagging and parsing results on a test set of 500 TLE sentences (9,591 original tokens, 9,700 corrected tokens). Results are provided for three different training regimes. The first regime uses the training portion of version 1.3 of the EWT, the UD English treebank, containing 12,543 sentences (204,586 tokens). The second training mode uses 4,124 training sentences (78,541 original tokens, 79,581 corrected tokens) from the TLE corpus. In the third setup we combine these two training corpora. The remaining 500 TLE sentences (9,549 original tokens, 9,695 corrected tokens) are allocated to a development set, not used in this experiment. Parsing of the test sentences was performed on predicted POS tags.","The EWT training regime, which uses out of domain texts written in standard English, provides the lowest performance on all the evaluation metrics. An additional factor which negatively affects performance in this regime are systematic differences in the EWT annotation of possessive pronouns, expletives and names compared to the UD guidelines, which are utilized in the TLE. In particular, the EWT annotates possessive pronoun UPOS as PRON rather than DET, which leads the UPOS results in this setup to be lower than the PTB POS results. Improved results are obtained using the TLE training data, which, despite its smaller size, is closer in genre and syntactic characteristics to the TLE test set. The strongest PTB POS tagging and parsing results are obtained by combining the EWT with the TLE training data, yielding 95.77 POS accuracy and a UAS of 90.3 on the original version of the TLE test set.","The dual annotation of sentences in their original and error corrected forms enables estimating the impact of grammatical errors on tagging and parsing by examining the performance gaps between the two sentence versions. Averaged across the three training conditions, the POS tagging accuracy on the original sentences is lower than the accuracy on the sentence corrections by 1.0 UPOS and 0.61 POS. Parsing performance degrades by 1.9 UAS, 1.59 LA and 2.21 LAS.","To further elucidate the influence of grammatical errors on parsing quality, table TABREF17 compares performance on tokens in the original sentences appearing inside grammatical error tags to those appearing outside such tags. Although grammatical errors may lead to tagging and parsing errors with respect to any element in the sentence, we expect erroneous tokens to be more challenging to analyze compared to grammatical tokens.","This comparison indeed reveals a substantial difference between the two types of tokens, with an average gap of 5.0 UPOS, 6.65 POS, 4.67 UAS, 6.56 LA and 7.39 LAS. Note that differently from the global measurements in the first experiment, this analysis, which focuses on the local impact of remove/replace errors, suggests a stronger effect of grammatical errors on the dependency labels than on the dependency structure.","Finally, we measure tagging and parsing performance relative to the fraction of sentence tokens marked with grammatical errors. Similarly to the previous experiment, this analysis focuses on remove/replace rather than insert errors.","Figure 1 presents the average sentential performance as a function of the percentage of tokens in the original sentence marked with grammatical errors. In this experiment, we train the parser on the EWT training set and test on the entire TLE corpus. Performance curves are presented for POS, UAS and LAS on the original and error corrected versions of the annotations. We observe that while the performance on the corrected sentences is close to constant, original sentence performance is decreasing as the percentage of the erroneous tokens in the sentence grows.","Overall, our results suggest a negative, albeit limited effect of grammatical errors on parsing. This outcome contrasts a study by Geertzen et al. geertzen2013 which reported a larger performance gap of 7.6 UAS and 8.8 LAS between sentences with and without grammatical errors. We believe that our analysis provides a more accurate estimate of this impact, as it controls for both sentence content and sentence length. The latter factor is crucial, since it correlates positively with the number of grammatical errors in the sentence, and negatively with parsing accuracy."]},{"idx":14,"heading":"Related Work","paragraphs":["Previous studies on learner language proposed several annotation schemes for both POS tags and syntax BIBREF14 , BIBREF5 , BIBREF6 , BIBREF15 . The unifying theme in these proposals is a multi-layered analysis aiming to decouple the observed language usage from conventional structures in the foreign language.","In the context of ESL, Dıaz et al. diaz2010towards propose three parallel POS tag annotations for the lexical, morphological and distributional forms of each word. In our work, we adopt the distinction between morphological word forms, which roughly correspond to our literal word readings, and distributional forms as the error corrected words. However, we account for morphological forms only when these constitute valid existing PTB POS tags and are contextually plausible. Furthermore, while the internal structure of invalid word forms is an interesting object of investigation, we believe that it is more suitable for annotation as word features rather than POS tags. Our treebank supports the addition of such features to the existing annotations.","The work of Ragheb and Dickinson dickinson2009dependency,ragheb2012defining,ragheb2013 proposes ESL annotation guidelines for POS tags and syntactic dependencies based on the CHILDES annotation framework. This approach, called “morphosyntactic dependencies” is related to our annotation scheme in its focus on surface structures. Differently from this proposal, our annotations are grounded in a parallel annotation of grammatical errors and include an additional layer of analysis for the corrected forms. Moreover, we refrain from introducing new syntactic categories and dependency relations specific to ESL, thereby supporting computational treatment of ESL using existing resources for standard English. At the same time, we utilize a multilingual formalism which, in conjunction with our literal annotation strategy, facilitates linking the annotations to native language syntax.","While the above mentioned studies focus on annotation guidelines, attention has also been drawn to the topic of parsing in the learner language domain. However, due to the shortage of syntactic resources for ESL, much of the work in this area resorted to using surrogates for learner data. For example, in Foster foster2007treebanks and Foster et al. foster2008 parsing experiments are carried out on synthetic learner-like data, that was created by automatic insertion of grammatical errors to well formed English text. In Cahill et al. cahill2014 a treebank of secondary level native students texts was used to approximate learner text in order to evaluate a parser that utilizes unlabeled learner data.","Syntactic annotations for ESL were previously developed by Nagata et al. nagata2011, who annotate an English learner corpus with POS tags and shallow syntactic parses. Our work departs from shallow syntax to full syntactic analysis, and provides annotations on a significantly larger scale. Furthermore, differently from this annotation effort, our treebank covers a wide range of learner native languages. An additional syntactic dataset for ESL, currently not available publicly, are 1,000 sentences from the EFCamDat dataset BIBREF8 , annotated with Stanford dependencies BIBREF19 . This dataset was used to measure the impact of grammatical errors on parsing by comparing performance on sentences with grammatical errors to error free sentences. The TLE enables a more direct way of estimating the magnitude of this performance gap by comparing performance on the same sentences in their original and error corrected versions. Our comparison suggests that the effect of grammatical errors on parsing is smaller that the one reported in this study."]},{"idx":15,"heading":"Conclusion","paragraphs":["We present the first large scale treebank of learner language, manually annotated and double-reviewed for POS tags and universal dependencies. The annotation is accompanied by a linguistically motivated framework for handling syntactic structures associated with grammatical errors. Finally, we benchmark automatic tagging and parsing on our corpus, and measure the effect of grammatical errors on tagging and parsing quality. The treebank will support empirical study of learner syntax in NLP, corpus linguistics and second language acquisition."]},{"idx":16,"heading":"Acknowledgements","paragraphs":["We thank Anna Korhonen for helpful discussions and insightful comments on this paper. We also thank Dora Alexopoulou, Andrei Barbu, Markus Dickinson, Sue Felshin, Jeroen Geertzen, Yan Huang, Detmar Meurers, Sampo Pyysalo, Roi Reichart and the anonymous reviewers for valuable feedback on this work. This material is based upon work supported by the Center for Brains, Minds, and Machines (CBMM), funded by NSF STC award CCF-1231216."]},{"idx":17,"heading":"Figures and Tables","paragraphs":["Table 1: Statistics of the TLE. Standard deviations are denoted in parenthesis.","Table 2: Inter-annotator agreement on the entire TLE corpus. Agreement is measured as the fraction of tokens that remain unchanged after an editing round. The four evaluation columns correspond to universal POS tags, PTB POS tags, unlabeled attachment, and dependency labels. Cohen’s Kappa scores (Cohen, 1960) for POS tags and dependency labels in all evaluation conditions are above 0.96.","Table 3: Tagging and parsing results on a test set of 500 sentences from the TLE corpus. UPOS and POS are accuracies on universal and PTB tags. UAS is unlabeled attachment score, LA is dependency label accuracy, and LAS is labeled attachment score. EWT is the English UD treebank. TLEorig are original sentences from the TLE. TLEcorr are the corresponding error corrected sentences.","Table 4: Tagging and parsing results on the original version of the TLE test set for tokens marked with grammatical errors (Ungrammatical) and tokens not marked for errors (Grammatical).","Figure 1: Mean per sentence POS accuracy, UAS and LAS of the Turbo parser, as a function of the percentage of original sentence tokens marked with grammatical errors. The parser is trained on the EWT corpus, and tested on all 5,124 sentences of the TLE. Points connected by continuous lines denote performance on original TLE sentences. Points connected by dashed lines denote performance on the corresponding error corrected sentences. The number of sentences whose errors fall within each percentage range appears below the range in parenthesis."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1605.07333","title":"Combining Recurrent and Convolutional Neural Networks for Relation Classification","sections":[{"idx":0,"heading":"Abstract","paragraphs":["This paper investigates two different neural architectures for the task of relation classification: convolutional neural networks and recurrent neural networks. For both models, we demonstrate the effect of different architectural choices. We present a new context representation for convolutional neural networks for relation classification (extended middle context). Furthermore, we propose connectionist bi-directional recurrent neural networks and introduce ranking loss for their optimization. Finally, we show that combining convolutional and recurrent neural networks using a simple voting scheme is accurate enough to improve results. Our neural models achieve state-of-the-art results on the SemEval 2010 relation classification task."]},{"idx":1,"heading":"Introduction","paragraphs":["Relation classification is the task of assigning sentences with two marked entities to a predefined set of relations. The sentence “We poured the <e1>milk</e1> into the <e2>pumpkin mixture</e2>.”, for example, expresses the relation Entity-Destination(e1,e2). While early research mostly focused on support vector machines or maximum entropy classifiers BIBREF0 , BIBREF1 , recent research showed performance improvements by applying neural networks (NNs) BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 on the benchmark data from SemEval 2010 shared task 8 BIBREF8 .","This study investigates two different types of NNs: recurrent neural networks (RNNs) and convolutional neural networks (CNNs) as well as their combination. We make the following contributions:","(1) We propose extended middle context, a new context representation for CNNs for relation classification. The extended middle context uses all parts of the sentence (the relation arguments, left of the relation arguments, between the arguments, right of the arguments) and pays special attention to the middle part.","(2) We present connectionist bi-directional RNN models which are especially suited for sentence classification tasks since they combine all intermediate hidden layers for their final decision. Furthermore, the ranking loss function is introduced for the RNN model optimization which has not been investigated in the literature for relation classification before.","(3) Finally, we combine CNNs and RNNs using a simple voting scheme and achieve new state-of-the-art results on the SemEval 2010 benchmark dataset."]},{"idx":2,"heading":"Related Work","paragraphs":["In 2010, manually annotated data for relation classification was released in the context of a SemEval shared task BIBREF8 . Shared task participants used, i.a., support vector machines or maximum entropy classifiers BIBREF0 , BIBREF1 . Recently, their results on this data set were outperformed by applying NNs BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 .","zeng2014 built a CNN based only on the context between the relation arguments and extended it with several lexical features. kim2014 and others used convolutional filters of different sizes for CNNs. nguyen applied this to relation classification and obtained improvements over single filter sizes. deSantos2015 replaced the softmax layer of the CNN with a ranking layer. They showed improvements and published the best result so far on the SemEval dataset, to our knowledge.","socher used another NN architecture for relation classification: recursive neural networks that built recursive sentence representations based on syntactic parsing. In contrast, zhang investigated a temporal structured RNN with only words as input. They used a bi-directional model with a pooling layer on top."]},{"idx":3,"heading":"Convolutional Neural Networks (CNN)","paragraphs":["CNNs perform a discrete convolution on an input matrix with a set of different filters. For NLP tasks, the input matrix represents a sentence: Each column of the matrix stores the word embedding of the corresponding word. By applying a filter with a width of, e.g., three columns, three neighboring words (trigram) are convolved. Afterwards, the results of the convolution are pooled. Following collobertWeston, we perform max-pooling which extracts the maximum value for each filter and, thus, the most informative n-gram for the following steps. Finally, the resulting values are concatenated and used for classifying the relation expressed in the sentence."]},{"idx":4,"heading":"Input: Extended Middle Context","paragraphs":["One of our contributions is a new input representation especially designed for relation classification. The contexts are split into three disjoint regions based on the two relation arguments: the left context, the middle context and the right context. Since in most cases the middle context contains the most relevant information for the relation, we want to focus on it but not ignore the other regions completely. Hence, we propose to use two contexts: (1) a combination of the left context, the left entity and the middle context; and (2) a combination of the middle context, the right entity and the right context. Due to the repetition of the middle context, we force the network to pay special attention to it. The two contexts are processed by two independent convolutional and max-pooling layers. After pooling, the results are concatenated to form the sentence representation. Figure FIGREF3 depicts this procedure. It shows an examplary sentence: “He had chest pain and <e1>headaches</e1> from <e2>mold</e2> in the bedroom.” If we only considered the middle context “from”, the network might be tempted to predict a relation like Entity-Origin(e1,e2). However, by also taking the left and right context into account, the model can detect the relation Cause-Effect(e2,e1). While this could also be achieved by integrating the whole context into the model, using the whole context can have disadvantages for longer sentences: The max pooling step can easily choose a value from a part of the sentence which is far away from the mention of the relation. With splitting the context into two parts, we reduce this danger. Repeating the middle context increases the chance for the max pooling step to pick a value from the middle context."]},{"idx":5,"heading":"Convolutional Layer","paragraphs":["Following previous work (e.g., BIBREF5 , BIBREF6 ), we use 2D filters spanning all embedding dimensions. After convolution, a max pooling operation is applied that stores only the highest activation of each filter. We apply filters with different window sizes 2-5 (multi-windows) as in BIBREF5 , i.e. spanning a different number of input words."]},{"idx":6,"heading":"Recurrent Neural Networks (RNN)","paragraphs":["Traditional RNNs consist of an input vector, a history vector and an output vector. Based on the representation of the current input word and the previous history vector, a new history is computed. Then, an output is predicted (e.g., using a softmax layer). In contrast to most traditional RNN architectures, we use the RNN for sentence modeling, i.e., we predict an output vector only after processing the whole sentence and not after each word. Training is performed using backpropagation through time BIBREF9 which unfolds the recurrent computations of the history vector for a certain number of time steps. To avoid exploding gradients, we use gradient clipping with a threshold of 10 BIBREF10 ."]},{"idx":7,"heading":"Input of the RNNs","paragraphs":["Initial experiments showed that using trigrams as input instead of single words led to superior results. Hence, at timestep INLINEFORM0 we do not only give word INLINEFORM1 to the model but the trigram INLINEFORM2 by concatenating the corresponding word embeddings."]},{"idx":8,"heading":"Connectionist Bi-directional RNNs","paragraphs":["Especially for relation classification, the processing of the relation arguments might be easier with knowledge of the succeeding words. Therefore in bi-directional RNNs, not only a history vector of word INLINEFORM0 is regarded but also a future vector. This leads to the following conditioned probability for the history INLINEFORM1 at time step INLINEFORM2 : DISPLAYFORM0","Thus, the network can be split into three parts: a forward pass which processes the original sentence word by word (Equation EQREF6 ); a backward pass which processes the reversed sentence word by word (Equation ); and a combination of both (Equation ). All three parts are trained jointly. This is also depicted in Figure FIGREF7 .","Combining forward and backward pass by adding their hidden layer is similar to BIBREF7 . We, however, also add a connection to the previous combined hidden layer with weight INLINEFORM0 to be able to include all intermediate hidden layers into the final decision of the network (see Equation ). We call this “connectionist bi-directional RNN”.","In our experiments, we compare this RNN with uni-directional RNNs and bi-directional RNNs without additional hidden layer connections."]},{"idx":9,"heading":"Word Representations","paragraphs":["Words are represented by concatenated vectors: a word embedding and a position feature vector.","Pretrained word embeddings. In this study, we used the word2vec toolkit BIBREF11 to train embeddings on an English Wikipedia from May 2014. We only considered words appearing more than 100 times and added a special PADDING token for convolution. This results in an embedding training text of about 485,000 terms and INLINEFORM0 tokens. During model training, the embeddings are updated.","Position features. We incorporate randomly initialized position embeddings similar to zeng2014, nguyen and deSantos2015. In our RNN experiments, we investigate different possibilities of integrating position information: position embeddings, position embeddings with entity presence flags (flags indicating whether the current word is one of the relation arguments), and position indicators BIBREF7 ."]},{"idx":10,"heading":"Objective Function: Ranking Loss","paragraphs":["Ranking. We applied the ranking loss function proposed in deSantos2015 to train our models. It maximizes the distance between the true label INLINEFORM0 and the best competitive label INLINEFORM1 given a data point INLINEFORM2 . The objective function is DISPLAYFORM0","with INLINEFORM0 and INLINEFORM1 being the scores for the classes INLINEFORM2 and INLINEFORM3 respectively. The parameter INLINEFORM4 controls the penalization of the prediction errors and INLINEFORM5 and INLINEFORM6 are margins for the correct and incorrect classes. Following deSantos2015, we set INLINEFORM7 . We do not learn a pattern for the class Other but increase its difference to the best competitive label by using only the second summand in Equation EQREF10 during training."]},{"idx":11,"heading":"Experiments and Results","paragraphs":["We used the relation classification dataset of the SemEval 2010 task 8 BIBREF8 . It consists of sentences which have been manually labeled with 19 relations (9 directed relations and one artificial class Other). 8,000 sentences have been distributed as training set and 2,717 sentences served as test set. For evaluation, we applied the official scoring script and report the macro F1 score which also served as the official result of the shared task.","RNN and CNN models were implemented with theano BIBREF12 , BIBREF13 . For all our models, we use L2 regularization with a weight of 0.0001. For CNN training, we use mini batches of 25 training examples while we perform stochastic gradient descent for the RNN. The initial learning rates are 0.2 for the CNN and 0.01 for the RNN. We train the models for 10 (CNN) and 50 (RNN) epochs without early stopping. As activation function, we apply tanh for the CNN and capped ReLU for the RNN. For tuning the hyperparameters, we split the training data into two parts: 6.5k (training) and 1.5k (development) sentences. We also tuned the learning rate schedule on dev.","Beside of training single models, we also report ensemble results for which we combined the presented single models with a voting process."]},{"idx":12,"heading":"Performance of CNNs","paragraphs":["As a baseline system, we implemented a CNN similar to the one described by zeng2014. It consists of a standard convolutional layer with filters with only one window size, followed by a softmax layer. As input it uses the middle context. In contrast to zeng2014, our CNN does not have an additional fully connected hidden layer. Therefore, we increased the number of convolutional filters to 1200 to keep the number of parameters comparable. With this, we obtain a baseline result of 73.0. After including 5 dimensional position features, the performance was improved to 78.6 (comparable to 78.9 as reported by zeng2014 without linguistic features).","In the next step, we investigate how this result changes if we successively add further features to our CNN: multi-windows for convolution (window sizes: 2,3,4,5 and 300 feature maps each), ranking layer instead of softmax and our proposed extended middle context. Table TABREF12 shows the results. Note that all numbers are produced by CNNs with a comparable number of parameters. We also report F1 for increasing the word embedding dimensionality from 50 to 400. The position embedding dimensionality is 5 in combination with 50 dimensional word embeddings and 35 with 400 dimensional word embeddings. Our results show that especially the ranking layer and the embedding size have an important impact on the performance."]},{"idx":13,"heading":"Performance of RNNs","paragraphs":["As a baseline for the RNN models, we apply a uni-directional RNN which predicts the relation after processing the whole sentence. With this model, we achieve an F1 score of 61.2 on the SemEval test set.","Afterwards, we investigate the impact of different position features on the performance of uni-directional RNNs (position embeddings, position embeddings concatenated with a flag indicating whether the current word is an entity or not, and position indicators BIBREF7 ). The results indicate that position indicators (i.e. artificial words that indicate the entity presence) perform the best on the SemEval data. We achieve an F1 score of 73.4 with them. However, the difference to using position embeddings with entity flags is not statistically significant.","Similar to our CNN experiments, we successively vary the RNN models by using bi-directionality, by adding connections between the hidden layers (“connectionist”), by applying ranking instead of softmax to predict the relation and by increasing the word embedding dimension to 400.","The results in Table TABREF14 show that all of these variations lead to statistically significant improvements. Especially the additional hidden layer connections and the integration of the ranking layer have a large impact on the performance."]},{"idx":14,"heading":"Combination of CNNs and RNNs","paragraphs":["Finally, we combine our CNN and RNN models using a voting process. For each sentence in the test set, we apply several CNN and RNN models presented in Tables TABREF12 and TABREF14 and predict the class with the most votes. In case of a tie, we pick one of the most frequent classes randomly. The combination achieves an F1 score of 84.9 which is better than the performance of the two NN types alone. It, thus, confirms our assumption that the networks provide complementary information: while the RNN computes a weighted combination of all words in the sentence, the CNN extracts the most informative n-grams for the relation and only considers their resulting activations."]},{"idx":15,"heading":"Comparison with State of the Art","paragraphs":["Table TABREF16 shows the results of our models ER-CNN (extended ranking CNN) and R-RNN (ranking RNN) in the context of other state-of-the-art models. Our proposed models obtain state-of-the-art results on the SemEval 2010 task 8 data set without making use of any linguistic features."]},{"idx":16,"heading":"Conclusion","paragraphs":["In this paper, we investigated different features and architectural choices for convolutional and recurrent neural networks for relation classification without using any linguistic features. For convolutional neural networks, we presented a new context representation for relation classification. Furthermore, we introduced connectionist recurrent neural networks for sentence classification tasks and performed the first experiments with ranking recurrent neural networks. Finally, we showed that even a simple combination of convolutional and recurrent neural networks improved results. With our neural models, we achieved new state-of-the-art results on the SemEval 2010 task 8 benchmark data."]},{"idx":17,"heading":"Acknowledgments","paragraphs":["Heike Adel is a recipient of the Google European Doctoral Fellowship in Natural Language Processing and this research is supported by this fellowship.","This research was also supported by Deutsche Forschungsgemeinschaft: grant SCHU 2246/4-2."]},{"idx":18,"heading":"Figures and Tables","paragraphs":["Figure 1: CNN with extended contexts","Figure 2: Connectionist bi-directional RNN","Table 1: F1 score of CNN and its components, * indicates statisticial significance compared to the result in the line above (z-test, p < 0.05)","Table 2: F1 score of RNN and its components, * indicates statisticial significance compared to the result in the line above (z-test, p < 0.05)","Table 3: State-of-the-art results for relation classification"]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1606.02891","title":"Edinburgh Neural Machine Translation Systems for WMT 16","sections":[{"idx":0,"heading":"Abstract","paragraphs":["We participated in the WMT 2016 shared news translation task by building neural translation systems for four language pairs, each trained in both directions: English<->Czech, English<->German, English<->Romanian and English<->Russian. Our systems are based on an attentional encoder-decoder, using BPE subword segmentation for open-vocabulary translation with a fixed vocabulary. We experimented with using automatic back-translations of the monolingual News corpus as additional training data, pervasive dropout, and target-bidirectional models. All reported methods give substantial improvements, and we see improvements of 4.3--11.2 BLEU over our baseline systems. In the human evaluation, our systems were the (tied) best constrained system for 7 out of 8 translation directions in which we participated."]},{"idx":1,"heading":"Introduction","paragraphs":["We participated in the WMT 2016 shared news translation task by building neural translation systems for four language pairs: English INLINEFORM0 Czech, English INLINEFORM1 German, English INLINEFORM2 Romanian and English INLINEFORM3 Russian. Our systems are based on an attentional encoder-decoder BIBREF0 , using BPE subword segmentation for open-vocabulary translation with a fixed vocabulary BIBREF1 . We experimented with using automatic back-translations of the monolingual News corpus as additional training data BIBREF2 , pervasive dropout BIBREF3 , and target-bidirectional models."]},{"idx":2,"heading":"Baseline System","paragraphs":["Our systems are attentional encoder-decoder networks BIBREF0 . We base our implementation on the dl4mt-tutorial, which we enhanced with new features such as ensemble decoding and pervasive dropout.","We use minibatches of size 80, a maximum sentence length of 50, word embeddings of size 500, and hidden layers of size 1024. We clip the gradient norm to 1.0 BIBREF4 . We train the models with Adadelta BIBREF5 , reshuffling the training corpus between epochs. We validate the model every 10000 minibatches via Bleu on a validation set (newstest2013, newstest2014, or half of newsdev2016 for EN INLINEFORM0 RO). We perform early stopping for single models, and use the 4 last saved models (with models saved every 30000 minibatches) for the ensemble results. Note that ensemble scores are the result of a single training run. Due to resource limitations, we did not train ensemble components independently, which could result in more diverse models and better ensembles.","Decoding is performed with beam search with a beam size of 12. For some language pairs, we used the AmuNMT C++ decoder as a more efficient alternative to the theano implementation of the dl4mt tutorial."]},{"idx":3,"heading":"Byte-pair encoding (BPE)","paragraphs":["To enable open-vocabulary translation, we segment words via byte-pair encoding (BPE) BIBREF1 . BPE, originally devised as a compression algorithm BIBREF6 , is adapted to word segmentation as follows:","First, each word in the training vocabulary is represented as a sequence of characters, plus an end-of-word symbol. All characters are added to the symbol vocabulary. Then, the most frequent symbol pair is identified, and all its occurrences are merged, producing a new symbol that is added to the vocabulary. The previous step is repeated until a set number of merge operations have been learned.","BPE starts from a character-level segmentation, but as we increase the number of merge operations, it becomes more and more different from a pure character-level model in that frequent character sequences, and even full words, are encoded as a single symbol. This allows for a trade-off between the size of the model vocabulary and the length of training sequences. The ordered list of merge operations, learned on the training set, can be applied to any text to segment words into subword units that are in-vocabulary in respect to the training set (except for unseen characters).","To increase consistency in the segmentation of the source and target text, we combine the source and target side of the training set for learning BPE. For each language pair, we learn 89500 merge operations."]},{"idx":4,"heading":"Synthetic Training Data","paragraphs":["WMT provides task participants with large amounts of monolingual data, both in-domain and out-of-domain. We exploit this monolingual data for training as described in BIBREF2 . Specifically, we sample a subset of the available target-side monolingual corpora, translate it automatically into the source side of the respective language pair, and then use this synthetic parallel data for training. For example, for EN INLINEFORM0 RO, the back-translation is performed with a RO INLINEFORM1 EN system, and vice-versa.","2015arXiv151106709S motivate the use of monolingual data with domain adaptation, reducing overfitting, and better modelling of fluency. We sample monolingual data from the News Crawl corpora, which is in-domain with respect to the test set.","The amount of monolingual data back-translated for each translation direction ranges from 2 million to 10 million sentences. Statistics about the amount of parallel and synthetic training data are shown in Table TABREF9 . With dl4mt, we observed a translation speed of about 200000 sentences per day (on a single Titan X GPU)."]},{"idx":5,"heading":"Pervasive Dropout","paragraphs":["For English INLINEFORM0 Romanian, we observed poor performance because of overfitting. To mitigate this, we apply dropout to all layers in the network, including recurrent ones.","Previous work dropped out different units at each time step. When applied to recurrent connections, this has the downside that it impedes the information flow over long distances, and DBLP:conf/icfhr/PhamBKL14 propose to only apply dropout to non-recurrent connections.","Instead, we follow the approach suggested by 2015arXiv151205287G, and use the same dropout mask at each time step. Our implementation differs from the recommendations by 2015arXiv151205287G in one respect: we also drop words at random, but we do so on a token level, not on a type level. In other words, if a word occurs multiple times in a sentence, we may drop out any number of its occurrences, and not just none or all.","In our English INLINEFORM0 Romanian experiments, we drop out full words (both on the source and target side) with a probability of 0.1. For all other layers, the dropout probability is set to 0.2."]},{"idx":6,"heading":"Target-bidirectional Translation","paragraphs":["We found that during decoding, the model would occasionally assign a high probability to words based on the target context alone, ignoring the source sentence. We speculate that this is an instance of the label bias problem BIBREF7 .","To mitigate this problem, we experiment with training separate models that produce the target text from right-to-left (r2l), and re-scoring the n-best lists that are produced by the main (left-to-right) models with these r2l models. Since the right-to-left model will see a complementary target context at each time step, we expect that the averaged probabilities will be more robust. In parallel to our experiments, this idea was published by liu2016.","We increase the size of the n-best-list to 50 for the reranking experiments.","A possible criticism of the l-r/r-l reranking approach is that the gains actually come from adding diversity to the ensemble, since we are now using two independent runs. However experiments in BIBREF8 show that a l-r/r-l reranking systems is stronger than an ensemble created from two independent l-r runs."]},{"idx":7,"heading":"English↔\\leftrightarrow German","paragraphs":["Table TABREF13 shows results for English INLINEFORM0 German. We observe improvements of 3.4–5.7 Bleu from training with a mix of parallel and synthetic data, compared to the baseline that is only trained on parallel data. Using an ensemble of the last 4 checkpoints gives further improvements (1.3–1.7 Bleu). Our submitted system includes reranking of the 50-best output of the left-to-right model with a right-to-left model – again an ensemble of the last 4 checkpoints – with uniform weights. This yields an improvements of 0.6–1.1 Bleu."]},{"idx":8,"heading":"English↔\\leftrightarrow Czech","paragraphs":["For English INLINEFORM0 Czech, we trained our baseline model on the complete WMT16 parallel training set (including CzEng 1.6pre BIBREF9 ), until we observed convergence on our heldout set (newstest2014). This took approximately 1M minibatches, or 3 weeks. Then we continued training the model on a new parallel corpus, comprising 8.2M sentences back-translated from the Czech monolingual news2015, 5 copies of news-commentary v11, and 9M sentences sampled from Czeng 1.6pre. The model used for back-translation was a neural MT model from earlier experiments, trained on WMT15 data. The training on this synthetic mix continued for a further 400,000 minibatches.","The right-left model was trained using a similar process, but with the target side of the parallel corpus reversed prior to training. The resulting model had a slightly lower Bleu score on the dev data than the standard left-right model. We can see in Table TABREF15 that back-translation improves performance by 2.2–2.8 Bleu, and that the final system (+r2l reranking) improves by 0.7–1.0 Bleu on the ensemble of 4, and 4.3–4.9 on the baseline.","For Czech INLINEFORM0 English the training process was similar to the above, except that we created the synthetic training data (back-translated from samples of news2015 monolingual English) in batches of 2.5M, and so were able to observe the effect of increasing the amount of synthetic data. After training a baseline model on all the WMT16 parallel set, we continued training with a parallel corpus consisting of 2 copies of the 2.5M sentences of back-translated data, 5 copies of news-commentary v11, and a matching quantity of data sampled from Czeng 1.6pre. After training this to convergence, we restarted training from the baseline model using 5M sentences of back-translated data, 5 copies of news-commentary v11, and a matching quantity of data sampled from Czeng 1.6pre. We repeated this with 7.5M sentences from news2015 monolingual, and then with 10M sentences of news2015. The back-translations were, as for English INLINEFORM1 Czech, created with an earlier NMT model trained on WMT15 data. Our final Czech INLINEFORM2 English was an ensemble of 8 systems – the last 4 save-points of the 10M synthetic data run, and the last 4 save-points of the 7.5M run. We show this as ensemble8 in Table TABREF15 , and the +synthetic results are on the last (i.e. 10M) synthetic data run.","We also show in Table TABREF16 how increasing the amount of back-translated data affects the results. We see that most of the gain from back-translation comes with the first batch, but increasing the amount of back-translated data does gradually improve performance."]},{"idx":9,"heading":"English↔\\leftrightarrow Romanian","paragraphs":["The results of our English INLINEFORM0 Romanian experiments are shown in Table TABREF18 . This language pair has the smallest amount of parallel training data, and we found dropout to be very effective, yielding improvements of 4–5 Bleu.","We found that the use of diacritics was inconsistent in the Romanian training (and development) data, so for Romanian INLINEFORM0 English we removed diacritics from the Romanian source side, obtaining improvements of 1.3–1.4 Bleu.","Synthetic training data gives improvements of 4.1–5.1 Bleu. for English INLINEFORM0 Romanian, we found that the best single system outperformed the ensemble of the last 4 checkpoints on dev, and we thus submitted the best single system as primary system."]},{"idx":10,"heading":"English↔\\leftrightarrow Russian","paragraphs":["For English INLINEFORM0 Russian, we cannot effectively learn BPE on the joint vocabulary because alphabets differ. We thus follow the approach described in BIBREF1 , first mapping the Russian text into Latin characters via ISO-9 transliteration, then learning the BPE operations on the concatenation of the English and latinized Russian training data, then mapping the BPE operations back into Cyrillic alphabet. We apply the Latin BPE operations to the English data (training data and input), and both the Cyrillic and Latin BPE operations to the Russian data.","Translation results are shown in Table TABREF21 . As for the other language pairs, we observe strong improvements from synthetic training data (4–4.4 Bleu). Ensembles yield another 1.1–1.7 Bleu."]},{"idx":11,"heading":"Shared Task Results","paragraphs":["Table TABREF22 shows the ranking of our submitted systems at the WMT16 shared news translation task. Our submissions are ranked (tied) first for 5 out of 8 translation directions in which we participated: EN INLINEFORM0 CS, EN INLINEFORM1 DE, and EN INLINEFORM2 RO. They are also the (tied) best constrained system for EN INLINEFORM3 RU and RO INLINEFORM4 EN, or 7 out of 8 translation directions in total.","Our models are also used in QT21-HimL-SysComb BIBREF10 , ranked 1–2 for EN INLINEFORM0 RO, and in AMU-UEDIN BIBREF11 , ranked 2–3 for EN INLINEFORM1 RU, and 1–2 for RU INLINEFORM2 EN."]},{"idx":12,"heading":"Conclusion","paragraphs":["We describe Edinburgh's neural machine translation systems for the WMT16 shared news translation task. For all translation directions, we observe large improvements in translation quality from using synthetic parallel training data, obtained by back-translating in-domain monolingual target-side data. Pervasive dropout on all layers was used for English INLINEFORM0 Romanian, and gave substantial improvements. For English INLINEFORM1 German and English INLINEFORM2 Czech, we trained a right-to-left model with reversed target side, and we found reranking the system output with these reversed models helpful."]},{"idx":13,"heading":"Acknowledgments","paragraphs":["This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreements 645452 (QT21), 644333 (TraMOOC) and 644402 (HimL)."]},{"idx":14,"heading":"Figures and Tables","paragraphs":["Table 1: Amount of parallel and synthetic training data (number of sentences, in millions) for EN* language pairs. For synthetic data, we separate the data according to whether the original monolingual language is English or not.","Table 2: English↔German translation results (BLEU) on dev (newstest2015) and test (newstest2016). Submitted system in bold.","Table 3: English↔Czech translation results (BLEU) on dev (newstest2015) and test (newstest2016). Submitted system in bold.","Table 5: English↔Romanian translation results (BLEU) on dev (newsdev2016), and test (newstest2016). Submitted system in bold.","Table 4: Czech→English translation results (BLEU) on dev (newstest2015) and test (newstest2016), after continued training with increasing amounts of back-translated synthetic data. For each row, training was continued from the baseline model until convergence.","Table 6: English↔Russian translation results (BLEU) on dev (newstest2015) and test (newstest2016). Submitted system in bold.","Table 7: Automatic (BLEU) and human ranking of our submitted systems (uedinnmt) at WMT16 shared news translation task. Automatic rankings are taken from http://matrix.statmt.org , only considering primary systems. Human rankings include anonymous online systems, and for EN↔CS, systems from the tuning task."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1606.02892","title":"Linguistic Input Features Improve Neural Machine Translation","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Neural machine translation has recently achieved impressive results, while using little in the way of external linguistic information. In this paper we show that the strong learning capability of neural MT models does not make linguistic features redundant; they can be easily incorporated to provide further improvements in performance. We generalize the embedding layer of the encoder in the attentional encoder--decoder architecture to support the inclusion of arbitrary features, in addition to the baseline word feature. We add morphological features, part-of-speech tags, and syntactic dependency labels as input features to English<->German, and English->Romanian neural machine translation systems. In experiments on WMT16 training and test sets, we find that linguistic input features improve model quality according to three metrics: perplexity, BLEU and CHRF3. An open-source implementation of our neural MT system is available, as are sample files and configurations."]},{"idx":1,"heading":"Introduction","paragraphs":["Neural machine translation has recently achieved impressive results BIBREF0 , BIBREF1 , while learning from raw, sentence-aligned parallel text and using little in the way of external linguistic information. However, we hypothesize that various levels of linguistic annotation can be valuable for neural machine translation. Lemmatisation can reduce data sparseness, and allow inflectional variants of the same word to explicitly share a representation in the model. Other types of annotation, such as parts-of-speech (POS) or syntactic dependency labels, can help in disambiguation. In this paper we investigate whether linguistic information is beneficial to neural translation models, or whether their strong learning capability makes explicit linguistic features redundant.","Let us motivate the use of linguistic features using examples of actual translation errors by neural MT systems. In translation out of English, one problem is that the same surface word form may be shared between several word types, due to homonymy or word formation processes such as conversion. For instance, close can be a verb, adjective, or noun, and these different meanings often have distinct translations into other languages. Consider the following English INLINEFORM0 German example:","For the English source sentence in Example SECREF4 (our translation in Example SECREF5 ), a neural MT system (our baseline system from Section SECREF4 ) mistranslates close as a verb, and produces the German verb schließen (Example SECREF6 ), even though close is an adjective in this sentence, which has the German translation nah. Intuitively, part-of-speech annotation of the English input could disambiguate between verb, noun, and adjective meanings of close.","As a second example, consider the following German INLINEFORM0 English example:","German main clauses have a verb-second (V2) word order, whereas English word order is generally SVO. The German sentence (Example UID7 ; English reference in Example UID8 ) topicalizes the predicate gefährlich 'dangerous', putting the subject die Route 'the route' after the verb. Our baseline system (Example UID9 ) retains the original word order, which is highly unusual in English, especially for prose in the news domain. A syntactic annotation of the source sentence could support the attentional encoder-decoder in learning which words in the German source to attend (and translate) first.","We will investigate the usefulness of linguistic features for the language pair German INLINEFORM0 English, considering the following linguistic features:","The inclusion of lemmas is motivated by the hope for a better generalization over inflectional variants of the same word form. The other linguistic features are motivated by disambiguation, as discussed in our introductory examples."]},{"idx":2,"heading":"Neural Machine Translation","paragraphs":["We follow the neural machine translation architecture by DBLP:journals/corr/BahdanauCB14, which we will briefly summarize here.","The neural machine translation system is implemented as an attentional encoder-decoder network with recurrent neural networks.","The encoder is a bidirectional neural network with gated recurrent units BIBREF3 that reads an input sequence INLINEFORM0 and calculates a forward sequence of hidden states INLINEFORM1 , and a backward sequence INLINEFORM2 . The hidden states INLINEFORM3 and INLINEFORM4 are concatenated to obtain the annotation vector INLINEFORM5 .","The decoder is a recurrent neural network that predicts a target sequence INLINEFORM0 . Each word INLINEFORM1 is predicted based on a recurrent hidden state INLINEFORM2 , the previously predicted word INLINEFORM3 , and a context vector INLINEFORM4 . INLINEFORM5 is computed as a weighted sum of the annotations INLINEFORM6 . The weight of each annotation INLINEFORM7 is computed through an alignment model INLINEFORM8 , which models the probability that INLINEFORM9 is aligned to INLINEFORM10 . The alignment model is a single-layer feedforward neural network that is learned jointly with the rest of the network through backpropagation.","A detailed description can be found in BIBREF0 , although our implementation is based on a slightly modified form of this architecture, released for the dl4mt tutorial. Training is performed on a parallel corpus with stochastic gradient descent. For translation, a beam search with small beam size is employed."]},{"idx":3,"heading":"Adding Input Features","paragraphs":["Our main innovation over the standard encoder-decoder architecture is that we represent the encoder input as a combination of features BIBREF4 .","We here show the equation for the forward states of the encoder (for the simple RNN case; consider BIBREF0 for GRU): DISPLAYFORM0","where INLINEFORM0 is a word embedding matrix, INLINEFORM1 , INLINEFORM2 are weight matrices, with INLINEFORM3 and INLINEFORM4 being the word embedding size and number of hidden units, respectively, and INLINEFORM5 being the vocabulary size of the source language.","We generalize this to an arbitrary number of features INLINEFORM0 : DISPLAYFORM0","where INLINEFORM0 is the vector concatenation, INLINEFORM1 are the feature embedding matrices, with INLINEFORM2 , and INLINEFORM3 is the vocabulary size of the INLINEFORM4 th feature. In other words, we look up separate embedding vectors for each feature, which are then concatenated. The length of the concatenated vector matches the total embedding size, and all other parts of the model remain unchanged."]},{"idx":4,"heading":"Linguistic Input Features","paragraphs":["Our generalized model of the previous section supports an arbitrary number of input features. In this paper, we will focus on a number of well-known linguistic features. Our main empirical question is if providing linguistic features to the encoder improves the translation quality of neural machine translation systems, or if the information emerges from training encoder-decoder models on raw text, making its inclusion via explicit features redundant. All linguistic features are predicted automatically; we use Stanford CoreNLP BIBREF5 , BIBREF6 , BIBREF7 to annotate the English input for English INLINEFORM0 German, and ParZu BIBREF8 to annotate the German input for German INLINEFORM1 English. We here discuss the individual features in more detail."]},{"idx":5,"heading":"Lemma","paragraphs":["Using lemmas as input features guarantees sharing of information between word forms that share the same base form. In principle, neural models can learn that inflectional variants are semantically related, and represent them as similar points in the continuous vector space BIBREF9 . However, while this has been demonstrated for high-frequency words, we expect that a lemmatized representation increases data efficiency; low-frequency variants may even be unknown to word-level models. With character- or subword-level models, it is unclear to what extent they can learn the similarity between low-frequency word forms that share a lemma, especially if the word forms are superficially dissimilar. Consider the following two German word forms, which share the lemma liegen `lie':","liegt `lies' (3.p.sg. present)","läge `lay' (3.p.sg. subjunctive II)","The lemmatisers we use are based on finite-state methods, which ensures a large coverage, even for infrequent word forms. We use the Zmorge analyzer for German BIBREF10 , BIBREF11 , and the lemmatiser in the Stanford CoreNLP toolkit for English BIBREF6 ."]},{"idx":6,"heading":"Subword Tags","paragraphs":["In our experiments, we operate on the level of subwords to achieve open-vocabulary translation with a fixed symbol vocabulary, using a segmentation based on byte-pair encoding (BPE) BIBREF12 . We note that in BPE segmentation, some symbols are potentially ambiguous, and can either be a separate word, or a subword segment of a larger word. Also, text is represented as a sequence of subword units with no explicit word boundaries, but word boundaries are potentially helpful to learn which symbols to attend to, and when to forget information in the recurrent layers. We propose an annotation of subword structure similar to popular IOB format for chunking and named entity recognition, marking if a symbol in the text forms the beginning (B), inside (I), or end (E) of a word. A separate tag (O) is used if a symbol corresponds to the full word."]},{"idx":7,"heading":"Morphological Features","paragraphs":["For German INLINEFORM0 English, the parser annotates the German input with morphological features. Different word types have different sets of features – for instance, nouns have case, number and gender, while verbs have person, number, tense and aspect – and features may be underspecified. We treat the concatenation of all morphological features of a word, using a special symbol for underspecified features, as a string, and treat each such string as a separate feature value."]},{"idx":8,"heading":"POS Tags and Dependency Labels","paragraphs":["In our introductory examples, we motivated POS tags and dependency labels as possible disambiguators. Each word is associated with one POS tag, and one dependency label. The latter is the label of the edge connecting a word to its syntactic head, or 'ROOT' if the word has no syntactic head."]},{"idx":9,"heading":"On Using Word-level Features in a Subword Model","paragraphs":["We segment rare words into subword units using BPE. The subword tags encode the segmentation of words into subword units, and need no further modification. All other features are originally word-level features. To annotate the segmented source text with features, we copy the word's feature value to all its subword units. An example is shown in Figure FIGREF26 ."]},{"idx":10,"heading":"Evaluation","paragraphs":["We evaluate our systems on the WMT16 shared translation task English INLINEFORM0 German. The parallel training data consists of about 4.2 million sentence pairs.","To enable open-vocabulary translation, we encode words via joint BPE BIBREF12 , learning 89500 merge operations on the concatenation of the source and target side of the parallel training data. We use minibatches of size 80, a maximum sentence length of 50, word embeddings of size 500, and hidden layers of size 1024. We clip the gradient norm to 1.0 BIBREF13 . We train the models with Adadelta BIBREF14 , reshuffling the training corpus between epochs. We validate the model every 10000 minibatches via Bleu and perplexity on a validation set (newstest2013).","For neural MT, perplexity is a useful measure of how well the model can predict a reference translation given the source sentence. Perplexity is thus a good indicator of whether input features provide any benefit to the models, and we report the best validation set perplexity of each experiment. To evaluate whether the features also increase translation performance, we report case-sensitive Bleu scores with mteval-13b.perl on two test sets, newstest2015 and newstest2016. We also report chrF3 BIBREF15 , a character n-gram F INLINEFORM0 score which was found to correlate well with human judgments, especially for translations out of English BIBREF16 . The two metrics may occasionally disagree, partly because they are highly sensitive to the length of the output. Bleu is precision-based, whereas chrF3 considers both precision and recall, with a bias for recall. For Bleu, we also report whether differences between systems are statistically significant according to a bootstrap resampling significance test BIBREF17 .","We train models for about a week, and report results for an ensemble of the 4 last saved models (with models saved every 12 hours). The ensemble serves to smooth the variance between single models.","Decoding is performed with beam search with a beam size of 12.","To ensure that performance improvements are not simply due to an increase in the number of model parameters, we keep the total size of the embedding layer fixed to 500. Table TABREF29 lists the embedding size we use for linguistic features – the embedding layer size of the word-level feature varies, and is set to bring the total embedding layer size to 500. If we include the lemma feature, we roughly split the embedding vector one-to-two between the lemma feature and the word feature. The table also shows the network vocabulary size; for all features except lemmas, we can represent all feature values in the network vocabulary – in the case of words, this is due to BPE segmentation. For lemmas, we choose the same vocabulary size as for words, replacing rare lemmas with a special UNK symbol.","2015arXiv151106709S report large gains from using monolingual in-domain training data, automatically back-translated into the source language to produce a synthetic parallel training corpus. We use the synthetic corpora produced in these experiments (3.6–4.2 million sentence pairs), and we trained systems which include this data to compare against the state of the art. We note that our experiments with this data entail a syntactic annotation of automatically translated data, which may be a source of noise. For the systems with synthetic data, we double the training time to two weeks.","We also evaluate linguistic features for the lower-resourced translation direction English INLINEFORM0 Romanian, with 0.6 million sentence pairs of parallel training data, and 2.2 million sentence pairs of synthetic parallel data. We use the same linguistic features as for English INLINEFORM1 German. We follow sennrich-wmt16 in the configuration, and use dropout for the English INLINEFORM2 Romanian systems. We drop out full words (both on the source and target side) with a probability of 0.1. For all other layers, the dropout probability is set to 0.2."]},{"idx":11,"heading":"Results","paragraphs":["Table TABREF32 shows our main results for German INLINEFORM0 English, and English INLINEFORM1 German. The baseline system is a neural MT system with only one input feature, the (sub)words themselves. For both translation directions, linguistic features improve the best perplexity on the development data (47.3 INLINEFORM2 46.2, and 54.9 INLINEFORM3 52.9, respectively). For German INLINEFORM4 English, the linguistic features lead to an increase of 1.5 Bleu (31.4 INLINEFORM5 32.9) and 0.5 chrF3 (58.0 INLINEFORM6 58.5), on the newstest2016 test set. For English INLINEFORM7 German, we observe improvements of 0.6 Bleu (27.8 INLINEFORM8 28.4) and 1.2 chrF3 (56.0 INLINEFORM9 57.2).","To evaluate the effectiveness of different linguistic features in isolation, we performed contrastive experiments in which only a single feature was added to the baseline. Results are shown in Table TABREF33 . Unsurprisingly, the combination of all features (Table TABREF32 ) gives the highest improvement, averaged over metrics and test sets, but most features are beneficial on their own. Subword tags give small improvements for English INLINEFORM0 German, but not for German INLINEFORM1 English. All other features outperform the baseline in terms of perplexity, and yield significant improvements in Bleu on at least one test set. The gain from different features is not fully cumulative; we note that the information encoded in different features overlaps. For instance, both the dependency labels and the morphological features encode the distinction between German subjects and accusative objects, the former through different labels (subj and obja), the latter through grammatical case (nominative and accusative).","We also evaluated adding linguistic features to a stronger baseline, which includes synthetic parallel training data. In addition, we compare our neural systems against phrase-based (PBSMT) and syntax-based (SBSMT) systems by BIBREF18 , all of which make use of linguistic annotation on the source and/or target side. Results are shown in Table TABREF34 . For German INLINEFORM0 English, we observe similar improvements in the best development perplexity (45.2 INLINEFORM1 44.1), test set Bleu (37.5 INLINEFORM2 38.5) and chrF3 (62.2 INLINEFORM3 62.8). Our test set Bleu is on par to the best submitted system to this year's WMT 16 shared translation task, which is similar to our baseline MT system, but which also uses a right-to-left decoder for reranking BIBREF19 . We expect that linguistic input features and bidirectional decoding are orthogonal, and that we could obtain further improvements by combining the two.","For English INLINEFORM0 German, improvements in development set perplexity carry over (49.7 INLINEFORM1 48.4), but we see only small, non-significant differences in Bleu and chrF3. While we cannot clearly account for the discrepancy between perplexity and translation metrics, factors that potentially lower the usefulness of linguistic features in this setting are the stronger baseline, trained on more data, and the low robustness of linguistic tools in the annotation of the noisy, synthetic data sets. Both our baseline neural MT systems and the systems with linguistic features substantially outperform phrase-based and syntax-based systems for both translation directions.","In the previous tables, we have reported the best perplexity. To address the question about the randomness in perplexity, and whether the best perplexity just happened to be lower for the systems with linguistic features, we show perplexity on our development set as a function of training time for different systems (Figure FIGREF35 ). We can see that perplexity is consistently lower for the systems trained with linguistic features.","Table TABREF36 shows results for a lower-resourced language pair, English INLINEFORM0 Romanian. With linguistic features, we observe improvements of 1.0 Bleu over the baseline, both for the systems trained on parallel data only (23.8 INLINEFORM1 24.8), and the systems which use synthetic training data (28.2 INLINEFORM2 29.2). According to Bleu, the best submission to WMT16 was a system combination by qt21syscomb2016. Our best system is competitive with this submission.","Table TABREF37 shows translation examples of our baseline, and the system augmented with linguistic features. We see that the augmented neural MT systems, in contrast to the respective baselines, successfully resolve the reordering for the German INLINEFORM0 English example, and the disambiguation of close for the English INLINEFORM1 German example."]},{"idx":12,"heading":"Related Work","paragraphs":["Linguistic features have been used in neural language modelling BIBREF4 , and are also used in other tasks for which neural models have recently been employed, such as syntactic parsing BIBREF7 . This paper addresses the question whether linguistic features on the source side are beneficial for neural machine translation. On the target side, linguistic features are harder to obtain for a generation task such as machine translation, since this would require incremental parsing of the hypotheses at test time, and this is possible future work.","Among others, our model incorporates information from a dependency annotation, but is still a sequence-to-sequence model. 2016arXiv160306075E propose a tree-to-sequence model whose encoder computes vector representations for each phrase in the source tree. Their focus is on exploiting the (unlabelled) structure of a syntactic annotation, whereas we are focused on the disambiguation power of the functional dependency labels.","Factored translation models are often used in phrase-based SMT BIBREF21 as a means to incorporate extra linguistic information. However, neural MT can provide a much more flexible mechanism for adding such information. Because phrase-based models cannot easily generalize to new feature combinations, the individual models either treat each feature combination as an atomic unit, resulting in data sparsity, or assume independence between features, for instance by having separate language models for words and POS tags. In contrast, we exploit the strong generalization ability of neural networks, and expect that even new feature combinations, e.g. a word that appears in a novel syntactic function, are handled gracefully.","One could consider the lemmatized representation of the input as a second source text, and perform multi-source translation BIBREF22 . The main technical difference is that in our approach, the encoder and attention layers are shared between features, which we deem appropriate for the types of features that we tested."]},{"idx":13,"heading":"Conclusion","paragraphs":["In this paper we investigate whether linguistic input features are beneficial to neural machine translation, and our empirical evidence suggests that this is the case.","We describe a generalization of the encoder in the popular attentional encoder-decoder architecture for neural machine translation that allows for the inclusion of an arbitrary number of input features. We empirically test the inclusion of various linguistic features, including lemmas, part-of-speech tags, syntactic dependency labels, and morphological features, into English INLINEFORM0 German, and English INLINEFORM1 Romanian neural MT systems. Our experiments show that the linguistic features yield improvements over our baseline, resulting in improvements on newstest2016 of 1.5 Bleu for German INLINEFORM2 English, 0.6 Bleu for English INLINEFORM3 German, and 1.0 Bleu for English INLINEFORM4 Romanian.","In the future, we expect several developments that will shed more light on the usefulness of linguistic (or other) input features, and whether they will establish themselves as a core component of neural machine translation. On the one hand, the machine learning capability of neural architectures is likely to increase, decreasing the benefit provided by the features we tested. On the other hand, there is potential to explore the inclusion of novel features for neural MT, which might prove to be even more helpful than the ones we investigated, and the features we investigated may prove especially helpful for some translation settings, such as very low-resourced settings and/or translation settings with a highly inflected source language."]},{"idx":14,"heading":"Acknowledgments","paragraphs":["This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreements 645452 (QT21), and 644402 (HimL)."]},{"idx":15,"heading":"Figures and Tables","paragraphs":["Figure 1: Original dependency tree for sentence Leonidas begged in the arena ., and our feature representation after BPE segmentation.","Table 1: Vocabulary size, and size of embedding layer of linguistic features, in system that includes all features, and contrastive experiments that add a single feature over the baseline. The embedding layer size of the word feature is set to bring the total size to 500.","Table 3: Contrastive experiments with individual linguistic features: best perplexity on dev (newstest2013), and BLEU and CHRF3 on test15 (newstest2015) and test16 (newstest2016). BLEU scores that are significantly different (p < 0.05) from respective baseline are marked with (*).","Table 2: German↔English translation results: best perplexity on dev (newstest2013), and BLEU and CHRF3 on test15 (newstest2015) and test16 (newstest2016). BLEU scores that are significantly different (p < 0.05) from respective baseline are marked with (*).","Table 4: German↔English translation results with additional, synthetic training data: best perplexity on dev (newstest2013), and BLEU and CHRF3 on test15 (newstest2015) and test16 (newstest2016). BLEU scores that are significantly different (p < 0.05) from respective baseline are marked with (*).","Table 6: Translation examples illustrating the effect of adding linguistic input features.","Figure 2: English→German (black) and German→English (red) development set perplexity as a function of training time (number of minibatches) with and without linguistic features.","Table 5: English→Romanian translation results: best perplexity on newsdev2016, and BLEU and CHRF3 on newstest2016. BLEU scores that are significantly different (p < 0.05) from respective baseline are marked with (*)."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1606.03676","title":"External Lexical Information for Multilingual Part-of-Speech Tagging","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Morphosyntactic lexicons and word vector representations have both proven useful for improving the accuracy of statistical part-of-speech taggers. Here we compare the performances of four systems on datasets covering 16 languages, two of these systems being feature-based (MEMMs and CRFs) and two of them being neural-based (bi-LSTMs). We show that, on average, all four approaches perform similarly and reach state-of-the-art results. Yet better performances are obtained with our feature-based models on lexically richer datasets (e.g. for morphologically rich languages), whereas neural-based results are higher on datasets with less lexical variability (e.g. for English). These conclusions hold in particular for the MEMM models relying on our system MElt, which benefited from newly designed features. This shows that, under certain conditions, feature-based approaches enriched with morphosyntactic lexicons are competitive with respect to neural methods."]},{"idx":1,"heading":"Introduction","paragraphs":["Part-of-speech tagging is now a classic task in natural language processing, for which many systems have been developed or adapted for a large variety of languages. Its aim is to associate each “word” with a morphosyntactic tag, whose granularity can range from a simple morphosyntactic category, or part-of-speech (hereafter PoS), to finer categories enriched with morphological features (gender, number, case, tense, mood, etc.).","The use of machine learning algorithms trained on manually annotated corpora has long become the standard way to develop PoS taggers. A large variety of algorithms have been used, such as (in approximative chronological order) bigram and trigram hidden Markov models BIBREF0 , BIBREF1 , BIBREF2 , decision trees BIBREF3 , BIBREF4 , maximum entropy Markov models (MEMMs) BIBREF5 and Conditional Random Fields (CRFs) BIBREF6 , BIBREF7 . With such machine learning algorithms, it is possible to build PoS taggers for any language, provided adequate training data is available.","As a complement to annotated corpora, it has previously been shown that external lexicons are valuable sources of information, in particular morphosyntactic lexicons, which provide a large inventory of (word, PoS) pairs. Such lexical information can be used in the form of constraints at tagging time BIBREF8 , BIBREF9 or during the training process as additional features combined with standard features extracted from the training corpus BIBREF10 , BIBREF11 , BIBREF12 .","In recent years, a different approach to modelling lexical information and integrating it into natural language processing systems has emerged, namely the use of vector representations for words or word sequences BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17 , BIBREF18 . Such representations, which are generally extracted from large amounts of raw text, have proved very useful for numerous tasks including PoS tagging, in particular when used in recurrent neural networks (RNNs) and more specifically in mono- or bi-directional, word-level and/or character-level long short-term memory networks (LSTMs) BIBREF19 , BIBREF16 , BIBREF17 , BIBREF20 .","Both approaches to representing lexical properties and to integrating them into a PoS tagger improve tagging results. Yet they rely on resources of different natures. The main advantage of word vectors is that they are built in an unsupervised way, only requiring large amounts of raw textual data. They also encode finer-grained information than usual morphosyntactic lexicons, most of which do not include any quantitative data, not even simple frequency information. Conversely, lexical resources often provide information about scarcely attested words, for which corpus-based approaches such as word vector representations are of limited relevance. Moreover, morphological or morphosyntactic lexicons already exist for a number of languages, including less-resourced langauges for which it might be difficult to obtain the large amounts of raw data necessary to extract word vector representations.","Our main goal is therefore to compare the respective impact of external lexicons and word vector representations on the accuracy of PoS models. This question has already been investigated for 6 languages by BIBREF18 using the state-of-the-art CRF-based tagging system MarMoT. The authors found that their best-performing word-vector-based PoS tagging models outperform their models that rely on morphosyntactic resources (lexicons or morphological analysers). In this paper, we report on larger comparison, carried out in a larger multilingual setting and comparing different tagging models. Using different 16 datasets, we compare the performances of two feature-based models enriched with external lexicons and of two LSTM-based models enriched with word vector representations. A secondary goal of our work is to compare the relative improvements linked to the use of external lexical information in the two feature-based models, which use different models (MEMM vs. CRF) and feature sets.","More specifically, our starting point is the MElt system BIBREF12 , an MEMM tagging system. We first briefly describe this system and the way we adapted it by integrating our own set of corpus-based and lexical features. We then introduce the tagging models we have trained for 16 different languages using our adapted version of MElt. These models are trained on the Universal Dependencies (v1.2) corpus set BIBREF21 , complemented by morphosyntactic lexicons. We compare the accuracy of our models with the scores obtained by the CRF-based system MarMoT BIBREF22 , BIBREF18 , retrained on the same corpora and the same external morphosyntactic lexicons. We also compare our results to those obtained by the best bidirectional LSTM models described by BIBREF20 , which both make use of Polyglot word vector representations published by BIBREF23 . We will show that an optimised enrichment of feature-based models with morphosyntactic lexicon results in significant accuracy gains. The macro-averaged accuracy of our enriched MElt models is above that of enriched MarMoT models and virtually identical to that of LSTMs enriched with word vector representations. More precisely, per-language results indicate that lexicons provide more useful information for languages with a high lexical variability (such as morphologically rich languages), whereas word vectors are more informative for languages with a lower lexical variability (such as English)."]},{"idx":2,"heading":"MElt","paragraphs":["MElt BIBREF12 is a tagging system based on maximum entropy Markov models (MEMM) BIBREF5 , a class of discriminative models that are suitable for sequence labelling BIBREF5 . The basic set of features used by MElt is given in BIBREF12 . It is a superset of the feature sets used by BIBREF5 and BIBREF24 and includes both local standard features (for example the current word itself and its prefixes and suffixes of length 1 to 4) and contextual standard features (for example the tag just assigned to the preceding word). In particular, with respect to Ratnaparkhi's feature set, MElt's basic feature set lifts the restriction that local standard features used to analyse the internal composition of the current word should only apply to rare words.","One of the advantages of feature-based models such as MEMMs and CRFs is that complementary information can be easily added in the form of additional features. This was investigated for instance by BIBREF25 , whose best-performing model for PoS tagging dialogues was obtained with a version of MElt extended with dialogue-specific features. Yet the motivation of MElt's developers was first and foremost to investigate the best way to integrate lexical information extracted from large-scale morphosyntactic lexical resources into their models, on top of the training data BIBREF12 . They showed that performances are better when this external lexical information is integrated in the form of additional lexical features than when the external lexicon is used as constraints at tagging time. These lexical features can also be divided into local lexical features (for example the list of possible tags known to the external lexicon for the current word) and contextual lexical features (for example the list of possible tags known to the external lexicon for surrounding words). In particular, lexical contextual features provide a means to model the right context of the current word, made of words that have not yet been tagged by the system but for which the lexicon often provides a list of possible tags. Moreover, tagging accuracy for out-of-vocabulary (OOV) words is improved, as a result of the fact that words unknown to the training corpus might be known to the external lexicon.","Despite a few experiments published with MElt on languages other than French BIBREF12 , BIBREF40 , BIBREF41 , the original feature set used by MElt (standard and lexical features) was designed and tested mostly on this language, by building and evaluating tagging models on a variant of the French TreeBank. Since our goal was to carry out experiments in a multilingual setting, we have decided to design our own set of features, using the standard MElt features as a starting point. With respect to the original MElt feature set, we have added new ones, such as prefixes and suffixes of the following word, as well as a hybrid contextual feature obtained by concatenating the tag predicted for the preceding word and the tag(s) provided by the external lexicon for the following word.","In order to select the best performing feature set, we carried out a series of experiments using the multilingual dataset provided during the SPMRL parsing shared task BIBREF42 . This included discarding useless or harmful features and selecting the maximal length of the prefixes and suffixes to be used as features, both for the current word and for the following word.","We incorporated in MElt the best performing feature set, described in Table TABREF1 . All models discussed in this paper are based on this feature set."]},{"idx":3,"heading":"Corpora","paragraphs":["We carried out our experiments on the Universal Dependencies v1.2 treebanks BIBREF21 , hereafter UD1.2, from which morphosyntactically annotated corpora can be trivially extracted. All UD1.2 corpora use a common tag set, the 17 universal PoS tags, which is an extension of the tagset proposed by BIBREF43 .","As our goal is to study the impact of lexical information for PoS tagging, we have restricted our experiments to UD1.2 corpora that cover languages for which we have morphosyntactic lexicons at our disposal, and for which BIBREF20 provide results. We considered UD1.2 corpora for the following 16 languages: Bulgarian, Croatian, Czech, Danish, English, French, German, Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish and Swedish. Although this language list contains only one non-Indo-European (Indonesian), four major Indo-European sub-families are represented (Germanic, Romance, Slavic, Indo-Iranian). Overall, the 16 languages considered in our experiments are typologically, morphologically and syntactically fairly diverse."]},{"idx":4,"heading":"Lexicons","paragraphs":["We generate our external lexicons using the set of source lexicons listed in Table TABREF3 . Since external lexical information is exploited via features, there is no need for the external lexicons and the annotated corpora to use the same PoS inventory. Therefore, for each language, we simply extracted from the corresponding lexicon the PoS of each word based on its morphological tags, by removing all information provided except for its coarsest-level category. We also added entries for punctuations when the source lexicons did not contain any.","We also performed experiments in which we retained the full original tags provided by the lexicons, with all morphological features included. On average, results were slightly better than those presented in the paper, although not statistically significantly. Moreover, the granularity of tag inventories in the lexicons is diverse, which makes it difficult to draw general conclusions about results based on full tags. This is why we only report results based on (coarse) PoS extracted from the original lexicons."]},{"idx":5,"heading":"Baseline models","paragraphs":["In order to assess the respective contributions of external lexicons and word vector representations, we first compared the results of the three above-mentioned systems when trained without such additional lexical information. Table TABREF11 provides the results of MElt and MarMoT retrained on UD1.2 corpora, together with the results publised on the same corpora by BIBREF20 , using their best model not enhanced by external word vector representations —i.e. the model they call INLINEFORM0 , which is a bidirectional LSTM that combines both word and character embeddings.","These results show that Plank et al.'s (2016) bi-LSTM performs extremely well, surpassed by MarMoT on only 3 out of 16 datasets (Czech, French and Italian), and by MElt only once (Indonesian)."]},{"idx":6,"heading":"Models enriched with external lexical information","paragraphs":["Table TABREF13 provides the results of four systems enriched with lexical information. The feature-based systems MElt and MarMoT, respectively based on MEMMs and CRFs, are extended with the lexical information provided by our morphosyntactic lexicons. This extension takes the form of additional features, as described in Section SECREF2 for MElt. The results reported by BIBREF20 for their bidirectional LSTM when initialised with Polyglot embeddings trained on full wikipedias are also included, together with their new system FREQBIN, also initialised with Polyglot embeddings. FREQBIN trains bi-LSTMs to predict for each input word both a PoS and a label that represents its log frequency in the training data. As they word it, “the idea behind this model is to make the representation predictive for frequency, which encourages the model not to share representations between common and rare words, thus benefiting the handling of rare tokens.”","The results, which are also displayed in Figures FIGREF14 and FIGREF15 , show that all systems reach very similar results on average, although discrepancies can be observed from one dataset to another, on which we shall comment shortly. The best performing system in terms of macro-average is MElt (96.60%). Both bi-LSTM systems reach the same score (96.58%), the difference with MElt's results being non significant, whereas MarMoT is only 0.14% behind (96.46%). Given the better baseline scores of the neural approaches, these results show that the benefit of using external lexicons in the feature-based models MElt and MarMoT are much higher than those using Polyglot word vector representations as initialisations for bi-LSTMs.","Yet these very similar overall results reflect a different picture when focusing on OOV tagging accuracy. The best models for OOV tagging accuracy are, by far, FREQBIN models, which are beaten by MarMoT and by MElt only once each (on English and Danish respectively). The comparison on OOV tagging between MElt and MarMoT shows that MElt performs better on average than MarMoT, despite the fact that MarMoT's baseline results were better than those reached by MElt. This shows that the information provided by external morphosyntactic lexicons is better exploited by MElt's lexical features than by those used by MarMoT. On the other hand, the comparison of both bi-LSTM-based approaches confirm that the FREQBIN models is better by over 10% absolute on OOV tagging accuracy (94.28% vs. 83.59%), with 65% lower error rate.","One of the important differences between the lexical information provided by an external lexicon and word vectors built from raw corpora, apart from the very nature of the lexical information provided, is the coverage and accuracy of this lexical information on rare words. All words in a morphosyntactic lexicon are associated with information of a same granularity and quality, which is not the case with word representations such as provided by Polyglot. Models that take advantage of external lexicons should therefore perform comparatively better on datasets containing a higher proportion of rarer words, provided the lexicons' coverage is high. In order to confirm this intuition, we have used a lexical richness metric based on the type/token ratio. Since this ratio is well-known for being sensitive to corpus length, we normalised it by computing it over the 60,000 first tokens of each training set. When this normalised type/token ratio is plotted against the difference between the results of MElt and both bi-LSTM-based models, the expected correlation is clearly visible (see Figure FIGREF16 ). This explains why MElt obtains better results on the morphologically richer Slavic datasets (average normalised type/token ratio: 0.28, average accuracy difference: 0.32 compared to both bi-LSTM+Polyglot and FREQBIN+Polyglot) and, at the other end of the spectrum, significantly worse results on the English dataset (normalised type/token ratio: 0.15, average accuracy difference: -0.56 compared to bi-LSTM+Polyglot, -0.57 compared to FREQBIN+Polyglot)."]},{"idx":7,"heading":"Conclusion","paragraphs":["Two main conclusions can be drawn from our comparative results. First, feature-based tagging models adequately enriched with external morphosyntactic lexicons perform, on average, as well as bi-LSTMs enriched with word embeddings. Per-language results show that the best accuracy levels are reached by feature-based models, and in particular by our improved version of the MEMM-based system MElt, on datasets with high lexical variability (in short, for morphologically rich languages), whereas neural-based results perform better on datatsets with lower lexical variability (e.g. for English).","We have only compared the contribution of morphosyntactic lexicons to feature-based models (MEMMs, CRFs) and that of word vector representations to bi-LSTM-based models as reported by BIBREF20 . As mentioned above, work on the contribution of word vector representations to feature-based approaches has been carried out by BIBREF18 . However, the exploitation of existing morphosyntactic or morphological lexicons in neural models is a less studied question. Improvements over the state of the art might be achieved by integrating lexical information both from an external lexicon and from word vector representations into tagging models.","In that regard, further work will be required to understand which class of models perform the best. An option would be to integrate feature-based models such as a CRF with an LSTM-based layer, following recent proposals such as the one proposed by BIBREF45 for named entity recognition."]},{"idx":8,"heading":"Figures and Tables","paragraphs":["Table 1: Feature set used by our MElt models. The current word is wi = w1i . . . w ni i . Previously assigned tags for the two previous words are ti−2 and ti−1. The tag to be predicted for the current word is ti, which can be assigned any tag T in the tagset. The lex function applied to a word returns the set of all tags known to the lexicon for this word, or the singleton {_unk_} if the word is unknown to the lexicon. Boolean functions used by the local standard features have self-explanatory names.","Table 2: Information about the morphosyntactic lexicons used as external sources of lexical information in our MElt and MarMoT models. The number of entries and tagset sizes refers to the morphosyntactic lexicons we extracted and used in our models, not to the original resources.","Table 3: Overall accuracy (in %) of baseline systems, i.e. MElt and MarMoT models trained without external lexicons, and Plank et al.’s (2016) ~c + ~w models, which do not make use of Polyglot embeddings. Best scores are highlighted for each corpus.","Table 4: Accuracy (in %) of the feature-based systems MElt and MarMoT as well as the two best LSTM-based systems by Plank et al. (2016) on UD1.2 datasets, which all use the 17 “universal PoS tags”. MElt and MarMoT models integrate the external lexicons listed in Table 2, whereas bidirectional LSTM-based systems rely on Polyglot word embeddings. Best scores overall and on OOV words are highlighted for each corpus.","Figure 1: Graphical visualisation of the overall tagging accuracies for all four types of enriched models. Detailed results are given in Table 4. Languages are sorted by increasing MElt’s overall tagging scores.","Figure 2: Graphical visualisation of the OOV tagging accuracies for all types of models enriched with external lexicons. Detailed results are given in Table 4. Languages are sorted by increasing MElt’s OOV tagging scores.","Figure 3: Difference between the tagging accuracy of lexicon-enhanced MElt models and each of the two types of Polyglot-enhanced neural bi-LSTM models plotted against training sets’ normalised token/type ratio."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1607.07514","title":"Tweet2Vec: Learning Tweet Embeddings Using Character-level CNN-LSTM Encoder-Decoder","sections":[{"idx":0,"heading":"Abstract","paragraphs":["We present Tweet2Vec, a novel method for generating general-purpose vector representation of tweets. The model learns tweet embeddings using character-level CNN-LSTM encoder-decoder. We trained our model on 3 million, randomly selected English-language tweets. The model was evaluated using two methods: tweet semantic similarity and tweet sentiment categorization, outperforming the previous state-of-the-art in both tasks. The evaluations demonstrate the power of the tweet embeddings generated by our model for various tweet categorization tasks. The vector representations generated by our model are generic, and hence can be applied to a variety of tasks. Though the model presented in this paper is trained on English-language tweets, the method presented can be used to learn tweet embeddings for different languages."]},{"idx":1,"heading":"Introduction","paragraphs":["In recent years, the micro-blogging site Twitter has become a major social media platform with hundreds of millions of users. The short (140 character limit), noisy and idiosyncratic nature of tweets make standard information retrieval and data mining methods ill-suited to Twitter. Consequently, there has been an ever growing body of IR and data mining literature focusing on Twitter. However, most of these works employ extensive feature engineering to create task-specific, hand-crafted features. This is time consuming and inefficient as new features need to be engineered for every task.","In this paper, we present Tweet2Vec, a method for generating general-purpose vector representation of tweets that can be used for any classification task. Tweet2Vec removes the need for expansive feature engineering and can be used to train any standard off-the-shelf classifier (e.g., logistic regression, svm, etc). Tweet2Vec uses a CNN-LSTM encoder-decoder model that operates at the character level to learn and generate vector representation of tweets. Our method is especially useful for natural language processing tasks on Twitter where it is particularly difficult to engineer features, such as speech-act classification and stance detection (as shown in our previous works on these topics BIBREF0 , BIBREF1 ).","There has been several works on generating embeddings for words, most famously Word2Vec by Mikolov et al. BIBREF2 ). There has also been a number of different works that use encoder-decoder models based on long short-term memory (LSTM) BIBREF3 , and gated recurrent neural networks (GRU) BIBREF4 . These methods have been used mostly in the context of machine translation. The encoder maps the sentence from the source language to a vector representation, while the decoder conditions on this encoded vector for translating it to the target language. Perhaps the work most related to ours is the work of Le and Mikolov le2014distributed, where they extended the Word2Vec model to generate representations for sentences (called ParagraphVec). However, these models all function at the word level, making them ill-suited to the extremely noisy and idiosyncratic nature of tweets. Our character-level model, on the other hand, can better deal with the noise and idiosyncrasies in tweets. We plan to make our model and the data used to train it publicly available to be used by other researchers that work with tweets."]},{"idx":2,"heading":"CNN-LSTM Encoder-Decoder","paragraphs":["In this section, we describe the CNN-LSTM encoder-decoder model that operates at the character level and generates vector representation of tweets. The encoder consists of convolutional layers to extract features from the characters and an LSTM layer to encode the sequence of features to a vector representation, while the decoder consists of two LSTM layers which predict the character at each time step from the output of encoder."]},{"idx":3,"heading":"Character-Level CNN Tweet Model","paragraphs":["Character-level CNN (CharCNN) is a slight variant of the deep character-level convolutional neural network introduced by Zhang et al BIBREF5 . In this model, we perform temporal convolutional and temporal max-pooling operations, which computes one-dimensional convolution and pooling functions, respectively, between input and output. Given a discrete input function INLINEFORM0 , a discrete kernel function INLINEFORM1 and stride INLINEFORM2 , the convolution INLINEFORM3 between INLINEFORM4 and INLINEFORM5 and pooling operation INLINEFORM6 of INLINEFORM7 is calculated as: DISPLAYFORM0 DISPLAYFORM1","where INLINEFORM0 is an offset constant.","We adapted this model, which employs temporal convolution and pooling operations, for tweets. The character set includes the English alphabets, numbers, special characters and unknown character. There are 70 characters in total, given below:","abcdefghijklmnopqrstuvwxyz0123456789","-,;.!?:'\"/\\|_#$%&^*~`+-=<>()[]{}","Each character in the tweets can be encoded using one-hot vector INLINEFORM0 . Hence, the tweets are represented as a binary matrix INLINEFORM1 with padding wherever necessary, where 150 is the maximum number of characters in a tweet (140 tweet characters and padding) and 70 is the size of the character set.","Each tweet, in the form of a matrix, is now fed into a deep model consisting of four 1-d convolutional layers. A convolution operation employs a filter INLINEFORM0 , to extract n-gram character feature from a sliding window of INLINEFORM1 characters at the first layer and learns abstract textual features in the subsequent layers. The convolution in the first layer operates on sliding windows of character (size INLINEFORM2 ), and the convolutions in deeper layers are defined in a similar way. Generally, for tweet INLINEFORM3 , a feature INLINEFORM4 at layer INLINEFORM5 is generated by: DISPLAYFORM0","where INLINEFORM0 , INLINEFORM1 is the bias at layer INLINEFORM2 and INLINEFORM3 is a rectified linear unit.","This filter INLINEFORM0 is applied across all possible windows of characters in the tweet to produce a feature map. The output of the convolutional layer is followed by a 1-d max-overtime pooling operation BIBREF6 over the feature map and selects the maximum value as the prominent feature from the current filter. In this way, we apply INLINEFORM1 filters at each layer. Pooling size may vary at each layer (given by INLINEFORM2 at layer INLINEFORM3 ). The pooling operation shrinks the size of the feature representation and filters out trivial features like unnecessary combination of characters. The window length INLINEFORM4 , number of filters INLINEFORM5 , pooling size INLINEFORM6 at each layer are given in Table TABREF6 .",".93",".91","We define INLINEFORM0 to denote the character-level CNN operation on input tweet matrix INLINEFORM1 . The output from the last convolutional layer of CharCNN(T) (size: INLINEFORM2 ) is subsequently given as input to the LSTM layer. Since LSTM works on sequences (explained in Section SECREF8 and SECREF11 ), pooling operation is restricted to the first two layers of the model (as shown in Table TABREF6 )."]},{"idx":4,"heading":"Long-Short Term Memory (LSTM)","paragraphs":["In this section we briefly describe the LSTM model BIBREF7 . Given an input sequence INLINEFORM0 ( INLINEFORM1 ), LSTM computes the hidden vector sequence INLINEFORM2 ( INLINEFORM3 ) and and output vector sequence INLINEFORM4 ( INLINEFORM5 ). At each time step, the output of the module is controlled by a set of gates as a function of the previous hidden state INLINEFORM6 and the input at the current time step INLINEFORM7 , the forget gate INLINEFORM8 , the input gate INLINEFORM9 , and the output gate INLINEFORM10 . These gates collectively decide the transitions of the current memory cell INLINEFORM11 and the current hidden state INLINEFORM12 . The LSTM transition functions are defined as follows: DISPLAYFORM0","Here, INLINEFORM0 is the INLINEFORM1 function that has an output in [0, 1], INLINEFORM2 denotes the hyperbolic tangent function that has an output in INLINEFORM3 , and INLINEFORM4 denotes the component-wise multiplication. The extent to which the information in the old memory cell is discarded is controlled by INLINEFORM5 , while INLINEFORM6 controls the extent to which new information is stored in the current memory cell, and INLINEFORM7 is the output based on the memory cell INLINEFORM8 . LSTM is explicitly designed for learning long-term dependencies, and therefore we choose LSTM after the convolution layer to learn dependencies in the sequence of extracted features. In sequence-to-sequence generation tasks, an LSTM defines a distribution over outputs and sequentially predicts tokens using a softmax function. DISPLAYFORM0","where INLINEFORM0 is the activation function. For simplicity, we define INLINEFORM1 to denote the LSTM operation on input INLINEFORM2 at time-step INLINEFORM3 and the previous hidden state INLINEFORM4 ."]},{"idx":5,"heading":"The Combined Model","paragraphs":["The CNN-LSTM encoder-decoder model draws on the intuition that the sequence of features (e.g. character and word n-grams) extracted from CNN can be encoded into a vector representation using LSTM that can embed the meaning of the whole tweet. Figure FIGREF7 illustrates the complete encoder-decoder model. The input and output to the model are the tweet represented as a matrix where each row is the one-hot vector representation of the characters. The procedure for encoding and decoding is explained in the following section.","Given a tweet in the matrix form T (size: INLINEFORM0 ), the CNN (Section SECREF2 ) extracts the features from the character representation. The one-dimensional convolution involves a filter vector sliding over a sequence and detecting features at different positions. The new successive higher-order window representations then are fed into LSTM (Section SECREF8 ). Since LSTM extracts representation from sequence input, we will not apply pooling after convolution at the higher layers of Character-level CNN model. The encoding procedure can be summarized as: DISPLAYFORM0 DISPLAYFORM1","where INLINEFORM0 is an extracted feature matrix where each row can be considered as a time-step for the LSTM and INLINEFORM1 is the hidden representation at time-step INLINEFORM2 . LSTM operates on each row of the INLINEFORM3 along with the hidden vectors from previous time-step to produce embedding for the subsequent time-steps. The vector output at the final time-step, INLINEFORM4 , is used to represent the entire tweet. In our case, the size of the INLINEFORM5 is 256.","The decoder operates on the encoded representation with two layers of LSTMs. In the initial time-step, the end-to-end output from the encoding procedure is used as the original input into first LSTM layer. The last LSTM decoder generates each character, INLINEFORM0 , sequentially and combines it with previously generated hidden vectors of size 128, INLINEFORM1 , for the next time-step prediction. The prediction of character at each time step is given by: DISPLAYFORM0","where INLINEFORM0 refers to the character at time-step INLINEFORM1 , INLINEFORM2 represents the one-hot vector of the character at time-step INLINEFORM3 . The result from the softmax is a decoded tweet matrix INLINEFORM4 , which is eventually compared with the actual tweet or a synonym-replaced version of the tweet (explained in Section SECREF3 ) for learning the parameters of the model."]},{"idx":6,"heading":"Data Augmentation & Training","paragraphs":["We trained the CNN-LSTM encoder-decoder model on 3 million randomly selected English-language tweets populated using data augmentation techniques, which are useful for controlling generalization error for deep learning models. Data augmentation, in our context, refers to replicating tweet and replacing some of the words in the replicated tweets with their synonyms. These synonyms are obtained from WordNet BIBREF8 which contains words grouped together on the basis of their meanings. This involves selection of replaceable words (example of non-replaceable words are stopwords, user names, hash tags, etc) from the tweet and the number of words INLINEFORM0 to be replaced. The probability of the number, INLINEFORM1 , is given by a geometric distribution with parameter INLINEFORM2 in which INLINEFORM3 . Words generally have several synonyms, thus the synonym index INLINEFORM4 , of a given word is also determined by another geometric distribution in which INLINEFORM5 . In our encoder-decoder model, we decode the encoded representation to the actual tweet or a synonym-replaced version of the tweet from the augmented data. We used INLINEFORM6 , INLINEFORM7 for our training. We also make sure that the POS tags of the replaced words are not completely different from the actual words. For regularization, we apply a dropout mechanism after the penultimate layer. This prevents co-adaptation of hidden units by randomly setting a proportion INLINEFORM8 of the hidden units to zero (for our case, we set INLINEFORM9 ).","To learn the model parameters, we minimize the cross-entropy loss as the training objective using the Adam Optimization algorithm BIBREF9 . It is given by DISPLAYFORM0","where p is the true distribution (one-hot vector representing characters in the tweet) and q is the output of the softmax. This, in turn, corresponds to computing the negative log-probability of the true class."]},{"idx":7,"heading":"Experiments","paragraphs":["We evaluated our model using two classification tasks: Tweet semantic relatedness and Tweet sentiment classification."]},{"idx":8,"heading":"Semantic Relatedness","paragraphs":["The first evaluation is based on the SemEval 2015-Task 1: Paraphrase and Semantic Similarity in Twitter BIBREF10 . Given a pair of tweets, the goal is to predict their semantic equivalence (i.e., if they express the same or very similar meaning), through a binary yes/no judgement. The dataset provided for this task contains 18K tweet pairs for training and 1K pairs for testing, with INLINEFORM0 of these pairs being paraphrases, and INLINEFORM1 non-paraphrases. We first extract the vector representation of all the tweets in the dataset using our Tweet2Vec model. We use two features to represent a tweet pair. Given two tweet vectors INLINEFORM2 and INLINEFORM3 , we compute their element-wise product INLINEFORM4 and their absolute difference INLINEFORM5 and concatenate them together (Similar to BIBREF11 ). We then train a logistic regression model on these features using the dataset. Cross-validation is used for tuning the threshold for classification. In contrast to our model, most of the methods used for this task were largely based on extensive use of feature engineering, or a combination of feature engineering with semantic spaces. Table 2 shows the performance of our model compared to the top four models in the SemEval 2015 competition, and also a model that was trained using ParagraphVec. Our model (Tweet2Vec) outperforms all these models, without resorting to extensive task-specific feature engineering.",".93",".91"]},{"idx":9,"heading":"Sentiment Classification","paragraphs":["The second evaluation is based on the SemEval 2015-Task 10B: Twitter Message Polarity Classification BIBREF12 . Given a tweet, the task is to classify it as either positive, negative or neutral in sentiment. The size of the training and test sets were 9,520 tweets and 2,380 tweets respectively ( INLINEFORM0 positive, INLINEFORM1 negative, and INLINEFORM2 neutral).","As with the last task, we first extract the vector representation of all the tweets in the dataset using Tweet2Vec and use that to train a logistic regression classifier using the vector representations. Even though there are three classes, the SemEval task is a binary task. The performance is measured as the average F1-score of the positive and the negative class. Table 3 shows the performance of our model compared to the top four models in the SemEval 2015 competition (note that only the F1-score is reported by SemEval for this task) and ParagraphVec. Our model outperforms all these models, again without resorting to any feature engineering.",".93",".91"]},{"idx":10,"heading":"Conclusion and Future Work","paragraphs":["In this paper, we presented Tweet2Vec, a novel method for generating general-purpose vector representation of tweets, using a character-level CNN-LSTM encoder-decoder architecture. To the best of our knowledge, ours is the first attempt at learning and applying character-level tweet embeddings. Our character-level model can deal with the noisy and peculiar nature of tweets better than methods that generate embeddings at the word level. Our model is also robust to synonyms with the help of our data augmentation technique using WordNet.","The vector representations generated by our model are generic, and thus can be applied to tasks of different nature. We evaluated our model using two different SemEval 2015 tasks: Twitter semantic relatedness, and sentiment classification. Simple, off-the-shelf logistic regression classifiers trained using the vector representations generated by our model outperformed the top-performing methods for both tasks, without the need for any extensive feature engineering. This was despite the fact that due to resource limitations, our Tweet2Vec model was trained on a relatively small set (3 million tweets). Also, our method outperformed ParagraphVec, which is an extension of Word2Vec to handle sentences. This is a small but noteworthy illustration of why our tweet embeddings are best-suited to deal with the noise and idiosyncrasies of tweets.","For future work, we plan to extend the method to include: 1) Augmentation of data through reordering the words in the tweets to make the model robust to word-order, 2) Exploiting attention mechanism BIBREF13 in our model to improve alignment of words in tweets during decoding, which could improve the overall performance."]},{"idx":11,"heading":"Figures and Tables","paragraphs":["Table 1: Layer Parameters of CharCNN","Table 2: Results of the paraphrase and semantic similarity in Twitter task.","Table 3: Results of Twitter sentiment classification task."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1608.01084","title":"To Swap or Not to Swap? Exploiting Dependency Word Pairs for Reordering in Statistical Machine Translation","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Reordering poses a major challenge in machine translation (MT) between two languages with significant differences in word order. In this paper, we present a novel reordering approach utilizing sparse features based on dependency word pairs. Each instance of these features captures whether two words, which are related by a dependency link in the source sentence dependency parse tree, follow the same order or are swapped in the translation output. Experiments on Chinese-to-English translation show a statistically significant improvement of 1.21 BLEU point using our approach, compared to a state-of-the-art statistical MT system that incorporates prior reordering approaches."]},{"idx":1,"heading":"Introduction","paragraphs":["Reordering in machine translation (MT) is a crucial process to get the correct translation output word order given an input source sentence, as word order reflects meaning. It remains a major challenge, especially for language pairs with a significant word order difference. Phrase-based MT systems BIBREF0 generally adopt a reordering model that predicts reordering based on the span of a phrase and that of the adjacent phrase BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 .","The above methods do not explicitly preserve the relationship between words in the source sentence, which reflects the sentence meaning. Word relationship in a sentence can be captured by its dependency parse tree, in which each word $w$ is a tree node connected to its head node $h_w$ , another word, indicating that the former is a dependent (child) of the latter.","Dependency parsing has been used for reordering in statistical machine translation (SMT). Its usage is well-known in the pre-ordering approach, where a source sentence is reordered before the actual translation. Dependency-based pre-ordering can be performed either by a rule-based approach based on manually specified human linguistic knowledge BIBREF5 , BIBREF6 , or by a learning approach BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 . Dependency parsing has also been used in reordering approaches integrated with decoding to determine the next source phrase to translate after translating the current source phrase BIBREF13 , BIBREF14 , BIBREF15 .","In this paper, we propose a novel reordering approach integrated with translation. We propose sparse feature functions based on the pre-ordering rules of BIBREF6 . However, in contrast to the manual rule-based pre-ordering approach of BIBREF6 , the weights of our sparse feature functions are automatically learned and used during the actual translation process, without an explicit pre-ordering step. Our approach detects and exploits the reordering of each dependency word pair in the source sentence during phrase-based decoding."]},{"idx":2,"heading":"Dependency Word Pair Features","paragraphs":["We define a set of sparse features based on dependency tree word pairs to be learned and used in a phrase-based SMT beam search decoding algorithm."]},{"idx":3,"heading":"Capturing Word Pair Ordering in Phrase-Based Beam Search","paragraphs":["The phrase-based approach is a state-of-the-art approach for SMT, taking phrases, defined as a sequence of one or more words, as its translation units. It is performed by a beam search algorithm BIBREF16 , in which the search process produces translation from left to right in the translation output. The search is organized into hypotheses, each of which represents an input sentence phrase covered and its possible translation.","As the beam search can choose input phrases in any order, the target-language phrase sequence in the translation output may not follow the original source sentence order. The sequence determines the translation output order and enables translation output reordering for language pairs with differences in word order, such as Chinese and English.","When a word $f_i$ in a source sentence ${\\bf f}=f_{1}^{N}$ is covered by a hypothesis, it is known that the words $\\lbrace f_{l}|f_{l} \\in f_{1}^{i-1} \\wedge \\lnot {translated(f_l)}\\rbrace $ on the left of $f_i$ and the words $\\lbrace f_{r}|f_{r} \\in f_{i+1}^{N} \\wedge \\lnot {translated(f_r)}\\rbrace $ on the right of $f_i$ that have not been translated will be translated after (appearing on the right of) the translation of $f_i$ . As $f_l$ is before $f_i$ in the source sentence, but the translation of $f_l$ is after the translation of ${\\bf f}=f_{1}^{N}$0 , the translations of ${\\bf f}=f_{1}^{N}$1 and ${\\bf f}=f_{1}^{N}$2 are swapped. Meanwhile, ${\\bf f}=f_{1}^{N}$3 is after ${\\bf f}=f_{1}^{N}$4 both in the source sentence and in the translation, therefore the translations of ${\\bf f}=f_{1}^{N}$5 and ${\\bf f}=f_{1}^{N}$6 are in-order. Internally within a phrase, the ordering of each of its words in the translation depends on the phrasal word alignment, which is stored in the phrase table.","As each word in the source sentence is a node of the source dependency parse tree, the above notion can be used for reordering based on the source dependency parse tree. Instead of capturing all pairwise relations, we are only interested in the relations between a word and its related words, defined collectively as its head, sibling, and child words in the dependency parse tree."]},{"idx":4,"heading":"Dependency Swap Features","paragraphs":["We define our dependency swap features, following the rule template definition for dependency swap rules BIBREF6 , which defines rule instances based on the word pairs with head-child or sibling relationship. However, the difference is that our approach does not require manually specifying which dependency labels are the conditions to swap words, but learns them automatically.","In our approach, each rule instance based on the above template becomes a Boolean sparse feature function BIBREF17 . The function parameters are the word pair specification and output order. While BIBREF6 caidependency-based2014 defined the rules only by the dependency labels, we define our feature functions for each word pair by the dependency labels, the POS tags, and the combination of both, resulting in a group of four feature functions for every word pair ordering. The dependency link label of a word $x$ is defined as the label of the link connecting $x$ to its head word. Henceforth for each word $x$ , $L(x)$ and $T(x)$ denote the dependency link label and POS tag of $x$ respectively.","Following the dependency swap rule template, we define two types of feature function templates, namely head-child and sibling. The head-child feature functions are equal to 1 if a head word $x_h$ and its child word $x_c$ (where $x_h$ is on the $p\\in \\lbrace left,right\\rbrace $ of $x_c$ in the source sentence) take a certain ordering $o$ (which can be in-order ( $io$ ) or swapped ( $sw$ )) in the translation output, and 0 otherwise. This group of feature functions is defined as","$$H_{hc}(x_h,x_c,p,o) = \\left[\\begin{array}{c} h_{hc}(L(x_h),L(x_c),p,o) \\\\ h_{hc}(T(x_h),T(x_c),p,o) \\\\ h_{hc}(L(x_h),T(x_c),p,o) \\\\ h_{hc}(T(x_h),L(x_c),p,o) \\end{array}\\right]$$ (Eq. 6)","Similarly, the sibling feature functions are equal to 1 if siblings $x_l$ and $x_r$ ( $x_l$ on the left of $x_r$ in the source sentence) take a certain ordering $o$ in the translation output, and 0 otherwise. This group of feature functions is defined as","$$H_{sib}(x_l,x_r,o) = \\left[\\begin{array}{c} h_{sib}(L(x_l),L(x_r),o) \\\\ h_{sib}(T(x_l),T(x_r),o) \\\\ h_{sib}(L(x_l),T(x_r),o) \\\\ h_{sib}(T(x_l),L(x_r),o) \\end{array}\\right]$$ (Eq. 7)","UTF8gbsn Each dependency swap feature that is set to 1 when a hypothesis is generated captures two source words, one is covered by the current hypothesis, while the other has not yet been translated. These word pairs have either head-child or sibling relationship as defined above. Figure 1 is an illustration of how word order is detected during beam search decoding and the swap features involved. When “发表 讲话 $\\rightarrow $ made a speech” is translated after “佐科威 $\\rightarrow $ Jokowi”, it is known that the head “发表” is before the child “讲话” ( $p=left$ ) and their translations also follow the same word order, setting the value of the following four features to 1: $H_{hc}(\\text{发表},\\text{讲话},left,io) = \\left[\\begin{array}{c} h_{hc}(root,dobj,left,io) \\\\ h_{hc}(VV,NN,left,io) \\\\ h_{hc}(root,NN,left,io) \\\\ h_{hc}(VV,dobj,left,io) \\end{array}\\right]$","On the other hand, the head “发表” is after the child “在” in the source sentence ( $p=right$ ), but “在” has not been translated. Therefore, the translation of “发表” will be swapped with that of “在”, setting the four features in $H_{hc}(\\text{发表},\\text{在},right,sw)$ to 1. Similarly, “讲话” is after its sibling “在” in the source sentence, but “在” has not been translated, resulting in the translation of the two words being swapped and setting the four features in $H_{sib}(\\text{在},\\text{讲话},sw)$ to 1. The features for “佐科威” such as $H_{hc}(\\text{发表},\\text{佐科威},right,io)$ and $H_{sib}(\\text{佐科威},\\text{讲话},io)$ are not set to 1 when the hypothesis “发表 讲话 $\\rightarrow $ made a speech” is generated. Those features were set to 1 with the previous hypothesis “佐科威 $\\rightarrow $ Jokowi”."]},{"idx":5,"heading":"Dependency Distortion Penalty","paragraphs":["To encourage the translation output that conforms to the dependency parse structure, we impose a penalty feature that discourages translation candidates not conforming to the dependency parse subtree BIBREF13 . This assigns a penalty if the translation of the current phrase results in a source dependency parse subtree to be split in the output translation.","UTF8gbsn The translation output shown in Figure 1 has each dependency parse subtree grouped together, therefore incurring no penalty. However, there can be a case when the translation has produced “Jokowi made a speech in”, then translate “昨天” to “yesterday”. This translation will break the cohesion of the source dependency parse subtree “在 北京” and is bad. This is the case when a dependency distortion penalty is incurred."]},{"idx":6,"heading":"Modified Future Cost","paragraphs":["Phrase-based SMT beam search decoding involves future cost, not accumulated over the translation hypotheses but used to reduce search space BIBREF16 . This predicts the cost to translate the remaining untranslated phrases.","Based on our dependency swap features, we incorporate the future cost for each untranslated dependency word pairs. The future cost assumes that the untranslated words will be ordered in the most likely ordering. However, if an untranslated word $x$ is an ancestor (in the dependency parse tree) of a word covered by the current hypothesis, the future cost assumes that $x$ precedes all of its related words. This follows the assumption that each subtree will have a contiguous translation BIBREF13 ."]},{"idx":7,"heading":"Other Sparse Features","paragraphs":["This section describes other sparse features from previous work to compare to our method."]},{"idx":8,"heading":"Sparse Reordering Orientation Features","paragraphs":["We incorporate sparse reordering orientation features BIBREF4 . The features are derived from the reordering orientation model, capturing the source position of the current phrase being translated with respect to the previously translated phrase(s), for which three orientation types are defined:","Monotone (M), if the current phrase and the previously translated unit are adjacent in the input sentence and the former follows the latter in it, e.g., “Beijing” with respect to the previous phrase “in” in Figure 1 .","Swapped (S), if the current phrase and the previously translated unit are adjacent in the input sentence but the former precedes the latter in it (example below).","Discontinuous (D), if the current phrase and the previously translated unit are not adjacent in the input sentence, e.g., “made a speech” with respect to the previous phrase “Jokowi” in Figure 1 .","UTF8gbsn BIBREF4 cherryimproved2013 designed his sparse reordering orientation model following the hierarchical reordering (HR) model BIBREF3 , capturing the relative position of the current phrase (covered by the current hypothesis) with respect to the largest chunk of contiguous source phrases that form a contiguous translation before this phrase. Therefore, in Figure 1 , when the decoding produces a phrase “yesterday” after “Beijing”, the orientation of “yesterday” is swapped instead of discontinuous, as “made a speech in Beijing” is formed by contiguous phrases “在 北京 发表 讲话”, which is adjacent to “昨天”.","While the original (non-sparse) reordering orientation model is based on the phrase orientation probability in the parallel training data and defines a single feature function on it, the sparse model defines one feature function for each reordering phenomenon during decoding. We define the sparse feature functions taking into account the phrase orientation $o\\in \\lbrace M,S,D\\rbrace $ during decoding and important locations $loc$ pertaining to the current phrase and the previous phrase with the following template:","$$h_{s\\_hr}(loc:=rep(loc),o)$$ (Eq. 14)","where locations $loc$ are the first and last words of the current source phrase ( $s_{first}$ , $s_{last}$ ), the previously translated unit ( $p_{first}$ , $p_{last}$ ), i.e., the largest contiguous chunk of phrases forming a contiguous translation, and the span between the current and the previous source phrase, or gap ( $g_{first}$ , $g_{last}$ ) only for discontinuous orientation. Each word in $loc$ is represented, $rep(loc)$ , by its POS tag, and the surface lexical form if it belongs to the 80-most frequent words in the training data.","UTF8gbsn Assuming that only “在” belongs to the top-80 words, when the phrase “yesterday” is generated after “Beijing”, the sparse reordering orientation features that are equal to 1 are shown in Figure 2 .","While the approach leverages the reordering orientation model by defining features on the context information, not just the current phrase, it does not capture dependency relation, by which the important relation between words in the source sentence is captured. Therefore, we introduce our sparse dependency swap features to enable the translation system to arrive at reordering decisions based on the source dependency relations."]},{"idx":9,"heading":"Dependency Path Features","paragraphs":["We also utilize dependency path features BIBREF15 for phrase-based SMT, defined over the shortest path of dependency parse tree links bridging the current and the previous source phrase. BIBREF15 changdiscriminative2009 defined the dependency path features on a maximum entropy phrase orientation classifier, trained on their word-aligned parallel text and labeled by the two possible phrase orderings in the translation output: in-order and swapped. Meanwhile, we use the features as sparse decoding features, with the following template:","$$h_{path}(shortest\\_path(p_{last},s_{first});o)$$ (Eq. 18)","where $o \\in \\lbrace in\\_order,swapped\\rbrace $ denotes the orientation of the two phrases in the translation.","Given a source sentence and its translation output, a path is defined between the last word of the previous source phrase $p_{last}$ and the first word of the current source phrase $s_{first}$ . Path traversal is always from left to right based on the source sentence word position. Therefore, if the current source phrase being translated is to the right of the previous source phrase, the traversal is from $p_{last}$ to $s_{first}$ . Otherwise, if the current source phrase is to the left of the previous source phrase, it is from $s_{first}$ to $p_{last}$ . In addition, path edges going against the dependency label arrow are distinguished from those following the arrow.","UTF8gbsn As an example, in Figure 1 , when the translation generates “发表 讲话 $\\rightarrow $ made a speech” after “佐科威 $\\rightarrow $ Jokowi”, the path is from “佐科威” to “发表”, consisting of a direct link following the arrow of $nsubj$ , resulting in the feature $h_{path}(nsubj,in\\_order)=1$ . However, when the translation generates “昨天 $\\rightarrow $ yesterday” after “北京 $\\rightarrow $ Beijing”, as “昨天” is before “北京” in the source sentence, the traversal is from “昨天” to “北京”, consisting of the link sequence $tmod,prepR,pobjR$ . As it goes against the arrows of $prep$ and $pobj$ , the suffix $R$ is added to distinguish it. This results in the feature $\\rightarrow $0 .","The approach leverages the phrase-based reordering by guiding the ordering of two adjacent phrases using dependency parse. However, the features do not capture the pairwise ordering of every word with its related word, as the features are induced only when the words are used in the two adjacent translation phrases."]},{"idx":10,"heading":"Data Set and Toolkits","paragraphs":["We built a phrase-based Chinese-to-English SMT system by using Moses BIBREF18 . Our parallel training text is a collection of parallel corpora from LDC, which we divide into older corpora and newer corpora. Due to the dominant older data, we duplicate the newer corpora of various domains by 10 times to achieve better domain balance. To reduce the possibility of alignment errors, parallel sentences in the corpora that are longer than 85 words in either Chinese (after word segmentation) or English are discarded. In the end, the final parallel text consists of around 8.8M sentence pairs, 228M Chinese tokens, and 254M English tokens (a token can be a word or punctuation symbol). We also added two dictionaries by concatenating them to our training parallel text. The total number of words in these two corpora is 1.81M for Chinese and 2.03M for English.","All Chinese sentences in the training, development, and test data are first word-segmented using a maximum entropy-based Chinese word segmenter BIBREF19 trained on the Chinese Treebank (CTB) scheme. Then the parallel corpus is word-aligned by GIZA++ BIBREF20 using IBM Models 1, 3, and 4 BIBREF21 . For building the phrase table, which follows word alignment, the maximum length of a phrase pair is set to 7 words for both the source and target sides.","The language model (LM) is a 5-gram model trained on the English side of the FBIS parallel corpus (LDC2003E14) and the monolingual corpus English Gigaword version 4 (LDC2009T13), consisting of 107M sentences and 3.8G tokens altogether. Each individual Gigaword sub-corpus is used to train a separate language model and so is the English side of FBIS. These individual language models are then interpolated to build one single large LM, via perplexity tuning on the development set.","Our translation development set is MTC corpus version 1 (LDC2002T01) and version 3 (LDC2004T07). This development set has 1,928 sentence pairs in total, 49K Chinese tokens and 58K English tokens on average across the four reference translations. Weight tuning is done by using the pairwise ranked optimization (PRO) algorithm BIBREF22 , which is also used to obtain weights of the sparse features to help determine the reordering.","For dependency sparse features, we parse the Chinese side of our development and test sets by the Mate parser, which jointly performs POS tagging and dependency parsing BIBREF23 , trained on Chinese Treebank (CTB) version 8.0 (LDC2013T21).","Our test set consists of the NIST MT evaluation sets from 2002 to 2006 and 2008 (LDC2010T10, LDC2010T11, LDC2010T12, LDC2010T14, LDC2010T17, LDC2010T21)."]},{"idx":11,"heading":"Baseline System","paragraphs":["UTF8gbsn We build a phrase-based baseline SMT system, which uses non-sparse phrase-based lexicalized reordering (PBLR), in which the reordering probability depends on the phrase being translated and its position with respect to the source position of the previously translated phrase BIBREF1 , BIBREF24 , and non-sparse hierarchical reordering (HR), in which the previous unit is not only the previous phrase, but the largest chunk of contiguous source phrases having contiguous translation BIBREF3 . In addition, a distortion limit is set such that the reordering cannot be longer than a certain distance. We set punctuation symbols as reordering constraint across which phrases cannot be reordered, as they form the natural boundaries between different clauses. We also use n-best Minimum Bayes Risk (MBR) decoding BIBREF25 instead of the default maximum a-posteriori (MAP) decoding."]},{"idx":12,"heading":"Our Approach","paragraphs":["To accommodate our sparse feature functions, our Moses code has been modified to read dependency-parsed input sentences and incorporate additional decoding features on top of our baseline, namely dependency distortion penalty (DDP) BIBREF13 , sparse dependency path features (Path) BIBREF15 , sparse reordering orientation following hierarchical reordering orientation (SHR) BIBREF4 , and our sparse dependency swap features (DS). DDP feature is a single penalty feature similar to the distortion penalty for distance-based reordering model BIBREF0 , while Path, SHR, and DS are sparse features, each instance of which captures a specific phenomenon during translation BIBREF17 .","We always couple DS with DDP. However, as the original Path features did not use DDP, we experiment with Path features in one setting that does not incorporate DDP and another that does. The SHR features are not coupled with DDP, following the original design, as the features are not defined on a dependency-parsed input sentence."]},{"idx":13,"heading":"Experimental Results","paragraphs":["The translation quality of the system outputs is measured by case-insensitive BLEU BIBREF26 , for which the brevity penalty is computed based on the shortest reference (NIST-BLEU). Statistical significance testing between systems is conducted by bootstrap resampling BIBREF27 .","Table 1 shows the experimental results. The distortion limit of all the systems is set to 14, which yields the best result on the development set for the baseline system. As shown in the table, the system with our DS features and DDP on top of the baseline is able to improve over the baseline system without and with DDP, by +1.09 and +0.93 BLEU points respectively. The individual contribution of the other dependency-based features (Path), without or with DDP, is inferior to our DS features. Nevertheless, coupling our DS features with Path features yields the best result (+1.21 and +1.05 BLEU points over the baseline without and with DDP).","The SHR features yield more improvement than Path features without DDP, and are comparable to the Path features with DDP. However, our DS features yield more improvement than SHR features. Our preliminary experiments indicate that adding these features on top of the system with DS does not improve over it."]},{"idx":14,"heading":"Discussions","paragraphs":["The reordering orientation models, i.e., PBLR and HR, only take into account the phrase pair generated by a hypothesis and not the related word properties. Sparse reordering orientation features BIBREF4 leverage this by capturing the previous phrase (or contiguous chunk) properties. Therefore, they are able to improve over the baseline. However, as the results suggest, dependency parse provides a more useful guidance to reorder a source sentence.","As shown in Figure 3 , the baseline phrase-based SMT system with the two reordering orientation models (PBLR and HR) produces an incorrect translation output that “the export is the main market”, which is not what the source sentence means. This is also the case with the system added with prior reordering approaches, which includes DDP BIBREF13 , Path BIBREF15 , their combination DDP+Path, and SHR BIBREF4 . Meanwhile, our reordering approach with DS features is able to output the correct translation “the main markets for the export”, as it penalizes the swap between the subject head “market” and the copula, which should not be swapped.","The dependency parse of a sentence can capture the relationship among the words (nodes) in it. Between each related word pair, there is a dependency label which specifies the head-modifier relationship. While the head-modifier relationships in a sentence hold across languages, their ordering may differ. For example, in Chinese, prepositional phrase (modifier) comes before the predicate (head) verb, while in English, they come in the other way round. This particular clue, provided by the source dependency parse, is useful in deciding the word order in the translation output, corresponding to a word in the source sentence.","Combining our sparse dependency swap features with sparse dependency path features BIBREF15 achieves the best experimental result. This can be attributed to the complementary nature of both types of sparse features. Path features are able to capture the subsequence of translation output phrases, i.e., which phrase follows another but not the relative position (right or left) of a source dependency word with respect to all its related words, while our swap features are designed to capture them."]},{"idx":15,"heading":"Related Work","paragraphs":["Source dependency trees have been exploited in phrase-based SMT, by modeling transition among dependency subtrees during translation BIBREF14 . However, this does not take into account the dependency label and the POS tag. Another work exploits source and target dependency trees for phrase-based MT output reranking BIBREF28 , instead of for translation decoding.","BIBREF15 changdiscriminative2009 introduced dependency path as a soft constraint based on the sequence of source dependency links traversed in phrase-based translation. It is used as features on a maximum entropy phrase orientation classifier, whose probability output is used as a decoding feature function. As the path can be arbitrarily long, it may not be represented sufficiently in the training samples. Our sparse feature definition can alleviate this as features are defined on two words. In addition, capturing word pairs instead of paths enables incorporation of other word properties such as POS tags.","BIBREF29 hunterexploiting2010 proposed a probability model to capture the offset of a word with respect to its head position in phrase-based MT. Their model does not take into account two sibling words sharing the same head. They reported negative result.","Meanwhile, BIBREF30 gaosoft2011 defined soft constraints on hierarchical phrase-based MT, which produce translation by bottom-up constituency parsing algorithm instead of beam search. Their soft constraint is also defined on a maximum entropy classifier instead of sparse features.","Prior work has also used constituency parse to guide reordering, by manually defining pre-ordering rules to reorder input sentences into the target language order before translation BIBREF31 , BIBREF32 , or to automatically learn those rules BIBREF33 , BIBREF34 . Each constituency parse tree node represents the phrase nesting instead of a word, resulting in a deeper structure, which is generally slower to produce."]},{"idx":16,"heading":"Conclusion","paragraphs":["We have presented a reordering approach for phrase-based SMT, guided by sparse dependency swap features. We have contributed a new approach for learning and performing reordering in phrase-based MT by the incorporation of dependency-based features. From our experiments, we have shown that utilizing source dependency parse for reordering sentences helps to significantly improve translation quality over a phrase-based baseline system with state-of-the-art reordering orientation models."]},{"idx":17,"heading":"Acknowledgments","paragraphs":["This research is supported by the Singapore National Research Foundation under its International Research Centre @ Singapore Funding Initiative and administered by the IDM Programme Office."]},{"idx":18,"heading":"Figures and Tables","paragraphs":["Figure 1: Illustrating example: (a) an aligned Chinese-English parallel sentence pair with Chinese dependency parse and a sequence of beam search hypotheses producing phrases “佐科威 → Jokowi”, “发表 讲话 → made a speech”, “在 → in”, “北京 → Beijing”, and “昨天 → yesterday” (each hypothesis is marked by a grey dashed box and a sequence number in parentheses) (b) sparse features that are equal to 1 when each hypothesis is generated.","Figure 2: Sparse reordering orientation (HR) features that are equal to 1 when decoding in Fig. 1a generates the phrase “昨天→ yesterday” after “北京→ Beijing”","Table 1: The results of our reordering approach using sparse dependency swap (DS) features, in BLEU scores (%) compared to the baseline (Base), on which features are added (∗: significant at p < 0.05; ∗∗: significant at p < 0.01). We also show the results of prior reordering methods, i.e., dependency distortion penalty (+DDP) (Cherry 2008), sparse dependency path features (+Path) (Chang et al. 2009), the combination of both (+DDP+Path), as well as sparse reordering orientation features (+SHR) (Cherry 2013). For all systems involving DDP and other features, comparison is also made to the system with only DDP. (†: significant at p < 0.05; ††: significant at p < 0.01). Note: All systems involving DS always incorporate DDP.","Figure 3: Translation output comparison between our baseline (Base), prior reordering work (+DDP, +Path, +DDP+Path, and +SHR), and our dependency-swap-driven reordering approach (+DDP+DS and +DDP+Path+DS)."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1608.01972","title":"Bridging the Gap: Incorporating a Semantic Similarity Measure for Effectively Mapping PubMed Queries to Documents","sections":[{"idx":0,"heading":"Abstract","paragraphs":["The main approach of traditional information retrieval (IR) is to examine how many words from a query appear in a document. A drawback of this approach, however, is that it may fail to detect relevant documents where no or only few words from a query are found. The semantic analysis methods such as LSA (latent semantic analysis) and LDA (latent Dirichlet allocation) have been proposed to address the issue, but their performance is not superior compared to common IR approaches. Here we present a query-document similarity measure motivated by the Word Mover's Distance. Unlike other similarity measures, the proposed method relies on neural word embeddings to calculate the distance between words. Our method is efficient and straightforward to implement. The experimental results on TREC and PubMed show that our approach provides significantly better performance than BM25. We also discuss the pros and cons of our approach and show that there is a synergy effect when the word embedding measure is combined with the BM25 function."]},{"idx":1,"heading":"Introduction","paragraphs":["In information retrieval (IR), queries and documents are typically represented by term vectors where each term is a content word and weighted by tf-idf, i.e. the product of the term frequency and the inverse document frequency, or other weighting schemes BIBREF0 . The similarity of a query and a document is then determined as a dot product or cosine similarity. Although this works reasonably, the traditional IR scheme often fails to find relevant documents when synonymous or polysemous words are used in a dataset, e.g. a document including only “neoplasm\" cannot be found when the word “cancer\" is used in a query. One solution of this problem is to use query expansion BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 or dictionaries, but these alternatives still depend on the same philosophy, i.e. queries and documents should share exactly the same words.","While the term vector model computes similarities in a sparse and high-dimensional space, the semantic analysis methods such as latent semantic analysis (LSA) BIBREF5 , BIBREF6 and latent Dirichlet allocation (LDA) BIBREF7 learn dense vector representations in a low-dimensional space. These methods choose a vector embedding for each term and estimate a similarity between terms by taking an inner product of their corresponding embeddings BIBREF8 . Since the similarity is calculated in a latent (semantic) space based on context, the semantic analysis approaches do not require having common words between a query and documents. However, it has been shown that LSA and LDA methods do not produce superior results in various IR tasks BIBREF9 , BIBREF10 , BIBREF11 and the classic ranking method, BM25 BIBREF12 , usually outperforms those methods in document ranking BIBREF13 , BIBREF14 .","Neural word embedding BIBREF15 , BIBREF16 is similar to the semantic analysis methods described above. It learns low-dimensional word vectors from text, but while LSA and LDA utilize co-occurrences of words, neural word embedding learns word vectors to predict context words BIBREF10 . Moreover, training of semantic vectors is derived from neural networks. Both co-occurrence and neural word embedding approaches have been used for lexical semantic tasks such as semantic relatedness (e.g. king and queen), synonym detection (e.g. cancer and carcinoma) and concept categorization (e.g. banana and pineapple belong to fruits) BIBREF10 , BIBREF17 . But, Baroni et al. Baroni2014 showed that neural word embedding approaches generally performed better on such tasks with less effort required for parameter optimization. The neural word embedding models have also gained popularity in recent years due to their high performance in NLP tasks BIBREF18 .","Here we present a query-document similarity measure using a neural word embedding approach. This work is particularly motivated by the Word Mover's Distance BIBREF19 . Unlike the common similarity measure taking query/document centroids of word embeddings, the proposed method evaluates a distance between individual words from a query and a document. Our first experiment was performed on the TREC 2006 and 2007 Genomics benchmark sets BIBREF20 , BIBREF21 , and the experimental results showed that our approach was better than BM25 ranking. This was solely based on matching queries and documents by the semantic measure and no other feature was used for ranking documents.","In general, conventional ranking models (e.g. BM25) rely on a manually designed ranking function and require heuristic optimization for parameters BIBREF22 , BIBREF23 . In the age of information explosion, this one-size-fits-all solution is no longer adequate. For instance, it is well known that links to a web page are an important source of information in web document search BIBREF24 , hence using the link information as well as the relevance between a query and a document is crucial for better ranking. In this regard, learning to rank BIBREF22 has drawn much attention as a scheme to learn how to combine diverse features. Given feature vectors of documents and their relevance levels, a learning to rank approach learns an optimal way of weighting and combining multiple features.","We argue that the single scores (or features) produced by BM25 and our proposed semantic measure complement each other, thus merging these two has a synergistic effect. To confirm this, we measured the impact on document ranking by combining BM25 and semantic scores using the learning to rank approach, LamdaMART BIBREF25 , BIBREF26 . Trained on PubMed user queries and their click-through data, we evaluated the search performance based on the most highly ranked 20 documents. As a result, we found that using our semantic measure further improved the performance of BM25.","Taken together, we make the following important contributions in this work. First, to the best of our knowledge, this work represents the first investigation of query-document similarity for information retrieval using the recently proposed Word Mover's Distance. Second, we modify the original Word Mover's Distance algorithm so that it is computationally less expensive and thus more practical and scalable for real-world search scenarios (e.g. biomedical literature search). Third, we measure the actual impact of neural word embeddings in PubMed by utilizing user queries and relevance information derived from click-through data. Finally, on TREC and PubMed datasets, our proposed method achieves stronger performance than BM25."]},{"idx":2,"heading":"Methods","paragraphs":["A common approach to computing similarity between texts (e.g. phrases, sentences or documents) is to take a centroid of word embeddings, and evaluate an inner product or cosine similarity between centroids BIBREF14 , BIBREF27 . This has found use in classification and clustering because they seek an overall topic of each document. However, taking a simple centroid is not a good approximator for calculating a distance between a query and a document BIBREF19 . This is mostly because queries tend to be short and finding the actual query words in documents is feasible and more accurate than comparing lossy centroids. Consistent with this, our approach here is to measure the distance between individual words, not the average distance between a query and a document."]},{"idx":3,"heading":"Word Mover's Distance","paragraphs":["Our work is based on the Word Mover's Distance between text documents BIBREF19 , which calculates the minimum cumulative distance that words from a document need to travel to match words from a second document. In this subsection, we outline the original Word Mover's Distance algorithm, and our adapted model is described in Section 2.2.","First, following Kusner et al. Kusner2015, documents are represented by normalized bag-of-words (BOW) vectors, i.e. if a word INLINEFORM0 appears INLINEFORM1 times in a document, the weight is DISPLAYFORM0","where INLINEFORM0 is number of words in the document. The higher the weight, the more important the word. They assume a word embedding so that each word INLINEFORM1 has an associated vector INLINEFORM2 . The dissimilarity INLINEFORM3 between INLINEFORM4 and INLINEFORM5 is then calculated by DISPLAYFORM0","The Word Mover's Distance makes use of word importance and the relatedness of words as we now describe.","Let INLINEFORM0 and INLINEFORM1 be BOW representations of two documents INLINEFORM2 and INLINEFORM3 . Let INLINEFORM4 be a flow matrix, where INLINEFORM5 denotes how much it costs to travel from INLINEFORM6 in INLINEFORM7 to INLINEFORM8 in INLINEFORM9 , and INLINEFORM10 is the number of unique words appearing in INLINEFORM11 and/or INLINEFORM12 . To entirely transform INLINEFORM13 to INLINEFORM14 , we ensure that the entire outgoing flow from INLINEFORM15 equals INLINEFORM16 and the incoming flow to INLINEFORM17 equals INLINEFORM18 . The Word Mover's Distance between INLINEFORM19 and INLINEFORM20 is then defined as the minimum cumulative cost required to move all words from INLINEFORM21 to INLINEFORM22 or vice versa, i.e. DISPLAYFORM0","The solution is attained by finding INLINEFORM0 that minimizes the expression in Eq. ( EQREF5 ). Kusner et al. Kusner2015 applied this to obtain nearest neighbors for document classification, i.e. k-NN classification and it produced outstanding performance among other state-of-the-art approaches. What we have just described is the approach given in Kusner et al. We will modify the word weights and the measure of the relatedness of words to better suit our application."]},{"idx":4,"heading":"Our Query-Document Similarity Measure","paragraphs":["While the prior work gives a hint that the Word Mover's Distance is a reasonable choice for evaluating a similarity between documents, it is uncertain how the same measure could be used for searching documents to satisfy a query. First, it is expensive to compute the Word Mover's Distance. The time complexity of solving the distance problem is INLINEFORM0 BIBREF28 . Second, the semantic space of queries is not the same as those of documents. A query consists of a small number of words in general, hence words in a query tend to be more ambiguous because of the restricted context. On the contrary, a text document is longer and more informational. Having this in mind, we realize that ideally two distinctive components could be employed for query-document search: 1) mapping queries to documents using a word embedding model trained on a document set and 2) mapping documents to queries using a word embedding model obtained from a query set. In this work, however, we aim to address the former, and the mapping of documents to queries remains as future work.","For our purpose, we will change the word weight INLINEFORM0 to incorporate inverse document frequency ( INLINEFORM1 ), i.e. DISPLAYFORM0","where INLINEFORM0 . INLINEFORM1 is the size of a document set and INLINEFORM2 is the number of documents that include the INLINEFORM3 th term. The rationale behind this is to weight words in such a way that common terms are given less importance. It is the idf factor normally used in tf-idf and BM25 BIBREF29 , BIBREF30 . In addition, our word embedding is a neural word embedding trained on the 25 million PubMed titles and abstracts.","Let INLINEFORM0 and INLINEFORM1 be BOW representations of a query INLINEFORM2 and a document INLINEFORM3 . INLINEFORM4 and INLINEFORM5 in Section 2.1 are now replaced by INLINEFORM6 and INLINEFORM7 , respectively. Since we want to have a higher score for documents relevant to INLINEFORM8 , INLINEFORM9 is redefined as a cosine similarity, i.e. DISPLAYFORM0","In addition, the problem we try to solve is the flow INLINEFORM0 . Hence, Eq. ( EQREF5 ) is rewritten as follows. DISPLAYFORM0","where INLINEFORM0 represents the word INLINEFORM1 in INLINEFORM2 . INLINEFORM3 in Eq. ( EQREF7 ) is unknown for queries, therefore we compute INLINEFORM4 based on the document collection. The optimal solution of the expression in Eq. ( EQREF9 ) is to map each word in INLINEFORM5 to the most similar word in INLINEFORM6 based on word embeddings. The time complexity for getting the optimal solution is INLINEFORM7 , where INLINEFORM8 is the number of unique query words and INLINEFORM9 is the number of unique document words. In general, INLINEFORM10 and evaluating the similarity between a query and a document can be implemented in parallel computation. Thus, the document ranking process can be quite efficient."]},{"idx":5,"heading":"Learning to Rank","paragraphs":["In our study, we use learning to rank to merge two distinctive features, BM25 scores and our semantic measures. This approach is trained and evaluated on real-world PubMed user queries and their responses based on click-through data BIBREF31 . While it is not common to use only two features for learning to rank, this approach is scalable and versatile. Adding more features subsequently should be straightforward and easy to implement. The performance result we obtain demonstrates the semantic measure is useful to rank documents according to users' interests.","We briefly outline learning to rank approaches BIBREF32 , BIBREF33 in this subsection. For a list of retrieved documents, i.e. for a query INLINEFORM0 and a set of candidate documents, INLINEFORM1 , we are given their relevancy judgements INLINEFORM2 , where INLINEFORM3 is a positive integer when the document INLINEFORM4 is relevant and 0 otherwise. The goal of learning to rank is to build a model INLINEFORM5 that can rank relevant documents near or at the top of the ranked retrieval list. To accomplish this, it is common to learn a function INLINEFORM6 , where INLINEFORM7 is a weight vector applied to the feature vector INLINEFORM8 . A part of learning involves learning the weight vector but the form of INLINEFORM9 may also require learning. For example, INLINEFORM10 may involve learned decision trees as in our application.","In particular, we use LambdaMART BIBREF25 , BIBREF26 for our experiments. LambdaMART is a pairwise learning to rank approach and is being used for PubMed relevance search. While the simplest approach (pointwise learning) is to train the function INLINEFORM0 directly, pairwise approaches seek to train the model to place correct pairs higher than incorrect pairs, i.e. INLINEFORM1 , where the document INLINEFORM2 is relevant and INLINEFORM3 is irrelevant. INLINEFORM4 indicates a margin. LambdaMART is a boosted tree version of LambdaRank BIBREF26 . An ensemble of LambdaMART, LambdaRank and logistic regression models won the Yahoo! learning to rank challenge BIBREF23 ."]},{"idx":6,"heading":"Results and Discussion","paragraphs":["Our resulting formula from the Word Mover's Distance seeks to find the closest terms for each query word. Figure FIGREF11 depicts an example with and without using our semantic matching. For the query, “negative pressure wound therapy\", a traditional way of searching documents is to find those documents which include the words “negative\", “pressure\", “wound\" and “therapy\". As shown in the figure, the words, “pressure\" and “therapy\", cannot be found by perfect string match. On the other hand, within the same context, the semantic measure finds the closest words “NPWT\" and “therapies\" for “pressure\" and “therapy\", respectively. Identifying abbreviations and singular/plural would help match the same words, but this example is to give a general idea about the semantic matching process. Also note that using dictionaries such as synonyms and abbreviations requires an additional effort for manual annotation.","In the following subsections, we describe the datasets and experiments, and discuss our results."]},{"idx":7,"heading":"Datasets","paragraphs":["To evaluate our word embedding approach, we used two scientific literature datasets: TREC Genomics data and PubMed. Table TABREF13 shows the number of queries and documents in each dataset. TREC represents the benchmark sets created for the TREC 2006 and 2007 Genomics Tracks BIBREF20 , BIBREF21 . The original task is to retrieve passages relevant to topics (i.e. queries) from full-text articles, but the same set can be utilized for searching relevant PubMed documents. We consider a PubMed document relevant to a TREC query if and only if the full-text of the document contains a passage judged relevant to that query by the TREC judges. Our setup is more challenging because we only use PubMed abstracts, not full-text articles, to find evidence.","This is the number of PubMed documents as of Apr. 6, 2017. This number and the actual number of documents used for our experiments may differ slightly.","Machine learning approaches, especially supervised ones such as learning to rank, are promising and popular nowadays. Nonetheless, they usually require a large set of training examples, and such datasets are particularly difficult to find in the biomedical domain. For this reason, we created a gold standard set based on real (anonymized) user queries and the actions users subsequently took, and named this the PubMed set.","To build the PubMed set, we collected one year's worth of search logs and restricted the set of queries to those where users requested the relevance order and which yielded at least 20 retrieved documents. This set contained many popular but duplicate queries. Therefore, we merged queries and summed up user actions for each of them. That is, for each document stored for each query, we counted the number of times it was clicked in the retrieved set (i.e. abstract click) and the number of times users requested full-text articles (i.e. full-text click). We considered the queries that appeared less than 10 times to be less informative because they were usually very specific, and we could not collect enough user actions for training. After this step, we further filtered out non-informational queries (e.g. author and journal names). As the result, 27,870 queries remained for the final set.","The last step for producing the PubMed set was to assign relevance scores to documents for each query. We will do this based on user clicks. It is known that click-through data is a useful proxy for relevance judgments BIBREF34 , BIBREF35 , BIBREF36 . Let INLINEFORM0 be the number of clicks to the abstract of a document INLINEFORM1 from the results page for the query INLINEFORM2 . Let INLINEFORM3 be the number of clicks from INLINEFORM4 's abstract page to its full-text, which result from the query INLINEFORM5 . Let INLINEFORM6 be the boost factor for documents without links to full-text articles. INLINEFORM7 is the indicator function such that INLINEFORM8 if the document INLINEFORM9 includes a link to full-text articles and INLINEFORM10 otherwise. We can then calculate the relevance, INLINEFORM11 , of a document for a given query: DISPLAYFORM0","INLINEFORM0 is the trade-off between the importance of abstract clicks and full-text clicks. The last term of the relevance function gives a slight boost to documents without full-text links, so that they get a better relevance (thus rank) than those for which full-text is available but never clicked, assuming they all have the same amount of abstract clicks. We manually tuned the parameters based on user behavior and log analyses, and used the settings, INLINEFORM1 and INLINEFORM2 .","Compared to the TREC Genomics set, the full PubMed set is much larger, including all 27 million documents in PubMed. While the TREC and PubMed sets share essentially the same type of documents, the tested queries are quite different. The queries in TREC are a question type, e.g. “what is the role of MMS2 in cancer?\" However, the PubMed set uses actual queries from PubMed users.","In our experiments, the TREC set was used for evaluating BM25 and the semantic measure separately and the PubMed set was used for evaluating the learning to rank approach. We did not use the TREC set for learning to rank due to the small number of queries. Only 62 queries and 162,259 documents are available in TREC, whereas the PubMed set consists of many more queries and documents."]},{"idx":8,"heading":"Word Embeddings and Other Experimental Setup","paragraphs":["We used the skip-gram model of word2vec BIBREF16 to obtain word embeddings. The alternative models such as GloVe BIBREF11 and FastText BIBREF37 are available, but their performance varies depending on tasks and is comparable to word2vec overall BIBREF38 , BIBREF39 . word2vec was trained on titles and abstracts from over 25 million PubMed documents. Word vector size and window size were set to 100 and 10, respectively. These parameters were optimized to produce high recall for synonyms BIBREF40 . Note that an independent set (i.e. synonyms) was used for tuning word2vec parameters, and the trained model is available online (https://www.ncbi.nlm.nih.gov/IRET/DATASET).","For experiments, we removed stopwords from queries and documents. BM25 was chosen for performance comparison and the parameters were set to INLINEFORM0 and INLINEFORM1 BIBREF41 . Among document ranking functions, BM25 shows a competitive performance BIBREF42 . It also outperforms co-occurrence based word embedding models BIBREF13 , BIBREF14 . For learning to rank approaches, 70% of the PubMed set was used for training and the rest for testing. The RankLib library (https://sourceforge.net/p/lemur/wiki/RankLib) was used for implementing LambdaMART and the PubMed experiments."]},{"idx":9,"heading":"TREC Experiments","paragraphs":["Table TABREF17 presents the average precision of tf-idf (TFIDF), BM25, word vector centroid (CENTROID) and our embedding approach on the TREC dataset. Average precision BIBREF43 is the average of the precisions at the ranks where relevant documents appear. Relevance judgements in TREC are based on the pooling method BIBREF44 , i.e. relevance is manually assessed for top ranking documents returned by participating systems. Therefore, we only used the documents that annotators reviewed for our evaluation BIBREF1 .","As shown in Table TABREF17 , BM25 performs better than TFIDF and CENTROID. CENTROID maps each query and document to a vector by taking a centroid of word embedding vectors, and the cosine similarity between two vectors is used for scoring and ranking documents. As mentioned earlier, this approach is not effective when multiple topics exist in a document. From the table, the embedding approach boosts the average precision of BM25 by 19% and 6% on TREC 2006 and 2007, respectively. However, CENTROID provides scores lower than BM25 and SEM approaches.","Although our approach outperforms BM25 on TREC, we do not claim that BM25 and other traditional approaches can be completely replaced with the semantic method. We see the semantic approach as a means to narrow the gap between words in documents and those in queries (or users' intentions). This leads to the next experiment using our semantic measure as a feature for ranking in learning to rank."]},{"idx":10,"heading":"PubMed Experiments","paragraphs":["For the PubMed dataset, we used learning to rank to combine BM25 and our semantic measure. An advantage of using learning to rank is its flexibility to add more features and optimize performance by learning their importance. PubMed documents are semi-structured, consisting of title, abstract and many more fields. Since our interest lies in text, we only used titles and abstracts, and applied learning to rank in two different ways: 1) to find semantically closest words in titles (BM25 + SEMTitle) and 2) to find semantically closest words in abstracts (BM25 + SEMAbstract). Although our semantic measure alone produces better ranking scores on the TREC set, this does not apply to user queries in PubMed. It is because user queries are often short, including around three words on average, and the semantic measure cannot differentiate documents when they include all query words.","Table TABREF19 shows normalized discounted cumulative gain (NDCG) scores for top 5, 10 and 20 ranked documents for each approach. NDCG BIBREF45 is a measure for ranking quality and it penalizes relevant documents appearing in lower ranks by adding a rank-based discount factor. In the table, reranking documents by learning to rank performs better than BM25 overall, however the larger gain is obtained from using titles (BM25 + SEMTitle) by increasing NDCG@20 by 23%. NDCG@5 and NDCG@10 also perform better than BM25 by 23% and 25%, respectively. It is not surprising that SEMTitle produces better performance than SEMAbstract. The current PubMed search interface does not allow users to see abstracts on the results page, hence users click documents mostly based on titles. Nevertheless, it is clear that the abstract-based semantic distance helps achieve better performance.","After our experiments for Table TABREF19 , we also assessed the efficiency of learning to rank (BM25 + SEMTitle) by measuring query processing speed in PubMed relevance search. Using 100 computing threads, 900 queries are processed per second, and for each query, the average processing time is 100 milliseconds, which is fast enough to be used in the production system."]},{"idx":11,"heading":"Conclusion","paragraphs":["We presented a word embedding approach for measuring similarity between a query and a document. Starting from the Word Mover's Distance, we reinterpreted the model for a query-document search problem. Even with the INLINEFORM0 flow only, the word embedding approach is already efficient and effective. In this setup, the proposed approach cannot distinguish documents when they include all query words, but surprisingly, the word embedding approach shows remarkable performance on the TREC Genomics datasets. Moreover, applied to PubMed user queries and click-through data, our semantic measure allows to further improves BM25 ranking performance. This demonstrates that the semantic measure is an important feature for IR and is closely related to user clicks.","While many deep learning solutions have been proposed recently, their slow training and lack of flexibility to adopt various features limit real-world use. However, our approach is more straightforward and can be easily added as a feature in the current PubMed relevance search framework. Proven by our PubMed search results, our semantic measure improves ranking performance without adding much overhead to the system."]},{"idx":12,"heading":"Acknowledgments","paragraphs":["This research was supported by the Intramural Research Program of the NIH, National Library of Medicine."]},{"idx":13,"heading":"Figures and Tables","paragraphs":["Table 1: Number of documents in the TREC and PubMed datasets.","Table 2: Average precision of BM25 and our embedding approach (EMBED) on the TREC set.","Table 3: Average precision of BM25 and our embedding approach (EMBED) on the PubMed set.","Table 4: Average precision of BM25 and our approach (EMBED) on the TREC set. All documents in the set are scored.","Figure 1: TREC performance changes depending on α."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1608.06757","title":"Robust Named Entity Recognition in Idiosyncratic Domains","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Named entity recognition often fails in idiosyncratic domains. That causes a problem for depending tasks, such as entity linking and relation extraction. We propose a generic and robust approach for high-recall named entity recognition. Our approach is easy to train and offers strong generalization over diverse domain-specific language, such as news documents (e.g. Reuters) or biomedical text (e.g. Medline). Our approach is based on deep contextual sequence learning and utilizes stacked bidirectional LSTM networks. Our model is trained with only few hundred labeled sentences and does not rely on further external knowledge. We report from our results F1 scores in the range of 84-94% on standard datasets."]},{"idx":1,"heading":"Introduction","paragraphs":["Information extraction tasks have become very important not only in the Web, but also for in-house enterprise settings. One of the crucial steps towards understanding natural language is named entity recognition (NER), which aims to extract mentions of entity names in text. NER is necessary for many higher-level tasks such as entity linking, relation extraction, building knowledge graphs, question answering and intent based search. In these scenarios, NER recall is critical, as candidates that are never generated can not be recovered later BIBREF0 ."]},{"idx":2,"heading":"Robust Contextual Word Labeling","paragraphs":["We abstract the task of NER as sequential word labeling problem. Figure FIGREF15 illustrates an example for sequential transformation of a sentence into word labels. We express each sentence in a document as a sequence of words: INLINEFORM0 , e.g. INLINEFORM1 Aspirin. We define a mention as the longest possible span of adjacent tokens that refer to a an entity or relevant concept of a real-world object, such as Aspirin (ASA). We further assume that mentions are non-recursive and non-overlapping. To encode boundaries of the mention span, we adapt the idea of ramshaw1995text, which has been adapted as BIO2 standard in the CoNLL2003 shared task BIBREF15 . We assign labels INLINEFORM2 to each token to mark begin, inside and outside of a mention from left to right. We use the input sequence INLINEFORM3 together with a target sequence INLINEFORM4 of the same length that contains a BIO2 label for each word: INLINEFORM5 , e.g. INLINEFORM6 B. To predict the most likely label INLINEFORM7 of a token regarding its context, we utilize recurrent neural networks."]},{"idx":3,"heading":"Robust Word Encoding Methods","paragraphs":["We have shown that most common errors for recall loss are misspellings, POS errors, capitalization, unseen words and irregular context. Therefore we generalize our model throughout three layers: robust word encoding, in-sentence word context and contextual sequence labeling.","Dictionary-based word vectorization methods suffer from sparse training sets, especially in the case of non-verbatim mentions, rare words, typing and capitalization errors. For example, the word2vec model of mikolov2013efficient generalizes insufficiently for rare words in idiosyncratic domains or for misspelled words, since for these words no vector representation is learned at training time. In the GENIA data set, we notice 27% unseen words (dictionary misses) in the pretrained word2vec model. As training data generation is expensive, we investigate a generic approach for the generation of word vectors. We use letter-trigram word hashing as introduced by huang2013learning. This technique goes beyond words and generates word vectors as a composite of discriminative three-letter “syllables”, that might also include misspellings. Therefore, it is robust against dictionary misses and has the advantage (despite its name) to group syntactically similar words in similar vector spaces. We compare this approach to word embedding models such as word2vec.","The most important features for NER are word shape properties, such as length, initial capitalization, all-word uppercase, in-word capitalization and use of numbers or punctuation BIBREF16 . Mixed-case word encodings implicitly include capitalization features. However, this approach impedes generalization, as words appear in various surface forms, e.g. capitalized at the beginning of sentences, uppercase in headlines, lowercase in social media text. The strong coherence between uppercase and lowercase characters – they might have identical semantics – is not encoded in the embedding. Therefore, we encode the words using lowercase letter-trigrams. To keep the surface information, we add flag bits to the vector that indicate initial capitalization, uppercase, lower case or mixed case."]},{"idx":4,"heading":"Deep Contextual Sequence Learning","paragraphs":["With sparse training data in the idiosyncratic domain, we expect input data with high variance. Therefore, we require a strong generalization for the syntactic and semantic representation of language. To reach into the high 80–90% NER F1 performance, long-range context-sensitive information is indispensable. We apply the computational model of recurrent neural networks, in particular long short-term memory networks (LSTMs) BIBREF17 , BIBREF18 to the problem of sequence labeling. Like neural feed-forward networks, LSTMs are able to learn complex parameters using gradient descent, but include additional recurrent connections between cells to influence weight updates over adjacent time steps. With their ability to memorize and forget over time, LSTMs have proven to generalize context-sensitive sequential data well BIBREF19 , BIBREF20 .","Figure FIGREF15 shows an unfolded representation of the steps through a sentence. We feed the LSTM with letter-trigram vectors INLINEFORM0 as input data, one word at a time. The hidden layer of the LSTM represents context from long range dependencies over the entire sentence from left to right. However, to achieve deeper contextual understanding over the boundaries of multi-word annotations and at the beginning of sentences, we require a backwards pass through the sentence. We therefore implement a bidirectional LSTM and feed the output of both directions into a second LSTM layer for combined label prediction.","For the use in the neural network, word encodings INLINEFORM0 and labels INLINEFORM1 are real-valued vectors. To predict the most likely label INLINEFORM2 of a token, we utilize a LSTM with input nodes INLINEFORM3 , input gates INLINEFORM4 , forget gate INLINEFORM5 , output gate INLINEFORM6 and internal state INLINEFORM7 . For the bidirectional case, all gates are duplicated and combined into forward state INLINEFORM8 and backward state INLINEFORM9 . The network is trained using backpropagation through time (BPTT) by adapting weights INLINEFORM10 and bias parameters INLINEFORM11 to fit the training examples. DISPLAYFORM0","We iterate over labeled sentences in mini-batches and update the weights accordingly. The network is then used to predict label probabilities INLINEFORM0 for unseen word sequences INLINEFORM1 ."]},{"idx":5,"heading":"Implementation of NER Components","paragraphs":["To show the impact of our bidirectional LSTM model, we measure annotation performance on three different neural network configurations. We implement all components using the Deeplearning4j framework. For preprocessing (sentence and word tokenization), we use Stanford CoreNLP BIBREF11 . We test the sequence labeler using three input encodings:","[noitemsep]","DICT: We build a dictionary over all words in the corpus and generate the input vector using 1-hot encoding for each word","EMB: We use the GoogleNews word2vec embeddings, which encodes each word as vector of size 300","TRI: we implement letter-trigram word hashing as described in Section SECREF14 .","During training and test, we group all tokens of a sentence as mini-batch. We evaluate three different neural network types to show the impact of the bidirectional sequence learner.","[noitemsep]","FF: As baseline, we train a non-sequential feed-forward model based on a fully connected multilayer perceptron network with 3 hidden layers of size 150 with relu activation, feeding into a 3-class softmax classifier. We train the model using backpropagation with stochastic gradient descent and a learning rate of 0.005.","LSTM: We use a configuration of a single feed-forward layer of size 150 with two additional layers of single-direction LSTM with 20 cells and a 3-class softmax classifier. We train the model using backpropagation-through-time (BPTT) with stochastic gradient descent and a learning rate of 0.005.","BLSTM: Our final configuration consists of a single feed-forward layer of size 150 with one bidirectional LSTM layer with 20 cells and an additional single-direction LSTM with 20 cells into a 3-class softmax classifier. The BLSTM model is trained the same way as the single-direction LSTM."]},{"idx":6,"heading":"Evaluation","paragraphs":["We evaluate nine configurations of our model on five gold standard evaluation data sets. We show that the combination of letter-trigram word hashing with bidirectional LSTM yields the best results and outperforms sequence learners based on dictionaries or word2vec. To highlight the generalization of our model to idiosyncratic domains, we run tests on common-typed data sets as well as on specialized medical documents. We compare our system on these data sets with specialized state-of-the-art systems."]},{"idx":7,"heading":"Evaluation Set Up","paragraphs":["We train two models with identical parameterization, each with 2000 randomly chosen labeled sentences from a standard data set. To show the effectiveness of the components, we evaluate different configurations of this setting with 2000 random sentences from the remaining set. The model was trained using Deeplearning4j with nd4j-x86 backend. Training the TRI+BLSTM configuration on a commodity Intel i7 notebook with 4 cores at 2.8GHz takes approximately 50 minutes.","Table TABREF33 gives an overview of the standard data sets we use for training. The GENIA Corpus BIBREF3 contains biomedical abstracts from the PubMed database. We use GENIA technical term annotations 3.02, which cover linguistic expressions to entities of interest in molecular biology, e.g. proteins, genes and cells. CoNLL2003 BIBREF14 is a standard NER dataset based on the Reuters RCV-1 news corpus. It covers named entities of type person, location, organization and misc.","For testing the overall annotation performance, we utilize CoNLL2003-testA and a 50 document split from GENIA. Additionally, we test on the complete KORE50 BIBREF21 , ACE2004 BIBREF22 and MSNBC data sets using the GERBIL evaluation framework BIBREF23 ."]},{"idx":8,"heading":"Measurements","paragraphs":["We measure precision, recall and F1 score of our DATEXIS-NER system and state-of-the-art annotators introduced in Section SECREF2 . For the comparison with black box systems, we evaluate annotation results using weak annotation match. For a more detailed in-system error analysis, we measure BIO2 labeling performance based on each token.","We measure the overall performance of mention annotation using the evaluation measures defined by cornolti2013framework, which are also used by ling2015design. Let INLINEFORM0 be a set of documents with gold standard mention annotations INLINEFORM1 with a total of INLINEFORM2 examples. Each mention INLINEFORM3 is defined by start position INLINEFORM4 and end position INLINEFORM5 in the source document INLINEFORM6 . To quantify the performance of the system, we compare INLINEFORM7 to the set of predicted annotations INLINEFORM8 with mentions INLINEFORM9 : DISPLAYFORM0","We compare using a weak annotation match: DISPLAYFORM0","We measure micro-averaged precision ( INLINEFORM0 ), recall ( INLINEFORM1 ) and NER-style ( INLINEFORM2 ) score: DISPLAYFORM0","Tuning the model configuration with annotation match measurement is not always feasible. We therefore measure INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 separately for each label class INLINEFORM4 in our classification model and calculate binary classification precision INLINEFORM5 , recall INLINEFORM6 and INLINEFORM7 scores. To avoid skewed results from the expectedly large INLINEFORM8 class, we use macro-averaging over the three classes: DISPLAYFORM0"]},{"idx":9,"heading":"Evaluation Results","paragraphs":["We now discuss the evaluation of our DATEXIS-NER system on common and idiosyncratic data.","Table TABREF35 shows the comparison of DATEXIS-NER with eight state-of-the-art annotators on four common news data sets. Both common and medical models are configured identically and trained on only 2000 labeled sentences, without any external prior knowledge. We observe that DATEXIS-NER achieves the highest recall scores of all tested annotators, with 95%–98% on all measured data sets. Moreover, DATEXIS-NER precision scores are equal or better than median. Overall, we achieve high micro-F1 scores of 84%–94% on news entity recognition, which is slightly better than the ontology-based Entityclassifier.eu NER and reveals a better generalization than the 3-type Stanford NER with distributional semantics. We notice that systems specialized on word-sense disambiguation (Babelfy, DBpedia Spotlight) don't perform well on “raw” untyped entity recognition tasks. The highest precision scores are reached by Stanford NER. We also notice a low precision of all annotators on the ACE2004 dataset and high variance in MSNBC performance, which are probably caused by differing annotation standards.","Table TABREF45 shows the results of biomedical entity recognition compared to the participants of the JNLPBA 2004 bio-entity recognition task BIBREF14 . We notice that for these well-written Medline abstracts, there is not such a strong skew between precision and recall. Our DATEXIS-NER system outperforms the HMM, MEMM, CRF and CDN based models with a micro-F1 score of 84%. However, the highly specialized GENIA chunker for LingPipe achieves higher scores. This chunker is a very simple generative model predictor that is based on a sliding window of two tokens, word shape and dictionaries. We interpret this score as strong overfitting using a dictionary of the well-defined GENIA terms. Therefore, this model will generalize hardly considering the simple model. We can confirm this presumption in the common data sets, where the MUC-6 trained HMM LingPipe chunker performs on average on unseen data.","We evaluate different configurations of the components that we describe in Section SECREF22 . Table TABREF47 shows the results of experiments on both CoNLL2003 and GENIA data sets. We report the highest macro-F1 scores for BIO2 labeling for the configuration of letter-trigram word vectors and bidirectional LSTM. We notice that dictionary-based word encodings (DICT) work well for idiosyncratic medical domains, whereas they suffer from high word ambiguity in the news texts. Pretrained word2vec embeddings (EMB) perform well on news data, but cannot adapt to the medical domain without retraining, because of a large number of unseen words. Therefore, word2vec generally achieves a high precision on news texts, but low recall on medical text. The letter-trigram approach (TRI) combines both word vector generalization and robustness towards idiosyncratic language.","We observe that the contextual LSTM model achieves scores throughout in the 85%–94% range and significantly outperforms the feed-forward (FF) baseline that shows a maximum of 75%. Bidirectional LSTMs can further improve label classification in both precision and recall."]},{"idx":10,"heading":"Discussion and Error Analysis","paragraphs":["We investigate different aspects of the DATEXIS-NER components by manual inspection of classification errors in the context of the document. For the error classes described in the introduction (false negative detections, false positives and invalid boundaries), we observe following causes:","In dictionary based configurations (e.g. 1-hot word vector encoding DICT), we observe false negative predictions caused by dictionary misses for words that do not exist in the training data. The cause can be rare unseen or novel words (e.g. T-prolymphocytic cells) or misspellings (e.g. strengthnend). These words yield a null vector result from the encoder and can therefore not be distinguished by the LSTM. The error increases when using word2vec, because these models are trained with stop words filtered out. This implicates that e.g. mentions surrounded by or containing a determiner (e.g. The Sunday Telegraph quoted Majorie Orr) are highly error prone towards the detection of their boundaries. We resolve this error by the letter-trigram approach. Unseen trigrams (e.g. thh) may still be missing in the word vector, but only affect single dimensions as opposed to the vector as a whole.","Surface forms encode important features for NER (e.g. capitalization of “new” in Alan Shearer was named as the new England captain / as New York beat the Angels). However, case-sensitive word vectorization methods yield a large amount of false positive predictions caused by incorrect capitalization in the input data. An uppercase headline (e.g. TENNIS - U.S. TEAM ON THE ROAD FOR 1997 FED CUP) is encoded completely different than a lowercase one (e.g. U.S. team on the road for Fed Cup). Because of that, we achieve best results with lowercase word vectors and additional surface form feature flags, as described in Section SECREF14 .","We observe mentions that are composed of co-occurring words with high ambiguity (e.g. degradation of IkB alpha in T cell lines). These groups encode strong syntagmatic word relations BIBREF24 that can be leveraged to resolve word sense and homonyms from sentence context. Therefore, correct boundaries in these groups can effectively be identified only with contextual models such as LSTMs.","Orthogonal to the previous problem, different words in a paradigmatic relation BIBREF24 can occur in the same context (e.g. cyclosporin A-treated cells / HU treated cells). These groups are efficiently represented in word2vec. However, letter-trigram vectors cannot encode paradigmatic groups and therefore require a larger training sample to capture these relations.","Often, synonyms can only be resolved regarding a larger document context than the local sentence context known by the LSTM. In these cases, word sense is redefined by a topic model local to the paragraph (e.g. sports: Tiger was lost in the woods after divorce.). This problem does not heavily affect NER recall, but is crucial for named entity disambiguation and coreference resolution.","The proposed DATEXIS-NER model is restricted to recognize boundaries of generic mentions in text. We evaluate the model on annotations of isolated types (e.g. persons, organizations, locations) for comparison purposes only, but we do not approach NER-style typing. Contrary, we approach to detect mentions without type information. The detection of specific types can be realized by training multiple independent models on a selection of labels per type and nesting the resulting annotations using a longest-span semantic type heuristic BIBREF25 ."]},{"idx":11,"heading":"Summary","paragraphs":["ling2015design show that the task of NER is not clearly defined and rather depends on a specific problem context. Contrary, most NER approaches are specifically trained on fixed datasets in a batch mode. Worse, they often suffer from poor recall BIBREF26 . Ideally, one could personalize the task of recognizing named entities, concepts or phrases according to the specific problem. “Personalizing” and adapting such annotators should happen with very limited human labeling effort, in particular for idiosyncratic domains with sparse training data.","Our work follows this line. From our results we report F1 scores between 84–94% when using bidirectional multi-layered LSTMs, letter-trigram word hashing and surface form features on only few hundred training examples.","This work is only a preliminary step towards the vision of personalizing annotation guidelines for NER BIBREF2 . In our future work, we will focus on additional important idiosyncratic domains, such as health, life science, fashion, engineering or automotive. For these domains, we will consider the process of detecting mentions and linking them to an ontology as a joint task and we will investigate simple and interactive workflows for creating robust personalized named entity linking systems."]},{"idx":12,"heading":"Figures and Tables","paragraphs":["Figure 1: Architecture of the LSTM network used for named entity recognition. The character stream “Aspirin has an antiplatelet effect.” is","Table 1: Overview of CoNLL2003 and GENIA training datasets and sizes of word encodings. We use 2000 sentences of each set for training.","Table 2: Comparison of annotators trained for common English news texts (micro-averaged scores on match per annotation span). The table shows micro-precision, recall and NER-style F1 for CoNLL2003, KORE50, ACE2004 and MSNBC datasets.","Table 3: Comparison of annotators trained for biomedical text. The table shows NER annotation results for 50 documents from the GENIA dataset.","Table 4: Comparison of nine configurations from our implementation (macro-averaged scores on BIO2 classification per token)."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1608.07836","title":"What to do about non-standard (or non-canonical) language in NLP","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Real world data differs radically from the benchmark corpora we use in natural language processing (NLP). As soon as we apply our technologies to the real world, performance drops. The reason for this problem is obvious: NLP models are trained on samples from a limited set of canonical varieties that are considered standard, most prominently English newswire. However, there are many dimensions, e.g., socio-demographics, language, genre, sentence type, etc. on which texts can differ from the standard. The solution is not obvious: we cannot control for all factors, and it is not clear how to best go beyond the current practice of training on homogeneous data from a single domain and language. In this paper, I review the notion of canonicity, and how it shapes our community's approach to language. I argue for leveraging what I call fortuitous data, i.e., non-obvious data that is hitherto neglected, hidden in plain sight, or raw data that needs to be refined. If we embrace the variety of this heterogeneous data by combining it with proper algorithms, we will not only produce more robust models, but will also enable adaptive language technology capable of addressing natural language variation."]},{"idx":1,"heading":"Introduction","paragraphs":["The publication of the Penn Treebank Wall Street Journal (WSJ) corpus in the late 80s has undoubtedly pushed NLP from symbolic computation to statistical approaches, which dominate our field up to this day. The WSJ has become the NLP benchmark dataset for many tasks (e.g., part-of-speech tagging, parsing, semantic role labeling, discourse parsing), and has developed into the de-facto “standard” in our field.","However, while it has advanced the field in so many ways, it has also introduced almost imperceptible biases: why is newswire considered more standard or more canonical than other text types? Journalists are trained writers who make fewer errors and adhere to a codified norm. But let us pause for a minute. If NLP had emerged only in the last decade, would newswire data still be our canon? Or would, say, Wikipedia be considered canonical? User-generated data is less standardized, but is highly available. If we take this thought further and start over today, maybe we would be in an `inverted' world: social media is standard and newswire with its `headlinese' is the `bad language' BIBREF0 . It is easy to collect large quantities of social media data. Whatever we consider canonical, all data comes with its biases, even more democratic media like Wikipedia carry their own peculiarities.","It seems that what is considered canonical hitherto is mostly a historical coincidence and motivated largely by availability of resources. Newswire has and actually still does dominate our field. For example, in Figure 1 , I plot domains versus languages for the treebank data in version 1.3 of the on-going Universal Dependencies project BIBREF1 . Almost all languages include newswire, except ancient languages (for obvious reasons), English (since most data comes from the Web Treebank) and Khazak, Chinese (Wikipedia). While including other domains and languages is highly desirable, it is impossible to find unbiased data. Let's be aware of this fact and try to collect enough biased data.","Processing non-canonical (or non-canonical) data is difficult. A series of papers document large drops in accuracy when moving across domains BIBREF3 , BIBREF4 . There is a large body of work focusing on correcting for domain differences. Typically, in domain adaptation (DA) the task is to adapt a model trained on some source domain to perform better on some new target domain. However, it is less clear what really folds into a domain. In Section \"The variety space\" , I will review the notion of domain and propose what I call variety space.","Is the annotation of non-canonical also more difficult, just like its processing appears to be? Processing and annotating are two aspects, and the difficulty in one, say processing, does not necessarily propagate the same way to annotation BIBREF5 . However, very little work exists on disentangling the two. The same is true for examining what really constitutes a domain. What remains is clear: the challenge is all about variations of data. Language continuously changes, for various reasons (different social groups, communicative purposes, changes over time), and so we will continuously face interesting challenges, both for processing and annotation.","In the remainder I will look at the NLP community's approach to face these challenges. I will outline one potential way to go about it, arguing for the use of fortuitous data, and end by returning to the question of domain."]},{"idx":2,"heading":"What to do about non-standard data","paragraphs":["There are generally three main approaches to go about non-standard data."]},{"idx":3,"heading":"Annotate more data","paragraphs":["Annotating more data is a first and intuitive solution. However, it is naïve, for several reasons.","Domain (whatever that means) and language (whatever that comprises) are two factors of text variation. Now take the cross-product between the two. We will never be able to create annotated data that spans all possible combinations. This is the problem of training data sparsity, illustrated in Figure 1 . The figure only shows a tiny subset of the world's languages, and a tiny fraction of potential domains out there. The problem is that most of the data that is available out there is unlabeled. Annotation requires time. At the same time, ways of communication change, so what we annotate today might be very distant to what we need to process tomorrow. We cannot just “annotate our way out\" BIBREF0 . Moreover, it might not be trivial to find the right annotators; annotation schemes might need adaptation as well BIBREF6 and tradeoffs for doing so need to be defined BIBREF7 .","What we need is quick ways to semi-automatically gather annotated data, and use more unsupervised and weakly supervised approaches."]},{"idx":4,"heading":"Bring training and test data closer to each other","paragraphs":["The second approach is based on the idea of making data resemble each other more. The first strategy here is normalization, that is, preprocess the input to make it closer to what our technology expects, e.g. han:baldwin:2013. A less known but similar approach is to artificially corrupt the training data to make it more similar to the expected target domain BIBREF8 . However, normalization implies “norm”, and as eisenstein:2013:bad remarks: whose norm are we targeting? (e.g., labor vs labour). Furthermore, he notes that it is surprisingly difficult to find a precise notion of the normalization task.","Corrupting the training data is a less explored endeavor. This second strategy though hinges on the assumption that one knows what to expect.","What we need are models that do provide nonsensical predictions on unexpected inputs, i.e., models that include invariant representations. For example, our models should be capable of learning similar representations for the same inherent concept, e.g., kiss vs :* or love vs $<$ 3. Recent shifts towards using sub-token level information can be seen as one step in this direction."]},{"idx":5,"heading":"Domain adaptation","paragraphs":["There is a large body of work on adapting models trained on some source domain to work better on some new target domain. Approaches range from feature augmentation, shared representation learning, instance weighting, to approaches that exploit representation induced from general background corpora. For an overview, see BIBREF9 , BIBREF10 . However, what all of these approaches have in common is an unrealistic assumption: they know the target domain. That is, researchers typically have a small amount of target data available that they can use to adapt their models.","An extreme case of adaptation is cross-lingual learning, whose goal is similar: adapt models trained on some source languages to languages in which few or no resources exist. Also here a large body of work assumes knowledge of the target language and requires some in-domain, typically parallel data. However, most work has focused on a restricted set of languages, only recently approaches emerged that aim to transfer from multiple sources to many target languages BIBREF11 .","What we need are methods that can adapt quickly to unknown domains and languages, without much assumptions on what to expect, and use multiple sources, rather than just one. In addition, our models need to detect when to trigger domain adaptation approaches.","In the next parts I will outline some possibilities to address these challenges. However, there are other important areas that I will not touch upon here (e.g., evaluation)."]},{"idx":6,"heading":"Fortuitous data","paragraphs":["What we need are models that are more robust, work better on unexpected input and can be trained from semi-automatically or weakly annotated data, from a variety of sources. In order to build such models, I argue that the key is to look for signal in non-obvious places, i.e., fortuitous data.","Fortuitous data is data out there that just waits to be harvested. It might be in plain sight, but is neglected (available but not used), or it is in raw form and first needs to be refined (almost ready but needs refinement). Availability and ease-of-use (or readiness) are therefore two important dimensions that define fortuitous data. Fortuitous data is the unintended yield of a process, a promising by-product or side benefit.","In the following I will outline potential sources of fortuitous data. An overview is given in Table 1 ."]},{"idx":7,"heading":"But what's in a domain?","paragraphs":["As already noted earlier BIBREF9 , there is no common ground on what constitutes a domain. blitzer2006domain attribute domain differences mostly to differences in vocabulary, biber explores differences between corpora from a sociolinguistics perspective. mcclosky:2010 considers it in a broader view: “By domain, we mean the style, genre, and medium of a document.” Terms such as genre, register, text type, domain, style are often used differently in different communities BIBREF20 , or interchangeably.","While there exists no definition of domain, work on domain adaptation is plentiful but mostly focused on assuming a dichotomy: source versus target, without much interest in how they differ. In fact, there is surprisingly little work on how texts vary and the consequence for NLP. It is established that out-of-vocabulary (OOV) tokens impact NLP performance. However, what are other factors?","Interest in this question re-emerged recently. For example, focusing on annotation difficulty, zeldes-simonson:2016 remark “that domain adaptation may be folding in sentence type effects”, motivated by earlier findings by silveira2014gold who remark that “[t]he most striking difference between the two types of data [Web and newswire] has to do with imperatives, which occur two orders of magnitude more often in the EWT [English Web Treebank].” A very recent paper examines word order properties and their impact on parsing taking a control experiment approach BIBREF21 . On another angle, it has been shown that tagging accuracy correlates with demographic factors such as age BIBREF22 .","I want to propose that `domain' is an overloaded term. Besides the mathematical definition, in NLP it is typically used to refer to some coherent data with respect to topic or genre. However, there are many other (including yet unknown factors) out there, such as demographic factors, communicational purpose, but also sentence type, style, medium, technology/medium, language, etc. At the same time, these categories are not sharply defined either. Rather than imposing hard categories, let us consider a Wittgensteinian view."]},{"idx":8,"heading":"The variety space","paragraphs":["I here propose to see a domain as variety in a high-dimensional variety space. Points in the space are the data instances, and regions form domains. A dataset $\\mathcal {D}$ is a sample from the variety space, conditioned on latent factors $V$ : $\\mathcal {D} \\sim P(X,Y|V)$","The variety space is a unknown high-dimensional space, whose dimensions (latent factors $V$ ) include (fuzzy) aspects such as language (or dialect), topic or genre, and social factors (age, gender, personality, etc.), amongst others. A domain is a variety that forms a region in this complicated network of similarities, with some members more prototypical than others. However, we have neither access to the number of latent factors nor to their types. This vision is inspired by the notion of prototype theory in Cognitive Science and Wittgenstein's graded notion of categories. Figure 2 shows a hypothetical example of this variety space.","Our datasets are subspaces of this high-dimensional space. Depending on our task, instances are sentences, documents etc. In the following I will use POS tagging as a running example to analyze what's in a domain, by referring to the datasets with the typically used categories."]},{"idx":9,"heading":"Conclusions","paragraphs":["Current NLP models still suffer dramatically when applied to non-canonical data, where canonicity is a relative notion; in our field, newswire was and still often is the de-facto standard, the canonical data we typically train our models on.","While newswire has advanced the field in so many ways, it has also introduced almost imperceptible biases. What we need is to be aware of such biases, collect enough biased data, and model variety. I argue that if we embrace the variety of this heterogeneous data by combining it with proper algorithms, in addition to including text covariates/latent factors, we will not only produce more robust models, but will also enable adaptive language technology capable of addressing natural language variation."]},{"idx":10,"heading":"Acknowledgments","paragraphs":["I would like to thank the organizers for the invitation to the keynote at KONVENS 2016. I am also grateful to Héctor Martínez Alonso, Dirk Hovy, Anders Johannsen, Zeljko Agić and Gertjan van Noord for valuable discussions and feedback on earlier drafts of this paper."]},{"idx":11,"heading":"Figures and Tables","paragraphs":["Figure 1: The problem of training data sparsity illustrated for parsing: available annotated data in languages and domains; subset of syntacticallyannotated treebanks from Universal Dependencies v1.3 for which domain/genre info was available.","Table 1: Typology of fortuitous data.","Figure 2: What’s in a domain? Domain is an overloaded term. I propose to use the term variety. A dataset is a sample from the variety space, a unknown high-dimensional space, whose dimensions contain (fuzzy) aspects such as language (or dialect), topic or genre, and social factors (age, gender, personality, etc.), amongst others. A domain forms a region in this space, with some members more prototypical than others.","Table 2: Tagging accuracy on various test set varieties (domains, languages and age groups; Tw=Twitter), using coarse POS (Petrov et al., 2012). OOV: out-of-vocabulary rate wrt WSJ.TRAIN. Accuracy is significantly correlated with OOV rate (ρ = -0.70).","Figure 3: Accuracy of the WSJ tagger on 10 bootstrap samples (k = 150). Above: Accuracy versus OOV rate, Below: Accuracy vs KL divergence (src and trg gold POS bigram distributions). Different Twitter samples (green and darkgreen) exhibit very different behavior; oct27 has many OOVs and a high KL div; FOSTER is much closer to WSJ in terms of KL div."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1608.08188","title":"Visual Question: Predicting If a Crowd Will Agree on the Answer","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Visual question answering (VQA) systems are emerging from a desire to empower users to ask any natural language question about visual content and receive a valid answer in response. However, close examination of the VQA problem reveals an unavoidable, entangled problem that multiple humans may or may not always agree on a single answer to a visual question. We train a model to automatically predict from a visual question whether a crowd would agree on a single answer. We then propose how to exploit this system in a novel application to efficiently allocate human effort to collect answers to visual questions. Specifically, we propose a crowdsourcing system that automatically solicits fewer human responses when answer agreement is expected and more human responses when answer disagreement is expected. Our system improves upon existing crowdsourcing systems, typically eliminating at least 20% of human effort with no loss to the information collected from the crowd."]},{"idx":1,"heading":"Introduction","paragraphs":["What would be possible if a person had an oracle that could immediately provide the answer to any question about the visual world? Sight-impaired users could quickly and reliably figure out the denomination of their currency and so whether they spent the appropriate amount for a product BIBREF0 . Hikers could immediately learn about their bug bites and whether to seek out emergency medical care. Pilots could learn how many birds are in their path to decide whether to change course and so avoid costly, life-threatening collisions. These examples illustrate several of the interests from a visual question answering (VQA) system, including tackling problems that involve classification, detection, and counting. More generally, the goal for VQA is to have a single system that can accurately answer any natural language question about an image or video BIBREF1 , BIBREF2 , BIBREF3 .","Entangled in the dream of a VQA system is an unavoidable issue that, when asking multiple people a visual question, sometimes they all agree on a single answer while other times they offer different answers (Figure FIGREF1 ). In fact, as we show in the paper, these two outcomes arise in approximately equal proportions in today's largest publicly-shared VQA benchmark that contains over 450,000 visual questions. Figure FIGREF1 illustrates that human disagreements arise for a variety of reasons including different descriptions of the same concept (e.g., “minor\" and “underage\"), different concepts (e.g., “ghost\" and “photoshop\"), and irrelevant responses (e.g., “no\").","Our goal is to account for whether different people would agree on a single answer to a visual question to improve upon today's VQA systems. We propose multiple prediction systems to automatically decide whether a visual question will lead to human agreement and demonstrate the value of these predictions for a new task of capturing the diversity of all plausible answers with less human effort.","Our work is partially inspired by the goal to improve how to employ crowds as the computing power at run-time. Towards satisfying existing users, gaining new users, and supporting a wide range of applications, a crowd-powered VQA system should be low cost, have fast response times, and yield high quality answers. Today's status quo is to assume a fixed number of human responses per visual question and so a fixed cost, delay, and potential diversity of answers for every visual question BIBREF2 , BIBREF0 , BIBREF4 . We instead propose to dynamically solicit the number of human responses based on each visual question. In particular, we aim to accrue additional costs and delays from collecting extra answers only when extra responses are needed to discover all plausible answers. We show in our experiments that our system saves 19 40-hour work weeks and $1800 to answer 121,512 visual questions, compared to today's status quo approach BIBREF0 .","Our work is also inspired by the goal to improve how to employ crowds to produce the information needed to train and evaluate automated methods. Specifically, researchers in fields as diverse as computer vision BIBREF2 , computational linguistics BIBREF1 , and machine learning BIBREF3 rely on large datasets to improve their VQA algorithms. These datasets include visual questions and human-supplied answers. Such data is critical for teaching machine learning algorithms how to answer questions by example. Such data is also critical for evaluating how well VQA algorithms perform. In general, “bigger\" data is better. Current methods to create these datasets assume a fixed number of human answers per visual question BIBREF2 , BIBREF4 , thereby either compromising on quality by not collecting all plausible answers or cost by collecting additional answers when they are redundant. We offer an economical way to spend a human budget to collect answers from crowd workers. In particular, we aim to actively allocate additional answers only to visual questions likely to have multiple answers.","The key contributions of our work are as follows:"]},{"idx":2,"heading":"Paper Overview","paragraphs":["The remainder of the paper is organized into four sections. We first describe a study where we investigate: 1) How much answer diversity arises for visual questions? and 2) Why do people disagree (Section SECREF4 )? Next, we explore the following two questions: 1) Given a novel visual question, can a machine correctly predict whether multiple independent members of a crowd would supply the same answer? and 2) If so, what insights does our machine-learned system reveal regarding what humans are most likely to agree about (Section SECREF5 )? In the following section, we propose a novel resource allocation system for efficiently capturing the diversity of all answers for a set of visual questions (Section SECREF6 ). Finally, we end with concluding remarks (Section SECREF7 )."]},{"idx":3,"heading":"VQA - Analysis of Answer (Dis)Agreements","paragraphs":["Our first aim is to answer the following questions: 1) How much answer diversity arises for visual questions? and 2) Why do people disagree?"]},{"idx":4,"heading":"Visual Question - Predicting If a Crowd Will (Dis)Agree on an Answer","paragraphs":["We now explore the following two questions: 1) Given a novel visual question, can a machine correctly predict whether multiple independent members of a crowd would supply the same answer? and 2) If so, what insights does our machine-learned system reveal regarding what humans are most likely to agree about?"]},{"idx":5,"heading":"Prediction Systems","paragraphs":["We pose the prediction task as a binary classification problem. Specifically, given an image and associated question, a system outputs a binary label indicating whether a crowd will agree on the same answer. Our goal is to design a system that can detect which visual questions to assign a disagreement label, regardless of the disagreement cause (e.g., subjectivity, ambiguity, difficulty). We implement both random forest and deep learning classifiers.","A visual question is assigned either an answer agreement or disagreement label. To assign labels, we employ 10 crowdsourced answers for each visual question. A visual question is assigned an answer agreement label when there is an exact string match for 9 of the 10 crowdsourced answers (after answer pre-preprocessing, as discussed in the previous section) and an answer disagreement label otherwise. Our rationale is to permit the possibility of up to one “careless/spam\" answer per visual question. The outcome of our labeling scheme is that a disagreement label is agnostic to the specific cause of disagreement and rather represents the many causes (described above).","For our first system, we use domain knowledge to guide the learning process. We compile a set of features that we hypothesize inform whether a crowd will arrive at an undisputed, single answer. Then we apply a machine learning tool to reveal the significance of each feature. We propose features based on the observation that answer agreement often arises when 1) a lay person's attention can be easily concentrated to a single, undisputed region in an image and 2) a lay person would find the requested task easy to address.","We employ five image-based features coming from the salient object subitizing BIBREF22 (SOS) method, which produces five probabilities that indicate whether an image contains 0, 1, 2, 3, or 4+ salient objects. Intuitively, the number of salient objects shows how many regions in an image are competing for an observer's attention, and so may correlate with the ease in identifying a region of interest. Moreover, we hypothesize this feature will capture our observation from the previous study that counting problems typically leads to disagreement for images showing many objects, and agreement otherwise.","We employ a 2,492-dimensional feature vector to represent the question-based features. One feature is the number of words in the question. Intuitively, a longer question offers more information and we hypothesize additional information makes a question more precise. The remaining features come from two one-hot vectors describing each of the first two words in the question. Each one-hot vector is created using the learned vocabularies that define all possible words at the first and second word location of a question respectively (using training data, as described in the next section). Intuitively, early words in a question inform the type of answers that might be possible and, in turn, possible reasons/frequency for answer disagreement. For example, we expect “why is\" to regularly elicit many opinions and so disagreement. This intuition about the beginning words of a question is also supported by our analysis in the previous section which shows that different answer types yield different biases of eliciting answer agreement versus disagreement.","We leverage a random forest classification model BIBREF23 to predict an answer (dis)agreement label for a given visual question. This model consists of an ensemble of decision tree classifiers. We train the system to learn the unique weighted combinations of the aforementioned 2,497 features that each decision tree applies to make a prediction. At test time, given a novel visual question, the trained system converts a 2,497 feature descriptor of the visual question into a final prediction that reflects the majority vote prediction from the ensemble of decision trees. The system returns the final prediction along with a probability indicating the system's confidence in that prediction. We employ the Matlab implementation of random forests, using 25 trees and the default parameters.","We next adapt a VQA deep learning architecture BIBREF24 to learn the predictive combination of visual and textual features. The question is encoded with a 1024-dimensional LSTM model that takes in a one-hot descriptor of each word in the question. The image is described with the 4096-dimensional output from the last fully connected layer of the Convolutional Neural Network (CNN), VGG16 BIBREF25 . The system performs an element-wise multiplication of the image and question features, after linearly transforming the image descriptor to 1024 dimensions. The final layer of the architecture is a softmax layer.","We train the system to predict (dis)agreement labels with training examples, where each example includes an image and question. At test time, given a novel visual question, the system outputs an unnormalized log probability indicating its belief in both the agreement and disagreement label. For our system's prediction, we convert the belief in the disagreement label into a normalized probability. Consequently, predicted values range from 0 to 1 with lower values reflecting greater likelihood for crowd agreement."]},{"idx":6,"heading":"Analysis of Prediction System","paragraphs":["We now describe our studies to assess the predictive power of our classification systems to decide whether visual questions will lead to answer (dis)agreement.","We capitalize on today's largest visual question answering dataset BIBREF2 to evaluate our prediction system, which includes 369,861 visual questions about real images. Of these, 248,349 visual questions (i.e., Training questions 2015 v1.0) are kept for for training and the remaining 121,512 visual questions (i.e., Validation questions 2015 v1.0) are employed for testing our classification system. This separation of training and testing samples enables us to estimate how well a classifier will generalize when applied to an unseen, independent set of visual questions.","To our knowledge, no prior work has directly addressed predicting answer (dis)agreement for visual questions. Therefore, we employ as a baseline a related VQA algorithm BIBREF24 , BIBREF2 which produces for a given visual question an answer with a confidence score. This system parallels the deep learning architecture we adapt. However, it predicts the system's uncertainty in its own answer, whereas we are interested in the humans' collective disagreement on the answer. Still, it is a useful baseline to see if an existing algorithm could serve our purpose.","We evaluate the predictive power of the classification systems based on each classifier's predictions for the 121,512 visual questions in the test dataset. We first show performance of the baseline and our two prediction systems using precision-recall curves. The goals are to achieve a high precision, to minimize wasting crowd effort when their efforts will be redundant, and a high recall, to avoid missing out on collecting the diversity of accepted answers from a crowd. We also report the average precision (AP), which indicates the area under a precision-recall curve. AP values range from 0 to 1 with better-performing prediction systems having larger values.","Figure FIGREF8 a shows precision-recall curves for all prediction systems. Both our proposed classification systems outperform the VQA Algorithm BIBREF2 baseline; e.g., Ours - RF yields a 12 percentage point improvement with respect to AP. This is interesting because it shows there is value in learning the disagreement task specifically, rather than employing an algorithm's confidence in its answers. More generally, our results demonstrate it is possible to predict whether a crowd will agree on a single answer from a given image and associated question. Despite the significant variety of questions and image content, and despite the variety of reasons for which the crowd can disagree, our learned model is able to produce quite accurate results.","We observe our Random Forest classifier outperforms our deep learning classifier; e.g., Ours: RF yields a three percentage point improvement with respect to AP while consistently yielding improved precision-recall values over Ours: LSTM-CNN (Figure FIGREF8 a). In general, deep learning systems hold promise to replace handcrafted features to pick out the discriminative features. Our baselines highlight a possible value in developing a different deep learning architecture for the problem of learning answer disagreement than applied for predicting answers to visual questions.","We show examples of prediction results where our top-performing RF classifier makes its most confident predictions (Figure FIGREF8 b). In these examples, the predictor expects human agreement for “what room... ?\" visual questions and disagreement for “why... ?\" visual questions. These examples highlight that the classifier may have a strong language prior towards making predictions, as we will discuss in the next section.","We now explore what makes a visual question lead to crowd answer agreement versus disagreement. We examine the influence of whether visual questions lead to the three types of answers (“yes/no\", “number\", “other\") for both our random forest (RF) and deep learning (DL) classification systems. We enrich our analysis by examining the predictive performance of both classifiers when they are trained and tested exclusively with image and question features respectively. Figure FIGREF9 shows precision-recall curves for both classification systems with question features alone (Q), image features alone (I), and both question and image features together (Q+I).","When comparing AP scores (Figure FIGREF9 ), we observe our Q+I predictors yield the greatest predictive performance for visual questions that lead to “other\" answers, followed by “number\" answers, and finally “yes/no\" answers. One possible reason for this finding is that the question wording strongly drives whether a crowd will disagree for “other\" visual questions, whereas some notion of common sense may be required to learn whether a crowd will agree for “yes/no\" visual questions (e.g., Figure FIGREF7 a vs Figure FIGREF7 g).","We observe that question-based features yield greater predictive performance than image-based features for all visual questions, when comparing AP scores for Q and I classification results (Figure FIGREF9 ). In fact, image features contribute to performance improvements only for our random forest classifier for visual questions that lead to “number\" answers, as illustrated by comparing AP scores for Our RF: Q+I and Our RF: Q (Figure FIGREF9 b). Our overall finding that most of the predictive power stems from language-based features parallels feature analysis findings in the automated VQA literature BIBREF2 , BIBREF21 . This does not mean, however, that the image content is not predictive. Further work improving visual content cues for VQA agreement is warranted.","Our findings suggest that our Random Forest classifier's overall advantage over our deep learning system arises because of counting questions, as indicated by higher AP scores (Figure FIGREF9 ). For example, the advantage of the initial higher precision (Figure FIGREF8 a; Ours: RF vs Ours: DL) is also observed for counting questions (Figure FIGREF9 b; Ours: RF - Q+I vs Ours: DL - Q+I). We hypothesize this advantage arises due to the strength of the Random Forest classifier in pairing the question prior (“How many?\") with the image-based SOS features that indicates the number of objects in an image. Specifically, we expect “how many\" to lead to agreement only for small counting problems."]},{"idx":7,"heading":"Capturing Answer Diversity with Less Effort","paragraphs":["We next present a novel resource allocation system for efficiently capturing the diversity of true answers for a batch of visual questions. Today's status quo is to either uniformly collect INLINEFORM0 answers for every visual question BIBREF2 or collect multiple answers where the number is determined by external crowdsourcing conditions BIBREF0 . Our system instead spends a human budget by predicting the number of answers to collect for each visual question based on whether multiple human answers are predicted to be redundant."]},{"idx":8,"heading":"Answer Collection System","paragraphs":["Suppose we have a budget INLINEFORM0 which we can allocate to collect extra answers for a subset of visual questions. Our system automatically decides to which visual questions to allocate the “extra\" answers in order to maximize captured answer diversity for all visual questions.","The aim of our system is to accrue additional costs and delays from collecting extra answers only when extra responses will provide more information. Towards this aim, our system involves three steps to collect answers for all INLINEFORM0 visual questions (Figure FIGREF11 a). First, the system applies our top-performing random forest classifier to every visual question in the batch. Then, the system ranks the INLINEFORM1 visual questions based on predicted scores from the classifier, from visual questions most confidently predicted to lead to answer “agreement\" from a crowd to those most confidently predicted to lead to answer “disagreement\" from a crowd. Finally, the system solicits more ( INLINEFORM2 ) human answers for the INLINEFORM3 visual questions predicted to reflect the greatest likelihood for crowd disagreement and fewer ( INLINEFORM4 ) human answers for the remaining visual questions. More details below."]},{"idx":9,"heading":"Analysis of Answer Collection System","paragraphs":["We now describe our studies to assess the benefit of our allocation system to reduce human effort to capture the diversity of all answers to visual questions.","We evaluate the impact of actively allocating extra human effort to answer visual questions as a function of the available budget of human effort. Specifically, for a range of budget levels, we compute the total measured answer diversity (as defined below) resulting for the batch of visual questions. The goal is to capture a large amount of answer diversity with little human effort.","We conduct our studies on the 121,512 test visual questions about real images (i.e., Validation questions 2015 v1.0). For each visual question, we establish the set of true answers as all unique answers which are observed at least twice in the 10 crowdsourced answers per visual question. We require agreement by two workers to avoid the possibility that “careless/spam\" answers are treated as ground truth.","We collect either the minimum of INLINEFORM0 answer per visual question or the maximum of INLINEFORM1 answers per visual question. Our number of answers roughly aligns with existing crowd-powered VQA systems, for example with VizWiz, “On average, participants received 3.3 (SD=1.8) answers for each question\" BIBREF0 . Our maximum number of answers also supports the possibility of capturing the maximum of three unique, valid answers typically observed in practice (recall study above). While more elaborate schemes for distributing responses may be possible, we will show this approach already proves quite effective in our experiments. We simulate answer collection by randomly selecting answers from the 10 crowd answers per visual question.","We compare our approach to the following baselines:","As in the previous section, we leverage the output confidence score from the publicly-shared model BIBREF24 learned from a LSTM-CNN deep learning architecture to rank the order of priority for visual questions to receive redundancy.","The system randomly prioritizes which images receive redundancy. This predictor illustrates the best a user can achieve today with crowd-powered systems BIBREF0 , BIBREF5 or with current dataset collection methods BIBREF2 , BIBREF4 .","We quantify the total diversity of answers captured by a resource allocation system for a batch of visual questions INLINEFORM0 as follows: DISPLAYFORM0","where INLINEFORM0 represents the set of all true answers for the INLINEFORM1 -th visual question, INLINEFORM2 represents the set of unique answers captured in the INLINEFORM3 answers collected for the INLINEFORM4 -th visual question, and INLINEFORM5 represents the set of unique answers captured in the INLINEFORM6 answers collected for the INLINEFORM7 -th visual question. Given no extra human budget, total diversity comes from the second term which indicates the diversity captured when only INLINEFORM8 answers are collected for every visual question. Given a maximum available extra human budget ( INLINEFORM9 ), total diversity comes from the first term which indicates the diversity captured when INLINEFORM10 answers are collected for every visual question. Given a partial extra human budget ( INLINEFORM11 ), the aim is to have perfect predictions such that the minimum number of answers ( INLINEFORM12 ) are allocated only for visual questions with one true answer so that all diverse answers are safely captured.","We measure diversity per visual question as the number of all true answers collected per visual question ( INLINEFORM0 ). Larger values reflect greater captured diversity. The motivation for this measure is to only give total credit to visual questions when all valid, unique human answers are collected.","Our system consistently offers significant gains over today's status quo approach (Figure FIGREF11 b). For example, our system accelerates the collection of 70% of the diversity by 21% over the Status Quo baseline. In addition, our system accelerates the collection of the 82% of diversity one would observe with VizWiz by 23% (i.e., average of 3.3 answers per visual question). In absolute terms, this means eliminating the collection of 92,180 answers with no loss to captured answer diversity. This translates to eliminating 19 40-hour work weeks and saving over $1800, assuming workers are paid $0.02 per answer and take 30 seconds to answer a visual question. Our approach fills an important gap in the crowdsourcing answer collection literature for targeting the allocation of extra answers only to visual questions where a diversity of answers is expected.","Figure FIGREF11 b also illustrates the advantage of our system over a related VQA algorithm BIBREF2 for our novel application of cost-sensitive answer collection from a crowd. As observed, relying on an algorithm's confidence in its answer offers a valuable indicator over today's status quo of passively budgeting. While we acknowledge this method is not intended for our task specifically, it still serves as an important baseline (as discussed above). We attribute the further performance gains of our prediction system to it directly predicting whether humans will disagree rather than predicting a property of a specific algorithm (e.g., confidence of the Antol et al. algorithm in its answer prediction)."]},{"idx":10,"heading":"Conclusions","paragraphs":["We proposed a new problem of predicting whether different people would answer with the same response to the same visual question. Towards motivating the practical implications for this problem, we analyzed nearly half a million visual questions and demonstrated there is nearly a 50/50 split between visual questions that lead to answer agreement versus disagreement. We observed that crowd disagreement arose for various types of answers (yes/no, counting, other) for many different reasons. We next proposed a system that automatically predicts whether a visual question will lead to a single versus multiple answers from a crowd. Our method outperforms a strong existing VQA system limited to estimating system uncertainty rather than crowd disagreement. Finally, we demonstrated how to employ the prediction system to accelerate the collection of diverse answers from a crowd by typically at least 20% over today's status quo of fixed redundancy allocation."]},{"idx":11,"heading":"Acknowledgments","paragraphs":["The authors gratefully acknowledge funding from the Office of Naval Research (ONR YIP N00014-12-1-0754) and National Science Foundation (IIS-1065390). We thank Dinesh Jayaraman, Yu-Chuan Su, Suyog Jain, and Chao-Yeh Chen for their assistance with experiments."]},{"idx":12,"heading":"Figures and Tables","paragraphs":["Fig. 1: Examples of visual questions and corresponding answers, when 10 different people are asked to answer the same question about an image (“visual question”). As observed, the crowd sometimes all agree on a single answer (top) and at other times offer different answers (bottom). Given a visual question, we aim to build a prediction system that automatically decides whether multiple people would give the same answer.","TABLE I: Correlation between answer agreement and visual questions that elicit three different types of answers for VQAs on real images and abstract scenes. Shown for each answer type is the percentage of visual questions that lead to at most one disagreement (row 1), unanimous agreement (row 2), and exactly one disagreement (row 3) from 10 crowdsourced answers. On average, across all answer types for both datasets, the crowd agrees on the answer for nearly half (i.e., 53%) of all VQAs. Moreover, we observe crowd disagreement arises often for all three answer types, highlighting that the significance of crowd disagreement is applicable across various types of visual questions.","Fig. 2: Summary of answer diversity outcomes showing how frequently different numbers of unique answers arise when asking ten crowd workers to answer a visual question for (a) 369,861 visual questions about real images and (a) 90,000 visual questions about abstract scenes. Results are shown based on different degrees of answer agreement required to make an answer valid: only one person has to offer the answer, at least two people must agree on the answer, and at least three people must agree on the answer. The visual questions most often elicit exactly one answer per visual question but also regularly elicit up to three different answers per visual question.","Fig. 3: Illustration of visual questions that lead humans to (dis)agree on a single answer. As observed, (a) unanimous answer agreement arises when images are simple, questions are precise, and questions are visually grounded. (b-i) Answer disagreement arises for a variety of reasons: (b) expert skill needed, (c) human mistakes, (d) ambiguous question, (e) ambiguous visual content, (f) insufficient visual evidence, (g) subjective question, (h) answer synonyms, and (i) varying answer granularity.","Fig. 4: (a) Precision-recall curves and average precision (AP) scores for all benchmarked systems. Our random forest (RF) and deep learning (DL) classifiers outperform a related automated VQA baseline, showing the importance in modeling human disagreement as opposed to system uncertainty. (b) Examples of prediction results from our top-performing RF classifier. Shown are the top three visual questions for the most confidently predicted instances that lead to answer disagreement and agreement along with the observed crowd answers. These examples illustrate a strong language prior for making predictions. (Best viewed on pdf.)","Fig. 5: Precision-recall curves and average precision (AP) scores for our random forest (RF) and deep learning (DL) classifiers with different features (Question Only - Q; Image Only - I; Q +I) for visual questions that lead to (a-c) three answer types.","Fig. 6: We propose a novel application of predicting the number of redundant answers to collect from the crowd per visual question to efficiently capture the diversity of all answers for all visual questions. (a) For a batch of visual questions, our system first produces a relative ordering using the predicted confidence in whether a crowd would agree on an answer (upper half). Then, the system allocates a minimum number of annotations to all visual questions (bottom, left half) and then the extra available human budget to visual questions most confidently predicted to lead to crowd disagreement (bottom, right half). (b) For 121,512 visual questions, we show results for our system, a related VQA algorithm, and today’s status quo of random predictions. Boundary conditions are one answer (leftmost) and five answers (rightmost) for all visual questions. Our approach typically accelerates the capture of answer diversity by over 20% from today’s Status Quo selection; e.g., 21% for 70% of the answer diversity and 23% for 86% of the answer diversity. This translates to saving over 19 40-hour work weeks and $1800, assuming 30 seconds and $0.02 per answer."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1610.00956","title":"Embracing data abundance: BookTest Dataset for Reading Comprehension","sections":[{"idx":0,"heading":"Abstract","paragraphs":["There is a practically unlimited amount of natural language data available. Still, recent work in text comprehension has focused on datasets which are small relative to current computing possibilities. This article is making a case for the community to move to larger data and as a step in that direction it is proposing the BookTest, a new dataset similar to the popular Children's Book Test (CBT), however more than 60 times larger. We show that training on the new data improves the accuracy of our Attention-Sum Reader model on the original CBT test data by a much larger margin than many recent attempts to improve the model architecture. On one version of the dataset our ensemble even exceeds the human baseline provided by Facebook. We then show in our own human study that there is still space for further improvement."]},{"idx":1,"heading":"Introduction","paragraphs":["Since humans amass more and more generally available data in the form of unstructured text it would be very useful to teach machines to read and comprehend such data and then use this understanding to answer our questions. A significant amount of research has recently focused on answering one particular kind of questions the answer to which depends on understanding a context document. These are cloze-style questions BIBREF0 which require the reader to fill in a missing word in a sentence. An important advantage of such questions is that they can be generated automatically from a suitable text corpus which allows us to produce a practically unlimited amount of them. That opens the task to notoriously data-hungry deep-learning techniques which now seem to outperform all alternative approaches.","Two such large-scale datasets have recently been proposed by researchers from Google DeepMind and Facebook AI: the CNN/Daily Mail dataset BIBREF1 and the Children's Book Test (CBT) BIBREF2 respectively. These have attracted a lot of attention from the research community BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 with a new state-of-the-art model coming out every few weeks.","However if our goal is a production-level system actually capable of helping humans, we want the model to use all available resources as efficiently as possible. Given that","we believe that if the community is striving to bring the performance as far as possible, it should move its work to larger data.","This thinking goes in line with recent developments in the area of language modelling. For a long time models were being compared on several \"standard\" datasets with publications often presenting minuscule improvements in performance. Then the large-scale One Billion Word corpus dataset appeared BIBREF15 and it allowed Jozefowicz et al. to train much larger LSTM models BIBREF16 that almost halved the state-of-the-art perplexity on this dataset.","We think it is time to make a similar step in the area of text comprehension. Hence we are introducing the BookTest, a new dataset very similar to the Children's Book test but more than 60 times larger to enable training larger models even in the domain of text comprehension. Furthermore the methodology used to create our data can later be used to create even larger datasets when the need arises thanks to further technological progress.","We show that if we evaluate a model trained on the new dataset on the now standard Children's Book Test dataset, we see an improvement in accuracy much larger than other research groups achieved by enhancing the model architecture itself (while still using the original CBT training data). By training on the new dataset, we reduce the prediction error by almost one third. On the named-entity version of CBT this brings the ensemble of our models to the level of human baseline as reported by Facebook BIBREF2 . However in the final section we show in our own human study that there is still room for improvement on the CBT beyond the performance of our model."]},{"idx":2,"heading":"Task Description","paragraphs":["A natural way of testing a reader's comprehension of a text is to ask her a question the answer to which can be deduced from the text. Hence the task we are trying to solve consists of answering a cloze-style question, the answer to which depends on the understanding of a context document provided with the question. The model is also provided with a set of possible answers from which the correct one is to be selected. This can be formalized as follows:","The training data consist of tuples INLINEFORM0 , where INLINEFORM1 is a question, INLINEFORM2 is a document that contains the answer to question INLINEFORM3 , INLINEFORM4 is a set of possible answers and INLINEFORM5 is the ground-truth answer. Both INLINEFORM6 and INLINEFORM7 are sequences of words from vocabulary INLINEFORM8 . We also assume that all possible answers are words from the vocabulary, that is INLINEFORM9 . In the CBT and CNN/Daily Mail datasets it is also true that the ground-truth answer INLINEFORM10 appears in the document. This is exploited by many machine learning models BIBREF2 , BIBREF4 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF10 , BIBREF11 , BIBREF12 , however some do not explicitly depend on this property BIBREF1 , BIBREF3 , BIBREF5 , BIBREF9"]},{"idx":3,"heading":"Current Landscape","paragraphs":["We will now briefly review what datasets for text comprehension have been published up to date and look at models which have been recently applied to solving the task we have just described."]},{"idx":4,"heading":"Datasets","paragraphs":["A crucial condition for applying deep-learning techniques is to have a huge amount of data available for training. For question answering this specifically means having a large number of document-question-answer triples available. While there is an unlimited amount of text available, coming up with relevant questions and the corresponding answers can be extremely labour-intensive if done by human annotators. There were efforts to provide such human-generated datasets, e.g. Microsoft's MCTest BIBREF17 , however their scale is not suitable for deep learning without pre-training on other data BIBREF18 (such as using pre-trained word embedding vectors).","Google DeepMind managed to avoid this scale issue with their way of generating document-question-answer triples automatically, closely followed by Facebook with a similar method. Let us now briefly introduce the two resulting datasets whose properties are summarized in Table TABREF8 .","These two datasets BIBREF1 exploit a useful feature of online news articles – many articles include a short summarizing sentence near the top of the page. Since all information in the summary sentence is also presented in the article body, we get a nice cloze-style question about the article contents by removing a word from the short summary.","The dataset's authors also replaced all named entities in the dataset by anonymous tokens which are further shuffled for each new batch. This forces the model to rely solely on information from the context document, not being able to transfer any meaning of the named entities between documents.","This restricts the task to one specific aspect of context-dependent question answering which may be useful however it moves the task further from the real application scenario, where we would like the model to use all information available to answer questions. Furthermore Chen et al. BIBREF5 have suggested that this can make about 17% of the questions unanswerable even by humans. They also claim that more than a half of the question sentences are mere paraphrases or exact matches of a single sentence from the context document. This raises a question to what extent the dataset can test deeper understanding of the articles.","The Children's Book Test BIBREF2 uses a different source - books freely available thanks to Project Gutenberg. Since no summary is available, each example consists of a context document formed from 20 consecutive sentences from the story together with a question formed from the subsequent sentence.","The dataset comes in four flavours depending on what type of word is omitted from the question sentence. Based on human evaluation done in BIBREF2 it seems that NE and CN are more context dependent than the other two types – prepositions and verbs. Therefore we (and all of the recent publications) focus only on these two word types.","Several new datasets related to the (now almost standard) ones above emerged recently. We will now briefly present them and explain how the dataset we are introducing in this article differs from them.","The LAMBADA dataset BIBREF19 is designed to measure progress in understanding common-sense questions about short stories that can be easily answered by humans but cannot be answered by current standard machine-learning models (e.g. plain LSTM language models). This dataset is useful for measuring the gap between humans and machine learning algorithms. However, by contrast to our BookTest dataset, it will not allow us to track progress towards the performance of the baseline systems or on examples where machine learning may show super-human performance. Also LAMBADA is just a diagnostic dataset and does not provide ready-to-use question-answering training data, just a plain-text corpus which may moreover include copyrighted books making its use potentially problematic for some purposes. We are providing ready training data consisting of copyright-free books only.","The SQuAD dataset BIBREF20 based on Wikipedia and the Who-did-What dataset BIBREF21 based on Gigaword news articles are factoid question-answering datasets where a multi-word answer should be extracted from a context document. This is in contrast to the previous datasets, including CNN/DM, CBT, LAMBADA and our new dataset, which require only single-word answers. Both these datasets however provide less than 130,000 training questions, two orders of magnitude less than our dataset does.","The Story Cloze Test BIBREF22 provides a crowd-sourced corpus of 49,255 commonsense stories for training and 3,744 testing stories with right and wrong endings. Hence the dataset is again rather small. Similarly to LAMBADA, the Story Cloze Test was designed to be easily answerable by humans.","In the WikiReading BIBREF23 dataset the context document is formed from a Wikipedia article and the question-answer pair is taken from the corresponding WikiData page. For each entity (e.g. Hillary Clinton), WikiData contain a number of property-value pairs (e.g. place of birth: Chicago) which form the datasets's question-answer pairs. The dataset is certainly relevant to the community, however the questions are of very limited variety with only 20 properties (and hence unique questions) covering INLINEFORM0 of the dataset. Furthermore many of the frequent properties are mentioned at a set spot within the article (e.g. the date of birth is almost always in brackets behind the name of a person) which may make the task easier for machines. We are trying to provide a more varied dataset.","Although there are several datasets related to task we are aiming to solve, they differ sufficiently for our dataset to bring new value to the community. Its biggest advantage is its size which can furthermore be easily upscaled without expensive human annotation. Finally while we are emphasizing the differences, models could certainly benefit from as diverse a collection of datasets as possible."]},{"idx":5,"heading":"Machine Learning Models","paragraphs":["A first major work applying deep-learning techniques to text comprehension was Hermann et al. BIBREF1 . This work was followed by the application of Memory Networks to the same task BIBREF2 . Later three models emerged around the same time BIBREF3 , BIBREF4 , BIBREF5 including our psr model BIBREF4 . The AS Reader inspired several subsequent models that use it as a sub-component in a diverse ensemble BIBREF8 ; extend it with a hierarchical structure BIBREF6 , BIBREF24 , BIBREF7 ; compute attention over the context document for every word in the query BIBREF10 or use two-way context-query attention mechanism for every word in the context and the query BIBREF11 that is similar in its spirit to models recently proposed in different domains, e.g. BIBREF25 in information retrieval. Other neural approaches to text comprehension are explored in BIBREF9 , BIBREF12 ."]},{"idx":6,"heading":"Possible Directions for Improvements","paragraphs":["Accuracy in any machine learning tasks can be enhanced either by improving a machine learning model or by using more in-domain training data. Current state of the art models BIBREF6 , BIBREF7 , BIBREF8 , BIBREF11 improve over AS Reader's accuracy on CBT NE and CN datasets by 1-2 percent absolute. This suggests that with current techniques there is only limited room for improvement on the algorithmic side.","The other possibility to improve performance is simply to use more training data. The importance of training data was highlighted by the frequently quoted Mercer's statement that “There is no data like more data.” The observation that having more data is often more important than having better algorithms has been frequently stressed since then BIBREF13 , BIBREF14 .","As a step in the direction of exploiting the potential of more data in the domain of text comprehension, we created a new dataset called BookTest similar to, but much larger than the widely used CBT and CNN/DM datasets."]},{"idx":7,"heading":"BookTest","paragraphs":["Similarly to the CBT, our BookTest dataset is derived from books available through project Gutenberg. We used 3555 copyright-free books to extract CN examples and 10507 books for NE examples, for comparison the CBT dataset was extracted from just 108 books.","When creating our dataset we follow the same procedure as was used to create the CBT dataset BIBREF2 . That is, we detect whether each sentence contains either a named entity or a common noun that already appeared in one of the preceding twenty sentences. This word is then replaced by a gap tag (XXXXX) in this sentence which is hence turned into a cloze-style question. The preceding 20 sentences are used as the context document. For common noun and named entity detection we use the Stanford POS tagger BIBREF27 and Stanford NER BIBREF28 .","The training dataset consists of the original CBT NE and CN data extended with new NE and CN examples. The new BookTest dataset hence contains INLINEFORM0 training examples and INLINEFORM1 tokens.","The validation dataset consists of INLINEFORM0 NE and INLINEFORM1 CN questions. We have one test set for NE and one for CN, each containing INLINEFORM2 examples. The training, validation and test sets were generated from non-overlapping sets of books.","When generating the dataset we removed all editions of books used to create CBT validation and test sets from our training dataset. Therefore the models trained on the BookTest corpus can be evaluated on the original CBT data and they can be compared with recent text-comprehension models utilizing this dataset BIBREF2 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 ."]},{"idx":8,"heading":"Baselines","paragraphs":["We will now use our psr model to evaluate the performance gain from increasing the dataset size."]},{"idx":9,"heading":"AS Reader","paragraphs":["In BIBREF4 we introduced the psr , which at the time of publication significantly outperformed all other architectures on the CNN, DM and CBT datasets. This model is built to leverage the fact that the answer is a single word from the context document. Similarly to many other models it uses attention over the document – intuitively a measure of how relevant each word is to answering the question. However while most previous models used this attention as weights to calculate a blended representation of the answer word, we simply sum the attention across all occurrences of each unique words and then simply select the word with the highest sum as the final answer. While simple, this trick seems both to improve accuracy and to speed-up training. It was adopted by many subsequent models BIBREF8 , BIBREF6 , BIBREF7 , BIBREF10 , BIBREF11 , BIBREF24 .","Let us now describe the model in more detail. Figure FIGREF21 may help you in understanding the following paragraphs.","The words from the document and the question are first converted into vector embeddings using a look-up matrix INLINEFORM0 . The document is then read by a bidirectional GRU network BIBREF29 . A concatenation of the hidden states of the forward and backward GRUs at each word is then used as a contextual embedding of this word, intuitively representing the context in which the word is appearing. We can also understand it as representing the set of questions to which this word may be an answer.","Similarly the question is read by a bidirectional GRU but in this case only the final hidden states are concatenated to form the question embedding.","The attention over each word in the context is then calculated as the dot product of its contextual embedding with the question embedding. This attention is then normalized by the softmax function and summed across all occurrences of each answer candidate. The candidate with most accumulated attention is selected as the final answer.","For a more detailed description of the model including equations check BIBREF4 . More details about the training setup and model hyperparameters can be found in the Appendix.","During our past experiments on the CNN, DM and CBT datasets BIBREF4 each unique word from the training, validation and test datasets had its row in the look-up matrix INLINEFORM0 . However as we radically increased the dataset size, this would result in an extremely large number of model parameters so we decided to limit the vocabulary size to INLINEFORM1 most frequent words. For each example, each unique out-of-vocabulary word is now mapped on one of 1000 anonymous tokens which are randomly initialized and untrained. Fixing the embeddings of these anonymous tags proved to significantly improve the performance.","While mostly using the original AS Reader model, we have also tried introducing a minor tweak in some instances of the model. We tried initializing the context encoder GRU's hidden state by letting the encoder read the question first before proceeding to read the context document. Intuitively this allows the encoder to know in advance what to look for when reading over the context document.","Including models of this kind in the ensemble helped to improve the performance."]},{"idx":10,"heading":"Results","paragraphs":["Table TABREF25 shows the accuracy of the psr and other architectures on the CBT validation and test data. The last two rows show the performance of the psr trained on the BookTest dataset; all the other models have been trained on the original CBT training data.","If we take the best psr ensemble trained on CBT as a baseline, improving the model architecture as in BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , continuing to use the original CBT training data, lead to improvements of INLINEFORM0 and INLINEFORM1 absolute on named entities and common nouns respectively. By contrast, inflating the training dataset provided a boost of INLINEFORM2 while using the same model. The ensemble of our models even exceeded the human baseline provided by Facebook BIBREF2 on the Common Noun dataset.","Our model takes approximately two weeks to converge when trained on the BookTest dataset on a single Nvidia Tesla K40 GPU."]},{"idx":11,"heading":"Discussion","paragraphs":["Embracing the abundance of data may mean focusing on other aspects of system design than with smaller data. Here are some of the challenges that we need to face in this situation.","Firstly, since the amount of data is practically unlimited – we could even generate them on the fly resulting in continuous learning similar to the Never-Ending Language Learning by Carnegie Mellon University BIBREF30 – it is now the speed of training that determines how much data the model is able to see. Since more training data significantly help the model performance, focusing on speeding up the algorithm may be more important than ever before. This may for instance influence the decision whether to use regularization such as dropout which does seem to somewhat improve the model performance, however usually at a cost of slowing down training.","Thanks to its simplicity, the psr seems to be training fast - for example around seven times faster than the models proposed by Chen et al. BIBREF5 . Hence the psr may be particularly suitable for training on large datasets.","The second challenge is how to generalize the performance gains from large data to a specific target domain. While there are huge amounts of natural language data in general, it may not be the case in the domain where we may want to ultimately apply our model.","Hence we are usually not facing a scenario of simply using a larger amount of the same training data, but rather extending training to a related domain of data, hoping that some of what the model learns on the new data will still help it on the original task.","This is highlighted by our observations from applying a model trained on the BookTest to Children's Book Test test data. If we move model training from joint CBT NE+CN training data to a subset of the BookTest of the same size (230k examples), we see a drop in accuracy of around 10% on the CBT test datasets.","Hence even though the Children's Book Test and BookTest datasets are almost as close as two disjoint datasets can get, the transfer is still very imperfect . Rightly choosing data to augment the in-domain training data is certainly a problem worth exploring in future work.","Our results show that given enough data the AS Reader was able to exceed the human performance on CBT CN reported by Facebook. However we hypothesized that the system is still not achieving its full potential so we decided to examine the room for improvement in our own small human study."]},{"idx":12,"heading":"Human Study","paragraphs":["After adding more data we have the performance on the CBT validation and test datasets soaring. However is there still potential for much further growth beyond the results we have observed?","We decided to explore the remaining space for improvement on the CBT by testing humans on a random subset of 50 named entity and 50 common noun validation questions that the psr ensemble could not answer correctly. These questions were answered by 10 non-native English speakers from our research laboratory, each on a disjoint subset of questions.. Participants had unlimited time to answer the questions and were told that these questions were not correctly answered by a machine, providing additional motivation to prove they are better than computers. The results of the human study are summarized in Table TABREF28 . They show that a majority of questions that our system could not answer so far are in fact answerable. This suggests that 1) the original human baselines might have been underestimated, however, it might also be the case that there are some examples that can be answered by machines and not by humans; 2) there is still space for improvement.","A system that would answer correctly every time when either our ensemble or human answered correctly would achieve accuracy over 92% percent on both validation and test NE datasets and over 96% on both CN datasets. Hence it still makes sense to use CBT dataset to study further improvements of text-comprehension systems."]},{"idx":13,"heading":"Conclusion","paragraphs":["Few ways of improving model performance are as solidly established as using more training data. Yet we believe this principle has been somewhat neglected by recent research in text comprehension. While there is a practically unlimited amount of data available in this field, most research was performed on unnecessarily small datasets.","As a gentle reminder to the community we have shown that simply infusing a model with more data can yield performance improvements of up to INLINEFORM0 where several attempts to improve the model architecture on the same training data have given gains of at most INLINEFORM1 compared to our best ensemble result. Yes, experiments on small datasets certainly can bring useful insights. However we believe that the community should also embrace the real-world scenario of data abundance.","The BookTest dataset we are proposing gives the reading-comprehension community an opportunity to make a step in that direction."]},{"idx":14,"heading":"Training Details","paragraphs":["The training details are similar to those in BIBREF4 however we are including them here for completeness.","To train the model we used stochastic gradient descent with the ADAM update rule BIBREF32 and learning rates of INLINEFORM0 , INLINEFORM1 and INLINEFORM2 . The best learning rate in our experiments was INLINEFORM3 . We minimized negative log-likelihood as the training objective.","The initial weights in the word-embedding matrix were drawn randomly uniformly from the interval INLINEFORM0 . Weights in the GRU networks were initialized by random orthogonal matrices BIBREF34 and biases were initialized to zero. We also used a gradient clipping BIBREF33 threshold of 10 and batches of sizes between 32 or 256. Increasing the batch from 32 to 128 seems to significantly improve performance on the large dataset - something we did not observe on the original CBT data. Increasing the batch size much above 128 is currently difficult due to memory constraints of the GPU.","During training we randomly shuffled all examples at the beginning of each epoch. To speed up training, we always pre-fetched 10 batches worth of examples and sorted them according to document length. Hence each batch contained documents of roughly the same length.","We also did not use pre-trained word embeddings.","We did not perform any text pre-processing since the datasets were already tokenized.","During training we evaluated the model performance every 12 hours and at the end of each epoch and stopped training when the error on the 20k BookTest validation set started increasing. We explored the hyperparameter space by training 67 different models The region of the parameter space that we explored together with the parameters of the model with best validation accuracy are summarized in Table TABREF29 .","Our model was implemented using Theano BIBREF31 and Blocks BIBREF35 .","The ensembles were formed by simply averaging the predictions from the constituent single models. These single models were selected using the following algorithm.","We started with the best performing model according to validation performance. Then in each step we tried adding the best performing model that had not been previously tried. We kept it in the ensemble if it did improve its validation performance and discarded it otherwise. This way we gradually tried each model once. We call the resulting model a greedy ensemble. We used the INLINEFORM0 BookTest validation dataset for this procedure.","The algorithm was offered 10 models and selected 5 of them for the final ensemble."]},{"idx":15,"heading":"Figures and Tables","paragraphs":["Table 1: Statistics on the 4 standard text comprehension datasets and our new BookTest dataset introduced in this paper. CBT CN stands for CBT Common Nouns and CBT NE stands for CBT Named Entites. Statistics were taken from (Hermann et al., 2015) and the statistics provided with the CBT data set.","Figure 1: Structure of the AS Reader model.","Table 2: Results of various architectures on the CBT test datasets.","Table 3: Accuracy of human participants on examples from validation set that were previously incorrectly answered by our system trained only on the BookTest data.","Table 4: The best hyperparameters and range of values that we tested on the BookTest train dataset. We report number of hidden units of the unidirectional GRU; the bidirectional GRU has twice as many hidden units."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1610.07809","title":"How Document Pre-processing affects Keyphrase Extraction Performance","sections":[{"idx":0,"heading":"Abstract","paragraphs":["The SemEval-2010 benchmark dataset has brought renewed attention to the task of automatic keyphrase extraction. This dataset is made up of scientific articles that were automatically converted from PDF format to plain text and thus require careful preprocessing so that irrevelant spans of text do not negatively affect keyphrase extraction performance. In previous work, a wide range of document preprocessing techniques were described but their impact on the overall performance of keyphrase extraction models is still unexplored. Here, we re-assess the performance of several keyphrase extraction models and measure their robustness against increasingly sophisticated levels of document preprocessing."]},{"idx":1,"heading":"Introduction","paragraphs":["This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details: http://creativecommons.org/licenses/by/4.0/","Recent years have seen a surge of interest in automatic keyphrase extraction, thanks to the availability of the SemEval-2010 benchmark dataset BIBREF0 . This dataset is composed of documents (scientific articles) that were automatically converted from PDF format to plain text. As a result, most documents contain irrelevant pieces of text (e.g. muddled sentences, tables, equations, footnotes) that require special handling, so as to not hinder the performance of keyphrase extraction systems. In previous work, these are usually removed at the preprocessing step, but using a variety of techniques ranging from simple heuristics BIBREF1 , BIBREF2 , BIBREF3 to sophisticated document logical structure detection on richly-formatted documents recovered from Google Scholar BIBREF4 . Under such conditions, it may prove difficult to draw firm conclusions about which keyphrase extraction model performs best, as the impact of preprocessing on overall performance cannot be properly quantified.","While previous work clearly states that efficient document preprocessing is a prerequisite for the extraction of high quality keyphrases, there is, to our best knowledge, no empirical evidence of how preprocessing affects keyphrase extraction performance. In this paper, we re-assess the performance of several state-of-the-art keyphrase extraction models at increasingly sophisticated levels of preprocessing. Three incremental levels of document preprocessing are experimented with: raw text, text cleaning through document logical structure detection, and removal of keyphrase sparse sections of the document. In doing so, we present the first consistent comparison of different keyphrase extraction models and study their robustness over noisy text. More precisely, our contributions are:"]},{"idx":2,"heading":"Dataset and Preprocessing","paragraphs":["The SemEval-2010 benchmark dataset BIBREF0 is composed of 244 scientific articles collected from the ACM Digital Library (conference and workshop papers). The input papers ranged from 6 to 8 pages and were converted from PDF format to plain text using an off-the-shelf tool. The only preprocessing applied is a systematic dehyphenation at line breaks and removal of author-assigned keyphrases. Scientific articles were selected from four different research areas as defined in the ACM classification, and were equally distributed into training (144 articles) and test (100 articles) sets. Gold standard keyphrases are composed of both author-assigned keyphrases collected from the original PDF files and reader-assigned keyphrases provided by student annotators.","Long documents such as those in the SemEval-2010 benchmark dataset are notoriously difficult to handle due to the large number of keyphrase candidates (i.e. phrases that are eligible to be keyphrases) that the systems have to cope with BIBREF6 . Furthermore, noisy textual content, whether due to format conversion errors or to unusable elements (e.g. equations), yield many spurious keyphrase candidates that negatively affect keyphrase extraction performance. This is particularly true for systems that make use of core NLP tools to select candidates, that in turn exhibit poor performance on degraded text. Filtering out irrelevant text is therefore needed for addressing these issues.","In this study, we concentrate our effort on re-assessing keyphrase extraction performance on three increasingly sophisticated levels of document preprocessing described below.","Table shows the average number of sentences and words along with the maximum possible recall for each level of preprocessing. The maximum recall is obtained by computing the fraction of the reference keyphrases that occur in the documents. We observe that the level 2 preprocessing succeeds in eliminating irrelevant text by significantly reducing the number of words (-19%) while maintaining a high maximum recall (-2%). Level 3 preprocessing drastically reduce the number of words to less than a quarter of the original amount while interestingly still preserving high recall."]},{"idx":3,"heading":"Keyphrase Extraction Models","paragraphs":["We re-implemented five keyphrase extraction models : the first two are commonly used as baselines, the third is a resource-lean unsupervised graph-based ranking approach, and the last two were among the top performing systems in the SemEval-2010 keyphrase extraction task BIBREF0 . We note that two of the systems are supervised and rely on the training set to build their classification models. Document frequency counts are also computed on the training set. Stemming is applied to allow more robust matching. The different keyphrase extraction models are briefly described below:","Each model uses a distinct keyphrase candidate selection method that provides a trade-off between the highest attainable recall and the size of set of candidates. Table summarizes these numbers for each model. Syntax-based selection heuristics, as used by TopicRank and WINGNUS, are better suited to prune candidates that are unlikely to be keyphrases. As for KP-miner, removing infrequent candidates may seem rather blunt, but it turns out to be a simple yet effective pruning method when dealing with long documents. For details on how candidate selection methods affect keyphrase extraction, please refer to BIBREF16 .","Apart from TopicRank that groups similar candidates into topics, the other models do not have any redundancy control mechanism. Yet, recent work has shown that up to 12% of the overall error made by state-of-the-art keyphrase extraction systems were due to redundancy BIBREF6 , BIBREF17 . Therefore as a post-ranking step, we remove redundant keyphrases from the ranked lists generated by all models. A keyphrase is considered redundant if it is included in another keyphrase that is ranked higher in the list."]},{"idx":4,"heading":"Experimental settings","paragraphs":["We follow the evaluation procedure used in the SemEval-2010 competition and evaluate the performance of each model in terms of f-measure (F) at the top INLINEFORM0 keyphrases. We use the set of combined author- and reader-assigned keyphrases as reference keyphrases. Extracted and reference keyphrases are stemmed to reduce the number of mismatches."]},{"idx":5,"heading":"Results","paragraphs":["The performances of the keyphrase extraction models at each preprocessing level are shown in Table . Overall, we observe a significant increase of performance for all models at levels 3, confirming that document preprocessing plays an important role in keyphrase extraction performance. Also, the difference of f-score between the models, as measured by the standard deviation INLINEFORM0 , gradually decreases with the increasing level of preprocessing. This result strengthens the assumption made in this paper, that performance variation across models is partly a function of the effectiveness of document preprocessing.","Somewhat surprisingly, the ordering of the two best models reverses at level 3. This showcases that rankings are heavily influenced by the preprocessing stage, despite the common lack of details and analysis it suffers from in explanatory papers. We also remark that the top performing model, namely KP-Miner, is unsupervised which supports the findings of BIBREF6 indicating that recent unsupervised approaches have rivaled their supervised counterparts in performance.","In an attempt to quantify the performance variation across preprocessing levels, we compute the standard deviation INLINEFORM0 for each model. Here we see that unsupervised models are more sensitive to noisy input, as revealed by higher standard deviations. We found two main reasons for this. First, using multiple discriminative features to rank keyphrase candidates adds inherent robustness to the models. Second, the supervision signal helps models to disregard noise.","In Table , we compare the outputs of the five models by measuring the percentage of valid keyphrases that are retrieved by all models at once for each preprocessing level. By these additional results, we aim to assess whether document preprocessing smoothes differences between models. We observe that the overlap between the outputs of the different models increases along with the level of preprocessing. This suggests that document preprocessing reduces the effect that the keyphrase extraction model in itself has on overall performance. In other words, the singularity of each model fades away gradually with increase in preprocessing effort."]},{"idx":6,"heading":"Reproducibility","paragraphs":["Being able to reproduce experimental results is a central aspect of the scientific method. While assessing the importance of the preprocessing stage for five approaches, we found that several results were not reproducible, as shown in Table .","We note that the trends for baselines and high ranking systems are opposite: compared to the published results, our reproduction of top systems under-performs and our reproduction of baselines over-performs. We hypothesise that this stems from a difference in hyperparameter tuning, including the ones that the preprocessing stage makes implicit. Competitors have strong incentives to correctly optimize hyperparameters, to achieve a high ranking and get more publicity for their work while competition organizers might have the opposite incentive: too strong a baseline might not be considered a baseline anymore.","We also observe that with this leveled preprocessing, the gap between baselines and top systems is much smaller, lessening again the importance of raw scores and rankings to interpret the shared task results and emphasizing the importance of understanding correctly the preprocessing stage."]},{"idx":7,"heading":"Additional experiments","paragraphs":["In the previous sections, we provided empirical evidence that document preprocessing weighs heavily on the outcome of keyphrase extraction models. This raises the question of whether further improvement might be gained from a more aggressive preprocessing. To answer this question, we take another step beyond content filtering and further abridge the input text from level 3 preprocessed documents using an unsupervised summarization technique. More specifically, we keep the title and abstract intact, as they are the two most keyphrase dense parts within scientific articles BIBREF4 , and select only the most content bearing sentences from the remaining contents. To do this, sentences are ordered using TextRank BIBREF14 and the less informative ones, as determined by their TextRank scores normalized by their lengths in words, are filtered out.","Finding the optimal subset of sentences from already shortened documents is however no trivial task as maximum recall linearly decreases with the number of sentences discarded. Here, we simply set the reduction ratio to 0.865 so that the average maximum recall on the training set does not lose more than 5%. Table shows the reduction in the average number of sentences and words compared to level 3 preprocessing.","The performances of the keyphrase extraction models at level 4 preprocessing are shown in Table . We note that two models, namely TopicRank and TF INLINEFORM0 IDF, lose on performance. These two models mainly rely on frequency counts to rank keyphrase candidates, which in turn become less reliable at level 4 because of the very short length of the documents. Other models however have their f-scores once again increased, thus indicating that further improvement is possible from more reductive document preprocessing strategies."]},{"idx":8,"heading":"Conclusion","paragraphs":["In this study, we re-assessed the performance of several keyphrase extraction models and showed that performance variation across models is partly a function of the effectiveness of the document preprocessing. Our results also suggest that supervised keyphrase extraction models are more robust to noisy input.","Given our findings, we recommend that future works use a common preprocessing to evaluate the interest of keyphrase extraction approaches. For this reason we make the four levels of preprocessing used in this study available for the community."]},{"idx":9,"heading":"Figures and Tables","paragraphs":["Table 1: Statistics computed at the different levels of document preprocessing on the training set.","Table 2: Maximum recall and average number of keyphrase candidates for each model.","Table 3: F-scores computed at the top 10 extracted keyphrases for the unsupervised (Unsup.) and supervised (Sup.) models at each preprocessing level. We also report the standard deviation across the five models for each level (σ1) and the standard deviation across the three levels for each model (σ2). α and β indicate significance at the 0.05 level using Student’s t-test against level 1 and level 2 respectively.","Table 5: Difference in f-score between our re-implementation and the original scores reported in (Hasan and Ng, 2014; Bougouin et al., 2013).","Table 6: Statistics computed at level 4 of document preprocessing on the training set. We also report the relative difference with respect to level 3 preprocessing.","Table 7: F-scores computed at the top 10 extracted keyphrases for the unsupervised (Unsup.) and supervised (Sup.) models at level 4 preprocessing. We also report the difference in f-score with level 3 preprocessing."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1611.01576","title":"Quasi-Recurrent Neural Networks","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Recurrent neural networks are a powerful tool for modeling sequential data, but the dependence of each timestep's computation on the previous timestep's output limits parallelism and makes RNNs unwieldy for very long sequences. We introduce quasi-recurrent neural networks (QRNNs), an approach to neural sequence modeling that alternates convolutional layers, which apply in parallel across timesteps, and a minimalist recurrent pooling function that applies in parallel across channels. Despite lacking trainable recurrent layers, stacked QRNNs have better predictive accuracy than stacked LSTMs of the same hidden size. Due to their increased parallelism, they are up to 16 times faster at train and test time. Experiments on language modeling, sentiment classification, and character-level neural machine translation demonstrate these advantages and underline the viability of QRNNs as a basic building block for a variety of sequence tasks."]},{"idx":1,"heading":"Introduction","paragraphs":["Recurrent neural networks (RNNs), including gated variants such as the long short-term memory (LSTM) BIBREF0 have become the standard model architecture for deep learning approaches to sequence modeling tasks. RNNs repeatedly apply a function with trainable parameters to a hidden state. Recurrent layers can also be stacked, increasing network depth, representational power and often accuracy. RNN applications in the natural language domain range from sentence classification BIBREF1 to word- and character-level language modeling BIBREF2 . RNNs are also commonly the basic building block for more complex models for tasks such as machine translation BIBREF3 , BIBREF4 , BIBREF5 or question answering BIBREF6 , BIBREF7 . Unfortunately standard RNNs, including LSTMs, are limited in their capability to handle tasks involving very long sequences, such as document classification or character-level machine translation, as the computation of features or states for different parts of the document cannot occur in parallel.","Convolutional neural networks (CNNs) BIBREF8 , though more popular on tasks involving image data, have also been applied to sequence encoding tasks BIBREF9 . Such models apply time-invariant filter functions in parallel to windows along the input sequence. CNNs possess several advantages over recurrent models, including increased parallelism and better scaling to long sequences such as those often seen with character-level language data. Convolutional models for sequence processing have been more successful when combined with RNN layers in a hybrid architecture BIBREF10 , because traditional max- and average-pooling approaches to combining convolutional features across timesteps assume time invariance and hence cannot make full use of large-scale sequence order information.","We present quasi-recurrent neural networks for neural sequence modeling. QRNNs address both drawbacks of standard models: like CNNs, QRNNs allow for parallel computation across both timestep and minibatch dimensions, enabling high throughput and good scaling to long sequences. Like RNNs, QRNNs allow the output to depend on the overall order of elements in the sequence. We describe QRNN variants tailored to several natural language tasks, including document-level sentiment classification, language modeling, and character-level machine translation. These models outperform strong LSTM baselines on all three tasks while dramatically reducing computation time."]},{"idx":2,"heading":"Model","paragraphs":["Each layer of a quasi-recurrent neural network consists of two kinds of subcomponents, analogous to convolution and pooling layers in CNNs. The convolutional component, like convolutional layers in CNNs, allows fully parallel computation across both minibatches and spatial dimensions, in this case the sequence dimension. The pooling component, like pooling layers in CNNs, lacks trainable parameters and allows fully parallel computation across minibatch and feature dimensions.","Given an input sequence INLINEFORM0 of INLINEFORM1 INLINEFORM2 -dimensional vectors INLINEFORM3 , the convolutional subcomponent of a QRNN performs convolutions in the timestep dimension with a bank of INLINEFORM4 filters, producing a sequence INLINEFORM5 of INLINEFORM6 -dimensional candidate vectors INLINEFORM7 . In order to be useful for tasks that include prediction of the next token, the filters must not allow the computation for any given timestep to access information from future timesteps. That is, with filters of width INLINEFORM8 , each INLINEFORM9 depends only on INLINEFORM10 through INLINEFORM11 . This concept, known as a masked convolution BIBREF11 , is implemented by padding the input to the left by the convolution's filter size minus one.","We apply additional convolutions with separate filter banks to obtain sequences of vectors for the elementwise gates that are needed for the pooling function. While the candidate vectors are passed through a INLINEFORM0 nonlinearity, the gates use an elementwise sigmoid. If the pooling function requires a forget gate INLINEFORM1 and an output gate INLINEFORM2 at each timestep, the full set of computations in the convolutional component is then: DISPLAYFORM0","where INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 , each in INLINEFORM3 , are the convolutional filter banks and INLINEFORM4 denotes a masked convolution along the timestep dimension. Note that if the filter width is 2, these equations reduce to the LSTM-like DISPLAYFORM0","Convolution filters of larger width effectively compute higher INLINEFORM0 -gram features at each timestep; thus larger widths are especially important for character-level tasks.","Suitable functions for the pooling subcomponent can be constructed from the familiar elementwise gates of the traditional LSTM cell. We seek a function controlled by gates that can mix states across timesteps, but which acts independently on each channel of the state vector. The simplest option, which BIBREF12 term “dynamic average pooling”, uses only a forget gate: DISPLAYFORM0","We term these three options f-pooling, fo-pooling, and ifo-pooling respectively; in each case we initialize INLINEFORM0 or INLINEFORM1 to zero. Although the recurrent parts of these functions must be calculated for each timestep in sequence, their simplicity and parallelism along feature dimensions means that, in practice, evaluating them over even long sequences requires a negligible amount of computation time.","A single QRNN layer thus performs an input-dependent pooling, followed by a gated linear combination of convolutional features. As with convolutional neural networks, two or more QRNN layers should be stacked to create a model with the capacity to approximate more complex functions."]},{"idx":3,"heading":"Variants","paragraphs":["Motivated by several common natural language tasks, and the long history of work on related architectures, we introduce several extensions to the stacked QRNN described above. Notably, many extensions to both recurrent and convolutional models can be applied directly to the QRNN as it combines elements of both model types.","Regularization An important extension to the stacked QRNN is a robust regularization scheme inspired by recent work in regularizing LSTMs.","The need for an effective regularization method for LSTMs, and dropout's relative lack of efficacy when applied to recurrent connections, led to the development of recurrent dropout schemes, including variational inference–based dropout BIBREF13 and zoneout BIBREF14 . These schemes extend dropout to the recurrent setting by taking advantage of the repeating structure of recurrent networks, providing more powerful and less destructive regularization.","Variational inference–based dropout locks the dropout mask used for the recurrent connections across timesteps, so a single RNN pass uses a single stochastic subset of the recurrent weights. Zoneout stochastically chooses a new subset of channels to “zone out” at each timestep; for these channels the network copies states from one timestep to the next without modification.","As QRNNs lack recurrent weights, the variational inference approach does not apply. Thus we extended zoneout to the QRNN architecture by modifying the pooling function to keep the previous pooling state for a stochastic subset of channels. Conveniently, this is equivalent to stochastically setting a subset of the QRNN's INLINEFORM0 gate channels to 1, or applying dropout on INLINEFORM1 : DISPLAYFORM0","Thus the pooling function itself need not be modified at all. We note that when using an off-the-shelf dropout layer in this context, it is important to remove automatic rescaling functionality from the implementation if it is present. In many experiments, we also apply ordinary dropout between layers, including between word embeddings and the first QRNN layer.","Densely-Connected Layers We can also extend the QRNN architecture using techniques introduced for convolutional networks. For sequence classification tasks, we found it helpful to use skip-connections between every QRNN layer, a technique termed “dense convolution” by BIBREF15 . Where traditional feed-forward or convolutional networks have connections only between subsequent layers, a “DenseNet” with INLINEFORM0 layers has feed-forward or convolutional connections between every pair of layers, for a total of INLINEFORM1 . This can improve gradient flow and convergence properties, especially in deeper networks, although it requires a parameter count that is quadratic in the number of layers.","When applying this technique to the QRNN, we include connections between the input embeddings and every QRNN layer and between every pair of QRNN layers. This is equivalent to concatenating each QRNN layer's input to its output along the channel dimension before feeding the state into the next layer. The output of the last layer alone is then used as the overall encoding result.","Encoder–Decoder Models To demonstrate the generality of QRNNs, we extend the model architecture to sequence-to-sequence tasks, such as machine translation, by using a QRNN as encoder and a modified QRNN, enhanced with attention, as decoder. The motivation for modifying the decoder is that simply feeding the last encoder hidden state (the output of the encoder's pooling layer) into the decoder's recurrent pooling layer, analogously to conventional recurrent encoder–decoder architectures, would not allow the encoder state to affect the gate or update values that are provided to the decoder's pooling layer. This would substantially limit the representational power of the decoder.","Instead, the output of each decoder QRNN layer's convolution functions is supplemented at every timestep with the final encoder hidden state. This is accomplished by adding the result of the convolution for layer INLINEFORM0 (e.g., INLINEFORM1 , in INLINEFORM2 ) with broadcasting to a linearly projected copy of layer INLINEFORM3 's last encoder state (e.g., INLINEFORM4 , in INLINEFORM5 ): DISPLAYFORM0","where the tilde denotes that INLINEFORM0 is an encoder variable. Encoder–decoder models which operate on long sequences are made significantly more powerful with the addition of soft attention BIBREF3 , which removes the need for the entire input representation to fit into a fixed-length encoding vector. In our experiments, we computed an attentional sum of the encoder's last layer's hidden states. We used the dot products of these encoder hidden states with the decoder's last layer's un-gated hidden states, applying a INLINEFORM1 along the encoder timesteps, to weight the encoder states into an attentional sum INLINEFORM2 for each decoder timestep. This context, and the decoder state, are then fed into a linear layer followed by the output gate: DISPLAYFORM0","where INLINEFORM0 is the last layer.","While the first step of this attention procedure is quadratic in the sequence length, in practice it takes significantly less computation time than the model's linear and convolutional layers due to the simple and highly parallel dot-product scoring function."]},{"idx":4,"heading":"Experiments","paragraphs":["We evaluate the performance of the QRNN on three different natural language tasks: document-level sentiment classification, language modeling, and character-based neural machine translation. Our QRNN models outperform LSTM-based models of equal hidden size on all three tasks while dramatically improving computation speed. Experiments were implemented in Chainer BIBREF16 ."]},{"idx":5,"heading":"Sentiment Classification","paragraphs":["We evaluate the QRNN architecture on a popular document-level sentiment classification benchmark, the IMDb movie review dataset BIBREF17 . The dataset consists of a balanced sample of 25,000 positive and 25,000 negative reviews, divided into equal-size train and test sets, with an average document length of 231 words BIBREF18 . We compare only to other results that do not make use of additional unlabeled data (thus excluding e.g., BIBREF19 ).","Our best performance on a held-out development set was achieved using a four-layer densely-connected QRNN with 256 units per layer and word vectors initialized using 300-dimensional cased GloVe embeddings BIBREF20 . Dropout of 0.3 was applied between layers, and we used INLINEFORM0 regularization of INLINEFORM1 . Optimization was performed on minibatches of 24 examples using RMSprop BIBREF21 with learning rate of INLINEFORM2 , INLINEFORM3 , and INLINEFORM4 .","Small batch sizes and long sequence lengths provide an ideal situation for demonstrating the QRNN's performance advantages over traditional recurrent architectures. We observed a speedup of 3.2x on IMDb train time per epoch compared to the optimized LSTM implementation provided in NVIDIA's cuDNN library. For specific batch sizes and sequence lengths, a 16x speed gain is possible. Figure FIGREF15 provides extensive speed comparisons.","In Figure FIGREF12 , we visualize the hidden state vectors INLINEFORM0 of the final QRNN layer on part of an example from the IMDb dataset. Even without any post-processing, changes in the hidden state are visible and interpretable in regards to the input. This is a consequence of the elementwise nature of the recurrent pooling function, which delays direct interaction between different channels of the hidden state until the computation of the next QRNN layer."]},{"idx":6,"heading":"Language Modeling","paragraphs":["We replicate the language modeling experiment of BIBREF2 and BIBREF13 to benchmark the QRNN architecture for natural language sequence prediction. The experiment uses a standard preprocessed version of the Penn Treebank (PTB) by BIBREF25 .","We implemented a gated QRNN model with medium hidden size: 2 layers with 640 units in each layer. Both QRNN layers use a convolutional filter width INLINEFORM0 of two timesteps. While the “medium” models used in other work BIBREF2 , BIBREF13 consist of 650 units in each layer, it was more computationally convenient to use a multiple of 32. As the Penn Treebank is a relatively small dataset, preventing overfitting is of considerable importance and a major focus of recent research. It is not obvious in advance which of the many RNN regularization schemes would perform well when applied to the QRNN. Our tests showed encouraging results from zoneout applied to the QRNN's recurrent pooling layer, implemented as described in Section SECREF5 .","The experimental settings largely followed the “medium” setup of BIBREF2 . Optimization was performed by stochastic gradient descent (SGD) without momentum. The learning rate was set at 1 for six epochs, then decayed by 0.95 for each subsequent epoch, for a total of 72 epochs. We additionally used INLINEFORM0 regularization of INLINEFORM1 and rescaled gradients with norm above 10. Zoneout was applied by performing dropout with ratio 0.1 on the forget gates of the QRNN, without rescaling the output of the dropout function. Batches consist of 20 examples, each 105 timesteps.","Comparing our results on the gated QRNN with zoneout to the results of LSTMs with both ordinary and variational dropout in Table TABREF14 , we see that the QRNN is highly competitive. The QRNN without zoneout strongly outperforms both our medium LSTM and the medium LSTM of BIBREF2 which do not use recurrent dropout and is even competitive with variational LSTMs. This may be due to the limited computational capacity that the QRNN's pooling layer has relative to the LSTM's recurrent weights, providing structural regularization over the recurrence.","Without zoneout, early stopping based upon validation loss was required as the QRNN would begin overfitting. By applying a small amount of zoneout ( INLINEFORM0 ), no early stopping is required and the QRNN achieves competitive levels of perplexity to the variational LSTM of BIBREF13 , which had variational inference based dropout of 0.2 applied recurrently. Their best performing variation also used Monte Carlo (MC) dropout averaging at test time of 1000 different masks, making it computationally more expensive to run.","When training on the PTB dataset with an NVIDIA K40 GPU, we found that the QRNN is substantially faster than a standard LSTM, even when comparing against the optimized cuDNN LSTM. In Figure FIGREF15 we provide a breakdown of the time taken for Chainer's default LSTM, the cuDNN LSTM, and QRNN to perform a full forward and backward pass on a single batch during training of the RNN LM on PTB. For both LSTM implementations, running time was dominated by the RNN computations, even with the highly optimized cuDNN implementation. For the QRNN implementation, however, the “RNN” layers are no longer the bottleneck. Indeed, there are diminishing returns from further optimization of the QRNN itself as the softmax and optimization overhead take equal or greater time. Note that the softmax, over a vocabulary size of only 10,000 words, is relatively small; for tasks with larger vocabularies, the softmax would likely dominate computation time.","It is also important to note that the cuDNN library's RNN primitives do not natively support any form of recurrent dropout. That is, running an LSTM that uses a state-of-the-art regularization scheme at cuDNN-like speeds would likely require an entirely custom kernel."]},{"idx":7,"heading":"Character-level Neural Machine Translation","paragraphs":["We evaluate the sequence-to-sequence QRNN architecture described in SECREF5 on a challenging neural machine translation task, IWSLT German–English spoken-domain translation, applying fully character-level segmentation. This dataset consists of 209,772 sentence pairs of parallel training data from transcribed TED and TEDx presentations, with a mean sentence length of 103 characters for German and 93 for English. We remove training sentences with more than 300 characters in English or German, and use a unified vocabulary of 187 Unicode code points.","Our best performance on a development set (TED.tst2013) was achieved using a four-layer encoder–decoder QRNN with 320 units per layer, no dropout or INLINEFORM0 regularization, and gradient rescaling to a maximum magnitude of 5. Inputs were supplied to the encoder reversed, while the encoder convolutions were not masked. The first encoder layer used convolutional filter width INLINEFORM1 , while the other encoder layers used INLINEFORM2 . Optimization was performed for 10 epochs on minibatches of 16 examples using Adam BIBREF28 with INLINEFORM3 , INLINEFORM4 , INLINEFORM5 , and INLINEFORM6 . Decoding was performed using beam search with beam width 8 and length normalization INLINEFORM7 . The modified log-probability ranking criterion is provided in the appendix.","Results using this architecture were compared to an equal-sized four-layer encoder–decoder LSTM with attention, applying dropout of 0.2. We again optimized using Adam; other hyperparameters were equal to their values for the QRNN and the same beam search procedure was applied. Table TABREF17 shows that the QRNN outperformed the character-level LSTM, almost matching the performance of a word-level attentional baseline."]},{"idx":8,"heading":"Related Work","paragraphs":["Exploring alternatives to traditional RNNs for sequence tasks is a major area of current research. Quasi-recurrent neural networks are related to several such recently described models, especially the strongly-typed recurrent neural networks (T-RNN) introduced by BIBREF12 . While the motivation and constraints described in that work are different, BIBREF12 's concepts of “learnware” and “firmware” parallel our discussion of convolution-like and pooling-like subcomponents. As the use of a fully connected layer for recurrent connections violates the constraint of “strong typing”, all strongly-typed RNN architectures (including the T-RNN, T-GRU, and T-LSTM) are also quasi-recurrent. However, some QRNN models (including those with attention or skip-connections) are not “strongly typed”. In particular, a T-RNN differs from a QRNN as described in this paper with filter size 1 and f-pooling only in the absence of an activation function on INLINEFORM0 . Similarly, T-GRUs and T-LSTMs differ from QRNNs with filter size 2 and fo- or ifo-pooling respectively in that they lack INLINEFORM1 on INLINEFORM2 and use INLINEFORM3 rather than sigmoid on INLINEFORM4 .","The QRNN is also related to work in hybrid convolutional–recurrent models. BIBREF31 apply CNNs at the word level to generate INLINEFORM0 -gram features used by an LSTM for text classification. BIBREF32 also tackle text classification by applying convolutions at the character level, with a stride to reduce sequence length, then feeding these features into a bidirectional LSTM. A similar approach was taken by BIBREF10 for character-level machine translation. Their model's encoder uses a convolutional layer followed by max-pooling to reduce sequence length, a four-layer highway network, and a bidirectional GRU. The parallelism of the convolutional, pooling, and highway layers allows training speed comparable to subword-level models without hard-coded text segmentation.","The QRNN encoder–decoder model shares the favorable parallelism and path-length properties exhibited by the ByteNet BIBREF33 , an architecture for character-level machine translation based on residual convolutions over binary trees. Their model was constructed to achieve three desired properties: parallelism, linear-time computational complexity, and short paths between any pair of words in order to better propagate gradient signals."]},{"idx":9,"heading":"Conclusion","paragraphs":["Intuitively, many aspects of the semantics of long sequences are context-invariant and can be computed in parallel (e.g., convolutionally), but some aspects require long-distance context and must be computed recurrently. Many existing neural network architectures either fail to take advantage of the contextual information or fail to take advantage of the parallelism. QRNNs exploit both parallelism and context, exhibiting advantages from both convolutional and recurrent neural networks. QRNNs have better predictive accuracy than LSTM-based models of equal hidden size, even though they use fewer parameters and run substantially faster. Our experiments show that the speed and accuracy advantages remain consistent across tasks and at both word and character levels.","Extensions to both CNNs and RNNs are often directly applicable to the QRNN, while the model's hidden states are more interpretable than those of other recurrent architectures as its channels maintain their independence across timesteps. We believe that QRNNs can serve as a building block for long-sequence tasks that were previously impractical with traditional RNNs."]},{"idx":10,"heading":"Beam search ranking criterion","paragraphs":["The modified log-probability ranking criterion we used in beam search for translation experiments is: DISPLAYFORM0","where INLINEFORM0 is a length normalization parameter BIBREF34 , INLINEFORM1 is the INLINEFORM2 th output character, and INLINEFORM3 is a “target length” equal to the source sentence length plus five characters. This reduces at INLINEFORM4 to ordinary beam search with probabilities: DISPLAYFORM0","and at INLINEFORM0 to beam search with probabilities normalized by length (up to the target length): DISPLAYFORM0","Conveniently, this ranking criterion can be computed at intermediate beam-search timesteps, obviating the need to apply a separate reranking on complete hypotheses."]},{"idx":11,"heading":"Figures and Tables","paragraphs":["Figure 1: Block diagrams showing the computation structure of the QRNN compared with typical LSTM and CNN architectures. Red signifies convolutions or matrix multiplications; a continuous block means that those computations can proceed in parallel. Blue signifies parameterless functions that operate in parallel along the channel/feature dimension. LSTMs can be factored into (red) linear blocks and (blue) elementwise blocks, but computation at each timestep still depends on the results from the previous timestep.","Figure 2: The QRNN encoder–decoder architecture used for machine translation experiments.","Table 1: Accuracy comparison on the IMDb binary sentiment classification task. All of our models use 256 units per layer; all layers other than the first layer, whose filter width may vary, use filter width k = 2. Train times are reported on a single NVIDIA K40 GPU. We exclude semi-supervised models that conduct additional training on the unlabeled portion of the dataset.","Figure 3: Visualization of the final QRNN layer’s hidden state vectors cLt in the IMDb task, with timesteps along the vertical axis. Colors denote neuron activations. After an initial positive statement “This movie is simply gorgeous” (off graph at timestep 9), timestep 117 triggers a reset of most hidden states due to the phrase “not exactly a bad story” (soon after “main weakness is its story”). Only at timestep 158, after “I recommend this movie to everyone, even if you’ve never played the game”, do the hidden units recover.","Table 2: Single model perplexity on validation and test sets for the Penn Treebank language modeling task. Lower is better. “Medium” refers to a two-layer network with 640 or 650 hidden units per layer. All QRNN models include dropout of 0.5 on embeddings and between layers. MC refers to Monte Carlo dropout averaging at test time.","Figure 4: Left: Training speed for two-layer 640-unit PTB LM on a batch of 20 examples of 105 timesteps. “RNN” and “softmax” include the forward and backward times, while “optimization overhead” includes gradient clipping, L2 regularization, and SGD computations. Right: Inference speed advantage of a 320-unit QRNN layer alone over an equal-sized cuDNN LSTM layer for data with the given batch size and sequence length. Training results are similar.","Table 3: Translation performance, measured by BLEU, and train speed in hours per epoch, for the IWSLT German-English spoken language translation task. All models were trained on in-domain data only, and use negative log-likelihood as the training criterion. Our models were trained for 10 epochs. The QRNN model uses k = 2 for all layers other than the first encoder layer."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1611.01884","title":"AC-BLSTM: Asymmetric Convolutional Bidirectional LSTM Networks for Text Classification","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Recently deeplearning models have been shown to be capable of making remarkable performance in sentences and documents classification tasks. In this work, we propose a novel framework called AC-BLSTM for modeling sentences and documents, which combines the asymmetric convolution neural network (ACNN) with the Bidirectional Long Short-Term Memory network (BLSTM). Experiment results demonstrate that our model achieves state-of-the-art results on five tasks, including sentiment analysis, question type classification, and subjectivity classification. In order to further improve the performance of AC-BLSTM, we propose a semi-supervised learning framework called G-AC-BLSTM for text classification by combining the generative model with AC-BLSTM."]},{"idx":1,"heading":"Introduction","paragraphs":["Deep neural models recently have achieved remarkable results in computer vision BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , and a range of NLP tasks such as sentiment classification BIBREF4 , BIBREF5 , BIBREF6 , and question-answering BIBREF7 . Convolutional neural networks (CNNs) and recurrent neural networks (RNNs) especially Long Short-term Memory Network (LSTM), are used wildly in natural language processing tasks. With increasing datas, these two methods can reach considerable performance by requiring only limited domain knowledge and easy to be finetuned to specific applications at the same time.","CNNs, which have the ability of capturing local correlations of spatial or temporal structures, have achieved excellent performance in computer vision and NLP tasks. And recently the emerge of some new techniques, such as Inception module BIBREF8 , Batchnorm BIBREF9 and Residual Network BIBREF3 have also made the performance even better. For sentence modeling, CNNs perform excellently in extracting n-gram features at different positions of a sentence through convolutional filters.","RNNs, with the ability of handling sequences of any length and capturing long-term dependencies, , have also achieved remarkable results in sentence or document modeling tasks. LSTMs BIBREF10 were designed for better remembering and memory accesses, which can also avoid the problem of gradient exploding or vanishing in the standard RNN. Be capable of incorporating context on both sides of every position in the input sequence, BLSTMs introduced in BIBREF11 , BIBREF12 have reported to achieve great performance in Handwriting Recognition BIBREF13 , and Machine Translation BIBREF14 tasks.","Generative adversarial networks (GANs) BIBREF15 are a class of generative models for learning how to produce images. Basically, GANs consist of a generator G and a discriminator D, which are trained based on game theory. G maps a input noise vector to an output image, while D takes in an image then outputs a prediction whether the input image is a sample generated by G. Recently, applications of GANs have shown that they can generate promising results BIBREF16 , BIBREF17 . Several recent papers have also extended GANs to the semi-supervised context BIBREF18 , BIBREF19 by simply increasing the dimension of the classifier output from INLINEFORM0 to INLINEFORM1 , which the samples of the extra class are generated by G.","In this paper, We proposed an end-to-end architecture named AC-BLSTM by combining the ACNN with the BLSTM for sentences and documents modeling. In order to make the model deeper, instead of using the normal convolution, we apply the technique proposed in BIBREF8 which employs a INLINEFORM0 convolution followed by a INLINEFORM1 convolution by spatial factorizing the INLINEFORM2 convolution. And we use the pretrained word2vec vectors BIBREF20 as the ACNN input, which were trained on 100 billion words of Google News to learn the higher-level representations of n-grams. The outputs of the ACNN are organized as the sequence window feature to feed into the multi-layer BLSTM. So our model does not rely on any other extra domain specific knowledge and complex preprocess, e.g. word segmentation, part of speech tagging and so on. We evaluate AC-BLSTM on sentence-level and document-level tasks including sentiment analysis, question type classification, and subjectivity classification. Experimental results demonstrate the effectiveness of our approach compared with other state-of-the-art methods. Further more, inspired by the ideas of extending GANs to the semi-supervised learning context by BIBREF18 , BIBREF19 , we propose a semi-supervised learning framework for text classification which further improve the performance of AC-BLSTM.","The rest of the paper is organized as follows. Section 2 presents a brief review of related work. Section 3 discusses the architecture of our AC-BLSTM and our semi-supervised framework. Section 4 presents the experiments result with comparison analysis. Section 5 concludes the paper."]},{"idx":2,"heading":"Related Work","paragraphs":["Deep learning models have made remarkable progress in various NLP tasks recently. For example, word embeddings BIBREF20 , BIBREF21 , question answearing BIBREF7 , sentiment analysis BIBREF22 , BIBREF23 , BIBREF24 , machine translation BIBREF25 and so on. CNNs and RNNs are two wildly used architectures among these models. The success of deep learning models for NLP mostly relates to the progress in learning distributed word representations BIBREF20 , BIBREF21 . In these mothods, instead of using one-hot vectors by indexing words into a vocabulary, each word is modeled as a low dimensional and dense vector which encodes both semantic and syntactic information of words.","Our model mostly relates to BIBREF4 which combines CNNs of different filter lengths and either static or fine-tuned word vectors, and BIBREF5 which stacks CNN and LSTM in a unified architecture with static word vectors. It is known that in computer vision, the deeper network architecture usually possess the better performance. We consider NLP also has this property. In order to make our model deeper, we apply the idea of asymmetric convolution introduced in BIBREF8 , which can reduce the number of the parameters, and increase the representation ability of the model by adding more nonlinearity. Then we stack the multi-layer BLSTM, which is cable of analysing the future as well as the past of every position in the sequence, on top of the ACNN. The experiment results also demonstrate the effectiveness of our model."]},{"idx":3,"heading":"AC-BLSTM Model","paragraphs":["In this section, we will introduce our AC-BLSTM architecture in detail. We first describe the ACNN which takes the word vector represented matrix of the sentence as input and produces higher-level presentation of word features. Then we introduce the BLSTM which can incorporate context on both sides of every position in the input sequence. Finally, we introduce the techniques to avoid overfitting in our model. An overall illustration of our architecture is shown in Figure FIGREF1 ."]},{"idx":4,"heading":"Asymmetric Convolution","paragraphs":["Let x INLINEFORM0 be the INLINEFORM1 -dimensional word vector corresponding to the INLINEFORM2 -th word in the sentence and INLINEFORM3 be the maximum length of the sentence in the dataset. Then the sentence with length INLINEFORM4 is represented as DISPLAYFORM0","For those sentences that are shorter than INLINEFORM0 , we simply pad them with space.","In general, let INLINEFORM0 in which INLINEFORM1 be the length of convolution filter. Then instead of employing the INLINEFORM2 convolution operation described in BIBREF4 , BIBREF5 , we apply the asymmetric convolution operation inspired by BIBREF8 to the input matrix which factorize the INLINEFORM3 convolution into INLINEFORM4 convolution followed by a INLINEFORM5 convolution. And in experiments, we found that employ this technique can imporve the performance. The following part of this subsection describe how we define the asymmetric convolution layer.","First, the convolution operation corresponding to the INLINEFORM0 convolution with filter w INLINEFORM1 is applied to each word x INLINEFORM2 in the sentence and generates corresponding feature m INLINEFORM3 DISPLAYFORM0","where INLINEFORM0 is element-wise multiplication, INLINEFORM1 is a bias term and INLINEFORM2 is a non-linear function such as the sigmoid, hyperbolic tangent, etc. In our case, we choose ReLU BIBREF26 as the nonlinear function. Then we get the feature map m INLINEFORM3 DISPLAYFORM0","After that, the second convolution operation of the asymmetric convolution layer corresponding to the INLINEFORM0 convolution with filter w INLINEFORM1 is applied to a window of INLINEFORM2 features in the feature map m INLINEFORM3 to produce the new feature c INLINEFORM4 and the feature map c INLINEFORM5 DISPLAYFORM0 DISPLAYFORM1","with c INLINEFORM0 . Where INLINEFORM1 , INLINEFORM2 and INLINEFORM3 are the same as described above.","As shown in Figure FIGREF1 , we simultaneously apply three asymmetric convolution layers to the input matrix, which all have the same number of filters denoted as INLINEFORM0 . Thus the output of the asymmetric convolution layer has INLINEFORM1 feature maps. To generate the input sequence of the BLSTM, for each output sequence of the second convolution operation in the aysmmetric convolution layer, we slice the feature maps by channel then obtained sequence of INLINEFORM2 new features c INLINEFORM3 where INLINEFORM4 . Then we concatanate c INLINEFORM5 , c INLINEFORM6 and c INLINEFORM7 to get the input feature for each time step DISPLAYFORM0","where INLINEFORM0 for INLINEFORM1 and INLINEFORM2 . In general, those c INLINEFORM3 where INLINEFORM4 and INLINEFORM5 must be dropped in order to maintain the same sequence length, which will cause the loss of some information. In our model, instead of simply cutting the sequence, we use a simple trick to obtain the same sequence length without losing the useful information as shown in Figure FIGREF2 . For each output sequence INLINEFORM6 obtained from the second convolution operation with filter length INLINEFORM7 , we take those c INLINEFORM8 where INLINEFORM9 then apply a fullyconnected layer to get a new feature, which has the same dimension of c INLINEFORM10 , to replace the ( INLINEFORM11 +1)-th feature in the origin sequence."]},{"idx":5,"heading":"Bidirectional Long Short-Term Memory Network","paragraphs":["First introduced in BIBREF10 and shown as a successful model recently, LSTM is a RNN architecture specifically designed to bridge long time delays between relevant input and target events, making it suitable for problems where long range context is required, such as handwriting recognition, machine translation and so on.","For many sequence processing tasks, it is useful to analyze the future as well as the past of a given point in the series. Whereas standard RNNs make use of previous context only, BLSTM BIBREF11 is explicitly designed for learning long-term dependencies of a given point on both side, which has also been shown to outperform other neural network architectures in framewise phoneme recognition BIBREF12 .","Therefore we choose BLSTM on top of the ACNN to learn such dependencies given the sequence of higher-level features. And single layer BLSTM can extend to multi-layer BLSTM easily. Finally, we concatenate all hidden state of all the time step of BLSTM, or concatenate the last layer of all the time step hidden state of multi-layer BLSTM, to obtain final representation of the text and we add a softmax layer on top of the model for classification."]},{"idx":6,"heading":"Semi-supervised Framework","paragraphs":["Our semi-supervised text classification framewrok is inspired by works BIBREF18 , BIBREF19 . We assume the original classifier classify a sample into one of INLINEFORM0 possible classes. So we can do semi-supervised learning by simply adding samples from a generative network G to our dataset and labeling them to an extra class INLINEFORM1 . And correspondingly the dimension of our classifier output increases from INLINEFORM2 to INLINEFORM3 . The configuration of our generator network G is inspired by the architecture proposed in BIBREF16 . And we modify the architecture to make it suitable to the text classification tasks. Table TABREF13 shows the configuration of each layer in the generator G. Lets assume the training batch size is INLINEFORM4 and the percentage of the generated samples among a batch training samples is INLINEFORM5 . At each iteration of the training process, we first generate INLINEFORM6 samples from the generator G then we draw INLINEFORM7 samples from the real dataset. We then perform gradient descent on the AC-BLSTM and generative net G and finally update the parameters of both nets."]},{"idx":7,"heading":"Regularization","paragraphs":["For model regularization, we employ two commonly used techniques to prevent overfitting during training: dropout BIBREF27 and batch normalization BIBREF9 . In our model, we apply dropout to the input feature of the BLSTM, and the output of BLSTM before the softmax layer. And we apply batch normalization to outputs of each convolution operation just before the relu activation. During training, after we get the gradients of the AC-BLSTM network, we first calculate the INLINEFORM0 INLINEFORM1 of all gradients and sum together to get INLINEFORM2 . Then we compare the INLINEFORM3 to 0.5. If the INLINEFORM4 is greater than 0.5, we let all the gradients multiply with INLINEFORM5 , else just use the original gradients to update the weights."]},{"idx":8,"heading":"Datasets","paragraphs":["We evaluate our model on various benchmarks. Stanford Sentiment Treebank (SST) is a popular sentiment classification dataset introduced by BIBREF33 . The sentences are labeled in a fine-grained way (SST-1): very negative, negative, neutral, positive, very positive. The dataset has been split into 8,544 training, 1,101 validation, and 2,210 testing sentences. By removing the neutral sentences, SST can also be used for binary classification (SST-2), which has been split into 6,920 training, 872 validation, and 1,821 testing. Since the data is provided in the format of sub-sentences, we train the model on both phrases and sentences but only test on the sentences as in several previous works BIBREF33 , BIBREF6 .","Movie Review Data (MR) proposed by BIBREF34 is another dataset for sentiment analysis of movie reviews. The dataset consists of 5,331 positive and 5,331 negative reviews, mostly in one sentence. We follow the practice of using 10-fold cross validation to report the result.","Furthermore, we apply AC-BLSTM on the subjectivity classification dataset (SUBJ) released by BIBREF35 . The dataset contains 5,000 subjective sentences and 5,000 objective sentences. We also follow the practice of using 10-fold cross validation to report the result.","We also benchmark our system on question type classification task (TREC) BIBREF36 , where sentences are questions in the following 6 classes: abbreviation, human, entity, description, location, numeric. The entire dataset consists of 5,452 training examples and 500 testing examples.","For document-level dataset, we use the sentiment classification dataset Yelp 2013 (YELP13) with user and product information, which is built by BIBREF22 . The dataset has been split into 62,522 training, 7,773 validation, and 8,671 testing documents. But in the experiment, we neglect the user and product information to make it consistent with the above experiment settings."]},{"idx":9,"heading":"Training and Implementation Details","paragraphs":["We implement our model based on Mxnet BIBREF37 - a C++ library, which is a deep learning framework designed for both efficiency and flexibility. In order to benefit from the efficiency of parallel computation of the tensors, we train our model on a Nvidia GTX 1070 GPU. Training is done through stochastic gradient descent over shuffled mini-batches with the optimizer RMSprop BIBREF38 . For all experiments, we simultaneously apply three asymmetric convolution operation with the second filter length INLINEFORM0 of 2, 3, 4 to the input, set the dropout rate to 0.5 before feeding the feature into BLSTM, and set the initial learning rate to 0.0001. But there are some hyper-parameters that are not the same for all datasets, which are listed in table TABREF14 . We conduct experiments on 3 datasets (MR, SST and SUBJ) to verify the effectiveness our semi-supervised framework. And the setting of INLINEFORM1 and INLINEFORM2 for different datasets are listed in table TABREF15 ."]},{"idx":10,"heading":"Word Vector Initialization","paragraphs":["We use the publicly available word2vec vectors that were trained on 100 billion words from Google News. The vectors have dimensionality of 300 and were trained using the continuous bag-of-words architecture BIBREF20 . Words not present in the set of pre-trained words are initialized from the uniform distribution [-0.25, 0.25]. We fix the word vectors and learn only the other parameters of the model during training."]},{"idx":11,"heading":"Results and Discussion","paragraphs":["We used standard train/test splits for those datasets that had them. Otherwise, we performed 10-fold cross validation. We repeated each experiment 10 times and report the mean accuracy. Results of our models against other methods are listed in table TABREF16 . To the best of our knowledge, AC-BLSTM achieves the best results on five tasks.","Compared to methods BIBREF4 and BIBREF5 , which inspired our model mostly, AC-BLSTM can achieve better performance which show that deeper model actually has better performance. By just employing the word2vec vectors, our model can achieve better results than BIBREF30 which combines multiple word embedding methods such as word2vec BIBREF20 , glove BIBREF21 and Syntactic embedding. And the AC-BLSTM performs better when trained with the semi-supervised framework, which proves the success of combining the generative net with AC-BLSTM.","The experiment results show that the number of the convolution filter and the lstm memory dimension should keep the same for our model. Also the configuration of hyper-parameters: number of the convolution filter, the lstm memory dimension and the lstm layer are quiet stable across datasets. If the task is simple, e.g. TREC, we just set number of convolution filter to 100, lstm memory dimension to 100 and lstm layer to 1. And as the task becomes complicated, we simply increase the lstm layer from 1 to 4. The SST-2 is a special case, we find that if we set the number of convolution filter and lstm memory dimension to 300 can get better result. And the dropout rate before softmax need to be tuned."]},{"idx":12,"heading":"Conclusions","paragraphs":["In this paper we have proposed AC-BLSTM: a novel framework that combines asymmetric convolutional neural network with bidirectional long short-term memory network. The asymmetric convolutional layers are able to learn phrase-level features. Then output sequences of such higher level representations are fed into the BLSTM to learn long-term dependencies of a given point on both side. To the best of our knowledge, the AC-BLSTM model achieves top performance on standard sentiment classification, question classification and document categorization tasks. And then we proposed a semi-supervised framework for text classification which further improve the performance of AC-BLSTM. In future work, we plan to explore the combination of multiple word embeddings which are described in BIBREF30 .","2pt"]},{"idx":13,"heading":"Figures and Tables","paragraphs":["Figure 1: Illustration of the AC-BLSTM architecture. The input is represented as a matrix where each row is a d-dimensional word vector. Then the ACNN is applied to obtain the feature maps, we apply three parallel asymmetric convolution operation on the input in our model, where k1, k2 and k3 stand for the length of the filter. And then the features with the same convolution window index from different convolution layer (different color) are concatenated to generate the input sequence of BLSTM. Finally all the hidden units of BLSTM are concatenated then apply a softmax layer to obtain the prediction output.","Figure 2: Illustration of how to deal with the incosistence output sequence length by compression.","Table 1: Experiment results of our AC-BLSTM model compared with other methods. Performance is measured in accuracy. CNN-non-static, CNN-multichannel: Convolutional Neural Network with fine-tuned word vectors and multi-channels (Kim, 2014). C-LSTM: Combining CNN and LSTM to model sentences (Zhou et al., 2015). Molding-CNN: A feature mapping operation based on tensor products on stacked vectors (Lei et al., 2015). UPNN(no UP): User Product Neural Network without using user and product information (Tang et al., 2015). DSCNN, DSCNN-Pretrain: Dependency Sensitive Convolutional Neural Networks and with pretraind sequence autoencoders (Zhang et al., 2016a). MG-CNN(w2v+Syn+Glv), MGNC-CNN(w2v+Glv), MGNC-CNN(w2v+Syn+Glv): Multi-group norm constraint CNN with w2v:word2vec, Glv:GloVe (Pennington et al., 2014) and Syn: Syntactic embedding (Zhang et al., 2016b). NSC+LA: Neural Sentiment Classification model with local semantic attention (Chen et al., 2016a). SequenceModel(no UP): A sequence modeling-based neural network without using user and product information (Chen et al., 2016b).","Table 2: Defferent hyper-parameters setting across datasets."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1611.02378","title":"A Surrogate-based Generic Classifier for Chinese TV Series Reviews","sections":[{"idx":0,"heading":"Abstract","paragraphs":["With the emerging of various online video platforms like Youtube, Youku and LeTV, online TV series' reviews become more and more important both for viewers and producers. Customers rely heavily on these reviews before selecting TV series, while producers use them to improve the quality. As a result, automatically classifying reviews according to different requirements evolves as a popular research topic and is essential in our daily life. In this paper, we focused on reviews of hot TV series in China and successfully trained generic classifiers based on eight predefined categories. The experimental results showed promising performance and effectiveness of its generalization to different TV series."]},{"idx":1,"heading":"Introduction","paragraphs":["With Web 2.0's development, more and more commercial websites, such as Amazon, Youtube and Youku, encourage users to post product reviews on their platforms BIBREF0 , BIBREF1 . These reviews are helpful for both readers and product manufacturers. For example, for TV or movie producers, online reviews indicates the aspects that viewers like and/or dislike. This information facilitates producers' production process. When producing future films TV series, they can tailer their shows to better accommodate consumers' tastes. For manufacturers, reviews may reveal customers' preference and feedback on product functions, which help manufacturers to improve their products in future development. On the other hand, consumers can evaluate the quality of product or TV series based on online reviews, which help them make final decisions of whether to buy or watch it. However, there are thousands of reviews emerging every day. Given the limited time and attention consumers have, it is impossible for them to allocate equal amount of attention to all the reviews. Moreover, some readers may be only interested in certain aspects of a product or TV series. It's been a waste of time to look at other irrelevant ones. As a result, automatic classification of reviews is essential for the review platforms to provide a better perception of the review contents to the users.","Most of the existing review studies focus on product reviews in English. While in this paper, we focus on reviews of hot Chinese movies or TV series, which owns some unique characteristics. First, Table TABREF1 shows Chinese movies' development BIBREF2 in recent years. The growth of box office and viewers is dramatically high in these years, which provides substantial reviewer basis for the movie/TV series review data. Moreover, the State Administration of Radio Film and Television also announced that the size of the movie market in China is at the 2nd place right after the North America market. In BIBREF2 , it also has been predicted that the movie market in China may eventually become the largest movie market in the world within the next 5-10 years. Therefore, it is of great interest to researchers, practitioners and investors to understand the movie market in China.","Besides flourishing of movie/TV series, there are differences of aspect focuses between product and TV series reviews. When a reviewer writes a movie/TV series review, he or she not only care about the TV elements like actor/actress, visual effect, dialogues and music, but also related teams consisted of director, screenwriter, producer, etc. However, with product reviews, few reviewers care about the corresponding backstage teams. What they do care and will comment about are only product related issues like drawbacks of the product functions, or which aspect of the merchandise they like or dislike. Moreover, most of recent researchers' work has been focused on English texts due to its simpler grammatical structure and less vocabulary, as compared with Chinese. Therefore, Chinese movie reviews not only provide more content based information, but also raise more technical challenges. With bloom of Chinese movies, automatic classification of Chinese movie reviews is really essential and meaningful.","In this paper, we proposed several strategies to make our classifiers generalizable to agnostic TV series. First, TV series roles' and actors/actresses' names are substituted by generic tags like role_i and player_j, where i and j defines their importance in this movie. On top of such kind of words, feature tokens are further manipulated by feature selection techniques like DRC or INLINEFORM0 , in order to make it more generic. We also experimented with different feature sizes with multiple classifiers in order to alleviate overfitting with high dimension features.","The remainder of this paper is organized as follows. Section 2 describes some related work. Section 3 states our problem and details our proposed procedure of approaching the problem. In Section 4, experimental results are provided and discussed. Finally, the conclusions are presented in Section 5."]},{"idx":2,"heading":"Related Work","paragraphs":["Since we are doing supervised learning task with text input, it is related with work of useful techniques like feature selections and supervised classifiers. Besides, there are only public movie review datasets in English right now, which is different from our language requirement. In the following of this section, we will first introduce some existing feature selection techniques and supervised classifiers we applied in our approach. Then we will present some relevant datasets that are normally used in movie review domain."]},{"idx":3,"heading":"Feature selection","paragraphs":["Feature selection, or variable selection is a very common strategy applied in machine learning domain, which tries to select a subset of relevant features from the whole set. There are mainly three purposes behind this. Smaller feature set or features with lower dimension can help researchers to understand or interpret the model they designed more easily. With fewer features, we can also improve the generalization of our model through preventing overfitting, and reduce the whole training time.","Document Relevance Correlation(DRC), proposed by W. Fan et al 2005 BIBREF3 , is a useful feature selection technique. The authors apply this approach to profile generation in digital library service and news-monitoring. They compared DRC with other well-known methods like Robertson's Selection Value BIBREF4 , and machine learning based ones like information gain BIBREF5 . Promising experimental results were shown to demonstrate the effectiveness of DRC as a feature selection in text field.","Another popular feature selection method is called INLINEFORM0 BIBREF6 , which is a variant of INLINEFORM1 test in statistics that tries to test the independence between two events. While in feature selection domain, the two events can be interpreted as the occurrence of feature variable and a particular class. Then we can rank the feature terms with respect to the INLINEFORM2 value. It has been proved to be very useful in text domain, especially with bag of words feature model which only cares about the appearance of each term."]},{"idx":4,"heading":"Supervised Classifier","paragraphs":["What we need is to classify each review into several generic categories that might be attractive to the readers, so classifier selection is also quite important in our problem. Supervised learning takes labeled training pairs and tries to learn an inferred function, which can be used to predict new samples. In this paper, our selection is based on two kinds of learning, i.e., discriminative and generative learning algorithms. And we choose three typical algorithms to compare. Bayes BIBREF7 , which is the representative of generative learning, will output the class with the highest probability that is generated through the bayes' rule. While for the discriminative classifiers like logistic regression BIBREF8 or Support Vector Machine BIBREF9 , final decisions are based on the classifier's output score, which is compared with some threshold to distinguish between different classes."]},{"idx":5,"heading":"TV series Review Dataset","paragraphs":["Dataset is another important factor influencing the performance of our classifiers. Most of the public available movie review data is in English, like the IMDB dataset collected by Pang/Lee 2004 BIBREF10 . Although it covers all kinds of movies in IMDB website, it only has labels related with the sentiment. Its initial goal was for sentiment analysis. Another intact movie review dataset is SNAP BIBREF11 , which consists of reviews from Amazon but only bearing rating scores. However, what we need is the content or aspect tags that are being discussed in each review. In addition, our review text is in Chinese. Therefore, it is necessary for us to build the review dataset by ourselves and label them into generic categories, which is one of as one of the contributions of this paper."]},{"idx":6,"heading":"Chinese TV series Review Classification","paragraphs":["Let INLINEFORM0 be a set of Chinese movie reviews with no categorical information. The ultimate task of movie review classification is to label them into different predefined categories as INLINEFORM1 . Starting from scratch, we need to collect such review set INLINEFORM2 from an online review website and then manually label them into generic categories INLINEFORM3 . Based on the collected dataset, we can apply natural language processing techniques to get raw text features and further learn the classifiers. In the following subsections, we will go through and elaborate all the subtasks shown in Figure FIGREF5 ."]},{"idx":7,"heading":"Building Dataset","paragraphs":["What we are interested in are the reviews of the hottest or currently broadcasted TV series, so we select one of the most influential movie and TV series sharing websites in China, Douban. For every movie or TV series, you can find a corresponding section in it. For the sake of popularity, we choose “The Journey of Flower”, “Nirvana in Fire” and “Good Time” as parts of our movie review dataset, which are the hottest TV series from summer to fall 2015. Reviews of each episode have been collected for the sake of dataset comprehensiveness.","Then we built the crawler written in python with the help of scrapy. Scrapy will create multiple threads to crawl information we need simultaneously, which saves us lots of time. For each episode, it collected both the short description of this episode and all the reviews under this post. The statistics of our TV series review dataset is shown in Table TABREF7 ."]},{"idx":8,"heading":"Basic Text Processing","paragraphs":["Based on the collected reviews, we are ready to build a rough classifier. Before feeding the reviews into a classifier, we applied two common procedures: tokenization and stop words removal for all the reviews. We also applied a common text processing technique to make our reviews more generic. We replaced the roles' and actors/actresses' names in the reviews with some common tokens like role_i, actor_j, where i and j are determined by their importance in this TV series. Therefore, we have the following inference DISPLAYFORM0","where INLINEFORM0 is a function which map a role's or actor's index into its importance. However, in practice, it is not a trivial task to infer the importance of all actors and actresses. We rely on data onBaidu Encyclopedia, which is the Chinese version of Wikipedia. For each movie or TV series, Baidu Encyclopedia has all the required information, which includes the level of importance for each role and actor in the show. Actor/actress in a leading role will be listed at first, followed by the ones in a supporting role and other players. Thus we can build a crawler to collect such information, and replace the corresponding words in reviews with generic tags.","Afterwards, word sequence of each review can be manipulated with tokenization and stop words removal. Each sequence is broken up into a vector of unigram-based tokens using NLPIR BIBREF12 , which is a very powerful tool supporting sentence segmentation in Chinese. Stop words are words that do not contribute to the meaning of the whole sentence and are usually filtered out before following data processing. Since our reviews are collected from online websites which may include lots of forum words, for this particular domain, we include common forum words in addition to the basic Chinese stop words. Shown below are some typical examples in English that are widely used in Chinese forums. INLINEFORM0","These two processes will help us remove significant amount of noise in the data."]},{"idx":9,"heading":"Topic Modelling and Labeling","paragraphs":["With volumes of TV series review data, it's hard for us to define generic categories without looking at them one by one. Therefore, it's necessary to run some unsupervised models to get an overview of what's being talked in the whole corpus. Here we applied Latent Dirichlet Allocation BIBREF13 , BIBREF14 to discover the main topics related to the movies and actors. In a nutshell, the LDA model assumes that there exists a hidden structure consisting of the topics appearing in the whole text corpus. The LDA algorithm uses the co-occurrence of observed words to learn this hidden structure. Mathematically, the model calculates the posterior distribution of the unobserved variables. Given a set of training documents, LDA will return two main outputs. The first is the list of topics represented as a set of words, which presumably contribute to this topic in the form of their weights. The second output is a list of documents with a vector of weight values showing the probability of a document containing a specific topic.","Based on the results from LDA, we carefully defined eight generic categories of movie reviews which are most representative in the dataset as shown in Table TABREF11 .","The purpose of this research is to classify each review into one of the above 8 categories. In order to build reasonable classifiers, first we need to obtain a labeled dataset. Each of the TV series reviews was labeled by at least two individuals, and only those reviews with the same assigned label were selected in our training and testing data. This approach ensures that reviews with human biases are filtered out. As a result, we have 5000 for each TV series that matches the selection criteria."]},{"idx":10,"heading":"Feature Selection","paragraphs":["After the labelled cleaned data has been generated, we are now ready to process the dataset. One problem is that the vocabulary size of our corpus will be quite large. This could result in overfitting with the training data. As the dimension of the feature goes up, the complexity of our model will also increase. Then there will be quite an amount of difference between what we expect to learn and what we will learn from a particular dataset. One common way of dealing with the issue is to do feature selection. Here we applied DRC and INLINEFORM0 mentioned in related work. First let's define a contingency table for each word INLINEFORM1 like in Table TABREF13 . If INLINEFORM2 , it means the appearance of word INLINEFORM3 .","Recall that in classical statistics, INLINEFORM0 is a method designed to measure the independence between two variables or events, which in our case is the word INLINEFORM1 and its relevance to the class INLINEFORM2 . Higher INLINEFORM3 value means higher correlations between them. Therefore, based on the definition of INLINEFORM4 in BIBREF6 and the above Table TABREF13 , we can represent the INLINEFORM5 value as below: DISPLAYFORM0","While for DRC method, it's based on Relevance Correlation Value, whose purpose is to measure the similarity between two distributions, i.e., binary distribution of word INLINEFORM0 's occurrence and documents' relevance to class INLINEFORM1 along all the training data. For a particular word INLINEFORM2 , its occurrence distribution along all the data can be represented as below (assume we have INLINEFORM3 reviews): DISPLAYFORM0","And we also know each review INLINEFORM0 's relevance with respect to INLINEFORM1 using the manually tagged labels. DISPLAYFORM0","where 0 means irrelevant and 1 means relevant. Therefore, we can calculate the similarity between these two vectors as DISPLAYFORM0","where INLINEFORM0 is called the Relevance Correlation Value for word INLINEFORM1 . Because INLINEFORM2 is either 1 or 0, with the notation in the contingency table, RCV can be simplified as DISPLAYFORM0","Then on top of RCV, they incorporate the probability of the presence of word INLINEFORM0 if we are given that the document is relevant. In this way, our final formula for computing DRC becomes DISPLAYFORM0","Therefore, we can apply the above two methods to all the word terms in our dataset and choose words with higher INLINEFORM0 or DRC values to reduce the dimension of our input features."]},{"idx":11,"heading":"Learning Classifiers","paragraphs":["Finally, we are going to train classifiers on top of our reduced generic features. As mentioned above, there are two kinds of learning algorithms, i.e., discriminant and generative classifiers. Based on Bayes rule, the optimal classifier is represented as INLINEFORM0","where INLINEFORM0 is the prior information we know about class INLINEFORM1 .","So for generative approach like Bayes, it will try to estimate both INLINEFORM0 and INLINEFORM1 . During testing time, we can just apply the above Bayes rule to predict INLINEFORM2 . Why do we call it naive? Remember that we assume that each feature is conditionally independent with each other. So we have DISPLAYFORM0","where we made the assumption that there are INLINEFORM0 words being used in our input. If features are binary, for each word INLINEFORM1 we may simply estimate the probability by DISPLAYFORM0","in which, INLINEFORM0 is a smoothing parameter in case there is no training sample for INLINEFORM1 and INLINEFORM2 outputs the number of a set. With all these probabilities computed, we can make decisions by whether DISPLAYFORM0","On the other hand, discriminant learning algorithms will estimate INLINEFORM0 directly, or learn some “discriminant” function INLINEFORM1 . Then by comparing INLINEFORM2 with some threshold, we can make the final decision. Here we applied two common classifiers logistic regression and support vector machine to classify movie reviews. Logistic regression squeezes the input feature into some interval between 0 and 1 by the sigmoid function, which can be treated as the probability INLINEFORM3 . DISPLAYFORM0","The Maximum A Posteriori of logistic regression with Gaussian priors on parameter INLINEFORM0 is defined as below INLINEFORM1","which is a concave function with respect to INLINEFORM0 , so we can use gradient ascent below to optimize the objective function and get the optimal INLINEFORM1 . DISPLAYFORM0","where INLINEFORM0 is a positive hyper parameter called learning rate. Then we can just use equation ( EQREF24 ) to distinguish between classes.","While for Support Vector Machine(SVM), its initial goal is to learn a hyperplane, which will maximize the margin between the two classes' boundary hyperplanes. Suppose the hyperplane we want to learn is INLINEFORM0","Then the soft-margin version of SVM is INLINEFORM0","where INLINEFORM0 is the slack variable representing the error w.r.t. datapoint INLINEFORM1 . If we represent the inequality constraints by hinge loss function DISPLAYFORM0","What we want to minimize becomes DISPLAYFORM0","which can be solved easily with a Quadratic Programming solver. With learned INLINEFORM0 and INLINEFORM1 , decision is made by determining whether DISPLAYFORM0","Based on these classifiers, we may also apply some kernel trick function on input feature to make originally linearly non-separable data to be separable on mapped space, which can further improve our classifier performance. What we've tried in our experiments are the polynomial and rbf kernels."]},{"idx":12,"heading":"Experimental Results and Discussion","paragraphs":["As our final goal is to learn a generic classifier, which is agnostic to TV series but can predict review's category reasonably, we did experiments following our procedures of building the classifier as discussed in section 1."]},{"idx":13,"heading":"Category Determining by LDA","paragraphs":["Before defining the categories of the movie reviews, we should first run some topic modeling method. Here we define categories with the help of LDA. With the number of topics being set as eight, we applied LDA on “The Journey of Flower”, which is the hottest TV series in 2015 summer. As we rely on LDA to guide our category definition, we didn't run it on other TV series. The results are shown in Figure FIGREF30 . Note that the input data here haven't been replaced with the generic tag like role_i or actor_j, as we want to know the specifics being talked by reviewers. Here we present it in the form of heat maps. For lines with brighter color, the corresponding topic is discussed more, compared with others on the same height for each review. As the original texts are in Chinese, the output of LDA are represented in Chinese as well.","We can see that most of the reviews are focused on discussing the roles and analyzing the plots in the movie, i.e., 6th and 7th topics in Figure FIGREF30 , while quite a few are just following the posts, like the 4th and 5th topic in the figure. Based on the findings, we generate the category definition shown in Table TABREF11 . Then 5000 out of each TV series reviews, with no label bias between readers, are selected to make up our final data set."]},{"idx":14,"heading":"Feature Size Comparison","paragraphs":["Based on INLINEFORM0 and DRC discussed in section 3.4, we can sort the importance of each word term. With different feature size, we can train the eight generic classifiers and get their performances on both training and testing set. Here we use SVM as the classifier to compare feature size's influence. Our results suggest that it performs best among the three. The results are shown in Figure FIGREF32 . The red squares represent the training accuracy, while the blue triangles are testing accuracies.","As shown in Figure FIGREF32 , it is easy for us to determine the feature size for each classifier. Also it's obvious that test accuracies of classifiers for plot, actor/actress, analysis, and thumb up or down, didn't increase much with adding more words. Therefore, the top 1000 words with respect to these classes are fixed as the final feature words. While for the rest of classifiers, they achieved top testing performances at the size of about 4000. Based on these findings, we use different feature sizes in our final classifiers."]},{"idx":15,"heading":"Generalization of Classifiers","paragraphs":["To prove the generalization of our classifiers, we use two of the TV series as training data and the rest as testing set. We compare them with classifiers trained without the replacement of generic tags like role_i or actor_j. So 3 sets of experiments are performed, and each are trained on top of Bayes, Logistic Regression and SVM. Average accuracies among them are reported as the performance measure for the sake of space limit. The results are shown in Table TABREF42 . “1”, “2” and “3” represent the TV series “The Journey of Flower”, “Nirvana in Fire” and “Good Time” respectively. In each cell, the left value represents accuracy of classifier without replacement of generic tags and winners are bolded.","From the above table, we can see with substitutions of generic tags in movie reviews, the top 5 classifiers have seen performance increase, which indicates the effectiveness of our method. However for the rest three classifiers, we didn't see an improvement and in some cases the performance seems decreased. This might be due to the fact that in the first five categories, roles' or actors' names are mentioned pretty frequently while the rest classes don't care much about these. But some specific names might be helpful in these categories' classification, so the performance has decreased in some degree."]},{"idx":16,"heading":"Conclusion","paragraphs":["In this paper, a surrogate-based approach is proposed to make TV series review classification more generic among reviews from different TV series. Based on the topic modeling results, we define eight generic categories and manually label the collected TV series' reviews. Then with the help of Baidu Encyclopedia, TV series' specific information like roles' and actors' names are substituted by common tags within TV series domain. Our experimental results showed that such strategy combined with feature selection did improve the performance of classifications. Through this way, one may build classifiers on already collected TV series reviews, and then successfully classify those from new TV series. Our approach has broad implications on processing movie reviews as well. Since movie reviews and TV series reviews share many common characteristics, this approach can be easily applied to understand movie reviews and help movie producers to better process and classify consumers' movie review with higher accuracy."]},{"idx":17,"heading":"Figures and Tables","paragraphs":["Table 1: Chinese Movies Box Office Statistics","Figure 1: Procedure of Building Generic Classifiers","Table 3: Categories of Movie Reviews","Table 4: Contingency Table for Word j","Figure 2: LDA results with 8 topics","Table 5: Performance of 8 Classifiers","Figure 3: Accuracy vs Feature size on 8 classifiers"]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1611.02988","title":"Distant supervision for emotion detection using Facebook reactions","sections":[{"idx":0,"heading":"Abstract","paragraphs":["We exploit the Facebook reaction feature in a distant supervised fashion to train a support vector machine classifier for emotion detection, using several feature combinations and combining different Facebook pages. We test our models on existing benchmarks for emotion detection and show that employing only information that is derived completely automatically, thus without relying on any handcrafted lexicon as it's usually done, we can achieve competitive results. The results also show that there is large room for improvement, especially by gearing the collection of Facebook pages, with a view to the target domain."]},{"idx":1,"heading":"Introduction","paragraphs":["This work is licenced under a Creative Commons Attribution 4.0 International Licence. Licence details: http://creativecommons.org/licenses/by/4.0/","In the spirit of the brevity of social media's messages and reactions, people have got used to express feelings minimally and symbolically, as with hashtags on Twitter and Instagram. On Facebook, people tend to be more wordy, but posts normally receive more simple “likes” than longer comments. Since February 2016, Facebook users can express specific emotions in response to a post thanks to the newly introduced reaction feature (see Section SECREF2 ), so that now a post can be wordlessly marked with an expression of say “joy\" or “surprise\" rather than a generic “like”.","It has been observed that this new feature helps Facebook to know much more about their users and exploit this information for targeted advertising BIBREF0 , but interest in people's opinions and how they feel isn't limited to commercial reasons, as it invests social monitoring, too, including health care and education BIBREF1 . However, emotions and opinions are not always expressed this explicitly, so that there is high interest in developing systems towards their automatic detection. Creating manually annotated datasets large enough to train supervised models is not only costly, but also—especially in the case of opinions and emotions—difficult, due to the intrinsic subjectivity of the task BIBREF2 , BIBREF3 . Therefore, research has focused on unsupervised methods enriched with information derived from lexica, which are manually created BIBREF3 , BIBREF4 . Since go2009twitter have shown that happy and sad emoticons can be successfully used as signals for sentiment labels, distant supervision, i.e. using some reasonably safe signals as proxies for automatically labelling training data BIBREF5 , has been used also for emotion recognition, for example exploiting both emoticons and Twitter hashtags BIBREF6 , but mainly towards creating emotion lexica. mohammad2015using use hashtags, experimenting also with highly fine-grained emotion sets (up to almost 600 emotion labels), to create the large Hashtag Emotion Lexicon. Emoticons are used as proxies also by hallsmarmulti, who use distributed vector representations to find which words are interchangeable with emoticons but also which emoticons are used in a similar context.","We take advantage of distant supervision by using Facebook reactions as proxies for emotion labels, which to the best of our knowledge hasn't been done yet, and we train a set of Support Vector Machine models for emotion recognition. Our models, differently from existing ones, exploit information which is acquired entirely automatically, and achieve competitive or even state-of-the-art results for some of the emotion labels on existing, standard evaluation datasets. For explanatory purposes, related work is discussed further and more in detail when we describe the benchmarks for evaluation (Section SECREF3 ) and when we compare our models to existing ones (Section SECREF5 ). We also explore and discuss how choosing different sets of Facebook pages as training data provides an intrinsic domain-adaptation method."]},{"idx":2,"heading":"Facebook reactions as labels","paragraphs":["For years, on Facebook people could leave comments to posts, and also “like” them, by using a thumbs-up feature to explicitly express a generic, rather underspecified, approval. A “like” could thus mean “I like what you said\", but also “I like that you bring up such topic (though I find the content of the article you linked annoying)\".","In February 2016, after a short trial, Facebook made a more explicit reaction feature available world-wide. Rather than allowing for the underspecified “like” as the only wordless response to a post, a set of six more specific reactions was introduced, as shown in Figure FIGREF1 : Like, Love, Haha, Wow, Sad and Angry. We use such reactions as proxies for emotion labels associated to posts.","We collected Facebook posts and their corresponding reactions from public pages using the Facebook API, which we accessed via the Facebook-sdk python library. We chose different pages (and therefore domains and stances), aiming at a balanced and varied dataset, but we did so mainly based on intuition (see Section SECREF4 ) and with an eye to the nature of the datasets available for evaluation (see Section SECREF5 ). The choice of which pages to select posts from is far from trivial, and we believe this is actually an interesting aspect of our approach, as by using different Facebook pages one can intrinsically tackle the domain-adaptation problem (See Section SECREF6 for further discussion on this). The final collection of Facebook pages for the experiments described in this paper is as follows: FoxNews, CNN, ESPN, New York Times, Time magazine, Huffington Post Weird News, The Guardian, Cartoon Network, Cooking Light, Home Cooking Adventure, Justin Bieber, Nickelodeon, Spongebob, Disney.","Note that thankful was only available during specific time spans related to certain events, as Mother's Day in May 2016.","For each page, we downloaded the latest 1000 posts, or the maximum available if there are fewer, from February 2016, retrieving the counts of reactions for each post. The output is a JSON file containing a list of dictionaries with a timestamp, the post and a reaction vector with frequency values, which indicate how many users used that reaction in response to the post (Figure FIGREF3 ). The resulting emotion vectors must then be turned into an emotion label.","In the context of this experiment, we made the simple decision of associating to each post the emotion with the highest count, ignoring like as it is the default and most generic reaction people tend to use. Therefore, for example, to the first post in Figure FIGREF3 , we would associate the label sad, as it has the highest score (284) among the meaningful emotions we consider, though it also has non-zero scores for other emotions. At this stage, we didn't perform any other entropy-based selection of posts, to be investigated in future work."]},{"idx":3,"heading":"Emotion datasets","paragraphs":["Three datasets annotated with emotions are commonly used for the development and evaluation of emotion detection systems, namely the Affective Text dataset, the Fairy Tales dataset, and the ISEAR dataset. In order to compare our performance to state-of-the-art results, we have used them as well. In this Section, in addition to a description of each dataset, we provide an overview of the emotions used, their distribution, and how we mapped them to those we obtained from Facebook posts in Section SECREF7 . A summary is provided in Table TABREF8 , which also shows, in the bottom row, what role each dataset has in our experiments: apart from the development portion of the Affective Text, which we used to develop our models (Section SECREF4 ), all three have been used as benchmarks for our evaluation."]},{"idx":4,"heading":"Affective Text dataset","paragraphs":["Task 14 at SemEval 2007 BIBREF7 was concerned with the classification of emotions and valence in news headlines. The headlines where collected from several news websites including Google news, The New York Times, BBC News and CNN. The used emotion labels were Anger, Disgust, Fear, Joy, Sadness, Surprise, in line with the six basic emotions of Ekman's standard model BIBREF8 . Valence was to be determined as positive or negative. Classification of emotion and valence were treated as separate tasks. Emotion labels were not considered as mututally exclusive, and each emotion was assigned a score from 0 to 100. Training/developing data amounted to 250 annotated headlines (Affective development), while systems were evaluated on another 1000 (Affective test). Evaluation was done using two different methods: a fine-grained evaluation using Pearson's r to measure the correlation between the system scores and the gold standard; and a coarse-grained method where each emotion score was converted to a binary label, and precision, recall, and f-score were computed to assess performance. As it is done in most works that use this dataset BIBREF3 , BIBREF4 , BIBREF9 , we also treat this as a classification problem (coarse-grained). This dataset has been extensively used for the evaluation of various unsupervised methods BIBREF2 , but also for testing different supervised learning techniques and feature portability BIBREF10 ."]},{"idx":5,"heading":"Fairy Tales dataset","paragraphs":["This is a dataset collected by alm2008affect, where about 1,000 sentences from fairy tales (by B. Potter, H.C. Andersen and Grimm) were annotated with the same six emotions of the Affective Text dataset, though with different names: Angry, Disgusted, Fearful, Happy, Sad, and Surprised. In most works that use this dataset BIBREF3 , BIBREF4 , BIBREF9 , only sentences where all annotators agreed are used, and the labels angry and disgusted are merged. We adopt the same choices."]},{"idx":6,"heading":"ISEAR","paragraphs":["The ISEAR (International Survey on Emotion Antecedents and Reactions BIBREF11 , BIBREF12 ) is a dataset created in the context of a psychology project of the 1990s, by collecting questionnaires answered by people with different cultural backgrounds. The main aim of this project was to gather insights in cross-cultural aspects of emotional reactions. Student respondents, both psychologists and non-psychologists, were asked to report situations in which they had experienced all of seven major emotions (joy, fear, anger, sadness, disgust, shame and guilt). In each case, the questions covered the way they had appraised a given situation and how they reacted. The final dataset contains reports by approximately 3000 respondents from all over the world, for a total of 7665 sentences labelled with an emotion, making this the largest dataset out of the three we use."]},{"idx":7,"heading":"Overview of datasets and emotions","paragraphs":["We summarise datasets and emotion distribution from two viewpoints. First, because there are different sets of emotions labels in the datasets and Facebook data, we need to provide a mapping and derive a subset of emotions that we are going to use for the experiments. This is shown in Table TABREF8 , where in the “Mapped” column we report the final emotions we use in this paper: anger, joy, sadness, surprise. All labels in each dataset are mapped to these final emotions, which are therefore the labels we use for training and testing our models.","Second, the distribution of the emotions for each dataset is different, as can be seen in Figure FIGREF9 .","In Figure FIGREF9 we also provide the distribution of the emotions anger, joy, sadness, surprise per Facebook page, in terms of number of posts (recall that we assign to a post the label corresponding to the majority emotion associated to it, see Section SECREF2 ). We can observe that for example pages about news tend to have more sadness and anger posts, while pages about cooking and tv-shows have a high percentage of joy posts. We will use this information to find the best set of pages for a given target domain (see Section SECREF5 )."]},{"idx":8,"heading":"Model","paragraphs":["There are two main decisions to be taken in developing our model: (i) which Facebook pages to select as training data, and (ii) which features to use to train the model, which we discuss below. Specifically, we first set on a subset of pages and then experiment with features. Further exploration of the interaction between choice of pages and choice of features is left to future work, and partly discussed in Section SECREF6 . For development, we use a small portion of the Affective data set described in Section SECREF4 , that is the portion that had been released as development set for SemEval's 2007 Task 14 BIBREF7 , which contains 250 annotated sentences (Affective development, Section SECREF4 ). All results reported in this section are on this dataset. The test set of Task 14 as well as the other two datasets described in Section SECREF3 will be used to evaluate the final models (Section SECREF4 )."]},{"idx":9,"heading":"Selecting Facebook pages","paragraphs":["Although page selection is a crucial ingredient of this approach, which we believe calls for further and deeper, dedicated investigation, for the experiments described here we took a rather simple approach. First, we selected the pages that would provide training data based on intuition and availability, then chose different combinations according to results of a basic model run on development data, and eventually tested feature combinations, still on the development set.","For the sake of simplicity and transparency, we first trained an SVM with a simple bag-of-words model and default parameters as per the Scikit-learn implementation BIBREF13 on different combinations of pages. Based on results of the attempted combinations as well as on the distribution of emotions in the development dataset (Figure FIGREF9 ), we selected a best model (B-M), namely the combined set of Time, The Guardian and Disney, which yields the highest results on development data. Time and The Guardian perform well on most emotions but Disney helps to boost the performance for the Joy class."]},{"idx":10,"heading":"Features","paragraphs":["In selecting appropriate features, we mainly relied on previous work and intuition. We experimented with different combinations, and all tests were still done on Affective development, using the pages for the best model (B-M) described above as training data. Results are in Table TABREF20 . Future work will further explore the simultaneous selection of features and page combinations.","We use a set of basic text-based features to capture the emotion class. These include a tf-idf bag-of-words feature, word (2-3) and character (2-5) ngrams, and features related to the presence of negation words, and to the usage of punctuation.","This feature is used in all unsupervised models as a source of information, and we mainly include it to assess its contribution, but eventually do not use it in our final model.","We used the NRC10 Lexicon because it performed best in the experiments by BIBREF10 , which is built around the emotions anger, anticipation, disgust, fear, joy, sadness, and surprise, and the valence values positive and negative. For each word in the lexicon, a boolean value indicating presence or absence is associated to each emotion. For a whole sentence, a global score per emotion can be obtained by summing the vectors for all content words of that sentence included in the lexicon, and used as feature.","As additional feature, we also included Word Embeddings, namely distributed representations of words in a vector space, which have been exceptionally successful in boosting performance in a plethora of NLP tasks. We use three different embeddings:","Google embeddings: pre-trained embeddings trained on Google News and obtained with the skip-gram architecture described in BIBREF14 . This model contains 300-dimensional vectors for 3 million words and phrases.","Facebook embeddings: embeddings that we trained on our scraped Facebook pages for a total of 20,000 sentences. Using the gensim library BIBREF15 , we trained the embeddings with the following parameters: window size of 5, learning rate of 0.01 and dimensionality of 100. We filtered out words with frequency lower than 2 occurrences.","Retrofitted embeddings: Retrofitting BIBREF16 has been shown as a simple but efficient way of informing trained embeddings with additional information derived from some lexical resource, rather than including it directly at the training stage, as it's done for example to create sense-aware BIBREF17 or sentiment-aware BIBREF18 embeddings. In this work, we retrofit general embeddings to include information about emotions, so that emotion-similar words can get closer in space. Both the Google as well as our Facebook embeddings were retrofitted with lexical information obtained from the NRC10 Lexicon mentioned above, which provides emotion-similarity for each token. Note that differently from the previous two types of embeddings, the retrofitted ones do rely on handcrafted information in the form of a lexical resource."]},{"idx":11,"heading":"Results on development set","paragraphs":["We report precision, recall, and f-score on the development set. The average f-score is reported as micro-average, to better account for the skewed distribution of the classes as well as in accordance to what is usually reported for this task BIBREF19 .","From Table TABREF20 we draw three main observations. First, a simple tf-idf bag-of-word mode works already very well, to the point that the other textual and lexicon-based features don't seem to contribute to the overall f-score (0.368), although there is a rather substantial variation of scores per class. Second, Google embeddings perform a lot better than Facebook embeddings, and this is likely due to the size of the corpus used for training. Retrofitting doesn't seem to help at all for the Google embeddings, but it does boost the Facebook embeddings, leading to think that with little data, more accurate task-related information is helping, but corpus size matters most. Third, in combination with embeddings, all features work better than just using tf-idf, but removing the Lexicon feature, which is the only one based on hand-crafted resources, yields even better results. Then our best model (B-M) on development data relies entirely on automatically obtained information, both in terms of training data as well as features."]},{"idx":12,"heading":"Results","paragraphs":["In Table TABREF26 we report the results of our model on the three datasets standardly used for the evaluation of emotion classification, which we have described in Section SECREF3 .","Our B-M model relies on subsets of Facebook pages for training, which were chosen according to their performance on the development set as well as on the observation of emotions distribution on different pages and in the different datasets, as described in Section SECREF4 . The feature set we use is our best on the development set, namely all the features plus Google-based embeddings, but excluding the lexicon. This makes our approach completely independent of any manual annotation or handcrafted resource. Our model's performance is compared to the following systems, for which results are reported in the referred literature. Please note that no other existing model was re-implemented, and results are those reported in the respective papers."]},{"idx":13,"heading":"Discussion, conclusions and future work","paragraphs":["We have explored the potential of using Facebook reactions in a distant supervised setting to perform emotion classification. The evaluation on standard benchmarks shows that models trained as such, especially when enhanced with continuous vector representations, can achieve competitive results without relying on any handcrafted resource. An interesting aspect of our approach is the view to domain adaptation via the selection of Facebook pages to be used as training data.","We believe that this approach has a lot of potential, and we see the following directions for improvement. Feature-wise, we want to train emotion-aware embeddings, in the vein of work by tang:14, and iacobacci2015sensembed. Retrofitting FB-embeddings trained on a larger corpus might also be successful, but would rely on an external lexicon.","The largest room for yielding not only better results but also interesting insights on extensions of this approach lies in the choice of training instances, both in terms of Facebook pages to get posts from, as well as in which posts to select from the given pages. For the latter, one could for example only select posts that have a certain length, ignore posts that are only quotes or captions to images, or expand posts by including content from linked html pages, which might provide larger and better contexts BIBREF23 . Additionally, and most importantly, one could use an entropy-based measure to select only posts that have a strong emotion rather than just considering the majority emotion as training label. For the former, namely the choice of Facebook pages, which we believe deserves the most investigation, one could explore several avenues, especially in relation to stance-based issues BIBREF24 . In our dataset, for example, a post about Chile beating Colombia in a football match during the Copa America had very contradictory reactions, depending on which side readers would cheer for. Similarly, the very same political event, for example, would get very different reactions from readers if it was posted on Fox News or The Late Night Show, as the target audience is likely to feel very differently about the same issue. This also brings up theoretical issues related more generally to the definition of the emotion detection task, as it's strongly dependent on personal traits of the audience. Also, in this work, pages initially selected on availability and intuition were further grouped into sets to make training data according to performance on development data, and label distribution. Another criterion to be exploited would be vocabulary overlap between the pages and the datasets.","Lastly, we could develop single models for each emotion, treating the problem as a multi-label task. This would even better reflect the ambiguity and subjectivity intrinsic to assigning emotions to text, where content could be at same time joyful or sad, depending on the reader."]},{"idx":14,"heading":"Acknowledgements","paragraphs":["In addition to the anonymous reviewers, we want to thank Lucia Passaro and Barbara Plank for insightful discussions, and for providing comments on draft versions of this paper."]},{"idx":15,"heading":"Figures and Tables","paragraphs":["Table 1: Emotion labels in existing datasets, Facebook, and resulting mapping for the experiments in this work. The last row indicates which role each dataset has in our experiments.","Figure 3: Emotion distribution in the datasets Figure 4: Emotion distribution per Facebook page","Table 2: Results on the development set (Affective development). avg f is the micro-averaged f-score.","Table 3: Results on test datasets according to Precision, Recall and F-score."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1701.03578","title":"Efficient Transfer Learning Schemes for Personalized Language Modeling using Recurrent Neural Network","sections":[{"idx":0,"heading":"Abstract","paragraphs":["In this paper, we propose an efficient transfer leaning methods for training a personalized language model using a recurrent neural network with long short-term memory architecture. With our proposed fast transfer learning schemes, a general language model is updated to a personalized language model with a small amount of user data and a limited computing resource. These methods are especially useful for a mobile device environment while the data is prevented from transferring out of the device for privacy purposes. Through experiments on dialogue data in a drama, it is verified that our transfer learning methods have successfully generated the personalized language model, whose output is more similar to the personal language style in both qualitative and quantitative aspects."]},{"idx":1,"heading":"Introduction","paragraphs":["Recently there has been a considerable interest in language modeling due to various academic and commercial demands. Academically, many studies have investigated this domain such as machine translation, chat-bot, message generation, image tagging and other language-related areas. Commercially, it can be used as a core technology for providing a new application on consumer products or services. For instance, an automatic message-reply prediction service can be launched in mobile devices, thus helping a user to send a reply message when he/she is not provided with a proper input interface.","To model the language of human dialogue, a recurrent neural network (RNN) structure is known to show the state of the arts performance with its ability to learn a sequential pattern of the data BIBREF0 . Among the RNN structures, a Long Short-Term Memory RNN (LSTM-RNN) and its variants are successfully used for language modeling tasks BIBREF1 , BIBREF2 . However, as a kind of deep learning technique, the LSTM-RNN and the RNN structure requires both a large number of data and huge computing power to train the model properly. Hence any attempts for applying the RNN structure to personalized language modeling are mainly constrained by the following two limitations. First, personal mobile devices contain private message data among close acquaintances, so users seldom agree to transfer their log out of the devices. This causes a limitation of gathering the whole user data to common computing spaces, where high-performance machines are available. Second, in relatively small computing machines, i.e., smart phone, it is not always-guaranteed to have enough resources to train a deep model within the devices.","To resolve these limitations, we propose fast transfer learning schemes. It trains a base model with a large dataset and copies its first n-many layers to the first n-many layers of a target model. Then the target model is fine-tuned with relatively small target data. Several learning schemes such as freezing a certain layer or adding a surplus layer are proposed for achieving the result. In experiments, we trained a general language model with huge corpus such as an Workshop on Statistical Machine Translation (WMT) data and a movie script data by using powerful computing machines, and then transferred the model to target environment for updating to be a personalized language model. With this approach, the final model can mimic target user's language style with proper syntax.","In the experiments, we trained the general language model with literary-style data and applied the transfer learning with spoken-style data. Then we evaluated the model output for sentence completion task in a qualitative and a quantitative manner. The test result showed that the model learned the style of the target language properly. Another test was conducted by training the general language model with the script of the drama, “Friends,\" and by applying transfer learning with main character corpora from the script to generate the personalized language model. The message-reply prediction task was evaluated with this model. The test result shows higher similarity between the output of the personalized language model and the same user dialogue than the one between the output of the personalized language model and other users' dialogues.","The contributions of this paper are as follows. First, we propose efficient transfer learning schemes for personalized language modeling, which is the first research on transfer learning for RNN based language models with privacy preserving. Second, we show the applicability of our research to the target scenario in the short message reply application by training the model in the similar environment to that of the mobile device, and highlight its test results."]},{"idx":2,"heading":"Architecture for Personalized Language Model","paragraphs":["As we are focusing on a personalized language modeling with the preservation of user data, we generate two types of language models. First is a sentence completion language model, which can complete sentences with a given n-many sequence of words. Second is a message-reply prediction language model, which can generate a response sentence for a given message. The output of both models implies user characteristics such as preferable vocabulary, sentence length, and other language-related patterns.","To achieve this result, we trained the language model with a large amount of general data in powerful computing environments, and then applied the transfer learning in relatively small computing environments. We assume that this method would be applied to mobile devices. As we are taking the preservation of privacy into consideration, the transferred model is retrained within the local environments such as mobile devices, and no personal data is sent out of the devices. This could have been accomplished using the proposed transfer learning schemes in RNN-LSTM architecture."]},{"idx":3,"heading":"Sentence Completion Language Model","paragraphs":["A sentence completion model completes a sentence with the given word sequence $X= \\lbrace x_1,x_2, \\dots , x_T\\rbrace $ , where $x_N$ is a word ( $N=1, 2, \\dots , T$ ). The model can predict the next word $x_{N+1}$ with given word sequence $x_{1:N}$ . By repeating the prediction until the output word reaches the end-of-sentence signal, “ $<eos>$ ,\" the whole sentence can be generated.","The model is similar to that of BIBREF3 , and we put the 1,000-dimension word-embedding layer right after the input layer. Then 3 deep LSTM layers with 100 LSTM cells each and without peephole connection are used for learning the sequential pattern of the sentences.","The output probability to the input sequence $X$ and the training objective are","$$\\begin{aligned} & p(Y|X)=\\prod _{t=1}^{T}p(y_t|x_{1:t-1}) \\\\ & \\textit {L}= -\\dfrac{1}{|T|}\\sum \\limits _{t=1}^{T} x_{t+1}\\log p(y_t|x_{1:t-1}), \\end{aligned}$$ (Eq. 3)","where $X$ is a word sequence in the sentence, $Y$ is a model output sequence $Y=\\lbrace y_1,y_2, \\dots , y_{T}\\rbrace $"]},{"idx":4,"heading":"Message-Reply Prediction Language Model","paragraphs":["A message-reply prediction model generates a response sentence for a given message. It is similar to the sentence completion language model except that the message sentence is encoded and used as a context information when the model generates a response word sequence. Our approach is inspired by the sequence-to-sequence learning research BIBREF0 that is successfully applied to a machine translation task. The message word sequence $X=\\lbrace x_1, x_2, \\dots , x_T\\rbrace $ is fed into the model, and the last hidden state is used as context information $c_T$ . With this context information, the next sequence word is predicted similarly to that in the sentence completion language model case. During implementation, we used 1,000-dimension word embedding and 3-deep LSTM layers with 100 LSTM cells in each layer. The output probability and the training objective are","$$\\begin{aligned} & p(Y|X)=\\prod _{t=1}^{T^{\\prime }}p(y_t|c_T,y_{1:t-1})\\\\ & L = -\\dfrac{1}{|T^{\\prime }|}|\\sum \\limits _{t=1}^{T^{\\prime }} z_t\\log p(y_t|c_T, y_{1:t-1}), \\end{aligned}$$ (Eq. 5)","where $X$ is a word sequence in the message sentence, $Z$ is a target word sequence in the response sentence $Z = \\lbrace z_1,z_2, \\dots , z_{T^{\\prime }}\\rbrace $ , $Y$ is a model output sequence $Y=\\lbrace y_1,y_2, \\dots , y_{T^{\\prime }}\\rbrace $ , $c_T$ is the encoding vector for the message sentence."]},{"idx":5,"heading":"Fast Transfer Learning Schemes","paragraphs":["To generate a personalized language model with a small amount of user data and limited computing resources, transfer learning is essential. In the private data preservation scenario, we investigate three fast transfer learning schemes. Each scheme is described below:","Scheme 1, relearn the whole layer: As a baseline, we retrain the whole model with private data only and compare the result with the two other schemes below. Because of the retraining of the LSTM layers in their entirety, this scheme requires more computing power than the other two schemes.","Scheme 2, surplus layer: After the training of the model with general data, a surplus layer is inserted between the output layer and the last of the deep LSTM layers. Then, with private data, we update only the parameters of the surplus layer in the transfer learning phase. We assume that a user's parlance could be modeled by learning additional features in the user's private data.","Scheme 3, fixed first n layers: After training the model with general data, we fix the parameters in the first n LSTM layers (layer 1 and layer 2 in our experiments) and train remaining parameters in the transfer learning phase. We assume that the user's parlance is a subset of the general pattern and the last layer plays the key role in determining this pattern."]},{"idx":6,"heading":"Measures","paragraphs":["The perplexity is one of the popular measures for a language model. It measures how well the language model predicts a sample. However, it is not good at measuring how well the output of the language model matches a target language style. Another measure, the BLEU score algorithm BIBREF4 , has been widely used for the automatic evaluation of the model output. However, it cannot be applied directly to measuring a quality of the personalized model output because it considers the similarity between one language and the target language. Other research was conducted on proving authorship and fraud in literature, for instance, Jane Austen's left-over novel with partially completed BIBREF5 . This research counted the occurrence of several words in the literature, compared their relative frequencies with those of the words in the target literature, and concluded that the target literature was a forgery. This approach could be applied to a text evaluation where a large amount of data is available and certain words are used more frequently. In spoken language, such as in the message-reply case, however, whole word distribution must be considered instead of considering the occurrence of several words, because the data is usually not enough than the literature case. So, we use a simple and efficient metric to measure the similarity between the user style and the output of the personalized model.","An output of a personalized language model can be measured by calculating the cross entropy between the word distribution of the model output and that of the target data. Word distribution can be acquired by normalizing a word histogram which is calculated based on word counts in the target corpus. Equation (3) shows the metric formulation.","$$\\begin{aligned} & Y_1=g( f_{LM}( M_i ) ), Y_2=g( T_i ) \\\\ & measure = Cross~Entropy(Y_1, Y_2), \\\\ \\end{aligned}$$ (Eq. 11)","where $M_i$ is a message $\\in {D_{test}}$ , $T_i$ is a corpus $\\in {D_{target}}$ , $f_{LM}$ is a language model, $g(\\cdot )$ calculates word distribution with given corpus, CrossEntropy(p, q) is $- \\sum _{x} p(x) \\log q(x)$ .","The characteristics of a user speech can mainly be distinguished by the word dictionary. Thus, this metric tries to measure the differences of the word dictionary among the comparing set. Table 1 shows the quantitative measure results from the dialogue set of the main characters in drama data from “Friends,\" a famous American television sitcom. In the figures, “character_1\" to “character_6\" are the main characters of the drama (Chandler, Joey, Monica, Phoebe, Rachel, and Ross, respectively). The dialogues were measured against one another by using the cross entropy metric. As shown in the table, the lower cross entropy value among the same character's dialogue was calculated, and the higher value was calculated among the different character's dialogues as expected. This result demonstrates that the cross entropy metric can be used to measure the similarities among the members of the set."]},{"idx":7,"heading":"Experiments","paragraphs":["We mainly conduct two types of experiments. The first one is a sentence completion experiment, and the other one is a message-reply prediction experiment. In the former case, we train a general language model with literary-style data and apply a proposed transfer learning scheme with spoken-style data to achieve a personalized language model. With this setting, the difference between general and personalized language models can be measured in a quantitative and a qualitative manner. For the latter case, we use dialogue-style data such as drama scripts to train a general language model. From the drama scripts, some characters' data are taken apart and are used to train the personalized language model. With this setting, the output of the personalized model is compared to the original dialogue of the same character."]},{"idx":8,"heading":"Literary-Style to Spoken-Style Sentence Completion","paragraphs":["We train a general language model of literary-style with the WMT'14 corpus. We then apply a transfer learning scheme with “Friends\" drama data for the model to learn the spoken-style language. Training the general language model took about 10 days then we spent another 4 hours training the personalized language model in each scheme. A “titan-X GPU\" and a “GeForce GT 730 GPU\" were used for these experiments. The latter GPU is one of the low-end GPU series of which computing power was similar to that of latest mobile GPUs such as “Qualcomm Adreno 530\" in “Samsung Galaxy S7\" or “NVIDIA Tegra K1\" in “Google Nexus 9\". For a vocabulary setting, we construct our dictionary as 50,002 words, including “ $<eos>$ \" to mark ends of sentence and “**unknown**\" to replace unconsidered vocabulary in the data. The out-of-vocabulary rate is about 3.5%.","The “general language model\" in Table 2 shows the sample output of the general language model trained with document-style data, and the “personal language model 1\" in Table 2 shows the sample output of the personalized language model trained with human-dialogue-style data. Scheme_1 to scheme_3 are relearn-whole, surplus layer, and fixed-n layer, respectively. Given input word sequence for the test was, “It is possible, however.\" As can be seen in the table, both outputs differ in length and style. The sentence completed using the general language model tends to be longer than that of obtained using the personalized language model. This result indicates that the personalized language model is properly trained with the spoken language characteristics because human dialogue is usually briefer than the language in official documents.","We also apply the transfer learning schemes with some of the English bible data. The same general language model, which involved previously training with the WMT'14 corpus for 10 days, is used. English bible data is added and employed in training for another 4 hours using proposed transfer learning schemes.","The “personalized language model 2\" in Table 2 shows the sample output of the personalized language model trained with another style of document data, English bible data. As shown in Table 2, the output of the personalized language model contains more bible-like vocabulary and sentence styles."]},{"idx":9,"heading":"General-Style to Personal-Style Message-Reply Prediction","paragraphs":["We simulate the message-reply prediction scenario using the drama corpus. The script of the drama, “Friends,\" is used to train a general language model, and two main character corpora are used to generate a personalized language model. For this message-reply prediction experiment, we use a vocabulary size of 18,107, and the out-of-vocabulary rate is about 3.5%. In the message-reply prediction case, pairwise data is generated by extracting the drama corpus of each character and concatenating two consecutive sentences of different characters to form one single message-reply sentence data. We insert the word “ $<eos>$ \" between the message and reply to mark the border separating them. This pairwise data is used for the training, and only the message part of the pairwise data is used for the message-reply prediction. During implementation, it took about a day to train the general language model with the “Friends\" corpus and another 4 hours to train the personalized language model with two main character corpora. The “titan-X GPU\" and the “GeForce GT 730 GPU\" was used for these experiments. Validation messages-reply sentences of 1,281 are randomly sampled from the “Friends\" corpus for tracking validation curve and another 753 test messages are prepared for predicting the responses. These data remained unseen from training phase. The word distributions of the model output from the test messages and the target corpus data are calculated to measure their similarity.","Figure 1 shows the validation curve while training. Perplexity values from various model output are plotted. The perplexity of baseline model, “scheme_1\", decreases until around epoch 10, and then it starts to increase because model is over-fitted to training data. The proposed “scheme_2\" and “scheme_3\", however, show continuous decreasing tendency and reach lower perplexity values compared to that of the baseline model. It is interesting that proposed methods achieve lower perplexity than baseline while saving computing power with reduced parameters.","Table 3 shows the performances of various models measured with the same validation dataset used in Figure 1. An unpruned n-gram language models using modified Kneser-Ney smoothing are used for performance comparisons BIBREF7 . The n-gram models were trained by using KenLM software package BIBREF8 . The chandler n-gram model was trained with “Chandler” corpus and the friends n-gram model was trained with “Friends” corpus. The proposed scheme_1 to scheme_3 were trained with “Chandler” corpus from “Friends” general language model. We see that our proposed schemes outperform the n-gram models (n=3 and 5).","To check the influence of training data size (number of sentences) in personalized language model, we trained the general language model (trained with “Friends\" corpus, message-reply prediction model) with different sizes of personal (“chandler\" and “rachel\") dataset. The proposed scheme_2 method was used for this test. Table 4 shows evaluation results of the trained models. Dataset '0' means the model is not trained with personal dataset. The perplexity shows lower value as we use more dataset in training, and it outperforms “friends 5-gram” model from the 2,000 dataset cases.","Table 5 indicates the cross entropy measure between the output of “scheme_1\" to “scheme_3\" model and that of the target corpus, the “friends\" drama corpus, the “chandler\" corpus, and the “bible\" corpus. It shows the similarity between the personalized model output and the target corpus as the number of training epoch increasing. The general model was pre-trained with the “Friends” corpus and the “Chandler” corpus was used training personalized model. Each Model is selected from various training epoch (0, 10, 20 and 40) and schemes, and test messages of 753 are used for the reply generation with the selected model used. As the table shows, the cross entropy measure has the highest value when the target corpus is the “bible” as expected because it is written in different style than dialogues in drama script. For the drama script case, the cross entropy measured with the “chandler\" corpus shows the lowest value among schemes. This result reveals that the personalized language model is trained properly from the general language model. Thus it is more similar in style to the target data corpus than the general language model. The “epoch 0\" case means the initial model state trained from general language corpus, “friends\" corpus. Thus cross entropy with “friends\" target corpus shows lower value than that of “chandler\" and “bible\" target corpus cases."]},{"idx":10,"heading":"Related Work","paragraphs":["Researchers have proposed language models using RNN, which learns the probability of next sequence data at the character or word level BIBREF9 , BIBREF3 . The proposed language models were tested on web corpora (i.e. Wikipedia, news articles) and qualitative examples showed their applicability. BIBREF0 proposed a sequence-to-sequence learning algorithm with RNN and long short-term memory (LSTM) architecture BIBREF1 , and BIBREF2 proposed RNN encoder-decoder architecture. Those studies were applied to the machine translation problem.","Recently, the RNN machine translation approach was extended to the short message generation problem BIBREF10 . Considering the message and response as a translation problem, the Neural Responding Machine achieved 40% accuracy for both contextually and syntactically proper response generations with twitter-like micro-blogging data BIBREF11 . Those studies were similar to our research in the sense that both target message-reply prediction language model using RNN. Our research, however, differs in that it updates a general language model to a personalized language model with user data separately, whereas the previous research trained a language model with the data, as a whole, in same place.","In the commercial sphere, Google recently released a smart-reply service that could generate a response to a given email by using a sequence-to-sequence learning model BIBREF12 . There was another trial on the generation of responses in technical troubleshooting discourses BIBREF13 . This research also required complete data in one place and did not provide a personalized model.","Moreover, many researchers have conducted studies on transfer learning. BIBREF14 , BIBREF15 suggested that a base-trained model with general data could be transferred to another domain. Recently, BIBREF16 showed, through experiments, that the lower layers tended to have general features whereas the higher layer tended to have specific features. However, none of this research was applied to an RNN language model.","To adapt a neural network model to an embedded system with limited resources, BIBREF17 BIBREF18 reduced the size of the model by pruning the unnecessary connections within it. It repeatedly tried to reduce the model size without accuracy degradation. This research inspired us to a considerable extent. It applied a neural model to mobile devices. However, the research focused on reducing the model size using a powerful machine and releasing the final model to an embedded system, whereas ours investigated how to train a model within mobile devices so that private user data could be kept."]},{"idx":11,"heading":"Conclusion","paragraphs":["We propose an efficient method for training a personalized model using the LSTM-RNN model. To preserve users' privacy, we suggest various transfer learning schemes so that the personalized language model can be generated within the user's local environment. The proposed schemes “surplus layer' and “fixed-n layer' shows higher generalization performance whereas it trains only reduced number of parameters than baseline model. The quantitative and qualitative test result indicate that the output of the model is similar to that of the user's style.","It is certain that our proposed method reveals the applicability of the RNN-based language model in a user device with the preservation of privacy. Furthermore, with our method the personalized language model can be generated with a smaller amount of user data than the huge amount of training data that is usually required in the traditional deep neural network discipline. In the future work, we aim to visualize the deep neural network and to investigate the specific relationship among users' language styles and the LSTM cells in the network. This approach seems likely to uncover enhanced learning schemes that require less data than was previously necessary."]},{"idx":12,"heading":"Figures and Tables","paragraphs":["Table 1: Quantitative measure result of dialogues among main characters. Character 1 to character 6 are Chandler, Joey, Monica, Phoebe, Rachel, and Ross, respectively. A lower value indicates that the two sets compared have similar distributions and are, thus, similar in style.","Table 2: Sample model output of general language model and personalized language model. The general language model used WMT’14 data, personalized language model 1 used “Friends” drama data, and personalized language model 2 used the English bible data. Scheme 1 to scheme 3 are relearn whole, surplus layer and fixed-n layer, respectively. The output was generated with the given input sequence, “It is possible, however”","Table 3: Performances of models measured with the same validation dataset used in Figure 1. The chandler n-gram model was trained with “Chandler” corpus and the friends n-gram model was trained with “Friends” corpus. The scheme 1 model is over-fitted to training data (see Figure 1), and the lowest value is 48.17.","Figure 1: Validation curve for each schemes. Scheme 1 is re-learn whole, scheme 2 is surplus layer and scheme 3 is fixed-n layer (train 3rd layer only).","Table 4: Performances of models with different number of sentences in training dataset (lower is better). “Friends” corpus was used pre-training the general model, and “Chandler” and “Rachel” corpus was used training the personalized model with the proposed scheme 2 method. Dataset ’0’ means the model is not trained with personal dataset.","Table 5: Cross entropy measure between the language model output and the training data corpus, the “Friends” drama corpus, the“Chandler” corpus and the “Bible” corpus. Scheme 1 to scheme 3 are relearn whole, surplus layer and fixed-n layer, respectively. The “epoch 0” case means the initial model state trained from general language corpus, “friends” corpus. Thus cross entropy with “friends” target corpus shows lower value than that of “chandler” and “bible” target corpus cases. The lower value indicates that the language model output is similar in style to the compared target corpus"]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1702.02367","title":"Iterative Multi-document Neural Attention for Multiple Answer Prediction","sections":[{"idx":0,"heading":"Abstract","paragraphs":["People have information needs of varying complexity, which can be solved by an intelligent agent able to answer questions formulated in a proper way, eventually considering user context and preferences. In a scenario in which the user profile can be considered as a question, intelligent agents able to answer questions can be used to find the most relevant answers for a given user. In this work we propose a novel model based on Artificial Neural Networks to answer questions with multiple answers by exploiting multiple facts retrieved from a knowledge base. The model is evaluated on the factoid Question Answering and top-n recommendation tasks of the bAbI Movie Dialog dataset. After assessing the performance of the model on both tasks, we try to define the long-term goal of a conversational recommender system able to interact using natural language and to support users in their information seeking processes in a personalized way."]},{"idx":1,"heading":"Motivation and Background","paragraphs":["We are surrounded by a huge variety of technological artifacts which “live” with us today. These artifacts can help us in several ways because they have the power to accomplish complex and time-consuming tasks. Unfortunately, common software systems can do for us only specific types of tasks, in a strictly algorithmic way which is pre-defined by the software designer. Machine Learning (ML), a branch of Artificial Intelligence (AI), gives machines the ability to learn to complete tasks without being explicitly programmed.","People have information needs of varying complexity, ranging from simple questions about common facts which can be found in encyclopedias, to more sophisticated cases in which they need to know what movie to watch during a romantic evening. These tasks can be solved by an intelligent agent able to answer questions formulated in a proper way, eventually considering user context and preferences.","Question Answering (QA) emerged in the last decade as one of the most promising fields in AI, since it allows to design intelligent systems which are able to give correct answers to user questions expressed in natural language. Whereas, recommender systems produce individualized recommendations as output and have the effect of guiding the user in a personalized way to interesting or useful objects in a large space of possible options. In a scenario in which the user profile (the set of user preferences) can be represented by a question, intelligent agents able to answer questions can be used to find the most appealing items for a given user, which is the classical task that recommender systems can solve. Despite the efficacy of classical recommender systems, generally they are not able to handle a conversation with the user so they miss the possibility of understanding his contextual information, emotions and feedback to refine the user profile and provide enhanced suggestions. Conversational recommender systems assist online users in their information-seeking and decision making tasks by supporting an interactive process BIBREF0 which could be goal oriented with the task of starting general and, through a series of interaction cycles, narrowing down the user interests until the desired item is obtained BIBREF1 .","In this work we propose a novel model based on Artificial Neural Networks to answer questions exploiting multiple facts retrieved from a knowledge base and evaluate it on a QA task. Moreover, the effectiveness of the model is evaluated on the top-n recommendation task, where the aim of the system is to produce a list of suggestions ranked according to the user preferences. After having assessed the performance of the model on both tasks, we try to define the long-term goal of a conversational recommender system able to interact with the user using natural language and to support him in the information seeking process in a personalized way.","In order to fulfill our long-term goal of building a conversational recommender system we need to assess the performance of our model on specific tasks involved in this scenario. A recent work which goes in this direction is reported in BIBREF2 , which presents the bAbI Movie Dialog dataset, composed by different tasks such as factoid QA, top-n recommendation and two more complex tasks, one which mixes QA and recommendation and one which contains turns of dialogs taken from Reddit. Having more specific tasks like QA and recommendation, and a more complex one which mixes both tasks gives us the possibility to evaluate our model on different levels of granularity. Moreover, the subdivision in turns of the more complex task provides a proper benchmark of the model capability to handle an effective dialog with the user.","For the task related to QA, a lot of datasets have been released in order to assess the machine reading and comprehension capabilities and a lot of neural network-based models have been proposed. Our model takes inspiration from BIBREF3 , which is able to answer Cloze-style BIBREF4 questions repeating an attention mechanism over the query and the documents multiple times. Despite the effectiveness on the Cloze-style task, the original model does not consider multiple documents as a source of information to answer questions, which is fundamental in order to extract the answer from different relevant facts. The restricted assumption that the answer is contained in the given document does not allow the model to provide an answer which does not belong to the document. Moreover, this kind of task does not expect multiple answers for a given question, which is important for the complex information needs required for a conversational recommender system.","According to our vision, the main outcomes of our work can be considered as building blocks for a conversational recommender system and can be summarized as follows:","The paper is organized as follows: Section SECREF2 describes our model, while Section SECREF3 summarizes the evaluation of the model on the two above-mentioned tasks and the comparison with respect to state-of-the-art approaches. Section SECREF4 gives an overview of the literature of both QA and recommender systems, while final remarks and our long-term vision are reported in Section SECREF5 ."]},{"idx":2,"heading":"Methodology","paragraphs":["Given a query INLINEFORM0 , an operator INLINEFORM1 that produces the set of documents relevant for INLINEFORM2 , where INLINEFORM3 is the set of all queries and INLINEFORM4 is the set of all documents. Our model defines a workflow in which a sequence of inference steps are performed in order to extract relevant information from INLINEFORM5 to generate the answers for INLINEFORM6 .","Following BIBREF3 , our workflow consists of three steps: (1) the encoding phase, which generates meaningful representations for query and documents; (2) the inference phase, which extracts relevant semantic relationships between the query and the documents by using an iterative attention mechanism and finally (3) the prediction phase, which generates a score for each candidate answer."]},{"idx":3,"heading":"Encoding phase","paragraphs":["The input of the encoding phase is given by a query INLINEFORM0 and a set of documents INLINEFORM1 . Both queries and documents are represented by a sequence of words INLINEFORM2 , drawn from a vocabulary INLINEFORM3 . Each word is represented by a continuous INLINEFORM4 -dimensional word embedding INLINEFORM5 stored in a word embedding matrix INLINEFORM6 .","The sequences of dense representations for INLINEFORM0 and INLINEFORM1 are encoded using a bidirectional recurrent neural network encoder with Gated Recurrent Units (GRU) as in BIBREF3 which represents each word INLINEFORM2 as the concatenation of a forward encoding INLINEFORM3 and a backward encoding INLINEFORM4 . From now on, we denote the contextual representation for the word INLINEFORM5 by INLINEFORM6 and the contextual representation for the word INLINEFORM7 in the document INLINEFORM8 by INLINEFORM9 . Differently from BIBREF3 , we build a unique representation for the whole set of documents INLINEFORM10 related to the query INLINEFORM11 by stacking each contextual representation INLINEFORM12 obtaining a matrix INLINEFORM13 , where INLINEFORM14 ."]},{"idx":4,"heading":"Inference phase","paragraphs":["This phase uncovers a possible inference chain which models meaningful relationships between the query and the set of related documents. The inference chain is obtained by performing, for each inference step INLINEFORM0 , the attention mechanisms given by the query attentive read and the document attentive read keeping a state of the inference process given by an additional recurrent neural network with GRU units. In this way, the network is able to progressively refine the attention weights focusing on the most relevant tokens of the query and the documents which are exploited by the prediction neural network to select the correct answers among the candidate ones.","Given the contextual representations for the query words INLINEFORM0 and the inference GRU state INLINEFORM1 , we obtain a refined query representation INLINEFORM2 (query glimpse) by performing an attention mechanism over the query at inference step INLINEFORM3 : INLINEFORM4","where INLINEFORM0 are the attention weights associated to the query words, INLINEFORM1 and INLINEFORM2 are respectively a weight matrix and a bias vector which are used to perform the bilinear product with the query token representations INLINEFORM3 . The attention weights can be interpreted as the relevance scores for each word of the query dependent on the inference state INLINEFORM4 at the current inference step INLINEFORM5 .","Given the query glimpse INLINEFORM0 and the inference GRU state INLINEFORM1 , we perform an attention mechanism over the contextual representations for the words of the stacked documents INLINEFORM2 : INLINEFORM3","where INLINEFORM0 is the INLINEFORM1 -th row of INLINEFORM2 , INLINEFORM3 are the attention weights associated to the document words, INLINEFORM4 and INLINEFORM5 are respectively a weight matrix and a bias vector which are used to perform the bilinear product with the document token representations INLINEFORM6 . The attention weights can be interpreted as the relevance scores for each word of the documents conditioned on both the query glimpse and the inference state INLINEFORM7 at the current inference step INLINEFORM8 . By combining the set of relevant documents in INLINEFORM9 , we obtain the probability distribution ( INLINEFORM10 ) over all the relevant document tokens using the above-mentioned attention mechanism.","The inference GRU state at the inference step INLINEFORM0 is updated according to INLINEFORM1 , where INLINEFORM2 and INLINEFORM3 are the results of a gating mechanism obtained by evaluating INLINEFORM4 for the query and the documents, respectively. The gating function INLINEFORM5 is defined as a 2-layer feed-forward neural network with a Rectified Linear Unit (ReLU) BIBREF5 activation function in the hidden layer and a sigmoid activation function in the output layer. The purpose of the gating mechanism is to retain useful information for the inference process about query and documents and forget useless one."]},{"idx":5,"heading":"Prediction phase","paragraphs":["The prediction phase, which is completely different from the pointer-sum loss reported in BIBREF3 , is able to generate, given the query INLINEFORM0 , a relevance score for each candidate answer INLINEFORM1 by using the document attention weights INLINEFORM2 computed in the last inference step INLINEFORM3 . The relevance score of each word INLINEFORM4 is obtained by summing the attention weights of INLINEFORM5 in each document related to INLINEFORM6 . Formally the relevance score for a given word INLINEFORM7 is defined as: INLINEFORM8","where INLINEFORM0 returns 0 if INLINEFORM1 , INLINEFORM2 otherwise; INLINEFORM3 returns the word in position INLINEFORM4 of the stacked documents matrix INLINEFORM5 and INLINEFORM6 returns the frequency of the word INLINEFORM7 in the documents INLINEFORM8 related to the query INLINEFORM9 . The relevance score takes into account the importance of token occurrences in the considered documents given by the computed attention weights. Moreover, the normalization term INLINEFORM10 is applied to the relevance score in order to mitigate the weight associated to highly frequent tokens.","The evaluated relevance scores are concatenated in a single vector representation INLINEFORM0 which is given in input to the answer prediction neural network defined as: INLINEFORM1","where INLINEFORM0 is the hidden layer size, INLINEFORM1 and INLINEFORM2 are weight matrices, INLINEFORM3 , INLINEFORM4 are bias vectors, INLINEFORM5 is the sigmoid function and INLINEFORM6 is the ReLU activation function, which are applied pointwise to the given input vector.","The neural network weights are supposed to learn latent features which encode relationships between the most relevant words for the given query to predict the correct answers. The outer sigmoid activation function is used to treat the problem as a multi-label classification problem, so that each candidate answer is independent and not mutually exclusive. In this way the neural network generates a score which represents the probability that the candidate answer is correct. Moreover, differently from BIBREF3 , the candidate answer INLINEFORM0 can be any word, even those which not belong to the documents related to the query.","The model is trained by minimizing the binary cross-entropy loss function comparing the neural network output INLINEFORM0 with the target answers for the given query INLINEFORM1 represented as a binary vector, in which there is a 1 in the corresponding position of the correct answer, 0 otherwise."]},{"idx":6,"heading":"Experimental evaluation","paragraphs":["The model performance is evaluated on the QA and Recs tasks of the bAbI Movie Dialog dataset using HITS@k evaluation metric, which is equal to the number of correct answers in the top- INLINEFORM0 results. In particular, the performance for the QA task is evaluated according to HITS@1, while the performance for the Recs task is evaluated according to HITS@100.","Differently from BIBREF2 , the relevant knowledge base facts, taken from the knowledge base in triple form distributed with the dataset, are retrieved by INLINEFORM0 implemented by exploiting the Elasticsearch engine and not according to an hash lookup operator which applies a strict filtering procedure based on word frequency. In our work, INLINEFORM1 returns at most the top 30 relevant facts for INLINEFORM2 . Each entity in questions and documents is recognized using the list of entities provided with the dataset and considered as a single word of the dictionary INLINEFORM3 .","Questions, answers and documents given in input to the model are preprocessed using the NLTK toolkit BIBREF6 performing only word tokenization. The question given in input to the INLINEFORM0 operator is preprocessed performing word tokenization and stopword removal.","The optimization method and tricks are adopted from BIBREF3 . The model is trained using ADAM BIBREF7 optimizer (learning rate= INLINEFORM0 ) with a batch size of 128 for at most 100 epochs considering the best model until the HITS@k on the validation set decreases for 5 consecutive times. Dropout BIBREF8 is applied on INLINEFORM1 and on INLINEFORM2 with a rate of INLINEFORM3 and on the prediction neural network hidden layer with a rate of INLINEFORM4 . L2 regularization is applied to the embedding matrix INLINEFORM5 with a coefficient equal to INLINEFORM6 . We clipped the gradients if their norm is greater than 5 to stabilize learning BIBREF9 . Embedding size INLINEFORM7 is fixed to 50. All GRU output sizes are fixed to 128. The number of inference steps INLINEFORM8 is set to 3. The size of the prediction neural network hidden layer INLINEFORM9 is fixed to 4096. Biases INLINEFORM10 and INLINEFORM11 are initialized to zero vectors. All weight matrices are initialized sampling from the normal distribution INLINEFORM12 . The ReLU activation function in the prediction neural network has been experimentally chosen comparing different activation functions such as sigmoid and tanh and taking the one which leads to the best performance. The model is implemented in TensorFlow BIBREF10 and executed on an NVIDIA TITAN X GPU.","Following the experimental design, the results in Table TABREF10 are promising because our model outperforms all other systems on both tasks except for the QA SYSTEM on the QA task. Despite the advantage of the QA SYSTEM, it is a carefully designed system to handle knowledge base data in the form of triples, but our model can leverage data in the form of documents, without making any assumption about the form of the input data and can be applied to different kind of tasks. Additionally, the model MEMN2N is a neural network whose weights are pre-trained on the same dataset without using the long-term memory and the models JOINT SUPERVISED EMBEDDINGS and JOINT MEMN2N are models trained across all the tasks of the dataset in order to boost performance. Despite that, our model outperforms the three above-mentioned ones without using any supplementary trick. Even though our model performance is higher than all the others on the Recs task, we believe that the obtained result may be improved and so we plan a further investigation. Moreover, the need for further investigation can be justified by the work reported in BIBREF11 which describes some issues regarding the Recs task.","Figure FIGREF11 shows the attention weights computed in the last inference step of the iterative attention mechanism used by the model to answer to a given question. Attention weights, represented as red boxes with variable color shades around the tokens, can be used to interpret the reasoning mechanism applied by the model because higher shades of red are associated to more relevant tokens on which the model focus its attention. It is worth to notice that the attention weights associated to each token are the result of the inference mechanism uncovered by the model which progressively tries to focus on the relevant aspects of the query and the documents which are exploited to generate the answers.","Given the question “what does Larenz Tate act in?” shown in the above-mentioned figure, the model is able to understand that “Larenz Tate” is the subject of the question and “act in” represents the intent of the question. Reading the related documents, the model associates higher attention weights to the most relevant tokens needed to answer the question, such as “The Postman”, “A Man Apart” and so on."]},{"idx":7,"heading":"Related work","paragraphs":["We think that it is necessary to consider models and techniques coming from research both in QA and recommender systems in order to pursue our desire to build an intelligent agent able to assist the user in decision-making tasks. We cannot fill the gap between the above-mentioned research areas if we do not consider the proposed models in a synergic way by virtue of the proposed analogy between the user profile (the set of user preferences) and the items to be recommended, as the question and the correct answers. The first work which goes in this direction is reported in BIBREF12 , which exploits movie descriptions to suggest appealing movies for a given user using an architecture tipically used for QA tasks. In fact, most of the research in the recommender systems field presents ad-hoc systems which exploit neighbourhood information like in Collaborative Filtering techniques BIBREF13 , item descriptions and metadata like in Content-based systems BIBREF14 . Recently presented neural network models BIBREF15 , BIBREF16 systems are able to learn latent representations in the network weights leveraging information coming from user preferences and item information.","In recent days, a lot of effort is devoted to create benchmarks for artificial agents to assess their ability to comprehend natural language and to reason over facts. One of the first attempt is the bAbI BIBREF17 dataset which is a synthetic dataset containing elementary tasks such as selecting an answer between one or more candidate facts, answering yes/no questions, counting operations over lists and sets and basic induction and deduction tasks. Another relevant benchmark is the one described in BIBREF18 , which provides CNN/Daily Mail datasets consisting of document-query-answer triples where an entity in the query is replaced by a placeholder and the system should identify the correct entity by reading and comprehending the given document. MCTest BIBREF19 requires machines to answer multiple-choice reading comprehension questions about fictional stories, directly tackling the high-level goal of open-domain machine comprehension. Finally, SQuAD BIBREF20 consists in a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage.","According to the experimental evaluations conducted on the above-mentioned datasets, high-level performance can be obtained exploiting complex attention mechanisms which are able to focus on relevant evidences in the processed content. One of the earlier approaches used to solve these tasks is given by the general Memory Network BIBREF21 , BIBREF22 framework which is one of the first neural network models able to access external memories to extract relevant information through an attention mechanism and to use them to provide the correct answer. A deep Recurrent Neural Network with Long Short-Term Memory units is presented in BIBREF18 , which solves CNN/Daily Mail datasets by designing two different attention mechanisms called Impatient Reader and Attentive Reader. Another way to incorporate attention in neural network models is proposed in BIBREF23 which defines a pointer-sum loss whose aim is to maximize the attention weights which lead to the correct answer."]},{"idx":8,"heading":"Conclusions and Future Work","paragraphs":["In this work we propose a novel model based on Artificial Neural Networks to answer questions with multiple answers by exploiting multiple facts retrieved from a knowledge base. The proposed model can be considered a relevant building block of a conversational recommender system. Differently from BIBREF3 , our model can consider multiple documents as a source of information in order to generate multiple answers which may not belong to the documents. As presented in this work, common tasks such as QA and top-n recommendation can be solved effectively by our model.","In a common recommendation system scenario, when a user enters a search query, it is assumed that his preferences are known. This is a stringent requirement because users cannot have a clear idea of their preferences at that point. Conversational recommender systems support users to fulfill their information needs through an interactive process. In this way, the system can provide a personalized experience dynamically adapting the user model with the possibility to enhance the generated predictions. Moreover, the system capability can be further enhanced giving explanations to the user about the given suggestions.","To reach our goal, we should improve our model by designing a INLINEFORM0 operator able to return relevant facts recognizing the most relevant information in the query, by exploiting user preferences and contextual information to learn the user model and by providing a mechanism which leverages attention weights to give explanations. In order to effectively train our model, we plan to collect real dialog data containing contextual information associated to each user and feedback for each dialog which represents if the user is satisfied with the conversation. Given these enhancements, we should design a system able to hold effectively a dialog with the user recognizing his intent and providing him the most suitable contents.","With this work we try to show the effectiveness of our architecture for tasks which go from pure question answering to top-n recommendation through an experimental evaluation without any assumption on the task to be solved. To do that, we do not use any hand-crafted linguistic features but we let the system learn and leverage them in the inference process which leads to the answers through multiple reasoning steps. During these steps, the system understands relevant relationships between question and documents without relying on canonical matching, but repeating an attention mechanism able to unconver related aspects in distributed representations, conditioned on an encoding of the inference process given by another neural network. Equipping agents with a reasoning mechanism like the one described in this work and exploiting the ability of neural network models to learn from data, we may be able to create truly intelligent agents."]},{"idx":9,"heading":"Acknowledgments","paragraphs":["This work is supported by the IBM Faculty Award \"Deep Learning to boost Cognitive Question Answering\". The Titan X GPU used for this research was donated by the NVIDIA Corporation."]},{"idx":10,"heading":"Figures and Tables","paragraphs":["Table 1: Comparison between our model and baselines from [6] on the QA and Recs tasks evaluated according to HITS@1 and HITS@100, respectively.","Figure 1: Attention weights q̃i and D̃qi computed by the neural network attention mechanisms at the last inference step T for each token. Higher shades correspond to higher relevance scores for the related tokens."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1703.04357","title":"Nematus: a Toolkit for Neural Machine Translation","sections":[{"idx":0,"heading":"Abstract","paragraphs":["We present Nematus, a toolkit for Neural Machine Translation. The toolkit prioritizes high translation accuracy, usability, and extensibility. Nematus has been used to build top-performing submissions to shared translation tasks at WMT and IWSLT, and has been used to train systems for production environments."]},{"idx":1,"heading":"Introduction","paragraphs":["Neural Machine Translation (NMT) BIBREF0 , BIBREF1 has recently established itself as a new state-of-the art in machine translation. We present Nematus, a new toolkit for Neural Machine Translation.","Nematus has its roots in the dl4mt-tutorial. We found the codebase of the tutorial to be compact, simple and easy to extend, while also producing high translation quality. These characteristics make it a good starting point for research in NMT. Nematus has been extended to include new functionality based on recent research, and has been used to build top-performing systems to last year's shared translation tasks at WMT BIBREF2 and IWSLT BIBREF3 .","Nematus is implemented in Python, and based on the Theano framework BIBREF4 . It implements an attentional encoder–decoder architecture similar to DBLP:journals/corr/BahdanauCB14. Our neural network architecture differs in some aspect from theirs, and we will discuss differences in more detail. We will also describe additional functionality, aimed to enhance usability and performance, which has been implemented in Nematus."]},{"idx":2,"heading":"Neural Network Architecture","paragraphs":["Nematus implements an attentional encoder–decoder architecture similar to the one described by DBLP:journals/corr/BahdanauCB14, but with several implementation differences. The main differences are as follows:","We will here describe some differences in more detail:","Given a source sequence INLINEFORM0 of length INLINEFORM1 and a target sequence INLINEFORM2 of length INLINEFORM3 , let INLINEFORM4 be the annotation of the source symbol at position INLINEFORM5 , obtained by concatenating the forward and backward encoder RNN hidden states, INLINEFORM6 , and INLINEFORM7 be the decoder hidden state at position INLINEFORM8 ."]},{"idx":3,"heading":"Training Algorithms","paragraphs":["By default, the training objective in Nematus is cross-entropy minimization on a parallel training corpus. Training is performed via stochastic gradient descent, or one of its variants with adaptive learning rate (Adadelta BIBREF14 , RmsProp BIBREF15 , Adam BIBREF16 ).","Additionally, Nematus supports minimum risk training (MRT) BIBREF17 to optimize towards an arbitrary, sentence-level loss function. Various MT metrics are supported as loss function, including smoothed sentence-level Bleu BIBREF18 , METEOR BIBREF19 , BEER BIBREF20 , and any interpolation of implemented metrics.","To stabilize training, Nematus supports early stopping based on cross entropy, or an arbitrary loss function defined by the user."]},{"idx":4,"heading":"Usability Features","paragraphs":["In addition to the main algorithms to train and decode with an NMT model, Nematus includes features aimed towards facilitating experimentation with the models, and their visualisation. Various model parameters are configurable via a command-line interface, and we provide extensive documentation of options, and sample set-ups for training systems.","Nematus provides support for applying single models, as well as using multiple models in an ensemble – the latter is possible even if the model architectures differ, as long as the output vocabulary is the same. At each time step, the probability distribution of the ensemble is the geometric average of the individual models' probability distributions. The toolkit includes scripts for beam search decoding, parallel corpus scoring and n-best-list rescoring.","Nematus includes utilities to visualise the attention weights for a given sentence pair, and to visualise the beam search graph. An example of the latter is shown in Figure FIGREF16 . Our demonstration will cover how to train a model using the command-line interface, and showing various functionalities of Nematus, including decoding and visualisation, with pre-trained models."]},{"idx":5,"heading":"Conclusion","paragraphs":["We have presented Nematus, a toolkit for Neural Machine Translation. We have described implementation differences to the architecture by DBLP:journals/corr/BahdanauCB14; due to the empirically strong performance of Nematus, we consider these to be of wider interest.","We hope that researchers will find Nematus an accessible and well documented toolkit to support their research. The toolkit is by no means limited to research, and has been used to train MT systems that are currently in production BIBREF21 .","Nematus is available under a permissive BSD license."]},{"idx":6,"heading":"Acknowledgments","paragraphs":["This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreements 645452 (QT21), 644333 (TraMOOC), 644402 (HimL) and 688139 (SUMMA)."]},{"idx":7,"heading":"Figures and Tables","paragraphs":["Table 1: Decoder phase differences","Figure 1: Search graph visualisation for DE→EN translation of \"Hallo Welt!\" with beam size 3."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1703.10344","title":"Automated News Suggestions for Populating Wikipedia Entity Pages","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Wikipedia entity pages are a valuable source of information for direct consumption and for knowledge-base construction, update and maintenance. Facts in these entity pages are typically supported by references. Recent studies show that as much as 20\\% of the references are from online news sources. However, many entity pages are incomplete even if relevant information is already available in existing news articles. Even for the already present references, there is often a delay between the news article publication time and the reference time. In this work, we therefore look at Wikipedia through the lens of news and propose a novel news-article suggestion task to improve news coverage in Wikipedia, and reduce the lag of newsworthy references. Our work finds direct application, as a precursor, to Wikipedia page generation and knowledge-base acceleration tasks that rely on relevant and high quality input sources. We propose a two-stage supervised approach for suggesting news articles to entity pages for a given state of Wikipedia. First, we suggest news articles to Wikipedia entities (article-entity placement) relying on a rich set of features which take into account the \\emph{salience} and \\emph{relative authority} of entities, and the \\emph{novelty} of news articles to entity pages. Second, we determine the exact section in the entity page for the input article (article-section placement) guided by class-based section templates. We perform an extensive evaluation of our approach based on ground-truth data that is extracted from external references in Wikipedia. We achieve a high precision value of up to 93\\% in the \\emph{article-entity} suggestion stage and upto 84\\% for the \\emph{article-section placement}. Finally, we compare our approach against competitive baselines and show significant improvements."]},{"idx":1,"heading":"Introduction","paragraphs":["Wikipedia is the largest source of open and collaboratively curated knowledge in the world. Introduced in 2001, it has evolved into a reference work with around 5m pages for the English Wikipedia alone. In addition, entities and event pages are updated quickly via collaborative editing and all edits are encouraged to include source citations, creating a knowledge base which aims at being both timely as well as authoritative. As a result, it has become the preferred source of information consumption about entities and events. Moreso, this knowledge is harvested and utilized in building knowledge bases like YAGO BIBREF0 and DBpedia BIBREF1 , and used in applications like text categorization BIBREF2 , entity disambiguation BIBREF3 , entity ranking BIBREF4 and distant supervision BIBREF5 , BIBREF6 .","However, not all Wikipedia pages referring to entities (entity pages) are comprehensive: relevant information can either be missing or added with a delay. Consider the city of New Orleans and the state of Odisha which were severely affected by cyclones Hurricane Katrina and Odisha Cyclone, respectively. While Katrina finds extensive mention in the entity page for New Orleans, Odisha Cyclone which has 5 times more human casualties (cf. Figure FIGREF2 ) is not mentioned in the page for Odisha. Arguably Katrina and New Orleans are more popular entities, but Odisha Cyclone was also reported extensively in national and international news outlets. This highlights the lack of important facts in trunk and long-tail entity pages, even in the presence of relevant sources. In addition, previous studies have shown that there is an inherent delay or lag when facts are added to entity pages BIBREF7 .","To remedy these problems, it is important to identify information sources that contain novel and salient facts to a given entity page. However, not all information sources are equal. The online presence of major news outlets is an authoritative source due to active editorial control and their articles are also a timely container of facts. In addition, their use is in line with current Wikipedia editing practice, as is shown in BIBREF7 that almost 20% of current citations in all entity pages are news articles. We therefore propose news suggestion as a novel task that enhances entity pages and reduces delay while keeping its pages authoritative.","Existing efforts to populate Wikipedia BIBREF8 start from an entity page and then generate candidate documents about this entity using an external search engine (and then post-process them). However, such an approach lacks in (a) reproducibility since rankings vary with time with obvious bias to recent news (b) maintainability since document acquisition for each entity has to be periodically performed. To this effect, our news suggestion considers a news article as input, and determines if it is valuable for Wikipedia. Specifically, given an input news article INLINEFORM0 and a state of Wikipedia, the news suggestion problem identifies the entities mentioned in INLINEFORM1 whose entity pages can improve upon suggesting INLINEFORM2 . Most of the works on knowledge base acceleration BIBREF9 , BIBREF10 , BIBREF11 , or Wikipedia page generation BIBREF8 rely on high quality input sources which are then utilized to extract textual facts for Wikipedia page population. In this work, we do not suggest snippets or paraphrases but rather entire articles which have a high potential importance for entity pages. These suggested news articles could be consequently used for extraction, summarization or population either manually or automatically – all of which rely on high quality and relevant input sources.","We identify four properties of good news recommendations: salience, relative authority, novelty and placement. First, we need to identify the most salient entities in a news article. This is done to avoid pollution of entity pages with only marginally related news. Second, we need to determine whether the news is important to the entity as only the most relevant news should be added to a precise reference work. To do this, we compute the relative authority of all entities in the news article: we call an entity more authoritative than another if it is more popular or noteworthy in the real world. Entities with very high authority have many news items associated with them and only the most relevant of these should be included in Wikipedia whereas for entities of lower authority the threshold for inclusion of a news article will be lower. Third, a good recommendation should be able to identify novel news by minimizing redundancy coming from multiple news articles. Finally, addition of facts is facilitated if the recommendations are fine-grained, i.e., recommendations are made on the section level rather than the page level (placement).","Approach and Contributions. We propose a two-stage news suggestion approach to entity pages. In the first stage, we determine whether a news article should be suggested for an entity, based on the entity's salience in the news article, its relative authority and the novelty of the article to the entity page. The second stage takes into account the class of the entity for which the news is suggested and constructs section templates from entities of the same class. The generation of such templates has the advantage of suggesting and expanding entity pages that do not have a complete section structure in Wikipedia, explicitly addressing long-tail and trunk entities. Afterwards, based on the constructed template our method determines the best fit for the news article with one of the sections.","We evaluate the proposed approach on a news corpus consisting of 351,982 articles crawled from the news external references in Wikipedia from 73,734 entity pages. Given the Wikipedia snapshot at a given year (in our case [2009-2014]), we suggest news articles that might be cited in the coming years. The existing news references in the entity pages along with their reference date act as our ground-truth to evaluate our approach. In summary, we make the following contributions."]},{"idx":2,"heading":"Related Work","paragraphs":["As we suggest a new problem there is no current work addressing exactly the same task. However, our task has similarities to Wikipedia page generation and knowledge base acceleration. In addition, we take inspiration from Natural Language Processing (NLP) methods for salience detection.","Wikipedia Page Generation is the problem of populating Wikipedia pages with content coming from external sources. Sauper and Barzilay BIBREF8 propose an approach for automatically generating whole entity pages for specific entity classes. The approach is trained on already-populated entity pages of a given class (e.g. `Diseases') by learning templates about the entity page structure (e.g. diseases have a treatment section). For a new entity page, first, they extract documents via Web search using the entity title and the section title as a query, for example `Lung Cancer'+`Treatment'. As already discussed in the introduction, this has problems with reproducibility and maintainability. However, their main focus is on identifying the best paragraphs extracted from the collected documents. They rank the paragraphs via an optimized supervised perceptron model for finding the most representative paragraph that is the least similar to paragraphs in other sections. This paragraph is then included in the newly generated entity page. Taneva and Weikum BIBREF12 propose an approach that constructs short summaries for the long tail. The summaries are called `gems' and the size of a `gem' can be user defined. They focus on generating summaries that are novel and diverse. However, they do not consider any structure of entities, which is present in Wikipedia.","In contrast to BIBREF8 and BIBREF12 , we actually focus on suggesting entire documents to Wikipedia entity pages. These are authoritative documents (news), which are highly relevant for the entity, novel for the entity and in which the entity is salient. Whereas relevance in Sauper and Barzilay is implicitly computed by web page ranking we solve that problem by looking at relative authority and salience of an entity, using the news article and entity page only. As Sauper and Barzilay concentrate on empty entity pages, the problem of novelty of their content is not an issue in their work whereas it is in our case which focuses more on updating entities. Updating entities will be more and more important the bigger an existing reference work is. Both the approaches in BIBREF8 and BIBREF12 (finding paragraphs and summarization) could then be used to process the documents we suggest further. Our concentration on news is also novel.","Knowledge Base Acceleration. In this task, given specific information extraction templates, a given corpus is analyzed in order to find worthwhile mentions of an entity or snippets that match the templates. Balog BIBREF9 , BIBREF10 recommend news citations for an entity. Prior to that, the news articles are classified for their appropriateness for an entity, where as features for the classification task they use entity, document, entity-document and temporal features. The best performing features are those that measure similarity between an entity and the news document. West et al. BIBREF13 consider the problem of knowledge base completion, through question answering and complete missing facts in Freebase based on templates, i.e. Frank_Zappa bornIn Baltymore, Maryland.","In contrast, we do not extract facts for pre-defined templates but rather suggest news articles based on their relevance to an entity. In cases of long-tail entities, we can suggest to add a novel section through our abstraction and generation of section templates at entity class level.","Entity Salience. Determining which entities are prominent or salient in a given text has a long history in NLP, sparked by the linguistic theory of Centering BIBREF14 . Salience has been used in pronoun and co-reference resolution BIBREF15 , or to predict which entities will be included in an abstract of an article BIBREF11 . Frequent features to measure salience include the frequency of an entity in a document, positioning of an entity, grammatical function or internal entity structure (POS tags, head nouns etc.). These approaches are not currently aimed at knowledge base generation or Wikipedia coverage extension but we postulate that an entity's salience in a news article is a prerequisite to the news article being relevant enough to be included in an entity page. We therefore use the salience features in BIBREF11 as part of our model. However, these features are document-internal — we will show that they are not sufficient to predict news inclusion into an entity page and add features of entity authority, news authority and novelty that measure the relations between several entities, between entity and news article as well as between several competing news articles."]},{"idx":3,"heading":"Terminology and Problem Definition","paragraphs":["We are interested in named entities mentioned in documents. An entity INLINEFORM0 can be identified by a canonical name, and can be mentioned differently in text via different surface forms. We canonicalize these mentions to entity pages in Wikipedia, a method typically known as entity linking. We denote the set of canonicalized entities extracted and linked from a news article INLINEFORM1 as INLINEFORM2 . For example, in Figure FIGREF7 , entities are canonicalized into Wikipedia entity pages (e.g. Odisha is canonicalized to the corresponding article). For a collection of news articles INLINEFORM3 , we further denote the resulting set of entities by INLINEFORM4 .","Information in an entity page is organized into sections and evolves with time as more content is added. We refer to the state of Wikipedia at a time INLINEFORM0 as INLINEFORM1 and the set of sections for an entity page INLINEFORM2 as its entity profile INLINEFORM3 . Unlike news articles, text in Wikipedia could be explicitly linked to entity pages through anchors. The set of entities explicitly referred in text from section INLINEFORM4 is defined as INLINEFORM5 . Furthermore, Wikipedia induces a category structure over its entities, which is exploited by knowledge bases like YAGO (e.g. Barack_Obama isA Person). Consequently, each entity page belongs to one or more entity categories or classes INLINEFORM6 . Now we can define our news suggestion problem below:","Definition 1 (News Suggestion Problem) Given a set of news articles INLINEFORM0 and set of Wikipedia entity pages INLINEFORM1 (from INLINEFORM2 ) we intend to suggest a news article INLINEFORM3 published at time INLINEFORM4 to entity page INLINEFORM5 and additionally to the most relevant section for the entity page INLINEFORM6 ."]},{"idx":4,"heading":"Approach Overview","paragraphs":["We approach the news suggestion problem by decomposing it into two tasks:","AEP: Article–Entity placement","ASP: Article–Section placement","In this first step, for a given entity-news pair INLINEFORM0 , we determine whether the given news article INLINEFORM1 should be suggested (we will refer to this as `relevant') to entity INLINEFORM2 . To generate such INLINEFORM3 pairs, we perform the entity linking process, INLINEFORM4 , for INLINEFORM5 .","The article–entity placement task (described in detail in Section SECREF16 ) for a pair INLINEFORM0 outputs a binary label (either `non-relevant' or `relevant') and is formalized in Equation EQREF14 . DISPLAYFORM0","In the second step, we take into account all `relevant' pairs INLINEFORM0 and find the correct section for article INLINEFORM1 in entity INLINEFORM2 , respectively its profile INLINEFORM3 (see Section SECREF30 ). The article–section placement task, determines the correct section for the triple INLINEFORM4 , and is formalized in Equation EQREF15 . DISPLAYFORM0","In the subsequent sections we describe in details how we approach the two tasks for suggesting news articles to entity pages."]},{"idx":5,"heading":"News Article Suggestion","paragraphs":["In this section, we provide an overview of the news suggestion approach to Wikipedia entity pages (see Figure FIGREF7 ). The approach is split into two tasks: (i) article-entity (AEP) and (ii) article-section (ASP) placement. For a Wikipedia snapshot INLINEFORM0 and a news corpus INLINEFORM1 , we first determine which news articles should be suggested to an entity INLINEFORM2 . We will denote our approach for AEP by INLINEFORM3 . Finally, we determine the most appropriate section for the ASP task and we denote our approach with INLINEFORM4 .","In the following, we describe the process of learning the functions INLINEFORM0 and INLINEFORM1 . We introduce features for the learning process, which encode information regarding the entity salience, relative authority and novelty in the case of AEP task. For the ASP task, we measure the overall fit of an article to the entity sections, with the entity being an input from AEP task. Additionally, considering that the entity profiles INLINEFORM2 are incomplete, in the case of a missing section we suggest and expand the entity profiles based on section templates generated from entities of the same class INLINEFORM3 (see Section UID34 )."]},{"idx":6,"heading":"Article–Entity Placement","paragraphs":["In this step we learn the function INLINEFORM0 to correctly determine whether INLINEFORM1 should be suggested for INLINEFORM2 , basically a binary classification model (0=`non-relevant' and 1=`relevant'). Note that we are mainly interested in finding the relevant pairs in this task. For every news article, the number of disambiguated entities is around 30 (but INLINEFORM3 is suggested for only two of them on average). Therefore, the distribution of `non-relevant' and `relevant' pairs is skewed towards the earlier, and by simply choosing the `non-relevant' label we can achieve a high accuracy for INLINEFORM4 . Finding the relevant pairs is therefore a considerable challenge.","An article INLINEFORM0 is suggested to INLINEFORM1 by our function INLINEFORM2 if it fulfills the following properties. The entity INLINEFORM3 is salient in INLINEFORM4 (a central concept), therefore ensuring that INLINEFORM5 is about INLINEFORM6 and that INLINEFORM7 is important for INLINEFORM8 . Next, given the fact there might be many articles in which INLINEFORM9 is salient, we also look at the reverse property, namely whether INLINEFORM10 is important for INLINEFORM11 . We do this by comparing the authority of INLINEFORM12 (which is a measure of popularity of an entity, such as its frequency of mention in a whole corpus) with the authority of its co-occurring entities in INLINEFORM13 , leading to a feature we call relative authority. The intuition is that for an entity that has overall lower authority than its co-occurring entities, a news article is more easily of importance. Finally, if the article we are about to suggest is already covered in the entity profile INLINEFORM14 , we do not wish to suggest redundant information, hence the novelty. Therefore, the learning objective of INLINEFORM15 should fulfill the following properties. Table TABREF21 shows a summary of the computed features for INLINEFORM16 .","Salience: entity INLINEFORM0 should be a salient entity in news article INLINEFORM1","Relative Authority: the set of entities INLINEFORM0 with which INLINEFORM1 co-occurs should have higher authority than INLINEFORM2 , making INLINEFORM3 important for INLINEFORM4","Novelty: news article INLINEFORM0 should provide novel information for entity INLINEFORM1 taking into account its profile INLINEFORM2","Baseline Features. As discussed in Section SECREF2 , a variety of features that measure salience of an entity in text are available from the NLP community. We reimplemented the ones in Dunietz and Gillick BIBREF11 . This includes a variety of features, e.g. positional features, occurrence frequency and the internal POS structure of the entity and the sentence it occurs in. Table 2 in BIBREF11 gives details.","Relative Entity Frequency. Although frequency of mention and positional features play some role in baseline features, their interaction is not modeled by a single feature nor do the positional features encode more than sentence position. We therefore suggest a novel feature called relative entity frequency, INLINEFORM0 , that has three properties.: (i) It rewards entities for occurring throughout the text instead of only in some parts of the text, measured by the number of paragraphs it occurs in (ii) it rewards entities that occur more frequently in the opening paragraphs of an article as we model INLINEFORM1 as an exponential decay function. The decay corresponds to the positional index of the news paragraph. This is inspired by the news-specific discourse structure that tends to give short summaries of the most important facts and entities in the opening paragraphs. (iii) it compares entity frequency to the frequency of its co-occurring mentions as the weight of an entity appearing in a specific paragraph, normalized by the sum of the frequencies of other entities in INLINEFORM2 . DISPLAYFORM0","where, INLINEFORM0 represents a news paragraph from INLINEFORM1 , and with INLINEFORM2 we indicate the set of all paragraphs in INLINEFORM3 . The frequency of INLINEFORM4 in a paragraph INLINEFORM5 is denoted by INLINEFORM6 . With INLINEFORM7 and INLINEFORM8 we indicate the number of paragraphs in which entity INLINEFORM9 occurs, and the total number of paragraphs, respectively.","Relative Authority. In this case, we consider the comparative relevance of the news article to the different entities occurring in it. As an example, let us consider the meeting of the Sudanese bishop Elias Taban with Hillary Clinton. Both entities are salient for the meeting. However, in Taban's Wikipedia page, this meeting is discussed prominently with a corresponding news reference, whereas in Hillary Clinton's Wikipedia page it is not reported at all. We believe this is not just an omission in Clinton's page but mirrors the fact that for the lesser known Taban the meeting is big news whereas for the more famous Clinton these kind of meetings are a regular occurrence, not all of which can be reported in what is supposed to be a selection of the most important events for her. Therefore, if two entities co-occur, the news is more relevant for the entity with the lower a priori authority.","The a priori authority of an entity (denoted by INLINEFORM0 ) can be measured in several ways. We opt for two approaches: (i) probability of entity INLINEFORM1 occurring in the corpus INLINEFORM2 , and (ii) authority assessed through centrality measures like PageRank BIBREF16 . For the second case we construct the graph INLINEFORM3 consisting of entities in INLINEFORM4 and news articles in INLINEFORM5 as vertices. The edges are established between INLINEFORM6 and entities in INLINEFORM7 , that is INLINEFORM8 , and the out-links from INLINEFORM9 , that is INLINEFORM10 (arrows present the edge direction).","Starting from a priori authority, we proceed to relative authority by comparing the a priori authority of co-occurring entities in INLINEFORM0 . We define the relative authority of INLINEFORM1 as the proportion of co-occurring entities INLINEFORM2 that have a higher a priori authority than INLINEFORM3 (see Equation EQREF28 . DISPLAYFORM0","As we might run the danger of not suggesting any news articles for entities with very high a priori authority (such as Clinton) due to the strict inequality constraint, we can relax the constraint such that the authority of co-occurring entities is above a certain threshold.","News Domain Authority. The news domain authority addresses two main aspects. Firstly, if bundled together with the relative authority feature, we can ensure that dependent on the entity authority, we suggest news from authoritative sources, hence ensuring the quality of suggested articles. The second aspect is in a news streaming scenario where multiple news domains report the same event — ideally only articles coming from authoritative sources would fulfill the conditions for the news suggestion task.","The news domain authority is computed based on the number of news references in Wikipedia coming from a particular news domain INLINEFORM0 . This represents a simple prior that a news article INLINEFORM1 is from domain INLINEFORM2 in corpus INLINEFORM3 . We extract the domains by taking the base URLs from the news article URLs.","An important feature when suggesting an article INLINEFORM0 to an entity INLINEFORM1 is the novelty of INLINEFORM2 w.r.t the already existing entity profile INLINEFORM3 . Studies BIBREF17 have shown that on comparable collections to ours (TREC GOV2) the number of duplicates can go up to INLINEFORM4 . This figure is likely higher for major events concerning highly authoritative entities on which all news media will report.","Given an entity INLINEFORM0 and the already added news references INLINEFORM1 up to year INLINEFORM2 , the novelty of INLINEFORM3 at year INLINEFORM4 is measured by the KL divergence between the language model of INLINEFORM5 and articles in INLINEFORM6 . We combine this measure with the entity overlap of INLINEFORM7 and INLINEFORM8 . The novelty value of INLINEFORM9 is given by the minimal divergence value. Low scores indicate low novelty for the entity profile INLINEFORM10 .","N(n|e) = n'Nt-1{DKL((n') || (n)) + DKL((N) || (n)).","DKL((n') || (n)). (1-) jaccard((n'),(n))} where INLINEFORM0 is the KL divergence of the language models ( INLINEFORM1 and INLINEFORM2 ), whereas INLINEFORM3 is the mixing weight ( INLINEFORM4 ) between the language models INLINEFORM5 and the entity overlap in INLINEFORM6 and INLINEFORM7 .","Here we introduce the evaluation setup and analyze the results for the article–entity (AEP) placement task. We only report the evaluation metrics for the `relevant' news-entity pairs. A detailed explanation on why we focus on the `relevant' pairs is provided in Section SECREF16 .","Baselines. We consider the following baselines for this task.","B1. The first baseline uses only the salience-based features by Dunietz and Gillick BIBREF11 .","B2. The second baseline assigns the value relevant to a pair INLINEFORM0 , if and only if INLINEFORM1 appears in the title of INLINEFORM2 .","Learning Models. We use Random Forests (RF) BIBREF23 . We learn the RF on all computed features in Table TABREF21 . The optimization on RF is done by splitting the feature space into multiple trees that are considered as ensemble classifiers. Consequently, for each classifier it computes the margin function as a measure of the average count of predicting the correct class in contrast to any other class. The higher the margin score the more robust the model.","Metrics. We compute precision P, recall R and F1 score for the relevant class. For example, precision is the number of news-entity pairs we correctly labeled as relevant compared to our ground truth divided by the number of all news-entity pairs we labeled as relevant.","The following results measure the effectiveness of our approach in three main aspects: (i) overall performance of INLINEFORM0 and comparison to baselines, (ii) robustness across the years, and (iii) optimal model for the AEP placement task.","Performance. Figure FIGREF55 shows the results for the years 2009 and 2013, where we optimized the learning objective with instances from year INLINEFORM0 and evaluate on the years INLINEFORM1 (see Section SECREF46 ). The results show the precision–recall curve. The red curve shows baseline B1 BIBREF11 , and the blue one shows the performance of INLINEFORM2 . The curve shows for varying confidence scores (high to low) the precision on labeling the pair INLINEFORM3 as `relevant'. In addition, at each confidence score we can compute the corresponding recall for the `relevant' label. For high confidence scores on labeling the news-entity pairs, the baseline B1 achieves on average a precision score of P=0.50, while INLINEFORM4 has P=0.93. We note that with the drop in the confidence score the corresponding precision and recall values drop too, and the overall F1 score for B1 is around F1=0.2, in contrast we achieve an average score of F1=0.67.","It is evident from Figure FIGREF55 that for the years 2009 and 2013, INLINEFORM0 significantly outperforms the baseline B1. We measure the significance through the t-test statistic and get a p-value of INLINEFORM1 . The improvement we achieve over B1 in absolute numbers, INLINEFORM2 P=+0.5 in terms of precision for the years between 2009 and 2014, and a similar improvement in terms of F1 score. The improvement for recall is INLINEFORM3 R=+0.4. The relative improvement over B1 for P and F1 is almost 1.8 times better, while for recall we are 3.5 times better. In Table TABREF58 we show the overall scores for the evaluation metrics for B1 and INLINEFORM4 . Finally, for B2 we achieve much poorer performance, with average scores of P=0.21, R=0.20 and F1=0.21.","Robustness. In Table TABREF58 , we show the overall performance for the years between 2009 and 2013. An interesting observation we make is that we have a very robust performance and the results are stable across the years. If we consider the experimental setup, where for year INLINEFORM0 we optimize the learning objective with only 74k training instances and evaluate on the rest of the instances, it achieves a very good performance. We predict with F1=0.68 the remaining 469k instances for the years INLINEFORM1 .","The results are particularly promising considering the fact that the distribution between our two classes is highly skewed. On average the number of `relevant' pairs account for only around INLINEFORM0 of all pairs. A good indicator to support such a statement is the kappa (denoted by INLINEFORM1 ) statistic. INLINEFORM2 measures agreement between the algorithm and the gold standard on both labels while correcting for chance agreement (often expected due to extreme distributions). The INLINEFORM3 scores for B1 across the years is on average INLINEFORM4 , while for INLINEFORM5 we achieve a score of INLINEFORM6 (the maximum score for INLINEFORM7 is 1).","In Figure FIGREF60 we show the impact of the individual feature groups that contribute to the superior performance in comparison to the baselines. Relative entity frequency from the salience feature, models the entity salience as an exponentially decaying function based on the positional index of the paragraph where the entity appears. The performance of INLINEFORM0 with relative entity frequency from the salience feature group is close to that of all the features combined. The authority and novelty features account to a further improvement in terms of precision, by adding roughly a 7%-10% increase. However, if both feature groups are considered separately, they significantly outperform the baseline B1."]},{"idx":7,"heading":"Article–Section Placement","paragraphs":["We model the ASP placement task as a successor of the AEP task. For all the `relevant' news entity pairs, the task is to determine the correct entity section. Each section in a Wikipedia entity page represents a different topic. For example, Barack Obama has the sections `Early Life', `Presidency', `Family and Personal Life' etc. However, many entity pages have an incomplete section structure. Incomplete or missing sections are due to two Wikipedia properties. First, long-tail entities miss information and sections due to their lack of popularity. Second, for all entities whether popular or not, certain sections might occur for the first time due to real world developments. As an example, the entity Germanwings did not have an `Accidents' section before this year's disaster, which was the first in the history of the airline.","Even if sections are missing for certain entities, similar sections usually occur in other entities of the same class (e.g. other airlines had disasters and therefore their pages have an accidents section). We exploit such homogeneity of section structure and construct templates that we use to expand entity profiles. The learning objective for INLINEFORM0 takes into account the following properties:","Section-templates: account for incomplete section structure for an entity profile INLINEFORM0 by constructing section templates INLINEFORM1 from an entity class INLINEFORM2","Overall fit: measures the overall fit of a news article to sections in the section templates INLINEFORM0","Given the fact that entity profiles are often incomplete, we construct section templates for every entity class. We group entities based on their class INLINEFORM0 and construct section templates INLINEFORM1 . For different entity classes, e.g. Person and Location, the section structure and the information represented in those section varies heavily. Therefore, the section templates are with respect to the individual classes in our experimental setup (see Figure FIGREF42 ). DISPLAYFORM0","Generating section templates has two main advantages. Firstly, by considering class-based profiles, we can overcome the problem of incomplete individual entity profiles and thereby are able to suggest news articles to sections that do not yet exist in a specific entity INLINEFORM0 . The second advantage is that we are able to canonicalize the sections, i.e. `Early Life' and `Early Life and Childhood' would be treated similarly.","To generate the section template INLINEFORM0 , we extract all sections from entities of a given type INLINEFORM1 at year INLINEFORM2 . Next, we cluster the entity sections, based on an extended version of k–means clustering BIBREF18 , namely x–means clustering introduced in Pelleg et al. which estimates the number of clusters efficiently BIBREF19 . As a similarity metric we use the cosine similarity computed based on the tf–idf models of the sections. Using the x–means algorithm we overcome the requirement to provide the number of clusters k beforehand. x–means extends the k–means algorithm, such that a user only specifies a range [ INLINEFORM3 , INLINEFORM4 ] that the number of clusters may reasonably lie in.","The learning objective of INLINEFORM0 is to determine the overall fit of a news article INLINEFORM1 to one of the sections in a given section template INLINEFORM2 . The template is pre-determined by the class of the entity for which the news is suggested as relevant by INLINEFORM3 . In all cases, we measure how well INLINEFORM4 fits each of the sections INLINEFORM5 as well as the specific entity section INLINEFORM6 . The section profiles in INLINEFORM7 represent the aggregated entity profiles from all entities of class INLINEFORM8 at year INLINEFORM9 .","To learn INLINEFORM0 we rely on a variety of features that consider several similarity aspects as shown in Table TABREF31 . For the sake of simplicity we do not make the distinction in Table TABREF31 between the individual entity section and class-based section similarities, INLINEFORM1 and INLINEFORM2 , respectively. Bear in mind that an entity section INLINEFORM3 might be present at year INLINEFORM4 but not at year INLINEFORM5 (see for more details the discussion on entity profile expansion in Section UID69 ).","Topic. We use topic similarities to ensure (i) that the content of INLINEFORM0 fits topic-wise with a specific section text and (ii) that it has a similar topic to previously referred news articles in that section. In a pre-processing stage we compute the topic models for the news articles, entity sections INLINEFORM1 and the aggregated class-based sections in INLINEFORM2 . The topic models are computed using LDA BIBREF20 . We only computed a single topic per article/section as we are only interested in topic term overlaps between article and sections. We distinguish two main features: the first feature measures the overlap of topic terms between INLINEFORM3 and the entity section INLINEFORM4 and INLINEFORM5 , and the second feature measures the overlap of the topic model of INLINEFORM6 against referred news articles in INLINEFORM7 at time INLINEFORM8 .","Syntactic. These features represent a mechanism for conveying the importance of a specific text snippet, solely based on the frequency of specific POS tags (i.e. NNP, CD etc.), as commonly used in text summarization tasks. Following the same intuition as in BIBREF8 , we weigh the importance of articles by the count of specific POS tags. We expect that for different sections, the importance of POS tags will vary. We measure the similarity of POS tags in a news article against the section text. Additionally, we consider bi-gram and tri-gram POS tag overlap. This exploits similarity in syntactical patterns between the news and section text.","Lexical. As lexical features, we measure the similarity of INLINEFORM0 against the entity section text INLINEFORM1 and the aggregate section text INLINEFORM2 . Further, we distinguish between the overall similarity of INLINEFORM3 and that of the different news paragraphs ( INLINEFORM4 which denotes the paragraphs of INLINEFORM5 up to the 5th paragraph). A higher similarity on the first paragraphs represents a more confident indicator that INLINEFORM6 should be suggested to a specific section INLINEFORM7 . We measure the similarity based on two metrics: (i) the KL-divergence between the computed language models and (ii) cosine similarity of the corresponding paragraph text INLINEFORM8 and section text.","Entity-based. Another feature set we consider is the overlap of named entities and their corresponding entity classes. For different entity sections, we expect to find a particular set of entity classes that will correlate with the section, e.g. `Early Life' contains mostly entities related to family, school, universities etc.","Frequency. Finally, we gather statistics about the number of entities, paragraphs, news article length, top– INLINEFORM0 entities and entity classes, and the frequency of different POS tags. Here we try to capture patterns of articles that are usually cited in specific sections."]},{"idx":8,"heading":"Evaluation Plan","paragraphs":["In this section we outline the evaluation plan to verify the effectiveness of our learning approaches. To evaluate the news suggestion problem we are faced with two challenges.","What comprises the ground truth for such a task ?","How do we construct training and test splits given that entity pages consists of text added at different points in time ?","Consider the ground truth challenge. Evaluating if an arbitrary news article should be included in Wikipedia is both subjective and difficult for a human if she is not an expert. An invasive approach, which was proposed by Barzilay and Sauper BIBREF8 , adds content directly to Wikipedia and expects the editors or other users to redact irrelevant content over a period of time. The limitations of such an evaluation technique is that content added to long-tail entities might not be evaluated by informed users or editors in the experiment time frame. It is hard to estimate how much time the added content should be left on the entity page. A more non-invasive approach could involve crowdsourcing of entity and news article pairs in an IR style relevance assessment setup. The problem of such an approach is again finding knowledgeable users or experts for long-tail entities. Thus the notion of relevance of a news recommendation is challenging to evaluate in a crowd setup.","We take a slightly different approach by making an assumption that the news articles already present in Wikipedia entity pages are relevant. To this extent, we extract a dataset comprising of all news articles referenced in entity pages (details in Section SECREF40 ). At the expense of not evaluating the space comprising of news articles absent in Wikipedia, we succeed in (i) avoiding restrictive assumptions about the quality of human judgments, (ii) being invasive and polluting Wikipedia, and (iii) deriving a reusable test bed for quicker experimentation.","The second challenge of construction of training and test set separation is slightly easier and is addressed in Section SECREF46 ."]},{"idx":9,"heading":"Datasets","paragraphs":["The datasets we use for our experimental evaluation are directly extracted from the Wikipedia entity pages and their revision history. The generated data represents one of the contributions of our paper. The datasets are the following:","Entity Classes. We focus on a manually predetermined set of entity classes for which we expect to have news coverage. The number of analyzed entity classes is 27, including INLINEFORM0 entities with at least one news reference. The entity classes were selected from the DBpedia class ontology. Figure FIGREF42 shows the number of entities per class for the years (2009-2014).","News Articles. We extract all news references from the collected Wikipedia entity pages. The extracted news references are associated with the sections in which they appear. In total there were INLINEFORM0 news references, and after crawling we end up with INLINEFORM1 successfully crawled news articles. The details of the news article distribution, and the number of entities and sections from which they are referred are shown in Table TABREF44 .","Article-Entity Ground-truth. The dataset comprises of the news and entity pairs INLINEFORM0 . News-entity pairs are relevant if the news article is referenced in the entity page. Non-relevant pairs (i.e. negative training examples) consist of news articles that contain an entity but are not referenced in that entity's page. If a news article INLINEFORM1 is referred from INLINEFORM2 at year INLINEFORM3 , the features are computed taking into account the entity profiles at year INLINEFORM4 .","Article-Section Ground-truth. The dataset consists of the triple INLINEFORM0 , where INLINEFORM1 , where we assume that INLINEFORM2 has already been determined as relevant. We therefore have a multi-class classification problem where we need to determine the section of INLINEFORM3 where INLINEFORM4 is cited. Similar to the article-entity ground truth, here too the features compute the similarity between INLINEFORM5 , INLINEFORM6 and INLINEFORM7 ."]},{"idx":10,"heading":"Data Pre-Processing","paragraphs":["We POS-tag the news articles and entity profiles INLINEFORM0 with the Stanford tagger BIBREF21 . For entity linking the news articles, we use TagMe! BIBREF22 with a confidence score of 0.3. On a manual inspection of a random sample of 1000 disambiguated entities, the accuracy is above 0.9. On average, the number of entities per news article is approximately 30. For entity linking the entity profiles, we simply follow the anchor text that refers to Wikipedia entities."]},{"idx":11,"heading":"Train and Testing Evaluation Setup","paragraphs":["We evaluate the generated supervised models for the two tasks, AEP and ASP, by splitting the train and testing instances. It is important to note that for the pairs INLINEFORM0 and the triple INLINEFORM1 , the news article INLINEFORM2 is referenced at time INLINEFORM3 by entity INLINEFORM4 , while the features take into account the entity profile at time INLINEFORM5 . This avoids any `overlapping' content between the news article and the entity page, which could affect the learning task of the functions INLINEFORM6 and INLINEFORM7 . Table TABREF47 shows the statistics of train and test instances. We learn the functions at year INLINEFORM8 and test on instances for the years greater than INLINEFORM9 . Please note that we do not show the performance for year 2014 as we do not have data for 2015 for evaluation."]},{"idx":12,"heading":"Article-Section Placement","paragraphs":["Here we show the evaluation setup for ASP task and discuss the results with a focus on three main aspects, (i) the overall performance across the years, (ii) the entity class specific performance, and (iii) the impact on entity profile expansion by suggesting missing sections to entities based on the pre-computed templates.","Baselines. To the best of our knowledge, we are not aware of any comparable approach for this task. Therefore, the baselines we consider are the following:","S1: Pick the section from template INLINEFORM0 with the highest lexical similarity to INLINEFORM1 : S1 INLINEFORM2","S2: Place the news into the most frequent section in INLINEFORM0","Learning Models. We use Random Forests (RF) BIBREF23 and Support Vector Machines (SVM) BIBREF24 . The models are optimized taking into account the features in Table TABREF31 . In contrast to the AEP task, here the scale of the number of instances allows us to learn the SVM models. The SVM model is optimized using the INLINEFORM0 loss function and uses the Gaussian kernels.","Metrics. We compute precision P as the ratio of news for which we pick a section INLINEFORM0 from INLINEFORM1 and INLINEFORM2 conforms to the one in our ground-truth (see Section SECREF40 ). The definition of recall R and F1 score follows from that of precision.","Figure FIGREF66 shows the overall performance and a comparison of our approach (when INLINEFORM0 is optimized using SVM) against the best performing baseline S2. With the increase in the number of training instances for the ASP task the performance is a monotonically non-decreasing function. For the year 2009, we optimize the learning objective of INLINEFORM1 with around 8% of the total instances, and evaluate on the rest. The performance on average is around P=0.66 across all classes. Even though for many classes the performance is already stable (as we will see in the next section), for some classes we improve further. If we take into account the years between 2010 and 2012, we have an increase of INLINEFORM2 P=0.17, with around 70% of instances used for training and the remainder for evaluation. For the remaining years the total improvement is INLINEFORM3 P=0.18 in contrast to the performance at year 2009.","On the other hand, the baseline S1 has an average precision of P=0.12. The performance across the years varies slightly, with the year 2011 having the highest average precision of P=0.13. Always picking the most frequent section as in S2, as shown in Figure FIGREF66 , results in an average precision of P=0.17, with a uniform distribution across the years.","Here we show the performance of INLINEFORM0 decomposed for the different entity classes. Specifically we analyze the 27 classes in Figure FIGREF42 . In Table TABREF68 , we show the results for a range of years (we omit showing all years due to space constraints). For illustration purposes only, we group them into four main classes ( INLINEFORM1 Person, Organization, Location, Event INLINEFORM2 ) and into the specific sub-classes shown in the second column in Table TABREF68 . For instance, the entity classes OfficeHolder and Politician are aggregated into Person–Politics.","It is evident that in the first year the performance is lower in contrast to the later years. This is due to the fact that as we proceed, we can better generalize and accurately determine the correct fit of an article INLINEFORM0 into one of the sections from the pre-computed templates INLINEFORM1 . The results are already stable for the year range INLINEFORM2 . For a few Person sub-classes, e.g. Politics, Entertainment, we achieve an F1 score above 0.9. These additionally represent classes with a sufficient number of training instances for the years INLINEFORM3 . The lowest F1 score is for the Criminal and Television classes. However, this is directly correlated with the insufficient number of instances.","The baseline approaches for the ASP task perform poorly. S1, based on lexical similarity, has a varying performance for different entity classes. The best performance is achieved for the class Person – Politics, with P=0.43. This highlights the importance of our feature choice and that the ASP cannot be considered as a linear function, where the maximum similarity yields the best results. For different entity classes different features and combination of features is necessary. Considering that S2 is the overall best performing baseline, through our approach INLINEFORM0 we have a significant improvement of over INLINEFORM1 P=+0.64.","The models we learn are very robust and obtain high accuracy, fulfilling our pre-condition for accurate news suggestions into the entity sections. We measure the robustness of INLINEFORM0 through the INLINEFORM1 statistic. In this case, we have a model with roughly 10 labels (corresponding to the number of sections in a template INLINEFORM2 ). The score we achieve shows that our model predicts with high confidence with INLINEFORM3 .","The last analysis is the impact we have on expanding entity profiles INLINEFORM0 with new sections. Figure FIGREF70 shows the ratio of sections for which we correctly suggest an article INLINEFORM1 to the right section in the section template INLINEFORM2 . The ratio here corresponds to sections that are not present in the entity profile at year INLINEFORM3 , that is INLINEFORM4 . However, given the generated templates INLINEFORM5 , we can expand the entity profile INLINEFORM6 with a new section at time INLINEFORM7 . In details, in the absence of a section at time INLINEFORM8 , our model trains well on similar sections from the section template INLINEFORM9 , hence we can predict accurately the section and in this case suggest its addition to the entity profile. With time, it is obvious that the expansion rate decreases at later years as the entity profiles become more `complete'.","This is particularly interesting for expanding the entity profiles of long-tail entities as well as updating entities with real-world emerging events that are added constantly. In many cases such missing sections are present at one of the entities of the respective entity class INLINEFORM0 . An obvious case is the example taken in Section SECREF16 , where the `Accidents' is rather common for entities of type Airline. However, it is non-existent for some specific entity instances, i.e Germanwings airline.","Through our ASP approach INLINEFORM0 , we are able to expand both long-tail and trunk entities. We distinguish between the two types of entities by simply measuring their section text length. The real distribution in the ground truth (see Section SECREF40 ) is 27% and 73% are long-tail and trunk entities, respectively. We are able to expand the entity profiles for both cases and all entity classes without a significant difference, with the only exception being the class Creative Work, where we expand significantly more trunk entities."]},{"idx":13,"heading":"Conclusion and Future Work","paragraphs":["In this work, we have proposed an automated approach for the novel task of suggesting news articles to Wikipedia entity pages to facilitate Wikipedia updating. The process consists of two stages. In the first stage, article–entity placement, we suggest news articles to entity pages by considering three main factors, such as entity salience in a news article, relative authority and novelty of news articles for an entity page. In the second stage, article–section placement, we determine the best fitting section in an entity page. Here, we remedy the problem of incomplete entity section profiles by constructing section templates for specific entity classes. This allows us to add missing sections to entity pages. We carry out an extensive experimental evaluation on 351,983 news articles and 73,734 entities coming from 27 distinct entity classes. For the first stage, we achieve an overall performance with P=0.93, R=0.514 and F1=0.676, outperforming our baseline competitors significantly. For the second stage, we show that we can learn incrementally to determine the correct section for a news article based on section templates. The overall performance across different classes is P=0.844, R=0.885 and F1=0.860.","In the future, we will enhance our work by extracting facts from the suggested news articles. Results suggest that the news content cited in entity pages comes from the first paragraphs. However, challenging task such as the canonicalization and chronological ordering of facts, still remain."]},{"idx":14,"heading":"Figures and Tables","paragraphs":["Figure 1: Comparing how cyclones are reported in Wikipedia entity pages.","Figure 2: News suggestion approach overview.","Table 1: Article–Entity placement feature summary.","Table 2: Feature types used in Fs for suggesting news articles into the entity sections. We compute the features for all s ∈ Ŝc(t− 1) as well as se(t− 1).","Figure 3: Number of entities with at least one news reference for different entity classes.","Table 3: News articles, entities and sections distribution across years.","Table 4: Number of instances for train and test in the AEP and ASP tasks.","Table 5: Article–Entity placement task performance.","Figure 4: Precision-Recall curve for the article–entity placement task, in blue is shown Fe, and in red is the baseline B1.","Figure 5: Feature analysis for the AEP placement task for t = 2009.","Figure 6: Article-Section performance averaged for all entity classes for Fs (using SVM) and S2.","Figure 7: Correctly suggested news articles for s ∈ Se(t) ∧ s /∈ Se(t− 1).","Table 6: Article-Section placement performance (with Fs learned through SVM) for the different entity classes. The results show the standard P/R/F1."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1704.03279","title":"Unfolding and Shrinking Neural Machine Translation Ensembles","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Ensembling is a well-known technique in neural machine translation (NMT) to improve system performance. Instead of a single neural net, multiple neural nets with the same topology are trained separately, and the decoder generates predictions by averaging over the individual models. Ensembling often improves the quality of the generated translations drastically. However, it is not suitable for production systems because it is cumbersome and slow. This work aims to reduce the runtime to be on par with a single system without compromising the translation quality. First, we show that the ensemble can be unfolded into a single large neural network which imitates the output of the ensemble system. We show that unfolding can already improve the runtime in practice since more work can be done on the GPU. We proceed by describing a set of techniques to shrink the unfolded network by reducing the dimensionality of layers. On Japanese-English we report that the resulting network has the size and decoding speed of a single NMT network but performs on the level of a 3-ensemble system."]},{"idx":1,"heading":"Introduction","paragraphs":["The top systems in recent machine translation evaluation campaigns on various language pairs use ensembles of a number of NMT systems BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 . Ensembling BIBREF7 , BIBREF8 of neural networks is a simple yet very effective technique to improve the accuracy of NMT. The decoder makes use of INLINEFORM0 NMT networks which are either trained independently BIBREF9 , BIBREF2 , BIBREF3 , BIBREF4 or share some amount of training iterations BIBREF10 , BIBREF1 , BIBREF5 , BIBREF6 . The ensemble decoder computes predictions from each of the individual models which are then combined using the arithmetic average BIBREF9 or the geometric average BIBREF5 .","Ensembling consistently outperforms single NMT by a large margin. However, the decoding speed is significantly worse since the decoder needs to apply INLINEFORM0 NMT models rather than only one. Therefore, a recent line of research transfers the idea of knowledge distillation BIBREF11 , BIBREF12 to NMT and trains a smaller network (the student) by minimizing the cross-entropy to the output of the ensemble system (the teacher) BIBREF13 , BIBREF14 . This paper presents an alternative to knowledge distillation as we aim to speed up decoding to be comparable to single NMT while retaining the boost in translation accuracy from the ensemble. In a first step, we describe how to construct a single large neural network which imitates the output of an ensemble of multiple networks with the same topology. We will refer to this process as unfolding. GPU-based decoding with the unfolded network is often much faster than ensemble decoding since more work can be done on the GPU. In a second step, we explore methods to reduce the size of the unfolded network. This idea is justified by the fact that ensembled neural networks are often over-parameterized and have a large degree of redundancy BIBREF15 , BIBREF16 , BIBREF17 . Shrinking the unfolded network leads to a smaller model which consumes less space on the disk and in the memory; a crucial factor on mobile devices. More importantly, the decoding speed on all platforms benefits greatly from the reduced number of neurons. We find that the dimensionality of linear embedding layers in the NMT network can be reduced heavily by low-rank matrix approximation based on singular value decomposition (SVD). This suggest that high dimensional embedding layers may be needed for training, but do not play an important role for decoding. The NMT network, however, also consists of complex layers like gated recurrent units BIBREF18 and attention BIBREF19 . Therefore, we introduce a novel algorithm based on linear combinations of neurons which can be applied either during training (data-bound) or directly on the weight matrices without using training data (data-free). We report that with a mix of the presented shrinking methods we are able to reduce the size of the unfolded network to the size of the single NMT network while keeping the boost in BLEU score from the ensemble. Depending on the aggressiveness of shrinking, we report either a gain of 2.2 BLEU at the same decoding speed, or a 3.4 INLINEFORM1 CPU decoding speed up with only a minor drop in BLEU compared to the original single NMT system. Furthermore, it is often much easier to stage a single NMT system than an ensemble in a commercial MT workflow, and it is crucial to be able to optimize quality at specific speed and memory constraints. Unfolding and shrinking address these problems directly."]},{"idx":2,"heading":"Unfolding KK Networks into a Single Large Neural Network","paragraphs":["The first concept of our approach is called unfolding. Unfolding is an alternative to ensembling of multiple neural networks with the same topology. Rather than averaging their predictions, unfolding constructs a single large neural net out of the individual models which has the same number of input and output neurons but larger inner layers. Our main motivation for unfolding is to obtain a single network with ensemble level performance which can be shrunk with the techniques in Sec. SECREF3 .","Suppose we ensemble two single layer feedforward neural nets as shown in Fig. FIGREF1 . Normally, ensembling is implemented by performing an isolated forward pass through the first network (Fig. SECREF2 ), another isolated forward pass through the second network (Fig. SECREF3 ), and averaging the activities in the output layers of both networks. This can be simulated by merging both networks into a single large network as shown in Fig. SECREF4 . The first neurons in the hidden layer of the combined network correspond to the hidden layer in the first single network, and the others to the hidden layer of the second network. A single pass through the combined network yields the same output as the ensemble if the output layer is linear (up to a factor 2). The weight matrices in the unfolded network can be constructed by stacking the corresponding weight matrices (either horizontally or vertically) in network 1 and 2. This kind of aggregation of multiple networks with the same topology is not only possible for single-layer feedforward architectures but also for complex networks consisting of multiple GRU layers and attention.","For a formal description of unfolding we address layers with indices INLINEFORM0 . The special layer 0 has a single neuron for modelling bias vectors. Layer 1 holds the input neurons and layer INLINEFORM1 is the output layer. We denote the size of a layer in the individual models as INLINEFORM2 . When combining INLINEFORM3 networks, the layer size INLINEFORM4 in the unfolded network is increased by factor INLINEFORM5 if INLINEFORM6 is an inner layer, and equal to INLINEFORM7 if INLINEFORM8 is the input or output layer. We denote the weight matrix between two layers INLINEFORM9 in the INLINEFORM10 -th individual model ( INLINEFORM11 ) as INLINEFORM12 , and the corresponding weight matrix in the unfolded network as INLINEFORM13 . We explicitly allow INLINEFORM14 and INLINEFORM15 to be non-consecutive or reversed to be able to model recurrent networks. We use the zero-matrix if layers INLINEFORM16 and INLINEFORM17 are not connected. The construction of the unfolded weight matrix INLINEFORM18 from the individual matrices INLINEFORM19 depends on whether the connected layers are inner layers or not. The complete formula is listed in Fig. FIGREF5 .","Unfolded NMT networks approximate but do not exactly match the output of the ensemble due to two reasons. First, the unfolded network synchronizes the attentions of the individual models. Each decoding step in the unfolded network computes a single attention weight vector. In contrast, ensemble decoding would compute one attention weight vector for each of the INLINEFORM0 input models. A second difference is that the ensemble decoder first applies the softmax at the output layer, and then averages the prediction probabilities. The unfolded network averages the neuron activities (i.e. the logits) first, and then applies the softmax function. Interestingly, as shown in Sec. SECREF4 , these differences do not have any impact on the BLEU score but yield potential speed advantages of unfolding since the computationally expensive softmax layer is only applied once."]},{"idx":3,"heading":"Shrinking the Unfolded Network","paragraphs":["After constructing the weight matrices of the unfolded network we reduce the size of it by iteratively shrinking layer sizes. In this section we denote the incoming weight matrix of the layer to shrink as INLINEFORM0 and the outgoing weight matrix as INLINEFORM1 . Our procedure is inspired by the method of Srinivas and Babu sparsify-datafree. They propose a criterion for removing neurons in inner layers of the network based on two intuitions. First, similarly to Hebb's learning rule, they detect redundancy by the principle neurons which fire together, wire together. If the incoming weight vectors INLINEFORM2 and INLINEFORM3 are exactly the same for two neurons INLINEFORM4 and INLINEFORM5 , we can remove the neuron INLINEFORM6 and add its outgoing connections to neuron INLINEFORM7 ( INLINEFORM8 ) without changing the output. This holds since the activity in neuron INLINEFORM14 will always be equal to the activity in neuron INLINEFORM15 . In practice, Srinivas and Babu use a distance measure based on the difference of the incoming weight vectors to search for similar neurons as exact matches are very rare.","The second intuition of the criterion used by Srinivas and Babu sparsify-datafree is that neurons with small outgoing weights contribute very little overall. Therefore, they search for a pair of neurons INLINEFORM0 according the following term and remove the INLINEFORM1 -th neuron. DISPLAYFORM0","Neuron INLINEFORM0 is selected for removal if (1) there is another neuron INLINEFORM1 which has a very similar set of incoming weights and if (2) INLINEFORM2 has a small outgoing weight vector. Their criterion is data-free since it does not require any training data. For further details we refer to Srinivas and Babu sparsify-datafree."]},{"idx":4,"heading":"Data-Free Neuron Removal","paragraphs":["Srinivas and Babu sparsify-datafree propose to add the outgoing weights of INLINEFORM0 to the weights of a similar neuron INLINEFORM1 to compensate for the removal of INLINEFORM2 . However, we have found that this approach does not work well on NMT networks. We propose instead to compensate for the removal of a neuron by a linear combination of the remaining neurons in the layer. Data-free shrinking assumes for the sake of deriving the update rule that the neuron activation function is linear. We now ask the following question: How can we compensate as well as possible for the loss of neuron INLINEFORM3 such that the impact on the output of the whole network is minimized? Data-free shrinking represents the incoming weight vector of neuron INLINEFORM4 ( INLINEFORM5 ) as linear combination of the incoming weight vectors of the other neurons. The linear factors can be found by satisfying the following linear system: DISPLAYFORM0","where INLINEFORM0 is matrix INLINEFORM1 without the INLINEFORM2 -th column. In practice, we use the method of ordinary least squares to find INLINEFORM3 because the system may be overdetermined. The idea is that if we mix the outputs of all neurons in the layer by the INLINEFORM4 -weights, we get the output of the INLINEFORM5 -th neuron. The row vector INLINEFORM6 contains the contributions of the INLINEFORM7 -th neuron to each of the neurons in the next layer. Rather than using these connections, we approximate their effect by adding some weight to the outgoing connections of the other neurons. How much weight depends on INLINEFORM8 and the outgoing weights INLINEFORM9 . The factor INLINEFORM10 which we need to add to the outgoing connection of the INLINEFORM11 -th neuron to compensate for the loss of the INLINEFORM12 -th neuron on the INLINEFORM13 -th neuron in the next layer is: DISPLAYFORM0","Therefore, the update rule for INLINEFORM0 is: DISPLAYFORM0","In the remainder we will refer to this method as data-free shrinking. Note that we recover the update rule of Srinivas and Babu sparsify-datafree by setting INLINEFORM0 to the INLINEFORM1 -th unit vector. Also note that the error introduced by our shrinking method is due to the fact that we ignore the non-linearity, and that the solution for INLINEFORM2 may not be exact. The method is error-free on linear layers as long as the residuals of the least-squares analysis in Eq. EQREF10 are zero.","The terminology of neurons needs some further elaboration for GRU layers which rather consist of update and reset gates and states BIBREF18 . On GRU layers, we treat the states as neurons, i.e. the INLINEFORM0 -th neuron refers to the INLINEFORM1 -th entry in the GRU state vector. Input connections to the gates are included in the incoming weight matrix INLINEFORM2 for estimating INLINEFORM3 in Eq. EQREF10 . Removing neuron INLINEFORM4 in a GRU layer means deleting the INLINEFORM5 -th entry in the states and both gate vectors."]},{"idx":5,"heading":"Data-Bound Neuron Removal","paragraphs":["Although we find our data-free approach to be a substantial improvement over the methods of Srinivas and Babu sparsify-datafree on NMT networks, it still leads to a non-negligible decline in BLEU score when applied to recurrent GRU layers. Our data-free method uses the incoming weights to identify similar neurons, i.e. neurons expected to have similar activities. This works well enough for simple layers, but the interdependencies between the states and the gates inside gated layers like GRUs or LSTMs are complex enough that redundancies cannot be found simply by looking for similar weights. In the spirit of Babaeizadeh et al. sparsify-noiseout, our data-bound version records neuron activities during training to estimate INLINEFORM0 . We compensate for the removal of the INLINEFORM1 -th neuron by using a linear combination of the output of remaining neurons with similar activity patterns. In each layer, we prune 40 neurons each 450 training iterations until the target layer size is reached. Let INLINEFORM2 be the matrix which holds the records of neuron activities in the layer since the last removal. For example, for the decoder GRU layer, a batch size of 80, and target sentence lengths of 20, INLINEFORM3 has INLINEFORM4 rows and INLINEFORM5 (the number of neurons in the layer) columns. Similarly to Eq. EQREF10 we find interpolation weights INLINEFORM6 using the method of least squares on the following linear system. DISPLAYFORM0","The update rule for the outgoing weight matrix is the same as for our data-free method (Eq. EQREF12 ). The key difference between data-free and data-bound shrinking is the way INLINEFORM0 is estimated. Data-free shrinking uses the similarities between incoming weights, and data-bound shrinking uses neuron activities recorded during training. Once we select a neuron to remove, we estimate INLINEFORM1 , compensate for the removal, and proceed with the shrunk network. Both methods are prior to any decoding and result in shrunk parameter files which are then loaded to the decoder. Both methods remove neurons rather than single weights.","The data-bound algorithm runs gradient-based optimization on the unfolded network. We use the AdaGrad BIBREF20 step rule, a small learning rate of 0.0001, and aggressive step clipping at 0.05 to avoid destroying useful weights which were learned in the individual networks prior to the construction of the unfolded network.","Our data-bound algorithm uses a data-bound version of the neuron selection criterion in Eq. EQREF8 which operates on the activity matrix INLINEFORM0 . We search for the pair INLINEFORM1 according the following term and remove neuron INLINEFORM2 . DISPLAYFORM0"]},{"idx":6,"heading":"Shrinking Embedding Layers with SVD","paragraphs":["The standard attention-based NMT network architecture BIBREF19 includes three linear layers: the embedding layer in the encoder, and the output and feedback embedding layers in the decoder. We have found that linear layers are particularly easy to shrink using low-rank matrix approximation. As before we denote the incoming weight matrix as INLINEFORM0 and the outgoing weight matrix as INLINEFORM1 . Since the layer is linear, we could directly connect the previous layer with the next layer using the product of both weight matrices INLINEFORM2 . However, INLINEFORM3 may be very large. Therefore, we approximate INLINEFORM4 as a product of two low rank matrices INLINEFORM5 and INLINEFORM6 ( INLINEFORM7 ) where INLINEFORM8 is the desired layer size. A very common way to find such a matrix factorization is using truncated singular value decomposition (SVD). The layer is eventually shrunk by replacing INLINEFORM9 with INLINEFORM10 and INLINEFORM11 with INLINEFORM12 ."]},{"idx":7,"heading":"Results","paragraphs":["The individual NMT systems we use as source for constructing the unfolded networks are trained using AdaDelta BIBREF21 on the Blocks/Theano implementation BIBREF22 , BIBREF23 of the standard attention-based NMT model BIBREF19 with: 1000 dimensional GRU layers BIBREF18 in both the decoder and bidrectional encoder; a single maxout output layer BIBREF24 ; and 620 dimensional embedding layers. We follow Sennrich et al. nmt-bpe and use subword units based on byte pair encoding rather than words as modelling units. Our SGNMT decoder BIBREF25 with a beam size of 12 is used in all experiments. Our primary corpus is the Japanese-English (Ja-En) ASPEC data set BIBREF26 . We select a subset of 500K sentence pairs to train our models as suggested by Neubig et al. sys-neubig-wat15. We report cased BLEU scores calculated with Moses' multi-bleu.pl to be strictly comparable to the evaluation done in the Workshop of Asian Translation (WAT). We also apply our method to the WMT data set for English-German (En-De), using the news-test2014 as a development set, and keeping news-test2015 and news-test2016 as test sets. En-De BLEU scores are computed using mteval-v13a.pl as in the WMT evaluation. We set the vocabulary sizes to 30K for Ja-En and 50K for En-De. We also report the size factor for each model which is the total number of model parameters (sum of all weight matrix sizes) divided by the number of parameters in the original NMT network (86M for Ja-En and 120M for En-De). We choose a widely used, simple ensembling method (prediction averaging) as our baseline. We feel that the prevalence of this method makes it a reasonable baseline for our experiments."]},{"idx":8,"heading":"Related Work","paragraphs":["The idea of pruning neural networks to improve the compactness of the models dates back more than 25 years BIBREF15 . The literature is therefore vast BIBREF28 . One line of research aims to remove unimportant network connections. The connections can be selected for deletion based on the second-derivative of the training error with respect to the weight BIBREF15 , BIBREF16 , or by a threshold criterion on its magnitude BIBREF29 . See et al. sparsify-nmt confirmed a high degree of weight redundancy in NMT networks.","In this work we are interested in removing neurons rather than single connections since we strive to shrink the unfolded network such that it resembles the layout of an individual model. We argued in Sec. SECREF4 that removing neurons rather than connections does not only improve the model size but also the memory footprint and decoding speed. As explained in Sec. SECREF9 , our data-free method is an extension of the approach by Srinivas and Babu sparsify-datafree; our extension performs significantly better on NMT networks. Our data-bound method (Sec. SECREF14 ) is inspired by Babaeizadeh et al. sparsify-noiseout as we combine neurons with similar activities during training, but we use linear combinations of multiple neurons to compensate for the loss of a neuron rather than merging pairs of neurons.","Using low rank matrices for neural network compression, particularly approximations via SVD, has been studied widely in the literature BIBREF30 , BIBREF31 , BIBREF32 , BIBREF33 , BIBREF34 . These approaches often use low rank matrices to approximate a full rank weight matrix in the original network. In contrast, we shrink an entire linear layer by applying SVD on the product of the incoming and outgoing weight matrices (Sec. SECREF18 ).","In this paper we mimicked the output of the high performing but cumbersome ensemble by constructing a large unfolded network, and shrank this network afterwards. Another approach, known as knowledge distillation, uses the large model (the teacher) to generate soft training labels for the smaller student network BIBREF11 , BIBREF12 . The student network is trained by minimizing the cross-entropy to the teacher. This idea has been applied to sequence modelling tasks such as machine translation and speech recognition BIBREF35 , BIBREF13 , BIBREF14 . Our approach can be computationally more efficient as the training set does not have to be decoded by the large teacher network.","Junczys-Dowmunt et al. averaging2,averaging1 reported gains from averaging the weight matrices of multiple checkpoints of the same training run. However, our attempts to replicate their approach were not successful. Averaging might work well when the behaviour of corresponding units is similar across networks, but that cannot be guaranteed when networks are trained independently."]},{"idx":9,"heading":"Conclusion","paragraphs":["We have described a generic method for improving the decoding speed and BLEU score of single system NMT. Our approach involves unfolding an ensemble of multiple systems into a single large neural network and shrinking this network by removing redundant neurons. Our best results on Japanese-English either yield a gain of 2.2 BLEU compared to the original single NMT network at about the same decoding speed, or a INLINEFORM0 CPU decoding speed up with only a minor drop in BLEU.","The current formulation of unfolding works for networks of the same topology as the concatenation of layers is only possible for analogous layers in different networks. Unfolding and shrinking diverse networks could be possible, for example by applying the technique only to the input and output layers or by some other scheme of finding associations between units in different models, but we leave this investigation to future work as models in NMT ensembles in current research usually have the same topology BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF6 ."]},{"idx":10,"heading":"Acknowledgments","paragraphs":["This work was supported by the U.K. Engineering and Physical Sciences Research Council (EPSRC grant EP/L027623/1)."]},{"idx":11,"heading":"Appendix: Probabilistic Interpretation of Data-Free and Data-Bound Shrinking","paragraphs":["Data-free and data-bound shrinking can be interpreted as setting the expected difference between network outputs before and after a removal operation to zero under different assumptions.","For simplicity, we focus our probabilistic treatment of shrinking on single layer feedforward networks. Such a network maps an input INLINEFORM0 to an output INLINEFORM1 . The INLINEFORM2 -th output INLINEFORM3 is computed according the following equation DISPLAYFORM0","where INLINEFORM0 is the incoming weight vector of the INLINEFORM1 -th hidden neuron (denoted as INLINEFORM2 in the main paper) and INLINEFORM3 the outgoing weight matrix of the INLINEFORM4 -dimensional hidden layer. We now remove the INLINEFORM5 -th neuron in the hidden layer and modify the outgoing weights to compensate for the removal: DISPLAYFORM0","where INLINEFORM0 is the output after the removal operation and INLINEFORM1 are the modified outgoing weights. Our goal is to choose INLINEFORM2 such that the expected error introduced by removing neuron INLINEFORM3 is zero: DISPLAYFORM0"]},{"idx":12,"heading":"Figures and Tables","paragraphs":["Figure 1: Unfolding mimics the output of the ensemble of two single layer feedforward networks.","Figure 2: General formula for unfolding weight matrices. The set InnerLayers := [2, D− 1] includes all layers except the input, output, and bias layer.","Table 1: Shrinking layers of the unfolded network on Ja-En to their original size.","Table 2: Compensating for neuron removal in the data-bound algorithm. Row (d) corresponds to row (f) in Tab. 1.","Table 3: Time measurements on Ja-En. Layers are shrunk to their size in the original NMT model.","Table 5: Our best models on Ja-En.","Figure 3: Impact of shrinking on the BLEU score.","Table 4: Layer sizes of our setups for Ja-En.","Table 6: Our best models on En-De."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1704.08390","title":"Duluth at SemEval-2017 Task 6: Language Models in Humor Detection","sections":[{"idx":0,"heading":"Abstract","paragraphs":["This paper describes the Duluth system that participated in SemEval-2017 Task 6 #HashtagWars: Learning a Sense of Humor. The system participated in Subtasks A and B using N-gram language models, ranking highly in the task evaluation. This paper discusses the results of our system in the development and evaluation stages and from two post-evaluation runs."]},{"idx":1,"heading":"Introduction","paragraphs":["Humor is an expression of human uniqueness and intelligence and has drawn attention in diverse areas such as linguistics, psychology, philosophy and computer science. Computational humor draws from all of these fields and is a relatively new area of study. There is some history of systems that are able to generate humor (e.g., BIBREF0 , BIBREF1 ). However, humor detection remains a less explored and challenging problem (e.g., BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 ).","SemEval-2017 Task 6 BIBREF6 also focuses on humor detection by asking participants to develop systems that learn a sense of humor from the Comedy Central TV show, @midnight with Chris Hardwick. Our system ranks tweets according to how funny they are by training N-gram language models on two different corpora. One consisting of funny tweets provided by the task organizers, and the other on a freely available research corpus of news data. The funny tweet data is made up of tweets that are intended to be humorous responses to a hashtag given by host Chris Hardwick during the program."]},{"idx":2,"heading":"Background","paragraphs":["Training Language Models (LMs) is a straightforward way to collect a set of rules by utilizing the fact that words do not appear in an arbitrary order; we in fact can gain useful information about a word by knowing the company it keeps BIBREF7 . A statistical language model estimates the probability of a sequence of words or an upcoming word. An N-gram is a contiguous sequence of N words: a unigram is a single word, a bigram is a two-word sequence, and a trigram is a three-word sequence. For example, in the tweet","tears in Ramen #SingleLifeIn3Words","“tears”, “in”, “Ramen” and “#SingleLifeIn3Words” are unigrams; “tears in”, “in Ramen” and “Ramen #SingleLifeIn3Words” are bigrams and “tears in Ramen” and “in Ramen #SingleLifeIn3Words” are trigrams.","An N-gram model can predict the next word from a sequence of N-1 previous words. A trigram Language Model (LM) predicts the conditional probability of the next word using the following approximation: DISPLAYFORM0","The assumption that the probability of a word depends only on a small number of previous words is called a Markov assumption BIBREF8 . Given this assumption the probability of a sentence can be estimated as follows: DISPLAYFORM0","In a study on how phrasing affects memorability, BIBREF9 take a language model approach to measure the distinctiveness of memorable movie quotes. They do this by evaluating a quote with respect to a “common language” model built from the newswire sections of the Brown corpus BIBREF10 . They find that movie quotes which are less like “common language” are more distinctive and therefore more memorable. The intuition behind our approach is that humor should in some way be memorable or distinct, and so tweets that diverge from a “common language” model would be expected to be funnier.","In order to evaluate how funny a tweet is, we train language models on two datasets: the tweet data and the news data. Tweets that are more probable according to the tweet data language model are ranked as being funnier. However, tweets that have a lower probability according to the news language model are considered the funnier since they are the least like the (unfunny) news corpus. We relied on both bigrams and trigrams when training our models.","We use KenLM BIBREF11 as our language modeling tool. Language models are estimated using modified Kneser-Ney smoothing without pruning. KenLM also implements a back-off technique so if an N-gram is not found, KenLM applies the lower order N-gram's probability along with its back-off weights."]},{"idx":3,"heading":"Method","paragraphs":["Our system estimated tweet probability using N-gram LMs. Specifically, it solved the comparison (Subtask A) and semi-ranking (Subtask B) subtasks in four steps:"]},{"idx":4,"heading":"Corpus Preparation and Pre-processing","paragraphs":["The tweet data was provided by the task organizers. It consists of 106 hashtag files made up of about 21,000 tokens. The hashtag files were further divided into a development set trial_dir of 6 hashtags and a training set of 100 hashtags train_dir. We also obtained 6.2 GB of English news data with about two million tokens from the News Commentary Corpus and the News Crawl Corpus from 2008, 2010 and 2011. Each tweet and each sentence from the news data is found on a single line in their respective files.","During the development of our system we trained our language models solely on the 100 hashtag files from train_dir and then evaluated our performance on the 6 hashtag files found in trial_dir. That data was formatted such that each tweet was found on a single line.","Pre-processing consists of two steps: filtering and tokenization. The filtering step was only for the tweet training corpus. We experimented with various filtering and tokenziation combinations during the development stage to determine the best setting.","Filtering removes the following elements from the tweets: URLs, tokens starting with the “@” symbol (Twitter user names), and tokens starting with the “#” symbol (Hashtags).","Tokenization: Text in all training data was split on white space and punctuation"]},{"idx":5,"heading":"Language Model Training","paragraphs":["Once we had the corpora ready, we used the KenLM Toolkit to train the N-gram language models on each corpus. We trained using both bigrams and trigrams on the tweet and news data. Our language models accounted for unknown words and were built both with and without considering sentence or tweet boundaries."]},{"idx":6,"heading":"Tweet Scoring","paragraphs":["After training the N-gram language models, the next step was scoring. For each hashtag file that needed to be evaluated, the logarithm of the probability was assigned to each tweet in the hashtag file based on the trained language model. The larger the probability, the more likely that tweet was according to the language model. Table 1 shows an example of two scored tweets from hashtag file Bad_Job_In_5_Words.tsv based on the tweet data trigram language model. Note that KenLM reports the log of the probability of the N-grams rather than the actual probabilities so the value closer to 0 (-19) has the higher probability and is associated with the tweet judged to be funnier."]},{"idx":7,"heading":"Tweet Prediction","paragraphs":["The system sorts all the tweets for each hashtag and orders them based on their log probability score, where the funniest tweet should be listed first. If the scores are based on the tweet language model then they are sorted in ascending order since the log probability value closest to 0 indicates the tweet that is most like the (funny) tweets model. However, if the log probability scores are based on the news data then they are sorted in descending order since the largest value will have the smallest probability associated with it and is therefore least like the (unfunny) news model.","For Subtask A, the system goes through the sorted list of tweets in a hashtag file and compares each pair of tweets. For each pair, if the first tweet was funnier than the second, the system would output the tweet_ids for the pair followed by a “1”. If the second tweet is funnier it outputs the tweet_ids followed by a “0”. For Subtask B, the system outputs all the tweet_ids for a hashtag file starting from the funniest."]},{"idx":8,"heading":"Experiments and Results","paragraphs":["In this section we present the results from our development stage (Table 2), the evaluation stage (Table 3), and two post-evaluation results (Table 3). Since we implemented both bigram and trigam language models during the development stage but only results from trigram language models were submitted to the task, we evaluated bigram language models in the post-evaluation stage. Note that the accuracy and distance measurements listed in Table 2 and Table 3 are defined by the task organizers BIBREF6 .","Table 2 shows results from the development stage. These results show that for the tweet data the best setting is to keep the # and @, omit sentence boundaries, be case sensitive, and ignore tokenization. While using these settings the trigram language model performed better on Subtask B (.887) and the bigram language model performed better on Subtask A (.548). We decided to rely on trigram language models for the task evaluation since the advantage of bigrams on Subtask A was very slight (.548 versus .543). For the news data, we found that the best setting was to perform tokenization, omit sentence boundaries, and to be case sensitive. Given that trigrams performed most effectively in the development stage, we decided to use those during the evaluation.","Table 3 shows the results of our system during the task evaluation. We submitted two runs, one with a trigram language model trained on the tweet data, and another with a trigram language model trained on the news data. In addition, after the evaluation was concluded we also decided to run the bigram language models as well. Contrary to what we observed in the development data, the bigram language model actually performed somewhat better than the trigram language model. In addition, and also contrary to what we observed with the development data, the news data proved generally more effective in the post–evaluation runs than the tweet data."]},{"idx":9,"heading":"Discussion and Future Work","paragraphs":["We relied on bigram and trigram language models because tweets are short and concise, and often only consist of just a few words.","The performance of our system was not consistent when comparing the development to the evaluation results. During development language models trained on the tweet data performed better. However during the evaluation and post-evaluation stage, language models trained on the news data were significantly more effective. We also observed that bigram language models performed slightly better than trigram models on the evaluation data. This suggests that going forward we should also consider both the use of unigram and character–level language models.","These results suggest that there are only slight differences between bigram and trigram models, and that the type and quantity of corpora used to train the models is what really determines the results.","The task description paper BIBREF6 reported system by system results for each hashtag. We were surprised to find that our performance on the hashtag file #BreakUpIn5Words in the evaluation stage was significantly better than any other system on both Subtask A (with accuracy of 0.913) and Subtask B (with distance score of 0.636). While we still do not fully understand the cause of these results, there is clearly something about the language used in this hashtag that is distinct from the other hashtags, and is somehow better represented or captured by a language model. Reaching a better understanding of this result is a high priority for future work.","The tweet data was significantly smaller than the news data, and so certainly we believe that this was a factor in the performance during the evaluation stage, where the models built from the news data were significantly more effective. Going forward we plan to collect more tweet data, particularly those that participate in #HashtagWars. We also intend to do some experiments where we cut the amount of news data and then build models to see how those compare.","While our language models performed well, there is some evidence that neural network models can outperform standard back-off N-gram models BIBREF12 . We would like to experiment with deep learning methods such as recurrent neural networks, since these networks are capable of forming short term memory and may be better suited for dealing with sequence data."]},{"idx":10,"heading":"Figures and Tables","paragraphs":["Table 1: Scored tweets according to the trigram LM. The log probability scores computed based on the trigram LM are shown in the third column.","Table 2: Development results based on trial dir data. The settings we chose to train LMs are in bold.","Table 3: Evaluation results (bold) and post-evaluation results based on evaluation dir data. The trigram LM trained on the news data ranked 4th place on Subtask A and 1st place on Subtask B."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1705.00571","title":"Lancaster A at SemEval-2017 Task 5: Evaluation metrics matter: predicting sentiment from financial news headlines","sections":[{"idx":0,"heading":"Abstract","paragraphs":["This paper describes our participation in Task 5 track 2 of SemEval 2017 to predict the sentiment of financial news headlines for a specific company on a continuous scale between -1 and 1. We tackled the problem using a number of approaches, utilising a Support Vector Regression (SVR) and a Bidirectional Long Short-Term Memory (BLSTM). We found an improvement of 4-6% using the LSTM model over the SVR and came fourth in the track. We report a number of different evaluations using a finance specific word embedding model and reflect on the effects of using different evaluation metrics."]},{"idx":1,"heading":"Introduction","paragraphs":["The objective of Task 5 Track 2 of SemEval semeval20175 was to predict the sentiment of news headlines with respect to companies mentioned within the headlines. This task can be seen as a finance-specific aspect-based sentiment task BIBREF0 . The main motivations of this task is to find specific features and learning algorithms that will perform better for this domain as aspect based sentiment analysis tasks have been conducted before at SemEval BIBREF1 .","Domain specific terminology is expected to play a key part in this task, as reporters, investors and analysts in the financial domain will use a specific set of terminology when discussing financial performance. Potentially, this may also vary across different financial domains and industry sectors. Therefore, we took an exploratory approach and investigated how various features and learning algorithms perform differently, specifically SVR and BLSTMs. We found that BLSTMs outperform an SVR without having any knowledge of the company that the sentiment is with respect to. For replicability purposes, with this paper we are releasing our source code and the finance specific BLSTM word embedding model."]},{"idx":2,"heading":"Related Work","paragraphs":["There is a growing amount of research being carried out related to sentiment analysis within the financial domain. This work ranges from domain-specific lexicons BIBREF2 and lexicon creation BIBREF3 to stock market prediction models BIBREF4 , BIBREF5 . BIBREF4 used a multi layer neural network to predict the stock market and found that incorporating textual features from financial news can improve the accuracy of prediction. BIBREF5 showed the importance of tuning sentiment analysis to the task of stock market prediction. However, much of the previous work was based on numerical financial stock market data rather than on aspect level financial textual data. In aspect based sentiment analysis, there have been many different techniques used to predict the polarity of an aspect as shown in SemEval-2016 task 5 BIBREF1 . The winning system BIBREF6 used many different linguistic features and an ensemble model, and the runner up BIBREF7 used uni-grams, bi-grams and sentiment lexicons as features for a Support Vector Machine (SVM). Deep learning methods have also been applied to aspect polarity prediction. BIBREF8 created a hierarchical BLSTM with a sentence level BLSTM inputting into a review level BLSTM thus allowing them to take into account inter- and intra-sentence context. They used only word embeddings making their system less dependent on extensive feature engineering or manual feature creation. This system outperformed all others on certain languages on the SemEval-2016 task 5 dataset BIBREF1 and on other languages performed close to the best systems. BIBREF9 also created an LSTM based model using word embeddings but instead of a hierarchical model it was a one layered LSTM with attention which puts more emphasis on learning the sentiment of words specific to a given aspect."]},{"idx":3,"heading":"Data","paragraphs":["The training data published by the organisers for this track was a set of headline sentences from financial news articles where each sentence was tagged with the company name (which we treat as the aspect) and the polarity of the sentence with respect to the company. There is the possibility that the same sentence occurs more than once if there is more than one company mentioned. The polarity was a real value between -1 (negative sentiment) and 1 (positive sentiment).","We additionally trained a word2vec BIBREF10 word embedding model on a set of 189,206 financial articles containing 161,877,425 tokens, that were manually downloaded from Factiva. The articles stem from a range of sources including the Financial Times and relate to companies from the United States only. We trained the model on domain specific data as it has been shown many times that the financial domain can contain very different language."]},{"idx":4,"heading":"System description","paragraphs":["Even though we have outlined this task as an aspect based sentiment task, this is instantiated in only one of the features in the SVR. The following two subsections describe the two approaches, first SVR and then BLSTM. Key implementation details are exposed here in the paper, but we have released the source code and word embedding models to aid replicability and further experimentation."]},{"idx":5,"heading":"SVR","paragraphs":["The system was created using ScitKit learn BIBREF11 linear Support Vector Regression model BIBREF12 . We experimented with the following different features and parameter settings:","For comparison purposes, we tested whether or not a simple whitespace tokeniser can perform just as well as a full tokeniser, and in this case we used Unitok.","We compared word-level uni-grams and bi-grams separately and in combination.","We tested different penalty parameters C and different epsilon parameters of the SVR.","We tested replacements to see if generalising words by inserting special tokens would help to reduce the sparsity problem. We placed the word replacements into three separate groups:","Company - When a company was mentioned in the input headline from the list of companies in the training data marked up as aspects, it was replaced by a company special token.","Positive - When a positive word was mentioned in the input headline from a list of positive words (which was created using the N most similar words based on cosine distance) to `excellent' using the pre-trained word2vec model.","Negative - The same as the positive group however the word used was `poor' instead of `excellent'.","In the positive and negative groups, we chose the words `excellent' and `poor' following BIBREF13 to group the terms together under non-domain specific sentiment words.","In order to incorporated the company as an aspect, we employed a boolean vector to represent the sentiment of the sentence. This was done in order to see if the system could better differentiate the sentiment when the sentence was the same but the company was different."]},{"idx":6,"heading":"BLSTM","paragraphs":["We created two different Bidirectional BIBREF14 Long Short-Term Memory BIBREF15 using the Python Keras library BIBREF16 with tensor flow backend BIBREF17 . We choose an LSTM model as it solves the vanishing gradients problem of Recurrent Neural Networks. We used a bidirectional model as it allows us to capture information that came before and after instead of just before, thereby allowing us to capture more relevant context within the model. Practically, a BLSTM is two LSTMs one going forward through the tokens the other in reverse order and in our models concatenating the resulting output vectors together at each time step.","The BLSTM models take as input a headline sentence of size L tokens where L is the length of the longest sentence in the training texts. Each word is converted into a 300 dimension vector using the word2vec model trained over the financial text. Any text that is not recognised by the word2vec model is represented as a vector of zeros; this is also used to pad out the sentence if it is shorter than L.","Both BLSTM models have the following similar properties:","Gradient clipping value of 5 - This was to help with the exploding gradients problem.","Minimised the Mean Square Error (MSE) loss using RMSprop with a mini batch size of 32.","The output activation function is linear.","The main difference between the two models is the use of drop out and when they stop training over the data (epoch). Both models architectures can be seen in figure FIGREF18 .","The BLSTMs do contain drop out in both the input and between the connections of 0.2 each. Finally the epoch is fixed at 25.","As can be seen from figure FIGREF18 , the drop out of 0.5 only happens between the layers and not the connections as in the SLSTM. Also the epoch is not fixed, it uses early stopping with a patience of 10. We expect that this model can generalise better than the standard one due to the higher drop out and that the epoch is based on early stopping which relies on a validation set to know when to stop training."]},{"idx":7,"heading":"Results","paragraphs":["We first present our findings on the best performing parameters and features for the SVRs. These were determined by cross validation (CV) scores on the provided training data set using cosine similarity as the evaluation metric. We found that using uni-grams and bi-grams performs best and using only bi-grams to be the worst. Using the Unitok tokeniser always performed better than simple whitespace tokenisation. The binary presence of tokens over frequency did not alter performance. The C parameter was tested for three values; 0.01, 0.1 and 1. We found very little difference between 0.1 and 1, but 0.01 produced much poorer results. The eplison parameter was tested for 0.001, 0.01 and 0.1 the performance did not differ much but the lower the higher the performance but the more likely to overfit. Using word replacements was effective for all three types (company, positive and negative) but using a value N=10 performed best for both positive and negative words. Using target aspects also improved results. Therefore, the best SVR model comprised of: Unitok tokenisation, uni- and bi- grams, word representation, C=0.1, eplison=0.01, company, positive, and negative word replacements and target aspects. DISPLAYFORM0","The main evaluation over the test data is based on the best performing SVR and the two BLSTM models once trained on all of the training data. The result table TABREF28 shows three columns based on the three evaluation metrics that the organisers have used. Metric 1 is the original metric, weighted cosine similarity (the metric used to evaluate the final version of the results, where we were ranked 5th; metric provided on the task website). This was then changed after the evaluation deadline to equation EQREF25 (which we term metric 2; this is what the first version of the results were actually based on, where we were ranked 4th), which then changed by the organisers to their equation as presented in BIBREF18 (which we term metric 3 and what the second version of the results were based on, where we were ranked 5th).","As you can see from the results table TABREF28 , the difference between the metrics is quite substantial. This is due to the system's optimisation being based on metric 1 rather than 2. Metric 2 is a classification metric for sentences with one aspect as it penalises values that are of opposite sign (giving -1 score) and rewards values with the same sign (giving +1 score). Our systems are not optimised for this because it would predict scores of -0.01 and true value of 0.01 as very close (within vector of other results) with low error whereas metric 2 would give this the highest error rating of -1 as they are not the same sign. Metric 3 is more similar to metric 1 as shown by the results, however the crucial difference is that again if you get opposite signs it will penalise more. We analysed the top 50 errors based on Mean Absolute Error (MAE) in the test dataset specifically to examine the number of sentences containing more than one aspect. Our investigation shows that no one system is better at predicting the sentiment of sentences that have more than one aspect (i.e. company) within them. Within those top 50 errors we found that the BLSTM systems do not know which parts of the sentence are associated to the company the sentiment is with respect to. Also they do not know the strength/existence of certain sentiment words."]},{"idx":8,"heading":"Conclusion and Future Work","paragraphs":["In this short paper, we have described our implemented solutions to SemEval Task 5 track 2, utilising both SVR and BLSTM approaches. Our results show an improvement of around 5% when using LSTM models relative to SVR. We have shown that this task can be partially represented as an aspect based sentiment task on a domain specific problem. In general, our approaches acted as sentence level classifiers as they take no target company into consideration. As our results show, the choice of evaluation metric makes a great deal of difference to system training and testing. Future work will be to implement aspect specific information into an LSTM model as it has been shown to be useful in other work BIBREF9 ."]},{"idx":9,"heading":"Acknowledgements","paragraphs":["We are grateful to Nikolaos Tsileponis (University of Manchester) and Mahmoud El-Haj (Lancaster University) for access to headlines in the corpus of financial news articles collected from Factiva. This research was supported at Lancaster University by an EPSRC PhD studentship."]},{"idx":10,"heading":"Figures and Tables","paragraphs":["Figure 1: Left hand side is the ELSTM model architecture and the right hand side shows the SLSTM. The numbers in the parenthesis represent the size of the output dimension where L is the length of the longest sentence.","Table 1: Results"]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1705.01265","title":"On the effectiveness of feature set augmentation using clusters of word embeddings","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Word clusters have been empirically shown to offer important performance improvements on various tasks. Despite their importance, their incorporation in the standard pipeline of feature engineering relies more on a trial-and-error procedure where one evaluates several hyper-parameters, like the number of clusters to be used. In order to better understand the role of such features we systematically evaluate their effect on four tasks, those of named entity segmentation and classification as well as, those of five-point sentiment classification and quantification. Our results strongly suggest that cluster membership features improve the performance."]},{"idx":1,"heading":"Introduction","paragraphs":["Many research attempts have proposed novel features that improve the performance of learning algorithms in particular tasks. Such features are often motivated by domain knowledge or manual labor. Although useful and often state-of-the-art, adapting such solutions on NLP systems across tasks can be tricky and time-consuming BIBREF0 . Therefore, simple yet general and powerful methods that perform well across several datasets are valuable BIBREF1 .","An approach that has become extremely popular lately in NLP tasks, is to train word embeddings in an unsupervised way. These embeddings are dense vectors that project words or short text spans like phrases in a vector space where dimensions are supposed to capture text properties. Such embeddings can then be used either as features with off-the-shelf algorithms like Support Vector Machines, or to initialize deep learning systems BIBREF2 . However, as shown in BIBREF3 linear architectures perform better in high-dimensional discrete spaces compared to continuous ones. The latter is probably the main reason of the high performance of the vector space model BIBREF4 in tasks like text classification with linear models like SVMs. Using linear algorithms, while taking advantage of the expressiveness of text embeddings is the focus of this work.","In this paper, we explore a hybrid approach, that uses text embeddings as a proxy to create features. Motivated by the argument that text embeddings manage to encode the semantics of text, we explore how clustering text embeddings can impact the performance of different NLP tasks. Although such an approach has been used in different studies during feature engineering, the selection of word vectors and the number of clusters remain a trial-end-error procedure. In this work we present an empirical evaluation across diverse tasks to verify whether and when such features are useful.","Word clusters have been used as features in various tasks like Part-of-Speech tagging and NER. Owoputi et al. Owoputi13 use Brown clusters BIBREF5 in a POS tagger showing that this type of features carry rich lexical knowledge as they can substitute lexical resources like gazetteers. Kiritchenko et al. KiritchenkoZM14 discusses their use on sentiment classification while Hee et al. HeeLH16 incorporate them in the task of irony detection in Twitter. Ritter et al. Ritter2011 inject also word clusters in a NER tagger. While these works show that word clusters are beneficial no clear guidelines can be concluded of how and when to use them.","In this work, we empirically demonstrate that using different types of embeddings on three NLP tasks with twitter data we manage to achieve better or near to the state-of-the art performance on three NLP tasks: (i) Named Entity Recognition (NER) segmentation, (ii) NER classification, (iii) fine-grained sentiment analysis and (iv) fine-grained sentiment quantification. For each of the three tasks, we achieve higher performance than without using features which indicates the effectiveness of the cluster membership features. Importantly, our evaluation compared to previous work BIBREF6 who focus on old and well studied datasets uses recent and challenging datasets composed by tweets. The obtained results across all the tasks permits us to reveal important aspects of the use of word clusters and therefore provide guidelines. Although our obtained scores are state-of-the-art, our analysis reveals that the performance in such tasks is far from perfect and, hence, identifies that there is still much space for improvement and future work."]},{"idx":2,"heading":"Word Clusters","paragraphs":["Word embeddings associate words with dense, low-dimensional vectors. Recently, several models have been proposed in order to obtain these embeddings. Among others, the skipgram (skipgram) model with negative sampling BIBREF7 , the continuous bag-of-words (cbow) model BIBREF7 and Glove (glove) BIBREF8 have been shown to be effective. Training those models requires no annotated data and can be done using big amounts of text. Such a model can be seen as a function INLINEFORM0 that projects a word INLINEFORM1 in a INLINEFORM2 -dimensional space: INLINEFORM3 , where INLINEFORM4 is predefined. Here, we focus on applications using data from Twitter, which pose several difficulties due to being particularly short, using creative vocabulary, abbreviations and slang.","For all the tasks in our experimental study, we use 36 millions English tweets collected between August and September 2017. A pre-processing step has been applied to replace URLs with a placeholder and to pad punctuation. The final vocabulary size was around 1.6 millions words. Additionally to the in-domain corpus we collected, we use GloVe vectors trained on Wikipedia articles in order to investigate the impact of out-of-domain word-vectors.","We cluster the embeddings with INLINEFORM0 -Means. The k-means clusters are initialized using “k-means++” as proposed in BIBREF9 , while the algorithm is run for 300 iterations. We try different values for INLINEFORM1 . For each INLINEFORM2 , we repeat the clustering experiment with different seed initialization for 10 times and we select the clustering result that minimizes the cluster inertia."]},{"idx":3,"heading":"Experimental Evaluation","paragraphs":["We evaluate the proposed approach for augmenting the feature space in four tasks: (i) NER segmentation, (ii) NER classification, (iii) fine-grained sentiment classification and (iv) fine-grained sentiment quantification. The next sections present the evaluation settings we used. For each of the tasks, we use the designated training sets to train the learning algorithms, and we report the scores of the evaluation measures used in the respective test parts."]},{"idx":4,"heading":"Named-Entity Recognition in Twitter","paragraphs":["NER concerns the classification of textual segments in a predefined set of categories, like persons, organization and locations. We use the data of the last competition in NER for Twitter which released as a part of the 2nd Workshop on Noisy User-generated Text BIBREF10 . More specifically, the organizers provided annotated tweets with 10 named-entity types (person, movie, sportsteam, product etc.) and the task comprised two sub-tasks: 1) the detection of entity bounds and 2) the classification of an entity into one of the 10 types. The evaluation measure for both sub-tasks is the F INLINEFORM0 measure.","The following is an example of a tweet which contains two named entities. Note that named entities may span several words in the text:","INLINEFORM0 tonite ... 90 's music .. oldskool night wiith INLINEFORM1","Our model for solving the task is a learning to search approach. More specifically we follow BIBREF11 which has been ranked 2nd among 10 participants in the aforementioned competition BIBREF10 . The model uses handcrafted features like n-grams, part-of-speech tags, capitalization and membership in gazetteers. The algorithm used belongs to the family of learning to search for structured prediction tasks BIBREF12 . These methods decompose the problem in a search space with states, actions and policies and then learn a hypothesis controlling a policy over the state-action space. The BIO encoding is used for attributing the corresponding labels to the tokens where B-type is used for the first token of the entity, I-type for inside tokens in case of multi-term entities and O for non entity tokens.","Tables TABREF6 and TABREF7 present the results for the different number of clusters across the three vector models used to induce the clusters. For all the experiments we keep the same parametrization for the learning algorithm and we present the performance of each run on the official test set.","Regarding the segmentation task we notice that adding word clusters as features improve the performance of the best model up to 1.1 F-score points while it boosts performance in the majority of cases. In only one case, for glove INLINEFORM0 vectors, there is a drop across all number of clusters used.","As for the number of clusters, the best results are generally obtained between 250 and 1000 classes for all word vector models. These dimensions seem to be sufficient for the three-class sub-task that we deal with. The different models of word vectors perform similarly and thus one cannot privilege a certain type of word vectors. Interestingly, the clusters learned on the Wikipedia GloVe vectors offer competitive performance with respect to the in-domain word vectors used for the other cases showing that one can rely to out-of-domain data for constructing such representations.","Concerning the classification task (Table TABREF7 ) we generally observe a drop in the performance of the tagger as we deal with 10 classes. This essentially corresponds to a multi-class problem with 21 classes: one for the non-entity type and two classes for each entity type. In this setting we notice that the best results are obtained in most cases for higher number of classes (1000 or 2000) possibly due to a better discriminatory power in higher dimensions. Note also, that in some cases the addition of word cluster features does not necessarily improve the performance. Contrary, it may degrade it as it is evident in the case of glove INLINEFORM0 word clusters. Like in the case of segmentation we do not observe a word vector model that clearly outperforms the rest. Finally, we note the same competitive performance of the Wikipedia word clusters and notably for the glove INLINEFORM1 clusters which obtain the best F1-score."]},{"idx":5,"heading":"Fine-grained Sentiment Analysis","paragraphs":["The task of fine grained sentiment classification consists in predicting the sentiment of an input text according to a five point scale (sentiment INLINEFORM0 {VeryNegative, Negative, Neutral, Positive, VeryPositive}). We use the setting of task 4 of SemEval2016 “Sentiment Analysis in Twitter” and the dataset released by the organizers for subtask 4 BIBREF13 .","In total, the training (resp. test) data consist of 9,070 (resp. 20,632) tweets.","The evaluation measure selected in BIBREF13 for the task in the macro-averaged Mean Absolute Error (MAE INLINEFORM0 ). It is a measure of error, hence lower values are better. The measure's goal is to take into account the order of the classes when penalizing the decision of a classifier. For instance, misclassifying a very negative example as very positive is a bigger mistake than classifying it as negative or neutral. Penalizing a classifier according to how far the predictions are from the true class is captured by MAE INLINEFORM1 BIBREF14 . Also, the advantage of using the macro- version instead of the standard version of the measure is the robustness against the class imbalance in the data.","Learning algorithm To demonstrate the efficiency of cluster membership features we rely on the system of BIBREF15 which was ranked 1st among 11 participants and uses a Logistic Regression as a learning algorithm. We follow the same feature extraction steps which consist of extracting n-gram and character n-gram features, part-of-speech counts as well as sentiment scores using standard sentiment lexicons such as the Bing Liu's BIBREF16 and the MPQA lexicons BIBREF17 . For the full description, we refer the interested reader to BIBREF15 .","To evaluate the performance of the proposed feature augmentation technique, we present in Table TABREF10 the macro-averaged Mean Absolute Error scores for different settings on the official test set of BIBREF13 . First, notice that the best score in the test data is achieved using cluster membership features, where the word embeddings are trained using the skipgram model. The achieved score improves the state-of-the art on the dataset, which to the best of our knowledge was by BIBREF15 . Also, note that the score on the test data improves for each type of embeddings used, which means that augmenting the feature space using cluster membership features helps the sentiment classification task.","Note, also, that using the clusters produced by the out-of-domain embeddings trained on wikipedia that were released as part of BIBREF8 performs surprisingly well. One might have expected their addition to hurt the performance. However, their value probably stems from the sheer amount of data used for their training as well as the relatively simple type of words (like awesome, terrible) which are discriminative for this task. Lastly, note that in each of the settings, the best results are achieved when the number of clusters is within INLINEFORM0 as in the NER tasks. Comparing the performance across the different embeddings, one cannot claim that a particular embedding performs better. It is evident though that augmenting the feature space with feature derived using the proposed method, preferably with in-domain data, helps the classification performance and reduces MAE INLINEFORM1 .","From the results of Table TABREF10 it is clear that the addition of the cluster membership features improves the sentiment classification performance. To better understand though why these clusters help, we manually examined a sample of the words associated with the clusters. To improve the eligibility of those results we first removed the hashtags and we filter the results using an English vocabulary. In Table TABREF11 we present sample words from two of the most characteristic clusters with respect to the task of sentiment classification. Notice how words with positive and negative meanings are put in the respective clusters."]},{"idx":6,"heading":"Fine-Grained Sentiment Quantification","paragraphs":["Quantification is the problem of estimating the prevalence of a class in a dataset. While classification concerns assigning a category to a single instance, like labeling a tweet with the sentiment it conveys, the goal of quantification is, given a set of instances, to estimate the relative frequency of single class. Therefore, sentiment quantification tries to answer questions like “Given a set of tweets about the new iPhone, what is the fraction of VeryPositive ones?”. In the rest, we show the effect of the features derived from the word embeddings clusters in the fine-grained classification problem, which was also part of the SemEval-2016 “Sentiment Analysis in Twitter” task BIBREF13 .","Learning Algorithm To perform the quantification task, we rely on a classify and count approach, which was shown effective in a related binary quantification problem BIBREF15 . The idea is that given a set of instances on a particular subject, one first classifies the instances and then aggregates the counts. To this end, we use the same feature representation steps and data with the ones used for fine grained classification (Section 3.2). Note that the data of the task are associated with subjects (described in full detail at BIBREF13 ), and, hence, quantification is performed for the tweets of a subject. For each of the five categories, the output of the approach is a 5-dimensional vector with the estimated prevalence of the categories.","The evaluation measure for the problem is the Earth Movers Distance (EMD) BIBREF18 . EMD is a measure of error, hence lower values are better. It assumes ordered categories, which in our problem is naturally defined. Further assuming that the distance of consecutive categories (e.g., Positive and VeryPositive) is 1, the measure is calculated by: INLINEFORM0","where INLINEFORM0 is number of categories (five in our case) and INLINEFORM1 and INLINEFORM2 are the true and predicted prevalence respectively BIBREF19 .","Results Table TABREF13 presents the results of augmenting the feature set with the proposed features. We use Logistic Regression as a base classifier for the classify and count approach. Notice the positive impact of the features in the performance in the task. Adding the features derived from clustering the embeddings consistently improves the performance. Interestingly, the best performance ( INLINEFORM0 ) is achieved using the out-of-domain vectors, as in the NER classification task. Also, notice how the approach improves over the state-of-the-art performance in the challenge ( INLINEFORM1 ) BIBREF13 , held by the method of BIBREF20 . The improvement over the method of BIBREF20 however, does not necessarily mean that classify and count performs better in the task. It implies that the feature set we used is richer, that in turn highlights the value of robust feature extraction mechanisms which is the subject of this paper."]},{"idx":7,"heading":"Conclusion","paragraphs":["We have shown empirically the effectiveness of incorporating cluster membership features in the feature extraction pipeline of Named-Entity recognition, sentiment classification and quantification tasks. Our results strongly suggest that incorporating cluster membership features benefit the performance in the tasks. The fact that the performance improvements are consistent in the four tasks we investigated, further highlights their usefulness, both for practitioners and researchers.","Although our study does not identify a clear winner with respect to the type of word vectors (skipgram, cbow, or GloVe), our findings suggest that one should first try skip-gram embeddings of low dimensionality ( INLINEFORM0 ) and high number of clusters (e.g., INLINEFORM1 ) as the results obtained using these settings are consistently competitive. Our results also suggest that using out-of-domain data, like Wikipedia articles in this case, to construct the word embeddings is a good practice, as the results we obtained with these vectors are also competitive. The positive of out-of-domain embeddings and their combination with in-domain ones remains to be further studied."]},{"idx":8,"heading":"Figures and Tables","paragraphs":["Table 1: Scores on F1-measure for named entities segmentation for the different word embeddings across different number of clusters. For each embedding type, we show its dimension and window size. For instance, glove40,w5 is 40-dimensional glove embeddings with window size 5.","Table 2: Results in terms of F1-score for named entities classification for the different word clusters across different number of clusters.","Table 3: MAEM scores (lower is better) for sentiment classification across different types of word embeddings and number of clusters.","Table 4: Sample from two clusters that were found useful for the sentiment classification. Words with positive or negative meaning are grouped together.","Table 5: Earth Movers Distance for fine-grained sentiment quantification across different types of word embeddings and number of clusters. The score in brackets denotes the best performance achieved in the challenge."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1705.01306","title":"Amobee at SemEval-2017 Task 4: Deep Learning System for Sentiment Detection on Twitter","sections":[{"idx":0,"heading":"Abstract","paragraphs":["This paper describes the Amobee sentiment analysis system, adapted to compete in SemEval 2017 task 4. The system consists of two parts: a supervised training of RNN models based on a Twitter sentiment treebank, and the use of feedforward NN, Naive Bayes and logistic regression classifiers to produce predictions for the different sub-tasks. The algorithm reached the 3rd place on the 5-label classification task (sub-task C)."]},{"idx":1,"heading":"Introduction","paragraphs":["Sentiment detection is the process of determining whether a text has a positive or negative attitude toward a given entity (topic) or in general. Detecting sentiment on Twitter—a social network where users interact via short 140-character messages, exchanging information and opinions—is becoming ubiquitous. Sentiment in Twitter messages (tweets) can capture the popularity level of political figures, ideas, brands, products and people. Tweets and other social media texts are challenging to analyze as they are inherently different; use of slang, mis-spelling, sarcasm, emojis and co-mentioning of other messages pose unique difficulties. Combined with the vast amount of Twitter data (mostly public), these make sentiment detection on Twitter a focal point for data science research.","SemEval is a yearly event in which teams compete in natural language processing tasks. Task 4 is concerned with sentiment analysis in Twitter; it contains five sub-tasks which include classification of tweets according to 2, 3 or 5 labels and quantification of sentiment distribution regarding topics mentioned in tweets; for a complete description of task 4 see BIBREF0 .","This paper describes our system and participation in all sub-tasks of SemEval 2017 task 4. Our system consists of two parts: a recurrent neural network trained on a private Twitter dataset, followed by a task-specific combination of model stacking and logistic regression classifiers.","The paper is organized as follows: section SECREF2 describes the training of RNN models, data being used and model selection; section SECREF3 describes the extraction of semantic features; section SECREF4 describes the task-specific workflows and scores. We review and summarize in section SECREF5 . Finally, section SECREF6 describes our future plans, mainly the development of an LSTM algorithm."]},{"idx":2,"heading":"RNN Models","paragraphs":["The first part of our system consisted of training recursive-neural-tensor-network (RNTN) models BIBREF1 ."]},{"idx":3,"heading":"Data","paragraphs":["Our training data for this part was created by taking a random sample from Twitter and having it manually annotated on a 5-label basis to produce fully sentiment-labeled parse-trees, much like the Stanford sentiment treebank. The sample contains twenty thousand tweets with sentiment distribution as following:"]},{"idx":4,"heading":"Preprocessing","paragraphs":["First we build a custom dictionary by means of crawling Wikipedia and extracting lists of brands, celebrities, places and names. The lists were then pruned manually. Then we define the following steps when preprocessing tweets:","Standard tokenization of the sentences, using the Stanford coreNLP tools BIBREF2 .","Word-replacement step using the Wiki dictionary with representative keywords.","Lemmatization, using coreNLP.","Emojis: removing duplicate emojis, clustering them according to sentiment and replacing them with representative keywords, e.g. “happy-emoji”.","Regex: removing duplicate punctuation marks, replacing URLs with a keyword, removing Camel casing.","Parsing: parts-of-speech and constituency parsing using a shift-reduce parser, which was selected for its speed over accuracy.","NER: using entity recognition annotator, replacing numbers, dates and locations with representative keywords.","Wiki: second step of word-replacement using our custom wiki dictionary."]},{"idx":5,"heading":"Training","paragraphs":["We used the Stanford coreNLP sentiment annotator, introduced by BIBREF1 . Words are initialized either randomly as INLINEFORM0 dimensional vectors, or given externally as word vectors. We used four versions of the training data; with and without lemmatization and with and without pre-trained word representations BIBREF3 ."]},{"idx":6,"heading":"Tweet Aggregation","paragraphs":["Twitter messages can be comprised of several sentences, with different and sometimes contrary sentiments. However, the trained models predict sentiment on individual sentences. We aggregated the sentiment for each tweet by taking a linear combination of the individual sentences comprising the tweet with weights having the following power dependency: DISPLAYFORM0","where INLINEFORM0 are numerical factors to be found, INLINEFORM1 are the fraction of known words, length of the sentence and polarity, respectively, with polarity defined by: DISPLAYFORM0","where vn, n, p, vp are the probabilities as assigned by the RNTN for very-negative, negative, positive and very-positive label for each sentence. We then optimized the parameters INLINEFORM0 with respect to the true labels."]},{"idx":7,"heading":"Model Selection","paragraphs":["After training dozens of models, we chose to combine only the best ones using stacking, namely combining the models output using a supervised learning algorithm. For this purpose, we used the Scikit-learn BIBREF4 recursive feature elimination (RFE) algorithm to find both the optimal number and the actual models, thus choosing the best five models. The models chosen include a representative from each type of the data we used and they were:","Training data without lemmatization step, with randomly initialized word-vectors of size 27.","Training data with lemmatization step, with pre-trained word-vectors of size 25.","3 sets of training data with lemmatization step, with randomly initialized word-vectors of sizes 24, 26.","The five models output is concatenated and used as input for the various tasks, as described in SECREF27 ."]},{"idx":8,"heading":"Features Extraction","paragraphs":["In addition to the RNN trained models, our system includes feature extraction step; we defined a set of lexical and semantical features to be extracted from the original tweets:","For this purpose, we used the Stanford deterministic coreference resolution system BIBREF5 , BIBREF6 ."]},{"idx":9,"heading":"Experiments","paragraphs":["The experiments were developed by using Scikit-learn machine learning library and Keras deep learning library with TensorFlow backend BIBREF7 . Results for all sub-tasks are summarized in table","TABREF26 ."]},{"idx":10,"heading":"General Workflow","paragraphs":["For each tweet, we first ran the RNN models and got a 5-category probability distribution from each of the trained models, thus a 25-dimensional vector. Then we extracted sentence features and concatenated them with the RNN vector. We then trained a Feedforward NN which outputs a 5-label probability distribution for each tweet. That was the starting point for each of the tasks; we refer to this process as the pipeline."]},{"idx":11,"heading":"Task A","paragraphs":["The goal of this task is to classify tweets sentiment into three classes (negative, neutral, positive) where the measured metric is a macro-averaged recall.","We used the SemEval 2017 task A data in the following way: using SemEval 2016 TEST as our TEST, partitioning the rest into TRAIN and DEV datasets. The test dataset went through the previously mentioned pipeline, getting a 5-label probability distribution.","We anticipated the sentiment distribution of the test data would be similar to the training data—as they may be drawn from the same distribution. Therefore we used re-sampling of the training dataset to obtain a skewed dataset such that a logistic regression would predict similar sentiment distributions for both the train and test datasets. Finally we trained a logistic regression on the new dataset and used it on the task A test set. We obtained a macro-averaged recall score of INLINEFORM0 and accuracy of INLINEFORM1 .","Apparently, our assumption about distribution similarity was misguided as one can observe in the next table."]},{"idx":12,"heading":"Tasks B, D","paragraphs":["The goals of these tasks are to classify tweets sentiment regarding a given entity as either positive or negative (task B) and estimate sentiment distribution for each entity (task D). The measured metrics are macro-averaged recall and KLD, respectively.","We started with the training data passing our pipeline. We calculated the mean distribution for each entity on the training and testing datasets. We trained a logistic regression from a 5-label to a binary distribution and predicted a positive probability for each entity in the test set. This was used as a prior distribution for each entity, modeled as a Beta distribution. We then trained a logistic regression where the input is a concatenation of the 5-labels with the positive component of the probability distribution of the entity's sentiment and the output is a binary prediction for each tweet. Then we chose the label—using the mean positive probability as a threshold. These predictions are submitted as task B. We obtained a macro-averaged recall score of INLINEFORM0 and accuracy of INLINEFORM1 .","Next, we took the predictions mean for each entity as the likelihood, modeled as a Binomial distribution, thus getting a Beta posterior distribution for each entity. These were submitted as task D. We obtained a score of INLINEFORM0 ."]},{"idx":13,"heading":"Tasks C, E","paragraphs":["The goals of these tasks are to classify tweets sentiment regarding a given entity into five classes—very negative, negative, neutral, positive, very positive—(task C) and estimate sentiment distribution over five classes for each entity (task E). The measured metrics are macro-averaged MAE and earth-movers-distance (EMD), respectively.","We first calculated the mean sentiment for each entity. We then used bootstrapping to generate a sample for each entity. Then we trained a logistic regression model which predicts a 5-label distribution for each entity. We modified the initial 5-label probability distribution for each tweet using the following formula: DISPLAYFORM0","where INLINEFORM0 are the current tweet and label, INLINEFORM1 is the sentiment prediction of the logistic regression model for an entity, INLINEFORM2 is the set of all tweets and INLINEFORM3 is the set of labels. We trained a logistic regression on the new distribution and the predictions were submitted as task C. We obtained a macro-averaged MAE score of INLINEFORM4 .","Next, we defined a loss function as follows: DISPLAYFORM0","where the probabilities are the predicted probabilities after the previous logistic regression step. Finally we predicted a label for each tweet according to the lowest loss, and calculated the mean sentiment for each entity. These were submitted as task E. We obtained a score of INLINEFORM0 ."]},{"idx":14,"heading":"Review and Conclusions","paragraphs":["In this paper we described our system of sentiment analysis adapted to participate in SemEval task 4. The highest ranking we reached was third place on the 5-label classification task. Compared with classification with 2 and 3 labels, in which we scored lower, and the fact we used similar workflow for tasks A, B, C, we speculate that the relative success is due to our sentiment treebank ranking on a 5-label basis. This can also explain the relatively superior results in quantification of 5 categories as opposed to quantification of 2 categories.","Overall, we have had some unique advantages and disadvantages in this competition. On the one hand, we enjoyed an additional twenty thousand tweets, where every node of the parse tree was labeled for its sentiment, and also had the manpower to manually prune our dictionaries, as well as the opportunity to get feedback from our clients. On the other hand, we did not use any user information and/or metadata from Twitter, nor did we use the SemEval data for training the RNTN models. In addition, we did not ensemble our models with any commercially or freely available pre-trained sentiment analysis packages."]},{"idx":15,"heading":"Future Work","paragraphs":["We have several plans to improve our algorithm and to use new data. First, we plan to extract more semantic features such as verb and adverb classes and use them in neural network models as additional input. Verb classification was used to improve sentiment detection BIBREF8 ; we plan to label verbs according to whether their sentiment changes as we change the tense, form and active/passive voice. Adverbs were also used to determine sentiment BIBREF9 ; we plan to classify adverbs into sentiment families such as intensifiers (“very”), diminishers (“slightly”), positive (“delightfully”) and negative (“shamefully”).","Secondly, we can use additional data from Twitter regarding either the users or the entities-of-interest.","Finally, we plan to implement a long short-term memory (LSTM) network BIBREF10 which trains on a sentence together with all the syntax and semantic features extracted from it. There is some work in the field of semantic modeling using LSTM, e.g. BIBREF11 , BIBREF12 . Our plan is to use an LSTM module to extend the RNTN model of BIBREF1 by adding the additional semantic data of each phrase and a reference to the entity-of-interest. An illustration of the computational graph for the proposed model is presented in figure FIGREF33 . The inputs/outputs are: INLINEFORM0 is a word vector representation of dimension INLINEFORM1 , INLINEFORM2 encodes the parts-of-speech (POS) tagging, syntactic category and an additional bit indicating whether the entity-of-interest is present in the expression—all encoded in a 7 dimensional vector, INLINEFORM3 is a control channel of dimension INLINEFORM4 , INLINEFORM5 is an output layer of dimension INLINEFORM6 and INLINEFORM7 is a sentiment vector of dimension INLINEFORM8 .","The module functions are defined as following: DISPLAYFORM0","where INLINEFORM0 is a matrix to be learnt, INLINEFORM1 denotes Hadamard (element-wise) product and INLINEFORM2 denotes concatenation. The functions INLINEFORM3 are the six NN computations, given by: DISPLAYFORM0","where INLINEFORM0 are the INLINEFORM1 dimensional word embedding, 6-bit encoding of the syntactic category and an indication bit of the entity-of-interest for the INLINEFORM2 th phrase, respectively, INLINEFORM3 encodes the inputs of a left descendant INLINEFORM4 and a right descendant INLINEFORM5 in a parse tree and INLINEFORM6 . Define INLINEFORM7 , then INLINEFORM8 is a tensor defining bilinear forms, INLINEFORM9 with INLINEFORM10 are indication functions for having the entity-of-interest on the left and/or right child and INLINEFORM11 are matrices to be learnt.","The algorithm processes each tweet according to its parse tree, starting at the leaves and going up combining words into expressions; this is different than other LSTM algorithms since the parsing data is used explicitly. As an example, figure FIGREF36 presents the simple sentence “Amobee is awesome” with its parsing tree. The leaves are given by INLINEFORM0 -dimensional word vectors together with their POS tagging, syntactic categories (if defined for the leaf) and an entity indicator bit. The computation takes place in the inner nodes; “is” and “awesome” are combined in a node marked by “VP” which is the phrase category. In terms of our terminology, “is” and “awesome” are the INLINEFORM1 nodes, respectively for “VP” node calculation. We define INLINEFORM2 as the cell's state for the left child, in this case the “is” node. Left and right are concatenated as input INLINEFORM3 and the metadata INLINEFORM4 is from the right child while INLINEFORM5 is the metadata from the left child. The second calculation takes place at the root “S”; the input INLINEFORM6 is now a concatenation of “Amobee” word vector, the input INLINEFORM7 holds the INLINEFORM8 output of the previous step in node “VP”; the cell state INLINEFORM9 comes from the “Amobee” node."]},{"idx":16,"heading":"Figures and Tables","paragraphs":["Table 1: Summary of evaluation results, metrics used and rank achieved, for all sub tasks. ρ is macro-averaged recall, MAEM is macro-averaged mean absolute error, KLD is Kullback-Leibler divergence and EMD is earth-movers distance.","Figure 1: LSTM module; round purple nodes are element-wise operations, turquoise rectangles are neural network layers, orange rhombus is a dim-reducing matrix, splitting line is duplication, merging lines is concatenation.","Figure 2: Constituency-based parse tree; the LSTM module runs on the internal nodes by concatenating the left and right nodes as its input."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1705.02023","title":"Senti17 at SemEval-2017 Task 4: Ten Convolutional Neural Network Voters for Tweet Polarity Classification","sections":[{"idx":0,"heading":"Abstract","paragraphs":["This paper presents Senti17 system which uses ten convolutional neural networks (ConvNet) to assign a sentiment label to a tweet. The network consists of a convolutional layer followed by a fully-connected layer and a Softmax on top. Ten instances of this network are initialized with the same word embeddings as inputs but with different initializations for the network weights. We combine the results of all instances by selecting the sentiment label given by the majority of the ten voters. This system is ranked fourth in SemEval-2017 Task4 over 38 systems with 67.4%"]},{"idx":1,"heading":"Introduction","paragraphs":["Polarity classification is the basic task of sentiment analysis in which the polarity of a given text should be classified into three categories: positive, negative or neutral. In Twitter where the tweet is short and written in informal language, this task needs more attention. SemEval has proposed the task of Message Polarity Classification in Twitter since 2013, the objective is to classify a tweet into one of the three polarity labels BIBREF0 .","We can remark that in 2013, 2014 and 2015 most best systems were based on a rich feature extraction process with a traditional classifier such as SVM BIBREF1 or Logistic regression BIBREF2 . In 2014, kimconvolutional2014 proposed to use one convolutional neural network for sentence classification, he fixed the size of the input sentence and concatenated its word embeddings for representing the sentence, this architecture has been exploited in many later works. severynunitn:2015 adapted the convolutional network proposed by kimconvolutional2014 for sentiment analysis in Twitter, their system was ranked second in SemEval-2015 while the first system BIBREF3 combined four systems based on feature extraction and the third ranked system used logistic regression with different groups of features BIBREF2 .","In 2016, we remark that the number of participations which use feature extraction systems were degraded, and the first four systems used Deep Learning, the majority used a convolutional network except the fourth one BIBREF4 . Despite of that, using Deep Learning for sentiment analysis in Twitter has not yet shown a big improvement in comparison to feature extraction, the fifth and sixth systems BIBREF5 in 2016 which were built upon feature extraction process were only (3 and 3.5% respectively) less than the first system. But We think that Deep Learning is a promising direction in sentiment analysis. Therefore, we proposed to use convolutional networks for Twitter polarity classification.","Our proposed system consists of a convolutional layer followed by fully connected layer and a softmax on top. This is inspired by kimconvolutional2014, we just added a fully connected layer. This architecture gives a good performance but it could be improved. Regarding the best system in 2016 BIBREF6 , it uses different word embeddings for initialisation then it combines the predictions of different nets using a meta-classifier, Word2vec and Glove have been used to vary the tweet representation.","In our work, we propose to vary the neural network weights instead of tweet representation which can get the same effect of varying the word embeddings, therefore we vary the initial weights of the network to produce ten different nets, a voting system over the these ten voters will decide the sentiment label for a tweet.","The remaining of this paper is organized as follows: Section 2 describes the system architecture, Section 3 presents our experiments and results and Section 4 is devoted for the conclusion."]},{"idx":2,"heading":"System Architecture","paragraphs":["The architecture of our convolutional neural net- work for sentiment classification is shown on Fig. 1. Our network is composed of a single convolutional layer followed by a non-linearity, max pooling, Dropout, fully connected layer and a soft-max classification layer. Here we describe this architecture:"]},{"idx":3,"heading":"Tweet Representation","paragraphs":["We first tokenize each tweet to get all terms using HappyTokenizer which captures the words, emoticons and punctuations. We also replace each web link by the term url and each user name by uuser. Then, we used Structured Skip-Gram embeddings (SSG) BIBREF7 which was compiled by BIBREF4 using 52 million tweets.","Each term in the tweet is replaced by its SSG embedding which is a vector of d dimensions, all term vectors are concatenated to form the input matrix where the number of rows is d and the number of columns is set to be maxl: the max tweet length in the training dataset. This 2-dim matrix is the input layer for the neural network."]},{"idx":4,"heading":"Convolutional Layers","paragraphs":["We connect the input matrix with different convolutional layers, each one applies a convolution operation between the input matrix and a filter of size m x d. This is an element-wise operation which creates f vectors of maxl-m+1 dimension where f is the number of filters or feature maps.","This layer is supposed to capture the common patterns among the training tweets which have the same filter size but occur at any position of the tweet. To capture the common patterns which have different sizes we have to use more than one layer therefore we defined 8 different layers connected to the input matrix with different filter sizes but the same number of feature maps."]},{"idx":5,"heading":"Activation Layer","paragraphs":["Each convolutional layer is typically followed by a non-linear activation function, RELU (Rectified Linear Unit ) layer will apply an element-wise operation to swap the negative numbers to 0. The output of a ReLU layer is the same size as the input, just with all the negative values removed. It speeds up the training and is supposed to produce more accurate results."]},{"idx":6,"heading":"Max-Pooling Layer","paragraphs":["This layer reduces the size of the output of activation layer, for each vector it selects the max value. Different variation of pooling layer can be used: average or k-max pooling."]},{"idx":7,"heading":"Dropout Layer","paragraphs":["Dropout is used after the max pooling to regularize the ConvNet and prevent overfitting. It assumes that we can still obtain a reasonable classification even when some of the neurones are dropped. Dropout consists in randomly setting a fraction p of input units to 0 at each update during training time."]},{"idx":8,"heading":"Fully Conected Layer","paragraphs":["We concatenate the results of all pooling layers after applying Dropout, these units are connected to a fully connected layer. This layer performs a matrix multiplication between its weights and the input units. A RELU non-linarity is applied on the results of this layer."]},{"idx":9,"heading":"Softmax Layer","paragraphs":["The output of the fully connected layer is passed to a Softmax layer. It computes the probability distribution over the labels in order to decide the most probable label for a tweet."]},{"idx":10,"heading":"Experiments and Results","paragraphs":["For training the network, we used about 30000 English tweets provided by SemEval organisers and the test set of 2016 which contains 12000 tweets as development set. The test set of 2017 is used to evaluate the system in SemEval-2017 competition. For implementing our system we used python and Keras.","We set the network parameters as follows: SSG embbeding size d is chosen to be 200, the tweet max legnth maxl is 99. For convolutional layers, we set the number of feature maps f to 50 and used 8 filter sizes (1,2,3,4,5,2,3,4). The p value of Dropout layer is set to 0.3. We used Nadam optimizer BIBREF8 to update the weights of the network and back-propogation algorithm to compute the gradients. The batch size is set to be 50 and the training data is shuffled after each iteration.","We create ten instances of this network, we randomly initialize them using the uniform distribution, we repeat the random initialization for each instance 100 times, then we pick the networks which gives the highest average recall score as it is considered the official measure for system ranking. If the top network of each instance gives more than 95% of its results identical to another chosen network, we choose the next top networks to make sure that the ten networks are enough different.","Thus, we have ten classifiers, we count the number of classifiers which give the positive, negative and neutral sentiment label to each tweet and select the sentiment label which have the highest number of votes. For each new tweet from the test set, we convert it to 2-dim matrix, if the tweet is longer than maxl, it will be truncated. We then feed it into the ten networks and pass the results to the voting system.","Official ranking: Our system is ranked fourth over 38 systems in terms of macro-average recall. Table 4 shows the results of our system on the test set of 2016 and 2017."]},{"idx":11,"heading":"Conclusion","paragraphs":["We presented our deep learning approach to Twitter sentiment analysis. We used ten convolutional neural network voters to get the polarity of a tweet, each voter has been trained on the same training data using the same word embeddings but different initial weights. The results demonstrate that our system is competitive as it is ranked forth in SemEval-2017 task 4-A."]},{"idx":12,"heading":"Figures and Tables","paragraphs":["Figure 1: Network architecture."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1705.04153","title":"Dynamic Compositional Neural Networks over Tree Structure","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Tree-structured neural networks have proven to be effective in learning semantic representations by exploiting syntactic information. In spite of their success, most existing models suffer from the underfitting problem: they recursively use the same shared compositional function throughout the whole compositional process and lack expressive power due to inability to capture the richness of compositionality. In this paper, we address this issue by introducing the dynamic compositional neural networks over tree structure (DC-TreeNN), in which the compositional function is dynamically generated by a meta network. The role of meta-network is to capture the metaknowledge across the different compositional rules and formulate them. Experimental results on two typical tasks show the effectiveness of the proposed models."]},{"idx":1,"heading":"Introduction","paragraphs":["Learning the distributed representation for long spans of text from its constituents has been a key step for various natural language processing (NLP) tasks, such as text classification BIBREF0 , BIBREF1 , semantic matching BIBREF2 , BIBREF3 , and machine translation BIBREF4 . Existing deep learning approaches take a compositional function with different forms to compose word vectors recursively until obtaining a sentential representation. Typically, these compositional functions involve recurrent neural networks BIBREF5 , BIBREF6 , convolutional neural networks BIBREF7 , BIBREF8 , and tree-structured neural networks BIBREF9 , BIBREF10 .","Among these methods, tree-structured neural networks (Tree-NNs) show theirs superior performance in many NLP tasks BIBREF11 , BIBREF12 . Following the syntactic tree structure, Tree-NNs assign a fixed-length vector to each word at the leaves of the tree, and combine word and phrase pairs recursively to create intermediate node vectors, eventually obtaining one final vector to represent the whole sentence.","However, these models have a major limitation in their inability to fully capture the richness of compositionality BIBREF13 . The same parameters are used for all kinds of semantic compositions, even though the compositions have different characteristics in nature. For example, the composition of the adjective and the noun differs significantly from the composition of the verb and the noun. Moreover, many semantic phenomena, such as semantic idiomaticity or transparency, call for more powerful compositional mechanisms BIBREF14 . Therefore, Tree-NNs suffer from the underfitting problem.","To alleviate this problem, some researchers propose to use multiple compositional functions, which are arranged beforehand according to some partition criterion BIBREF11 , BIBREF13 , BIBREF15 . Intuitively, using different parameters for different types of compositions has the potential to greatly reduce underfitting. BIBREF13 [ BIBREF13 ] defined different compositional functions in terms of syntactic categories, and a suitable compositional function is selected based on the syntactic categories. BIBREF15 [ BIBREF15 ] introduced multiple compositional functions and during compositional phase, a proper one is selected based on the input information. Although these models accomplished their mission to a certain extent, they still suffer from the following three challenges. First, the predefined compositional functions cannot cover all the compositional rules; Second, they require more learnable parameters, suffering from the problem of overfitting; Third, it is difficult to determine a universal criterion for semantic composition based solely on syntactic categories.","In this paper, we propose dynamic compositional neural networks over tree structure, in which a meta network is used to generate the context-specific parameters of a dynamic compositional network. Specifically, we construct our models based on two kinds of tree-structured neural networks: recursive neural network (Tree-RecNN) BIBREF11 and tree-structure long short-term memory neural network (Tree-LSTM) BIBREF9 . Our work is inspired by recent work on dynamic parameter prediction BIBREF16 , BIBREF17 , BIBREF18 . The meta network is used to extract the shared meta-knowledge across different compositional rules and to dynamically generate the context-specific compositional function. Thus, the compositional function of our models varies with positions, contexts and samples. The dynamic compositional network then applies those context-specific parameters to the current input information. Both meta and dynamic networks are differentiable such that the overall networks can be trained in an end-to-end fashion. Additional, to reduce the complexity of the whole networks, we define the dynamic weight matrix in a manner simulating low-rank matrix decomposition.","We evaluate our models on two typical tasks: text classification and text semantic matching. The results show that our models are more expressive due to their learning to learn nature, yet without increasing the number of model's parameters. Moreover, we find certain composition operations can be learned implicitly by meta TreeNN, such as the composition of noun phrases and verb phrases.","The contributions of the paper can be summed up as follows."]},{"idx":2,"heading":"Tree-Structured Neural Network","paragraphs":["In this section, we briefly describe the tree-structured neural networks.","The idea of tree-structured neural networks for natural language processing (NLP) is to train a deep learning model with a grammatical tree structure BIBREF19 that can be applied to phrases and sentences. At every node in the tree, the contexts of the left and right children are combined by a compositional function. The parameters of the compositional function are shared across all nodes in the tree. The layer computed at the top node gives a representation for the whole sentence."]},{"idx":3,"heading":"Vanilla Recursive Neural Network","paragraphs":["The simplest member of tree-structured NN is the vanilla recursive neural network BIBREF13 , in which the representation of parent is calculated by weighted linear combination of the child vectors.","Formally, given a binary constituency tree $T$ induced by a sentence, each non-leaf node corresponds to a phrase. We refer to $\\mathbf {h}_{j} \\in \\mathbb {R}^{d}$ as the hidden state of each node $j$ , and let $\\mathbf {h}_{j}^{l}$ , $\\mathbf {h}_{j}^{r}$ denote the left and right child representations respectively.","$$\\mathbf {h}_{j} = \\tanh \\left( \\mathbf {W} \\begin{bmatrix} \\mathbf {h}_{j}^{l} \\\\ \\mathbf {h}_{j}^{r} \\\\ \\end{bmatrix} + \\mathbf {b} \\right),$$ (Eq. 5)","where $\\mathbf {W} \\in \\mathbb {R}^{d \\times 2d}$ is a learnable compositional matrix, $\\mathbf {b}$ is the bias vector."]},{"idx":4,"heading":"Tree LSTM","paragraphs":["Tree LSTM BIBREF9 is a generalization of LSTMs to tree-structured network topologies. In this model, the compositional function is an LSTM unit, and the hidden state $h_j$ of each node can be computed as follows: we refer to $\\mathbf {h}_{j}$ and $\\mathbf {c}_{j}$ as the hidden state and memory cell of each node $j$ . The transition equations of node $j$ are as follows:","$$\\begin{bmatrix} \\mathbf {\\tilde{c}}_{j} \\\\ \\mathbf {o}_{j} \\\\ \\mathbf {i}_{j} \\\\ \\mathbf {f}_{j}^{l} \\\\ \\mathbf {f}_{j}^{r} \\end{bmatrix} &= \\begin{bmatrix} \\tanh \\\\ \\sigma \\\\ \\sigma \\\\ \\sigma \\\\ \\sigma \\end{bmatrix} \\left( \\mathbf {W} \\begin{bmatrix} \\mathbf {x}_{j} \\\\ \\mathbf {h}_{j}^{l} \\\\ \\mathbf {h}_{j}^{r} \\\\ \\end{bmatrix} + \\mathbf {b} \\right),\\\\ \\mathbf {c}_{j} &= \\mathbf {\\tilde{c}}_{j} \\odot \\mathbf {i}_{j} + \\mathbf {c}_{j}^{l} \\odot \\mathbf {f}_{j}^{l}+ \\mathbf {c}_{j}^{r} \\odot \\mathbf {f}_{j}^{r} , \\\\ \\mathbf {h}_{j} &= \\mathbf {o}_{j} \\odot \\tanh \\left( \\mathbf {c}_{j} \\right),$$ (Eq. 8)","where $\\mathbf {x}_j \\in \\mathbb {R}^{d}$ denotes the input vector and is non-zero if and only if it is a leaf node. The superscript $l$ and $r$ represent the left child and right child respectively. $\\sigma $ represents the logistic sigmoid function and $\\odot $ denotes element-wise multiplication. $\\mathbf {W} \\in \\mathbb {R}^{5d \\times 3d}$ and $\\mathbf {b} \\in \\mathbb {R}^{5d}$ are learnable parameters."]},{"idx":5,"heading":"Dynamic Compositional Neural Network","paragraphs":["In the above two tree-structured NNs, the compositional function is shared across all nodes in the tree, which results in underfitting since the semantic compositions have great diversities. To address this problem, we propose two dynamic compositional neural networks over tree structure, which dynamically generate different parameters for different types of compositions. Figure 2 shows an illustration of the dynamic compositional neural network, consisting of two components: (1) meta network and (2) basic network with dynamic parameters.","Specifically, we propose two meta networks to generate the context-specific compositional functions for RecNN and TreeLSTM respectively."]},{"idx":6,"heading":"Meta Network for RecNN","paragraphs":["For RecNN, we replace the static parameters $\\mathbf {W}$ and $\\mathbf {b}$ in Eq.( 5 ) with the dynamic parameters $\\mathbf {W}(\\mathbf {z}_{j})$ and $\\mathbf {b}(\\mathbf {z}_{j})$ , which are generated by a meta network. The meta network is a smaller RecNN, and the hidden state $\\hat{\\mathbf {h}}_{j}\\in \\mathbb {R}^{m}$ of node $j$ in meta network is defined as","$$\\hat{\\mathbf {h}}_{j} &= \\tanh \\left( \\mathbf {W}_{m} \\mathcal {H}_{j} + \\mathbf {b}_{m} \\right),\\\\ \\mathbf {z}_{j} &= \\mathbf {W}_{z} \\hat{\\mathbf {h}}_{j}$$ (Eq. 11)","where $\\mathcal {H}_{j} = \\mathbf {h}_{j}^{l}\\oplus \\mathbf {h}_{j}^{r}\\oplus \\mathbf {\\hat{h}}_{j}^{l}\\oplus \\mathbf {\\hat{h}}_{j}^{r} \\in \\mathbb {R}^{2m+2d}$ , $\\mathbf {W}_{m} \\in \\mathbb {R}^{m \\times (2d+2m)}$ and $\\mathbf {b}_{m} \\in \\mathbb {R}^{m}$ are parameters of meta RecNN; $\\mathbf {W}_{z} \\in \\mathbb {R}^{z \\times m}$ is a scale matrix.","To reduce the number of the parameters, we define the dynamic parameters with a low-rank factorized representation of the weights, analogous to the Singular Value Decomposition. The dynamic parameters $\\mathbf {W}(\\mathbf {z}_{j})$ and $\\mathbf {b}(\\mathbf {z}_{j})$ of the basic RecNN are computed by:","$$\\mathbf {W}(\\mathbf {z}_{j}) &= \\begin{bmatrix} P_l \\mathbf {D}(\\mathbf {z}_j)Q_l \\\\ P_r \\mathbf {D}(\\mathbf {z}_j)Q_r \\\\ \\end{bmatrix} \\\\ \\mathbf {b}(\\mathbf {z}_{j}) &= \\begin{bmatrix} B_l\\mathbf {z}_j \\\\ B_r\\mathbf {z}_j \\\\ \\end{bmatrix}$$ (Eq. 12)","where $P \\in \\mathbb {R}^{d \\times z}$ , $Q \\in \\mathbb {R}^{z \\times d}$ , and $\\mathbf {D}(\\mathbf {z}_t) \\in \\mathbb {R}^{z \\times z}$ is the diagonal matrix of $\\mathbf {z}$ .","Thus, our dynamic RecNN needs $(6dz + mz)$ parameters, while the vanilla RecNN has $(2d^2+d)$ parameters. With a small $z$ and $m$ , our dynamic RecNN needs less parameters than the vanilla RecNN. For example, if we set $d =100$ and $z=m=20$ , our model needs $12,400$ parameters while the vanilla model needs $20,100$ parameters."]},{"idx":7,"heading":"Meta Network for TreeLSTM","paragraphs":["Likewise, we also use a smaller meta network to generate the static parameters $\\mathbf {W}$ and $\\mathbf {b}$ in Eq.( 8 ) with the dynamic parameters $\\mathbf {W}(\\mathbf {z}_{j})$ and $\\mathbf {b}(\\mathbf {z}_{j})$ . The meta network is a smaller TreeLSTM, and the hidden state $\\hat{\\mathbf {h}}_{j}\\in \\mathbb {R}^{m}$ of node $j$ in meta network is defined as","$$\\begin{bmatrix} \\mathbf {\\hat{g}}_{j} \\\\ \\mathbf {\\hat{o}}_{j} \\\\ \\mathbf {\\hat{i}}_{j} \\\\ \\mathbf {\\hat{f}}_{j}^{l} \\\\ \\mathbf {\\hat{f}}_{j}^{r} \\end{bmatrix} &= \\begin{bmatrix} \\tanh \\\\ \\sigma \\\\ \\sigma \\\\ \\sigma \\\\ \\sigma \\end{bmatrix} \\left( \\mathbf {W}_{m} \\begin{bmatrix} \\mathbf {x}_{j} \\\\ \\mathcal {H}_{j} \\\\ \\end{bmatrix} + \\mathbf {b}_{m} \\right) ,\\\\ \\mathbf {\\hat{c}}_{j} &= \\mathbf {\\hat{g}}_{j} \\odot \\mathbf {\\hat{i}}_{j} + \\mathbf {\\hat{g}}_{j}^{l} \\odot \\mathbf {\\hat{f}}_{j}^{l}+ \\mathbf {\\hat{g}}_{j}^{r} \\odot \\mathbf {\\hat{f}}_{j}^{r} , \\\\ \\mathbf {\\hat{h}}_{j} &= \\mathbf {\\hat{o}}_{j} \\odot \\tanh \\left( \\mathbf {\\hat{c}}_{j} \\right), \\\\ \\mathbf {z}_{j} &= \\mathbf {W}_{z} \\hat{\\mathbf {h}}_{j}$$ (Eq. 14)","where $\\mathcal {H}_{j} = \\mathbf {h}_{j}^{l}\\oplus \\mathbf {h}_{j}^{r}\\oplus \\mathbf {\\hat{h}}_{j}^{l}\\oplus \\mathbf {\\hat{h}}_{j}^{r} \\in \\mathbb {R}^{2m+2d}$ ; $\\mathbf {W}_{m} \\in \\mathbb {R}^{5m \\times (3d+2m)}$ and $\\mathbf {b}_{m} \\in \\mathbb {R}^{m}$ are parameters of meta TreeLSTM; $\\mathbf {W}_{z} \\in \\mathbb {R}^{z \\times m}$ is a scale matrix.","The dynamic parameters $\\mathbf {W}(\\mathbf {z}_{j})$ and $\\mathbf {b}(\\mathbf {z}_{j})$ of basic TreeLSTM are computed by:","$$\\mathbf {W}(\\mathbf {z}_{j}) &= \\left[\\mathbf {W}^{g}, \\mathbf {W}^{i}, \\mathbf {W}^{f^l},\\mathbf {W}^{f^r},\\mathbf {W}^{o}\\right] \\\\ \\mathbf {b}(\\mathbf {z}_{j}) &= \\left[\\mathbf {b}^{g}, \\mathbf {b}^{i}, \\mathbf {b}^{f^l},\\mathbf {b}^{f^r},\\mathbf {b}^{o}\\right],$$ (Eq. 15)","where for $*\\in \\lbrace c,o,i,f^l,f^r\\rbrace $ ,","$$\\mathbf {W}^{*}(\\mathbf {z}_{j}) &= \\begin{bmatrix} P_x^{*} \\mathbf {D}(\\mathbf {z}_t)Q_x^{*} \\\\ P_l^{*} \\mathbf {D}(\\mathbf {z}_t)Q_l^{*} \\\\ P_r^{*} \\mathbf {D}(\\mathbf {z}_t)Q_r^{*} \\\\ \\end{bmatrix}, \\\\ \\mathbf {b}^{*}(\\mathbf {z}_{j}) &= \\begin{bmatrix} B_x^{*}\\mathbf {z}_t \\\\ B_l^{*}\\mathbf {z}_t \\\\ B_r^{*}\\mathbf {z}_t \\\\ \\end{bmatrix},$$ (Eq. 16)","where $P \\in \\mathbb {R}^{5d \\times z}$ , $Q \\in \\mathbb {R}^{z \\times 3d}$ , $B \\in \\mathbb {R}^{5d \\times z}$ , and $\\mathbf {D}(\\mathbf {z}_t) \\in \\mathbb {R}^{z \\times z}$ is the diagonal matrix of $\\mathbf {z}$ .","With a small $z$ and $m$ , our dynamic TreeLSTM needs a similar amount of parameters compared to the standard TreeLSTM."]},{"idx":8,"heading":"Application of Dynamic Compositional Neural Networks","paragraphs":["In this section, we describe two specific models to show the applications of dynamic compositional neural networks for two typical tasks in NLP."]},{"idx":9,"heading":"Text Classification","paragraphs":["The purpose of text classification is that, given a sentence $x$ , the model should predict labels $\\hat{y}$ from a pre-defined label set $\\mathcal {Y}$ . From the description in the previous section, we can compute the distributed representation $\\mathbf {h}_j$ of the phrase at node $j$ of a tree:","$$\\mathbf {h}_{j} &= \\operatornamewithlimits{DC-TreeNN}(\\mathbf {x}_j, \\mathbf {h}_{j}^l, \\mathbf {h}_{j}^r,\\theta )$$ (Eq. 18)","After this recursive process, the hidden state $\\mathbf {h}_R$ at the root node is used as the sentential representation, which then followed by a softmax classifier to predict the probability distribution over classes.","$${\\hat{\\mathbf {y}}} = \\operatornamewithlimits{softmax}(\\mathbf {W}_t \\mathbf {h}_R + \\mathbf {b}_t)$$ (Eq. 19)","where ${\\hat{\\mathbf {y}}}$ is prediction probabilities, $\\mathbf {W}_t$ and $\\mathbf {b}_t$ are the parameters of the classifier.","We evaluate our models on five different datasets. The detailed statistics about the five datasets are listed in Table 2 . Each dataset is briefly described as follows.","SST The movie reviews with two classes (negative, positive) in the Stanford Sentiment Treebank BIBREF23 .","MR The movie reviews with two classes BIBREF24 .","QC The TREC questions dataset involves six different question types. BIBREF25 .","SUBJ Subjectivity dataset where the goal is to classify each instance (snippet) as being subjective or objective. BIBREF26","IE Idiom enhanced sentiment classification. BIBREF27 . Each sentence contains at least one idiom.","As shown in Table 3 , DC-TreeLSTM consistently outperforms RecNN, MV-RecNN, RNTN, and TreeLSTM by a large margin while achieving comparable results to the CNN and using much fewer parameters.(The number of parameters in our models is approximately 10K while in CNN the number of parameters is about 400K). Compared with RecNN, DC-RecNN performs better, indicating the effectiveness of the dynamic compositional function. Additionally, both DC-RecNN and DC-TreeLSTM achieve substantial improvement on IE dataset, which covers the richness of compositionality (idiomaticity). We attribute the success on IE to its power in modeling more complicated compositionality."]},{"idx":10,"heading":"Text Semantic Matching","paragraphs":["Among many natural language processing (NLP) tasks, a common problem is modelling the relevance of a pair of texts. In this section, we show how to effectively use the dynamic compositional neural networks to model the semantic relationship between two sentences.","As shown in Figure 1 , given two sentences $x_a$ and $x_b$ , the representation of each sentence $\\mathbf {h}_{R}$ can be computed by one basic TreeNN.","$$\\mathbf {h}_{R}^{(a)} &= \\operatornamewithlimits{DC-TreeNN}(\\mathbf {x}_R, \\mathbf {h}_{R}^l, \\mathbf {h}_{R}^r,\\theta _{a}) \\\\ \\mathbf {h}_{R}^{(b)} &= \\operatornamewithlimits{DC-TreeNN}(\\mathbf {x}_R, \\mathbf {h}_{R}^l, \\mathbf {h}_{R}^r,\\theta _{b})$$ (Eq. 21)","where $R$ denotes the root node of a tree. $\\theta _{a}$ and $\\theta _{b}$ are generated by a shared meta TreeNN. Then, the representation of each sentence will be fed into a multi-layer perceptron to obtain a unified representation for the final relationship classification.","The sample-specific but shared meta TreeNN ensures that, on the one hand we can dynamically model the diversity of semantic compositionality, on the other hand we can capture the general rules across different samples.","We choose the dataset of Sentences Involving Compositional Knowledge (SICK), which is proposed by BIBREF30 [ BIBREF30 ] aiming at evaluation of compositional distributional semantic models. The dataset consists of 9927 sentence pairs in a 4500/500/4927 train/dev/test split, in which each sentence pairs are pre-defined into three labels: “entailment”,“contradiction” and “neutral”.","Our results are summarized in Table 4 , where the performance of NBOW, LSTM, RecNN, and RNTN are reported by BIBREF31 , BIBREF32 . For fair comparison, we train our models with the same setting. We can see both DC-RecNN and DC-TreeLSTM outperform competitor models, in which DC-RecNN (DC-TreeLSTM) achieves 3% (2.7%) improvements than RecNN (TreeLSTM). We think this breakthrough is basically attributed to the dynamic compositional mechanism, which enables our models to capture various syntactic patterns (As we will discuss later) therefore can more accurately understand sentences."]},{"idx":11,"heading":"Experiment","paragraphs":["To make a comprehensive evaluation, we assess our model on five text classification tasks and a semantic matching task."]},{"idx":12,"heading":"Training and Hyperparameters","paragraphs":["Given a sentence (or sentence pair) and its label, the output of a neural network is the probabilities of the different classes. The parameters of the network are trained to minimise the cross-entropy of the predicted and true label distributions. To minimize the objective, we use stochastic gradient descent with the diagonal variant of AdaGrad BIBREF20 .","The word embeddings for all of the models are initialized with GloVe vectors BIBREF21 . The other parameters are initialized by randomly sampling from uniform distribution in $[-0.1, 0.1]$ .","The final hyper-parameters are as follows. The initial learning rate is $0.1$ . The regularization weight of the parameters is $1E{-5}$ and the others are listed as Table 1 .","For all the sentences in the datasets, we parse them with constituency parser BIBREF22 to obtain the trees for our models and some competitor models."]},{"idx":13,"heading":"Competitor Methods","paragraphs":["RecNN BIBREF11 : Recursive neural network with standard compositional function.","RNTN BIBREF23 : The RNTN is a recursive neural network with neural tensor layer, which can model strong interactions between two constituents.","MV-RecNN BIBREF11 : The MV-RecNN is to represent every word and longer phrase in a parse tree as both a vector and a matrix in order to model rich compositionality.","TreeLSTM BIBREF9 : Recursive neural network with Long Short-Term Memory unit."]},{"idx":14,"heading":"Discussion and Qualitative Analysis","paragraphs":["In our models, the latent vector $\\mathbf {z}$ controls the process of predicting network's parameters and its dimensionality determines the number of model's parameters. Next, we will investigate how the controlling vector $\\mathbf {z}$ influences the performance of our models.","Figure 4 shows the accuracies of DC-RecNN across the different dimensions of $[5, 10, \\dots , 50]$ for the controlling vector $\\mathbf {z}$ on five datasets. We get the following findings:","For all five datasets, the model can achieve considerable performances even when the size of vector $\\mathbf {z}$ is reduced to 5. Particularly, for the dataset QC, the model obtains $87.0\\%$ accuracy with a pretty small meta Tree-RecNN, suggesting a smaller meta network can be used for generating a more powerful compositional function to effectively model sentence.","When dealing with the dataset with more labels, larger vector size leads to a better performance. For example, the performance on IE and QC datasets reaches the maximum when the size of $z$ equals 40, while for the other three datasets MR, SST and SUBJ, the model obtains the best performance with the value of 30, 30 and 20 respectively.","As described in previous sections, we know the compositional function is changed cross child nodes over a tree, which is controlled by a latent vector $z$ . To get an intuitive understanding of how the controlling vector $z$ works, we design an experiment to examine the neuron's behaviours of $\\mathbf {z}$ on each node. More concretely, we refer to $z_{jk}$ as the activation of the $k$ -neuron at node $j$ , where $j \\in \\lbrace 1,\\ldots ,N\\rbrace $ and $k \\in \\lbrace 1,\\ldots , z\\rbrace $ . Then we randomly sample some sentences on the development set from the datasets we used. By visualizing the latent vector $\\mathbf {z}_{j}$ and analyzing the maximum activation, we can find what kinds of patterns the current neuron focuses on.","Table 5 illustrates multiple interpretable neurons and some representative words or phrases which can activate these neurons. We can observe that:","For some simple tasks such as text classification, meta network will integrate useful semantic information into the the generation process of compositional function. These semantic bias before composition are task-specific.","For example, the 21- $st$ neuron is more sensitive to emotional terms, which can be understood as a sentinel, telling the basic neural network that an informative phrase is coming, more attention should be paid in the process of composition. Figure 5 -(a) shows a visualization. We can see in this sentence, the neuron has realized that this idiomatic collocation “in stitches” is a key pattern, which is crucial for the final sentiment prediction.","For more complicated tasks such as semantic matching, a well-grounded understanding of the syntactic structure is crucial. In this context, we find that a meta network could capture some syntactic information. For example, the 27- $th$ neuron monitors phrases constructed by light-verb. As shown in Figure 5 -(b), the verb phrase “taking off” has been attended for forthcoming compositional operation, which is more useful for judging the semantic relation between the sentence pair “An airplane is taking off/A plane is landing”."]},{"idx":15,"heading":"Related Work","paragraphs":["One thread of related work is the exploration of different kinds of compositional function over tree structures. BIBREF11 [ BIBREF11 ] proposed the recursive neural network with standard compositional function. After that, some extensions are introduced to enhance the expressive power of compositional function, such as MV-RecNN BIBREF23 , SU-RNN BIBREF13 , RNTN BIBREF23 , while these models suffer from the problem of hard-coded compositional operations and overfitting.","Another thread of work is the idea of using one network to direct the learning of another network BIBREF16 . BIBREF33 [ BIBREF33 ] introduce a meta neural network to provide another network with a step size and a direction vector, which is helpful for parameter optimization. BIBREF16 [ BIBREF16 ] propose the dynamic filter network to implicitly learn a variety of filtering operations. BIBREF17 [ BIBREF17 ] introduce a learnet for one-shot learning, which can predict the parameters of a second network given a single exemplar. BIBREF18 [ BIBREF18 ] propose the model hypernetwork, which uses a small network to generate the weights for a larger network.","Different from these models, we employ the idea of parameter generation to address the limitation of weight-sharing or partially sharing paradigm of tree-based compositional models."]},{"idx":16,"heading":"Conclusion","paragraphs":["In this work, we introduce a meta neural network, which can generate a compositional network to dynamically compose constituents over tree structure. The parameters of compositional function vary from position to position and from sample to sample, allowing for more sophisticated operations on the input.","To evaluate our models, we choose two typical NLP tasks involving six datasets. The qualitative and quantitative experiment results demonstrate the effectiveness of our models."]},{"idx":17,"heading":"Acknowledgments","paragraphs":["We would like to thank the anonymous reviewers for their valuable comments and thank Kaiyu Qian, Jiachen Xu, Jifan Chen for useful discussions. This work was partially funded by National Natural Science Foundation of China (No. 61532011 and 61672162), Shanghai Municipal Science and Technology Commission (No. 16JC1420401)."]},{"idx":18,"heading":"Figures and Tables","paragraphs":["Figure 1: Illustration of Vanilla Tree Structure Network. NP and VP represent noun and verb phrases respectively. θ denotes the shared parameters of compositional function.","Figure 2: Dynamic Compositional Neural Network. θ denotes the context-dependent parameters generated by meta TreeNN.","Figure 3: Illustration of DC-TreeNN Matching Network.","Table 1: Hyper-parameters for our models on all tasks. m, d denote the size of hidden state in meta and basic TreeNN respectively. e and z represent the size of embedding vector x and controlling vector z. The settings of our two proposed models on all datasets are the same except SICK: the numbers inside and outside parentheses correspond to DC-RecNN and DC-TreeLSTM respectively.","Table 2: Statistics of the five mainstream datasets for text classification. Lavg denotes the average length of documents; |V| denotes the size of vocabulary.","Table 3: Accuracies of our models on five datasets against state-of-the-art neural models. DCNN: Dynamic Convolutional Neural Network [Kalchbrenner et al., 2014; Denil et al., 2014]. CNN-multichannel: Convolutional Neural Network [Kim, 2014].","Table 4: Evaluation results of our models on the SICK train and test sets.","Figure 4: Performances of DC-RecNN with the different sizes of latent vector z on five development datasets: IE, MR, SST, SUBJ, and QC. Y-axis represents the accuracy(%), and X-axis represents dimensionality of z.","Table 5: Multiple interpretable neurons and the words/phrases captured by these neurons. The last column gives the explanations of corresponding neuron’s behaviours.","Figure 5: The two heat maps describe the behaviours of neurons z21 and z27 from DC-TreeNN."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1705.10272","title":"Who's to say what's funny? A computer using Language Models and Deep Learning, That's Who!","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Humor is a defining characteristic of human beings. Our goal is to develop methods that automatically detect humorous statements and rank them on a continuous scale. In this paper we report on results using a Language Model approach, and outline our plans for using methods from Deep Learning."]},{"idx":1,"heading":"Introduction","paragraphs":["Computational humor is an emerging area of research that ties together ideas from psychology, linguistics, and cognitive science. Humor generation is the problem of automatically creating humorous statements (e.g., BIBREF0 , BIBREF1 ). Humor detection seeks to identify humor in text, and is sometimes cast as a binary classification problem that decides if some input is humorous or not (e.g., BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 ). However, our focus is on the continuous and subjective aspects of humor.","We learn a particular sense of humor from a data set of tweets which are geared towards a certain style of humor BIBREF6 . This data consists of humorous tweets which have been submitted in response to hashtag prompts provided during the Comedy Central TV show @midnight with Chris Hardwick. Since not all jokes are equally funny, we use Language Models and methods from Deep Learning to allow potentially humorous statements to be ranked relative to each other."]},{"idx":2,"heading":"Language Models","paragraphs":["We used traditional Ngram language models as our first approach for two reasons : First, Ngram language models can learn a certain style of humor by using examples of that as the training data for the model. Second, they assign a probability to each input they are given, making it possible to rank statements relative to each other. Thus, Ngram language models make relative rankings of humorous statements based on a particular style of humor, thereby accounting for the continuous and subjective nature of humor.","We began this research by participating in SemEval-2017 Task 6 #HashtagWars: Learning a Sense of Humor BIBREF7 . This included two subtasks : Pairwise Comparison (Subtask A) and Semi-ranking (Subtask B). Pairwise comparison asks a system to choose the funnier of two tweets. Semi-ranking requires that each of the tweets associated with a particular hashtag be assigned to one of the following categories : top most funny tweet, next nine most funny tweets, and all remaining tweets.","Our system estimated tweet probabilities using Ngram language models. We created models from two different corpora - a collection of funny tweets from the @midnight program, and a corpus of news data that is freely available for research. We scored tweets by assigning them a probability based on each model. Tweets that have a higher probability according to the funny tweet model are considered funnier since they are more like the humorous training data. However, tweets that have a lower probability according to the news language model are viewed as funnier since they are least like the (unfunny) news corpus. We took a standard approach to language modeling and used bigrams and trigrams as features in our models. We used KenLM BIBREF8 with modified Kneser-Ney smoothing and a back-off technique as our language modeling tool.","Table 1 shows our results for both data sets when trained on bigrams and trigrams. The accuracy and distance measures are defined by the task organizers BIBREF7 . We seek high accuracy in picking the funnier tweet (Subtask A) and low distance (from the gold standard) in organizing the tweets into categories (Subtask B).","These results show that models trained on the news data have a significant advantage over the tweets model, and that bigram models performed slightly better than trigrams. We submitted trigram models trained on news and tweets to the official evaluation of SemEval-2017 Task 6. The trigram language models trained on the news data placed fourth in Subtask A and first in Subtask B.","We believe that the significant advantage of the news data over the tweet data is caused by the much larger quantity of news data available. The tweet data only consists of approximately 21,000 tweets, whereas the news data totals approximately 6.2 GB of text. In the future we intend to collect more tweet data, especially those participating in the ongoing #HashtagWars staged nightly by @midnight. We also plan to experiment with equal amounts of tweet data and news data, to see if one has an inherent advantage over the other.","Our language models performed better in the pairwise comparison, but it is clear that more investigation is needed to improve the semi-ranking results. We believe that Deep Learning may overcome some of the limits of Ngram language models, and so will explore those next."]},{"idx":3,"heading":"Deep Learning","paragraphs":["One limitation of our language model approach is the large number of out of vocabulary words we encounter. This problem can not be solved by increasing the quantity of training data because humor relies on creative use of language. For example, jokes often include puns based on invented words, e.g., a singing cat makes beautiful meowsic. BIBREF6 suggests that character–based Convolutional Neural Networks (CNNs) are an effective solution for these situations since they are not dependent on observing tokens in training data. Previous work has also shown the CNNs are effective tools for language modeling, even in the presence of complex morphology BIBREF9 . Other recent work has shown that Recurrent Neural Networks (RNNs), in particular Long Short–Term Memory networks (LSTMs), are effective in a wide range of language modeling tasks (e.g., BIBREF10 , BIBREF11 ). This seems to be due to their ability to capture long distance dependencies, which is something that Ngram language models can not do.","BIBREF6 finds that external knowledge is necessary to detect humor in tweet based data. This might include information about book and movie titles, song lyrics, biographies of celebrities etc. and is necessary given the reliance on current events and popular culture in making certain kinds of jokes.","We believe that Deep Learning techniques potentially offer improved handling of unknown words, long distance dependencies in text, and non–linear relationships among words and concepts. Moving forward we intend to explore a variety of these ideas and describe those briefly below."]},{"idx":4,"heading":"Future Work","paragraphs":["Our current language model approach is effective but does not account for out of vocabulary words nor long distance dependencies. CNNs in combination with LSTMs seem to be a particularly promising way to overcome these limitations (e.g., BIBREF12 ) which we will explore and compare to our existing results.","After evaluating CNNs and LSTMs we will explore how to include domain knowledge in these models. One possibility is to create word embeddings from domain specific materials and provide those to the CNNs along with more general text. Another is to investigate the use of Tree–Structured LSTMs BIBREF13 . These have the potential advantage of preserving non-linear structure in text, which may be helpful in recognizing some of the unusual variations of words and concepts that are characteristic of humor."]},{"idx":5,"heading":"Figures and Tables","paragraphs":["Table 1: Experimental results"]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1706.02222","title":"Gated Recurrent Neural Tensor Network","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Recurrent Neural Networks (RNNs), which are a powerful scheme for modeling temporal and sequential data need to capture long-term dependencies on datasets and represent them in hidden layers with a powerful model to capture more information from inputs. For modeling long-term dependencies in a dataset, the gating mechanism concept can help RNNs remember and forget previous information. Representing the hidden layers of an RNN with more expressive operations (i.e., tensor products) helps it learn a more complex relationship between the current input and the previous hidden layer information. These ideas can generally improve RNN performances. In this paper, we proposed a novel RNN architecture that combine the concepts of gating mechanism and the tensor product into a single model. By combining these two concepts into a single RNN, our proposed models learn long-term dependencies by modeling with gating units and obtain more expressive and direct interaction between input and hidden layers using a tensor product on 3-dimensional array (tensor) weight parameters. We use Long Short Term Memory (LSTM) RNN and Gated Recurrent Unit (GRU) RNN and combine them with a tensor product inside their formulations. Our proposed RNNs, which are called a Long-Short Term Memory Recurrent Neural Tensor Network (LSTMRNTN) and Gated Recurrent Unit Recurrent Neural Tensor Network (GRURNTN), are made by combining the LSTM and GRU RNN models with the tensor product. We conducted experiments with our proposed models on word-level and character-level language modeling tasks and revealed that our proposed models significantly improved their performance compared to our baseline models."]},{"idx":1,"heading":"Introduction","paragraphs":["Modeling temporal and sequential data, which is crucial in machine learning, can be applied in many areas, such as speech and natural language processing. Deep neural networks (DNNs) have garnered interest from many researchers after being successfully applied in image classification BIBREF0 and speech recognition BIBREF1 . Another type of neural network, called a recurrent neural network (RNN) is also widely used for speech recognition BIBREF2 , machine translation BIBREF3 , BIBREF4 and language modeling BIBREF5 , BIBREF6 . RNNs have achieved many state-of-the-art results. Compared to DNNs, they have extra parameters for modeling the relationships of previous or future hidden states with current input where the RNN parameters are shared for each input time-step.","Generally, RNNs can be separated by a simple RNN without gating units, such as the Elman RNN BIBREF7 , the Jordan RNN BIBREF8 , and such advanced RNNs with gating units as the Long-Short Term Memory (LSTM) RNN BIBREF9 and the Gated Recurrent Unit (GRU) RNN BIBREF4 . A simple RNN usually adequate to model some dataset and a task with short-term dependencies like slot filling for spoken language understanding BIBREF10 . However, for more difficult tasks like language modeling and machine translation where most predictions need longer information and a historical context from each sentence, gating units are needed to achieve good performance. With gating units for blocking and passing information from previous or future hidden layer, we can learn long-term information and recursively backpropagate the error from our prediction without suffering from vanishing or exploding gradient problems BIBREF9 . In spite of this situation, the concept of gating mechanism does not provide an RNN with a more powerful way to model the relation between the current input and previous hidden layer representations.","Most interactions inside RNNs between current input and previous (or future) hidden states are represented using linear projection and addition and are transformed by the nonlinear activation function. The transition is shallow because no intermediate hidden layers exist for projecting the hidden states BIBREF11 . To get a more powerful representation on the hidden layer, Pascanu et al. BIBREF11 modified RNNs with an additional nonlinear layer from input to the hidden layer transition, hidden to hidden layer transition and also hidden to output layer transition. Socher et al. BIBREF12 , BIBREF13 proposed another approach using a tensor product for calculating output vectors given two input vectors. They modified a Recursive Neural Network (RecNN) to overcome those limitations using more direct interaction between two input layers. This architecture is called a Recursive Neural Tensor Network (RecNTN), which uses a tensor product between child input vectors to represent the parent vector representation. By adding the tensor product operation to calculate their parent vector, RecNTN significantly improves the performance of sentiment analysis and reasoning on entity relations tasks compared to standard RecNN architecture. However, those models struggle to learn long-term dependencies because the do not utilize the concept of gating mechanism.","In this paper, we proposed a new RNN architecture that combine the gating mechanism and tensor product concepts to incorporate both advantages in a single architecture. Using the concept of such gating mechanisms as LSTMRNN and GRURNN, our proposed architecture can learn temporal and sequential data with longer dependencies between each input time-step than simple RNNs without gating units and combine the gating units with tensor products to represent the hidden layer with more powerful operation and direct interaction. Hidden states are generated by the interaction between current input and previous (or future) hidden states using a tensor product and a non-linear activation function allows more expressive model representation. We describe two different models based on LSTMRNN and GRURNN. LSTMRNTN is our proposed model for the combination between a LSTM unit with a tensor product inside its cell equation and GRURNTN is our name for a GRU unit with a tensor product inside its candidate hidden layer equation.","In Section \"Background\" , we provide some background information related to our research. In Section \"Proposed Architecture\" , we describe our proposed RNN architecture in detail. We evaluate our proposed RNN architecture on word-level and character-level language modeling tasks and reported the result in Section \"Experiment Settings\" . We present related works in Section \"Related Work\" . Section \"Conclusion\" summarizes our paper and provides some possible future improvements."]},{"idx":2,"heading":"Recurrent Neural Network","paragraphs":["A Recurrent Neural Network (RNN) is one kind of neural network architecture for modeling sequential and temporal dependencies BIBREF2 . Typically, we have input sequence $\\mathbf {x}=(x_1,...,x_{T})$ and calculate hidden vector sequence $\\mathbf {h}=(h_1,...,h_{T})$ and output vector sequence $\\mathbf {y}=(y_1,...,y_T)$ with RNNs. A standard RNN at time $t$ -th is usually formulated as:","$$h_t &=& f(x_t W_{xh} + h_{t-1} W_{hh} + b_h) \\\\ y_t &=& g(h_t W_{hy} + b_y).$$ (Eq. 2)","where $W_{xh}$ represents the input layer to the hidden layer weight matrix, $W_{hh}$ represents hidden to hidden layer weight matrix, $W_{hy}$ represents the hidden to the output weight matrix, $b_h$ and $b_y$ represent bias vectors for the hidden and output layers. $f(\\cdot )$ and $g(\\cdot )$ are nonlinear activation functions such as sigmoid or tanh."]},{"idx":3,"heading":"Gated Recurrent Neural Network","paragraphs":["Simple RNNs are hard to train to capture long-term dependencies from long sequential datasets because the gradient can easily explode or vanish BIBREF14 , BIBREF15 . Because the gradient (usually) vanishes after several steps, optimizing a simple RNN is more complicated than standard neural networks. To overcome the disadvantages of simple RNNs, several researches have been done. Instead of using a first-order optimization method, one approach optimized the RNN using a second-order Hessian Free optimization BIBREF16 . Another approach, which addressed the vanishing and exploding gradient problem, modified the RNN architecture with additional parameters to control the information flow from previous hidden layers using the gating mechanism concept BIBREF9 . A gated RNN is a special recurrent neural network architecture that overcomes this weakness of a simple RNN by introducing gating units. There are variants from RNN with gating units, such as Long Short Term Memory (LSTM) RNN and Gated Recurrent Unit (GRU) RNN. In the following sections, we explain both LSTMRNN and GRURNN in more detail.","A Long Short Term Memory (LSTM) BIBREF9 is a gated RNN with three gating layers and memory cells. The gating layers are used by the LSTM to control the existing memory by retaining the useful information and forgetting the unrelated information. Memory cells are used for storing the information across time. The LSTM hidden layer at time $t$ is defined by the following equations BIBREF17 :","$$i_t &=& \\sigma (x_t W_{xi} + h_{t-1} W_{hi} + c_{t-1} W_{ci} + b_i) \\\\ f_t &=& \\sigma (x_t W_{xf} + h_{t-1} W_{hf} + c_{t-1} W_{cf} + b_f) \\\\ c_t &=& f_t \\odot c_{t-1} + i_t \\odot \\tanh (x_t W_{xc} + h_{t-1} W_{hc} + b_c) \\\\ o_t &=& \\sigma (x_t W_{xo} + h_{t-1} W_{ho} + c_t W_{co} + b_o) \\\\ h_t &=& o_t \\odot \\tanh (c_t)$$ (Eq. 6)","where $\\sigma (\\cdot )$ is sigmoid activation function and $i_t, f_t, o_t$ and $c_t$ are respectively the input gates, the forget gates, the output gates and the memory cells at time-step $t$ . The input gates keep the candidate memory cell values that are useful for memory cell computation, and the forget gates keep the previous memory cell values that are useful for calculating the current memory cell. The output gates filter which the memory cell values that are useful for the output or next hidden layer input.","A Gated Recurrent Unit (GRU) BIBREF4 is a gated RNN with similar properties to a LSTM. However, there are several differences: a GRU does not have separated memory cells BIBREF18 , and instead of three gating layers, it only has two gating layers: reset gates and update gates. The GRU hidden layer at time $t$ is defined by the following equations BIBREF4 :","$$r_t &=& \\sigma (x_t W_{xr} + h_{t-1} W_{hr} + b_r)\\\\ z_t &=& \\sigma (x_t W_{xz} + h_{t-1} W_{hz} + b_r)\\\\ \\tilde{h_t} &=& f(x_t W_{xh} + (r_t \\odot h_{t-1}) W_{hh} + b_h)\\\\ h_t &=& (1 - z_t) \\odot h_{t-1} + z_t \\odot \\tilde{h_t}$$ (Eq. 9)","where $\\sigma (\\cdot )$ is a sigmoid activation function, $r_t, z_t$ are reset and update gates, $\\tilde{h_t}$ is the candidate hidden layer values and $h_t$ is the hidden layer values at time- $t$ . The reset gates determine which previous hidden layer value is useful for generating the current candidate hidden layer. The update gates keeps the previous hidden layer values or replaced by new candidate hidden layer values. In spite of having one fewer gating layer, the GRU can match LSTM's performance and its convergence speed convergence sometimes outperformed LSTM BIBREF18 ."]},{"idx":4,"heading":"Recursive Neural Tensor Network","paragraphs":["A Recursive Neural Tensor Network (RecNTN) is a variant of a Recursive Neural Network (RecNN) for modeling input data with variable length properties and tree structure dependencies between input features BIBREF19 . To compute the input representation with RecNN, the input must be parsed into a binary tree where each leaf node represents input data. Then, the parent vectors are computed in a bottom-up fashion, following the above computed tree structure whose information can be built using external computation tools (i.e., syntactic parser) or some heuristic from our dataset observations.","Given Fig. 4 , $p_1$ , $p_2$ and $y$ was defined by:","$$ p_1 &=& f\\left( \\begin{bmatrix} x_1 & x_2 \\end{bmatrix} W + b \\right) \\\\ p_2 &=& f\\left( \\begin{bmatrix} p_1 & x_3 \\end{bmatrix} W + b \\right) \\\\ y &=& g\\left( p_2 W_y + b_y \\right)$$ (Eq. 13)","where $f(\\cdot )$ is nonlinear activation function, such as sigmoid or tanh, $g(\\cdot )$ depends on our task, $W \\in \\mathbb {R}^{2d \\times d}$ is the weight parameter for projecting child input vectors $x_1, x_2, x_3 \\in \\mathbb {R}^{d}$ into the parent vector, $W_y$ is a weight parameter for computing output vector, and $b, b_y$ are biases. If we want to train RecNN for classification tasks, $g(\\cdot )$ can be defined as a softmax function.","However, standard RecNNs have several limitations, where two vectors only implicitly interact with addition before applying a nonlinear activation function on them BIBREF12 and standard RecNNs are not able to model very long-term dependency on tree structures. Zhu et al. BIBREF20 proposed the gating mechanism into standard RecNN model to solve the latter problem. For the former limitation, the RecNN performance can be improved by adding more interaction between the two input vectors. Therefore, a new architecture called a Recursive Neural Tensor Network (RecNTN) tried to overcome the previous problem by adding interaction between two vectors using a tensor product, which is connected by tensor weight parameters. Each slice of the tensor weight can be used to capture the specific pattern between the left and right child vectors. For RecNTN, value $p_1$ from Eq. 13 and is defined by:","$$p_1 &=& f\\left( \\begin{bmatrix} x_1 & x_2 \\end{bmatrix} W_{tsr}^{[1:d]} \\begin{bmatrix} x_1 \\\\ x_2 \\end{bmatrix} + \\begin{bmatrix} x_1 & x_2 \\end{bmatrix} W + b \\right) \\\\ p_2 &=& f\\left( \\begin{bmatrix} p_1 & x_3 \\end{bmatrix} W_{tsr}^{[1:d]} \\begin{bmatrix} p_1 \\\\ x_3 \\end{bmatrix} + \\begin{bmatrix} p_1 & x_3 \\end{bmatrix} W + b \\right)$$ (Eq. 15)","where $W_{tsr}^{[1:d]} \\in \\mathbb {R}^{2d \\times 2d \\times d}$ is the tensor weight to map the tensor product between two children vectors. Each slice $W_{tsr}^{[i]}$ is a matrix $\\mathbb {R}^{2d \\times 2d}$ . For more details, we visualize the calculation for $p_1$ in Fig. 5 ."]},{"idx":5,"heading":"Gated Recurrent Unit Recurrent Neural Tensor Network (GRURNTN)","paragraphs":["Previously in Sections \"Experiment Settings\" and \"Recursive Neural Tensor Network\" , we discussed that the gating mechanism concept can helps RNNs learn long-term dependencies from sequential input data and that adding more powerful interaction between the input and hidden layers simultaneously with the tensor product operation in a bilinear form improves neural network performance and expressiveness. By using tensor product, we increase our model expressiveness by using second-degree polynomial interactions, compared to first-degree polynomial interactions on standard dot product followed by addition in common RNNs architecture. Therefore, in this paper we proposed a Gated Recurrent Neural Tensor Network (GRURNTN) to combine these two advantages into an RNN architecture. In this architecture, the tensor product operation is applied between the current input and previous hidden layer multiplied by the reset gates for calculating the current candidate hidden layer values. The calculation is parameterized by tensor weight. To construct a GRURNTN, we defined the formulation as:","$$r_t &=& \\sigma (x_t W_{xr} + h_{t-1} W_{hr} + b_r) \\nonumber \\\\ z_t &=& \\sigma (x_t W_{xz} + h_{t-1} W_{hz} + b_z) \\nonumber \\\\ \\tilde{h_t} &=& f\\left( \\begin{bmatrix} x_t & (r \\odot h_{t-1}) \\end{bmatrix} W_{tsr}^{[1:d]} \\begin{bmatrix} x_t \\\\ (r \\odot h_{t-1}) \\end{bmatrix} \\right. \\nonumber \\\\ & & \\left. + x_t W_{xh} + (r_t \\odot h_{t-1}) W_{hh} + b_h \\right) \\\\ h_t &=& (1 - z_t) \\odot h_{t-1} + z_t \\odot \\tilde{h_t} \\nonumber $$ (Eq. 17)","where $W_{tsr}^{[1:d]} \\in \\mathbb {R}^{(i+d) \\times (i+d) \\times d}$ is a tensor weight for mapping the tensor product between the input-hidden layer, $i$ is the input layer size, and $d$ is the hidden layer size. Alternatively, in this paper we use a simpler bilinear form for calculating $\\tilde{h_t}$ :","$$\\tilde{h_t} &=& f\\left( \\begin{bmatrix} x_t \\end{bmatrix} W_{tsr}^{[1:d]} \\begin{bmatrix} (r_t \\odot h_{t-1}) \\end{bmatrix}^{\\intercal } \\right. \\nonumber \\\\ & & \\left. + x_t W_{xh} + (r_t \\odot h_{t-1}) W_{hh} + b_h \\right) $$ (Eq. 18)","where $W_{tsr}^{[i:d]} \\in \\mathbb {R}^{i \\times d \\times d}$ is a tensor weight. Each slice $W_{tsr}^{[i]}$ is a matrix $\\mathbb {R}^{i \\times d}$ . The advantage of this asymmetric version is that we can still maintain the interaction between the input and hidden layers through a bilinear form. We reduce the number of parameters from the original neural tensor network formulation by using this asymmetric version. Fig. 6 visualizes the $\\tilde{h_t}$ calculation in more detail."]},{"idx":6,"heading":"LSTM Recurrent Neural Tensor Network (LSTMRNTN)","paragraphs":["As with GRURNTN, we also applied the tensor product operation for the LSTM unit to improve its performance. In this architecture, the tensor product operation is applied between the current input and the previous hidden layers to calculate the current memory cell. The calculation is parameterized by the tensor weight. We call this architecture a Long Short Term Memory Recurrent Neural Tensor Network (LSTMRNTN). To construct an LSTMRNTN, we defined its formulation:","$$i_t &=& \\sigma (x_t W_{xi} + h_{t-1} W_{hi} + c_{t-1} W_{ci} + b_i) \\nonumber \\\\ f_t &=& \\sigma (x_t W_{xf} + h_{t-1} W_{hf} + c_{t-1} W_{cf} + b_f) \\nonumber \\\\ \\tilde{c_t} &=& \\tanh \\left( \\begin{bmatrix} x_t \\end{bmatrix} W_{tsr}^{[1:d]} \\begin{bmatrix} h_{t-1} \\end{bmatrix} \\right. \\nonumber \\\\ & & \\left. + x_t W_{xc} + h_{t-1} W_{hc} + b_c \\right) \\\\ c_t &=& f_t \\odot c_{t-1} + i_t \\odot \\tilde{c_t} \\\\ o_t &=& \\sigma (x_t W_{xo} + h_{t-1} W_{ho} + c_t W_{co} + b_o) \\nonumber \\\\ h_t &=& o_t \\odot \\tanh (c_t) \\nonumber $$ (Eq. 21)","where $W_{tsr}^{[1:d]} \\in R^{i \\times d \\times d}$ is a tensor weight to map the tensor product between current input $x_t$ and previous hidden layer $h_{t-1}$ into our candidate cell $\\tilde{c_t}$ . Each slice $W_{tsr}^{[i]}$ is a matrix $\\mathbb {R}^{i \\times d}$ . Fig. 7 visualizes the $\\tilde{c_t}$ calculation in more detail."]},{"idx":7,"heading":"Optimizing Tensor Weight using Backpropagation Through Time","paragraphs":["In this section, we explain how to train the tensor weight for our proposed architecture. Generally, we use backpropagation to train most neural network models BIBREF21 . For training an RNN, researchers tend to use backpropagation through time (BPTT) where the recurrent operation is unfolded as a feedforward neural network along with the time-step when we backpropagate the error BIBREF22 , BIBREF23 . Sometimes we face a performance issue when we unfold our RNN on such very long sequences. To handle that issue, we can use the truncated BPTT BIBREF5 to limit the number of time-steps when we unfold our RNN during backpropagation.","Assume we want to do segment classification BIBREF24 with an RNN trained as function $f : \\mathbf {x} \\rightarrow \\mathbf {y}$ , where $\\mathbf {x} = (x_1,...,x_T)$ as an input sequence and $\\mathbf {y} = (y_1,...,y_T)$ is an output label sequence. In this case, probability output label sequence $y$ , given input sequence $\\mathbf {x}$ , is defined as:","$$P(\\mathbf {y}|\\mathbf {x}) = \\prod _{i=1}^{T}P(y_i | x_1,..,x_i)$$ (Eq. 24)","Usually, we transform likelihood $P(\\mathbf {y}|\\mathbf {x})$ into a negative log-likelihood:","$$E(\\theta ) &=& -\\log P(\\mathbf {y}|\\mathbf {x}) = -\\log \\left(\\prod _{i=1}^{T} P(y_{i}|x_1,..,x_i)\\right) \\\\ &=& -\\sum _{i=1}^{T} \\log P(y_i | x_1,..,x_i)$$ (Eq. 25)","and our objective is to minimize the negative log-likelihood w.r.t all weight parameters $\\theta $ . To optimize $W_{tsr}^{[1:d]}$ weight parameters, we need to find derivative $E(\\theta )$ w.r.t $W_{tsr}^{[1:d]}$ :","$$\\frac{\\partial E(\\theta )}{\\partial W_{tsr}^{[1:d]}} &=& \\sum _{i=1}^{T} \\frac{\\partial E_i(\\theta )}{\\partial W_{tsr}^{[1:d]}} \\nonumber $$ (Eq. 26)","For applying backpropagation through time, we need to unfold our GRURNTN and backpropagate the error from $E_i(\\theta )$ to all candidate hidden layer $\\tilde{h_j}$ to accumulate $W_{tsr}^{[1..d]}$ gradient where $j \\in [1..i]$ . If we want to use the truncated BPTT to ignore the history past over $K$ time-steps, we can limit $j \\in [max(1, i-K) .. i]$ . We define the standard BPTT on GRURNTN to calculate $\\partial E_i(\\theta ) / \\partial W_{tsr}^{[1..d]}$ :","$$\\frac{\\partial E_i(\\theta )}{\\partial W_{tsr}^{[1:d]}} &=& \\sum _{j=1}^{i} \\frac{\\partial E_i(\\theta )}{\\partial \\tilde{h_j}} \\frac{\\partial \\tilde{h_j}}{\\partial W_{tsr}^{[1:d]}} \\nonumber \\\\ &=& \\sum _{j=1}^{i} \\frac{\\partial E_i(\\theta )}{\\partial \\tilde{h_j}}\\frac{\\partial \\tilde{h_j}}{\\partial a_j} \\frac{\\partial a_j}{\\partial W_{tsr}^{[1:d]}} \\nonumber \\\\ &=& \\sum _{j=1}^{i} \\frac{\\partial E_i(\\theta )}{\\partial \\tilde{h_j}} f^{\\prime }(a_j) \\begin{bmatrix} x_j \\end{bmatrix}^{\\intercal } \\begin{bmatrix} (r_j \\odot h_{j-1}) \\end{bmatrix} $$ (Eq. 27)","where","$$ a_j &=& \\left( \\begin{bmatrix} x_j \\end{bmatrix} W_{tsr}^{[1:d]} \\begin{bmatrix} (r_j \\odot h_{j-1}) \\end{bmatrix}^{\\intercal } \\right. \\nonumber \\\\ & & \\left. + x_j W_{xh} + (r_j \\odot h_{j-1}) W_{hh} + b_h \\right) \\nonumber $$ (Eq. 28)","and $f^{\\prime }(\\cdot )$ is a function derivative from our activation function :","$$f^{\\prime }(a_j) = {\\left\\lbrace \\begin{array}{ll} (1-f(a_j)^2), & \\text{if } f(\\cdot ) \\text{ is $\\tanh $ function} \\\\ f(a_j)(1-f(a_j)), & \\text{if } f(\\cdot ) \\text{ is sigmoid function} \\end{array}\\right.} \\nonumber $$ (Eq. 29)","For LSTMRNTN, we also need to unfold our LSTMRNN and backpropagate the error from $E_i(\\theta )$ to all cell layers $c_j$ to accumulate $W_{tsr}^{[1..d]}$ gradients where $j \\in [1..i]$ . We define the standard BPTT on LSTMRNTN to calculate $\\partial E_i(\\theta ) / \\partial W_{tsr}^{[1..d]}$ :","$$\\frac{\\partial E_i(\\theta )}{\\partial W_{tsr}^{[1:d]}} &=& \\sum _{j=1}^{i} \\frac{\\partial E_i{(\\theta )}}{\\partial c_j} \\frac{\\partial c_j}{\\partial W_{tsr}^{[1:d]}} \\nonumber \\\\ & = & \\sum _{j=1}^{i} \\frac{\\partial E_i{(\\theta )}}{\\partial c_j} \\frac{\\partial c_j}{\\partial \\tanh (a_j)} \\frac{\\partial \\tanh (a_j)}{\\partial a_j} \\frac{\\partial a_j}{\\partial W_{tsr}^{[1:d]}} \\nonumber \\\\ & = & \\sum _{j=1}^{i} \\frac{\\partial E_i{(\\theta )}}{\\partial c_j} i_j (1-\\tanh ^2(a_j)) \\begin{bmatrix} x_j \\end{bmatrix}^{\\intercal } \\begin{bmatrix} h_{j-1} \\end{bmatrix} $$ (Eq. 30)","where","$$ a_j &=& \\left(\\begin{bmatrix} x_j \\end{bmatrix} W_{tsr}^{[1:d]} \\begin{bmatrix} h_{j-1} \\end{bmatrix} + x_j W_{xc} + h_{j-1} W_{hc} + b_c \\right) $$ (Eq. 31)",". In both proposed models, we can see partial derivative ${\\partial E_i(\\theta )} / {\\partial W_{tsr}^{[1:d]}}$ in Eqs. 27 and 30 , the derivative from the tensor product w.r.t the tensor weight parameters depends on the values of our input and hidden layers. Then all the slices of tensor weight derivative are multiplied by the error from their corresponding pre-activated hidden unit values. From these derivations, we are able to see where each slice of tensor weight is learned more directly from their input and hidden layer values compared by using standard addition operations. After we accumulated every parameter's gradients from all the previous time-steps, we use a stochastic gradient optimization method such as AdaGrad BIBREF25 to optimize our model parameters."]},{"idx":8,"heading":"Experiment Settings","paragraphs":["Next we evaluate our proposed GRURNTN and LSTMRNTN models against baselines GRURNN and LSTMRNN with two different tasks and datasets."]},{"idx":9,"heading":"Datasets and Tasks","paragraphs":["We used a PennTreeBank (PTB) corpus, which is a standard benchmark corpus for statistical language modeling. A PTB corpus is a subset of the WSJ corpus. In this experiment, we followed the standard preprocessing step that was done by previous research BIBREF23 . The PTB dataset is divided as follows: a training set from sections 0-20 with total 930.000 words, a validation set from sections 21-22 with total 74.000 words, and a test set from sections 23-24 with total 82.000 words. The vocabulary is limited to the 10.000 most common words, and all words outside are mapped into a \" $<$ unk $>$ \" token. We used the preprocessed PTB corpus from the RNNLM-toolkit website.","We did two different language modeling tasks. First, we experimented on a word-level language model where our RNN predicts the next word probability given the previous words and current word. We used perplexity (PPL) to measure our RNN performance for word-level language modeling. The formula for calculating the PPL of word sequence $X$ is defined by:","$$PPL = 2^{-\\frac{1}{N}\\sum _{i=1}^{N} \\log _2 P(X_i|X_{1..{i-1}})}$$ (Eq. 35)","Second, we experimented on a character-level language model where our RNN predicts the next character probability given the previous characters and current character. We used the average number of bits-per-character (BPC) to measure our RNN performance for character-level language modeling. The formula for calculating the BPC of character sequence $X$ is defined by:","$$BPC = -\\frac{1}{N}\\left(\\sum _{i=1}^{N}\\log _2{p(X_i|X_{1..{i-1}})} \\right)$$ (Eq. 36)"]},{"idx":10,"heading":"Experiment Models","paragraphs":["In this experiment, we compared the performance from our baseline models GRURNN and LSTMRNN with our proposed GRURNTN and LSTMRNTN models. We used the same dimensions for the embedding matrix to represent the words and characters as the vectors of real numbers.","For the word-level language modeling task, we used 256 hidden units for GRURNTN and LSTMRNTN, 860 for GRURNN, and 740 for LSTMRNN. All of these models use 128 dimensions for word embedding. We used dropout regularization with $p=0.5$ dropout probability for GRURNTN and LSTMRNTN and $p=0.6$ for our baseline model. The total number of free parameters for GRURNN and GRURNTN were about 12 million and about 13 million for LSTMRNN and LSTMRNTN.","For the character-level language modeling task, we used 256 hidden units for GRURNTN and LSTMRNTN, 820 for GRURNN, and 600 for LSTMRNTN. All of these models used 32 dimensions for character embedding. We used dropout regularization with $p=0.25$ dropout probability. The total number of free parameters for GRURNN and GRURNTN was about 2.2 million and about 2.6 million for LSTMRNN and LSTMRNTN.","We constrained our baseline GRURNN to have a similar number of parameters as the GRURNTN model for a fair comparison. We also applied such constraints on our baseline LSTMRNN to LSTMRNTN model.","For all the experiment scenarios, we used AdaGrad for our stochastic gradient optimization method with mini-batch training and a batch size of 15 sentences. We multiplied our learning rate with a decay factor of 0.5 when the cost from the development set for current epoch is greater than previous epoch. We also used a rescaling trick on the gradient BIBREF26 when the norm was larger than 5 to avoid the issue of exploding gradients. For initializing the parameters, we used the orthogonal weight initialization trick BIBREF27 on every model."]},{"idx":11,"heading":"Character-level Language Modeling","paragraphs":["In this section, we report our experiment results on PTB character-level language modeling using our baseline models GRURNN and LSTMRNN as well as our proposed models GRURNTN and LSTMRNTN. Fig. 8 shows performance comparisons from every model based on the validation set's BPC per epoch. In this experiment, GRURNN made faster progress than LSTMRNN, but eventually LSTMRNN converged into a better BPC based on the development set. Our proposed model GRURNTN made faster and quicker progress than LSTMRNTN and converged into a similar BPC in the last epoch. Both proposed models produced lower BPC than our baseline models from the first epoch to the last epoch.","Table 1 shows PTB test set BPC among our baseline models, our proposed models and several published results. Our proposed model GRURNTN and LSTMRNTN outperformed both baseline models. GRURNTN reduced the BPC from 1.39 to 1.33 (0.06 absolute / 4.32% relative BPC) from the baseline GRURNN, and LSTMRNTN reduced the BPC from 1.37 to 1.34 (0.03 absolute / 2.22% relative BPC) from the baseline LSTMRNN. Overall, GRURNTN slightly outperformed LSTMRNTN, and both proposed models outperformed all of the baseline models on the character-level language modeling task."]},{"idx":12,"heading":"Word-level Language Modeling","paragraphs":["In this section, we report our experiment results on PTB word-level language modeling using our baseline models GRURNN and LSTMRNN and our proposed models GRURNTN and LSTMRNTN. Fig. 9 compares the performance from every models based on the validation set's PPL per epoch. In this experiment, GRURNN made faster progress than LSTMRNN. Our proposed GRURNTN's progress was also better than LSTMRNTN. The best model in this task was GRURNTN, which had a consistently lower PPL than the other models.","Table 1 shows the PTB test set PPL among our baseline models, proposed models, and several published results. Both our proposed models outperformed their baseline models. GRURNTN reduced the perplexity from 97.78 to 87.38 (10.4 absolute / 10.63% relative PPL) over the baseline GRURNN and LSTMRNTN reduced the perplexity from 108.26 to 96.97 (11.29 absolute / 10.42% relative PPL) over the baseline LSTMRNN. Overall, LSTMRNTN improved the LSTMRNN model and its performance closely resembles the baseline GRURNN. However, GRURNTN outperformed all the baseline models as well as the other models by a large margin."]},{"idx":13,"heading":"Related Work","paragraphs":["Representing hidden states with deeper operations was introduced just a few years ago BIBREF11 . In these works, Pascanu et al. BIBREF11 use additional nonlinear layers for representing the transition from input to hidden layers, hidden to hidden layers, and hidden to output layers. They also improved the RNN architecture by a adding shortcut connection in the deep transition by skipping the intermediate layers. Another work from BIBREF33 proposed a new RNN design for a stacked RNN model called Gated Feedback RNN (GFRNN), which adds more connections from all the previous time-step stacked hidden layers into the current hidden layer computations. Despite adding additional transition layers and connection weight from previous hidden layers, all of these models still represent the input and hidden layer relationships by using linear projection, addition and nonlinearity transformation.","On the tensor-based models, Irsoy et al. BIBREF34 proposed a simple RNN with a tensor product between the input and hidden layers. Such architecture resembles RecNTN, given a parse tree with a completely unbalanced tree on one side. Another work from BIBREF35 also use tensor products for representing hidden layers on DNN. By splitting the weight matrix into two parallel weight matrices, they calculated two parallel hidden layers and combined the pair of hidden layers using a tensor product. However, since not all of those models use a gating mechanism, the tensor parameters and tensor product operation can not be fully utilized because of the vanishing (or exploding) gradient problem.","On the recurrent neural network-based model, Sutskever et al. BIBREF30 proposed multiplicative RNN (mRNN) for character-level language modeling using tensor as the weight parameters. They proposed two different models. The first selected a slice of tensor weight based on the current character input, and the second improved the first model with factorization for constructing a hidden-to-hidden layer weight. However, those models fail to fully utilize the tensor weight with the tensor product. After they selected the weight matrix based on the current input information, they continue to use linear projection, addition, and nonlinearity for interacting between the input and hidden layers.","To the best of our knowledge, none of these works combined the gating mechanism and tensor product concepts into a single neural network architecture. In this paper, we built a new RNN by combining gating units and tensor products into a single RNN architecture. We expect that our proposed GRURNTN and LSTMRNTN architecture will improve the RNN performance for modeling temporal and sequential datasets."]},{"idx":14,"heading":"Conclusion","paragraphs":["We presented a new RNN architecture by combining the gating mechanism and tensor product concepts. Our proposed architecture can learn long-term dependencies from temporal and sequential data using gating units as well as more powerful interaction between the current input and previous hidden layers by introducing tensor product operations. From our experiment on the PennTreeBank corpus, our proposed models outperformed the baseline models with a similar number of parameters in character-level language modeling and word-level language modeling tasks. In a character-level language modeling task, GRURNTN obtained 0.06 absolute (4.32% relative) BPC reduction over GRURNN, and LSTMRNTN obtained 0.03 absolute (2.22% relative) BPC reduction over LSTMRNN. In a word-level language modeling task, GRURNTN obtained 10.4 absolute (10.63% relative) PPL reduction over GRURNN, and LSTMRNTN obtained 11.29 absolute (10.42% relative PPL) reduction over LSTMRNN. In the future, we will investigate the possibility of combining our model with other stacked RNNs architecture, such as Gated Feedback RNN (GFRNN). We would also like to explore other possible tensor operations and integrate them with our RNN architecture. By applying these ideas together, we expect to gain further performance improvement. Last, for further investigation we will apply our proposed models to other temporal and sequential tasks, such as speech recognition and video recognition."]},{"idx":15,"heading":"Acknowledgements","paragraphs":["Part of this research was supported by JSPS KAKENHI Grant Number 26870371."]},{"idx":16,"heading":"Figures and Tables","paragraphs":["Fig. 2. Long Short Term Memory Unit.","Fig. 1. Recurrent Neural Network","Fig. 4. Computation for parent values p1 and p2 was done in a bottom-up fashion. Visible node leaves x1, x2, and x3 are processed based on the given binary tree structure.","Fig. 5. Calculating vector p1 from left input x1 and right input x2 based on Eq. 15","Fig. 3. Gated Recurrent Unit","Fig. 6. Calculating candidate hidden layer h̃t from current input xt and previous hidden layer multiplied by reset gate r · ht1 based on Eq. 18","Fig. 7. Calculating candidate cell layer c̃t from current input xt and previous hidden layer ht−1 based on Eq. 19","TABLE I PennTreeBank test set BPC","Fig. 8. Comparison among GRURNN, GRURNTN, LSTMRNN, and LSTMRNTN bits-per-character (BPC) per epoch on PTB validation set. Note : a lower BPC score is better","TABLE II PennTreeBank test set PPL","Fig. 9. Comparison among GRURNN, GRURNTN, LSTMRNN and LSTMRNTN perplexity (PPL) per epoch on the PTB validation set. Note : a lower PPL value is better"]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1706.06894","title":"Stance Detection in Turkish Tweets","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Stance detection is a classification problem in natural language processing where for a text and target pair, a class result from the set {Favor, Against, Neither} is expected. It is similar to the sentiment analysis problem but instead of the sentiment of the text author, the stance expressed for a particular target is investigated in stance detection. In this paper, we present a stance detection tweet data set for Turkish comprising stance annotations of these tweets for two popular sports clubs as targets. Additionally, we provide the evaluation results of SVM classifiers for each target on this data set, where the classifiers use unigram, bigram, and hashtag features. This study is significant as it presents one of the initial stance detection data sets proposed so far and the first one for Turkish language, to the best of our knowledge. The data set and the evaluation results of the corresponding SVM-based approaches will form plausible baselines for the comparison of future studies on stance detection."]},{"idx":1,"heading":"Introduction","paragraphs":["Stance detection (also called stance identification or stance classification) is one of the considerably recent research topics in natural language processing (NLP). It is usually defined as a classification problem where for a text and target pair, the stance of the author of the text for that target is expected as a classification output from the set: {Favor, Against, Neither} BIBREF0 .","Stance detection is usually considered as a subtask of sentiment analysis (opinion mining) BIBREF1 topic in NLP. Both are mostly performed on social media texts, particularly on tweets, hence both are important components of social media analysis. Nevertheless, in sentiment analysis, the sentiment of the author of a piece of text usually as Positive, Negative, and Neutral is explored while in stance detection, the stance of the author of the text for a particular target (an entity, event, etc.) either explicitly or implicitly referred to in the text is considered. Like sentiment analysis, stance detection systems can be valuable components of information retrieval and other text analysis systems BIBREF0 .","Previous work on stance detection include BIBREF2 where a stance classifier based on sentiment and arguing features is proposed in addition to an arguing lexicon automatically compiled. The ultimate approach performs better than distribution-based and uni-gram-based baseline systems BIBREF2 . In BIBREF3 , the authors show that the use of dialogue structure improves stance detection in on-line debates. In BIBREF4 , Hasan and Ng carry out stance detection experiments using different machine learning algorithms, training data sets, features, and inter-post constraints in on-line debates, and draw insightful conclusions based on these experiments. For instance, they find that sequence models like HMMs perform better at stance detection when compared with non-sequence models like Naive Bayes (NB) BIBREF4 . In another related study BIBREF5 , the authors conclude that topic-independent features can be exploited for disagreement detection in on-line dialogues. The employed features include agreement, cue words, denial, hedges, duration, polarity, and punctuation BIBREF5 . Stance detection on a corpus of student essays is considered in BIBREF6 . After using linguistically-motivated feature sets together with multivalued NB and SVM as the learning models, the authors conclude that they outperform two baseline approaches BIBREF6 . In BIBREF7 , the author claims that Wikipedia can be used to determine stances about controversial topics based on their previous work regarding controversy extraction on the Web.","Among more recent related work, in BIBREF8 stance detection for unseen targets is studied and bidirectional conditional encoding is employed. The authors state that their approach achieves state-of-the art performance rates BIBREF8 on SemEval 2016 Twitter Stance Detection corpus BIBREF0 . In BIBREF9 , a stance-community detection approach called SCIFNET is proposed. SCIFNET creates networks of people who are stance targets, automatically from the related document collections BIBREF9 using stance expansion and refinement techniques to arrive at stance-coherent networks. A tweet data set annotated with stance information regarding six predefined targets is proposed in BIBREF10 where this data set is annotated through crowdsourcing. The authors indicate that the data set is also annotated with sentiment information in addition to stance, so it can help reveal associations between stance and sentiment BIBREF10 . Lastly, in BIBREF0 , SemEval 2016's aforementioned shared task on Twitter Stance Detection is described. Also provided are the results of the evaluations of 19 systems participating in two subtasks (one with training data set provided and the other without an annotated data set) of the shared task BIBREF0 .","In this paper, we present a tweet data set in Turkish annotated with stance information, where the corresponding annotations are made publicly available. The domain of the tweets comprises two popular football clubs which constitute the targets of the tweets included. We also provide the evaluation results of SVM classifiers (for each target) on this data set using unigram, bigram, and hashtag features.","To the best of our knowledge, the current study is the first one to target at stance detection in Turkish tweets. Together with the provided annotated data set and the corresponding evaluations with the aforementioned SVM classifiers which can be used as baseline systems, our study will hopefully help increase social media analysis studies on Turkish content.","The rest of the paper is organized as follows: In Section SECREF2 , we describe our tweet data set annotated with the target and stance information. Section SECREF3 includes the details of our SVM-based stance classifiers and their evaluation results with discussions. Section SECREF4 includes future research topics based on the current study, and finally Section SECREF5 concludes the paper with a summary."]},{"idx":2,"heading":"A Stance Detection Data Set","paragraphs":["We have decided to consider tweets about popular sports clubs as our domain for stance detection. Considerable amounts of tweets are being published for sports-related events at every instant. Hence we have determined our targets as Galatasaray (namely Target-1) and Fenerbahçe (namely, Target-2) which are two of the most popular football clubs in Turkey. As is the case for the sentiment analysis tools, the outputs of the stance detection systems on a stream of tweets about these clubs can facilitate the use of the opinions of the football followers by these clubs.","In a previous study on the identification of public health-related tweets, two tweet data sets in Turkish (each set containing 1 million random tweets) have been compiled where these sets belong to two different periods of 20 consecutive days BIBREF11 . We have decided to use one of these sets (corresponding to the period between August 18 and September 6, 2015) and firstly filtered the tweets using the possible names used to refer to the target clubs. Then, we have annotated the stance information in the tweets for these targets as Favor or Against. Within the course of this study, we have not considered those tweets in which the target is not explicitly mentioned, as our initial filtering process reveals.","For the purposes of the current study, we have not annotated any tweets with the Neither class. This stance class and even finer-grained classes can be considered in further annotation studies. We should also note that in a few tweets, the target of the stance was the management of the club while in some others a particular footballer of the club is praised or criticised. Still, we have considered the club as the target of the stance in all of the cases and carried out our annotations accordingly.","At the end of the annotation process, we have annotated 700 tweets, where 175 tweets are in favor of and 175 tweets are against Target-1, and similarly 175 tweets are in favor of and 175 are against Target-2. Hence, our data set is a balanced one although it is currently limited in size. The corresponding stance annotations are made publicly available at http://ceng.metu.edu.tr/ INLINEFORM0 e120329/ Turkish_Stance_Detection_Tweet_Dataset.csv in Comma Separated Values (CSV) format. The file contains three columns with the corresponding headers. The first column is the tweet id of the corresponding tweet, the second column contains the name of the stance target, and the last column includes the stance of the tweet for the target as Favor or Against.","To the best of our knowledge, this is the first publicly-available stance-annotated data set for Turkish. Hence, it is a significant resource as there is a scarcity of annotated data sets, linguistic resources, and NLP tools available for Turkish. Additionally, to the best of our knowledge, it is also significant for being the first stance-annotated data set including sports-related tweets, as previous stance detection data sets mostly include on-line texts on political/ethical issues."]},{"idx":3,"heading":"Stance Detection Experiments Using SVM Classifiers","paragraphs":["It is emphasized in the related literature that unigram-based methods are reliable for the stance detection task BIBREF2 and similarly unigram-based models have been used as baseline models in studies such as BIBREF0 . In order to be used as a baseline and reference system for further studies on stance detection in Turkish tweets, we have trained two SVM classifiers (one for each target) using unigrams as features. Before the extraction of unigrams, we have employed automated preprocessing to filter out the stopwords in our annotated data set of 700 tweets. The stopword list used is the list presented in BIBREF12 which, in turn, is the slightly extended version of the stopword list provided in BIBREF13 .","We have used the SVM implementation available in the Weka data mining application BIBREF14 where this particular implementation employs the SMO algorithm BIBREF15 to train a classifier with a linear kernel. The 10-fold cross-validation results of the two classifiers are provided in Table TABREF1 using the metrics of precision, recall, and F-Measure.","The evaluation results are quite favorable for both targets and particularly higher for Target-1, considering the fact that they are the initial experiments on the data set. The performance of the classifiers is better for the Favor class for both targets when compared with the performance results for the Against class. This outcome may be due to the common use of some terms when expressing positive stance towards sports clubs in Turkish tweets. The same percentage of common terms may not have been observed in tweets during the expression of negative stances towards the targets. Yet, completely the opposite pattern is observed in stance detection results of baseline systems given in BIBREF0 , i.e., better F-Measure rates have been obtained for the Against class when compared with the Favor class BIBREF0 . Some of the baseline systems reported in BIBREF0 are SVM-based systems using unigrams and ngrams as features similar to our study, but their data sets include all three stance classes of Favor, Against, and Neither, while our data set comprises only tweets classified as belonging to Favor or Against classes. Another difference is that the data sets in BIBREF0 have been divided into training and test sets, while in our study we provide 10-fold cross-validation results on the whole data set. On the other hand, we should also note that SVM-based sentiment analysis systems (such as those given in BIBREF16 ) have been reported to achieve better F-Measure rates for the Positive sentiment class when compared with the results obtained for the Negative class. Therefore, our evaluation results for each stance class seem to be in line with such sentiment analysis systems. Yet, further experiments on the extended versions of our data set should be conducted and the results should again be compared with the stance detection results given in the literature.","We have also evaluated SVM classifiers which use only bigrams as features, as ngram-based classifiers have been reported to perform better for the stance detection problem BIBREF0 . However, we have observed that using bigrams as the sole features of the SVM classifiers leads to quite poor results. This observation may be due to the relatively limited size of the tweet data set employed. Still, we can conclude that unigram-based features lead to superior results compared to the results obtained using bigrams as features, based on our experiments on our data set. Yet, ngram-based features may be employed on the extended versions of the data set to verify this conclusion within the course of future work.","With an intention to exploit the contribution of hashtag use to stance detection, we have also used the existence of hashtags in tweets as an additional feature to unigrams. The corresponding evaluation results of the SVM classifiers using unigrams together the existence of hashtags as features are provided in Table TABREF2 .","When the results given in Table TABREF2 are compared with the results in Table TABREF1 , a slight decrease in F-Measure (0.5%) for Target-1 is observed, while the overall F-Measure value for Target-2 has increased by 1.8%. Although we could not derive sound conclusions mainly due to the relatively small size of our data set, the increase in the performance of the SVM classifier Target-2 is an encouraging evidence for the exploitation of hashtags in a stance detection system. We leave other ways of exploiting hashtags for stance detection as a future work.","To sum up, our evaluation results are significant as reference results to be used for comparison purposes and provides evidence for the utility of unigram-based and hashtag-related features in SVM classifiers for the stance detection problem in Turkish tweets."]},{"idx":4,"heading":"Future Prospects","paragraphs":["Future work based on the current study includes the following:"]},{"idx":5,"heading":"Conclusion","paragraphs":["Stance detection is a considerably new research area in natural language processing and is considered within the scope of the well-studied topic of sentiment analysis. It is the detection of stance within text towards a target which may be explicitly specified in the text or not. In this study, we present a stance-annotated tweet data set in Turkish where the targets of the annotated stances are two popular sports clubs in Turkey. The corresponding annotations are made publicly-available for research purposes. To the best of our knowledge, this is the first stance detection data set for the Turkish language and also the first sports-related stance-annotated data set. Also presented in this study are SVM classifiers (one for each target) utilizing unigram and bigram features in addition to using the existence of hashtags as another feature. 10-fold cross validation results of these classifiers are presented which can be used as reference results by prospective systems. Both the annotated data set and the classifiers with evaluations are significant since they are the initial contributions to stance detection problem in Turkish tweets."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1706.08568","title":"Neural Question Answering at BioASQ 5B","sections":[{"idx":0,"heading":"Abstract","paragraphs":["This paper describes our submission to the 2017 BioASQ challenge. We participated in Task B, Phase B which is concerned with biomedical question answering (QA). We focus on factoid and list question, using an extractive QA model, that is, we restrict our system to output substrings of the provided text snippets. At the core of our system, we use FastQA, a state-of-the-art neural QA system. We extended it with biomedical word embeddings and changed its answer layer to be able to answer list questions in addition to factoid questions. We pre-trained the model on a large-scale open-domain QA dataset, SQuAD, and then fine-tuned the parameters on the BioASQ training set. With our approach, we achieve state-of-the-art results on factoid questions and competitive results on list questions."]},{"idx":1,"heading":"Introduction","paragraphs":["BioASQ is a semantic indexing, question answering (QA) and information extraction challenge BIBREF0 . We participated in Task B of the challenge which is concerned with biomedical QA. More specifically, our system participated in Task B, Phase B: Given a question and gold-standard snippets (i.e., pieces of text that contain the answer(s) to the question), the system is asked to return a list of answer candidates.","The fifth BioASQ challenge is taking place at the time of writing. Five batches of 100 questions each were released every two weeks. Participating systems have 24 hours to submit their results. At the time of writing, all batches had been released.","The questions are categorized into different question types: factoid, list, summary and yes/no. Our work concentrates on answering factoid and list questions. For factoid questions, the system's responses are interpreted as a ranked list of answer candidates. They are evaluated using mean-reciprocal rank (MRR). For list questions, the system's responses are interpreted as a set of answers to the list question. Precision and recall are computed by comparing the given answers to the gold-standard answers. F1 score, i.e., the harmonic mean of precision and recall, is used as the official evaluation measure .","Most existing biomedical QA systems employ a traditional QA pipeline, similar in structure to the baseline system by weissenborn2013answering. They consist of several discrete steps, e.g., named-entity recognition, question classification, and candidate answer scoring. These systems require a large amount of resources and feature engineering that is specific to the biomedical domain. For example, OAQA BIBREF1 , which has been very successful in last year's challenge, uses a biomedical parser, entity tagger and a thesaurus to retrieve synonyms.","Our system, on the other hand, is based on a neural network QA architecture that is trained end-to-end on the target task. We build upon FastQA BIBREF2 , an extractive factoid QA system which achieves state-of-the-art results on QA benchmarks that provide large amounts of training data. For example, SQuAD BIBREF3 provides a dataset of $\\approx 100,000$ questions on Wikipedia articles. Our approach is to train FastQA (with some extensions) on the SQuAD dataset and then fine-tune the model parameters on the BioASQ training set.","Note that by using an extractive QA network as our central component, we restrict our system's responses to substrings in the provided snippets. This also implies that the network will not be able to answer yes/no questions. We do, however, generalize the FastQA output layer in order to be able to answer list questions in addition to factoid questions."]},{"idx":2,"heading":"Model","paragraphs":["Our system is a neural network which takes as input a question and a context (i.e., the snippets) and outputs start and end pointers to tokens in the context. At its core, we use FastQA BIBREF2 , a state-of-the-art neural QA system. In the following, we describe our changes to the architecture and how the network is trained."]},{"idx":3,"heading":"Network architecture","paragraphs":["In the input layer, the context and question tokens are mapped to high-dimensional word vectors. Our word vectors consists of three components, which are concatenated to form a single vector:","GloVe embedding: We use 300-dimensional GloVe embeddings BIBREF4 which have been trained on a large collection of web documents.","Character embedding: This embedding is computed by a 1-dimensional convolutional neural network from the characters of the words, as introduced by seo2016bidirectional.","Biomedical Word2Vec embeddings: We use the biomedical word embeddings provided by biomedicalword2vec. These are 200-dimensional Word2Vec embeddings BIBREF5 which were trained on $\\approx 10$ million PubMed abstracts.","To the embedding vectors, we concatenate a one-hot encoding of the question type (list or factoid). Note that these features are identical for all tokens.","Following our embedding layer, we invoke FastQA in order to compute start and end scores for all context tokens. Because end scores are conditioned on the chosen start, there are $O(n^2)$ end scores where $n$ is the number of context tokens. We denote the start index by $i \\in [1, n]$ , the end index by $j \\in [i, n]$ , the start scores by $y_{start}^{i}$ , and end scores by $y_{end}^{i, j}$ .","In our output layer, the start, end, and span probabilities are computed as:","$$p_{start}^i = \\sigma (y_{start}^i)$$ (Eq. 8)","$$p_{end}^{i, \\cdot } = softmax(y_{end}^{i, \\cdot })$$ (Eq. 9)","where $\\sigma $ denotes the sigmoid function. By computing the start probability via the sigmoid rather than softmax function (as used in FastQA), we enable the model to output multiple spans as likely answer spans. This generalizes the factoid QA network to list questions."]},{"idx":4,"heading":"Training & decoding","paragraphs":["We define our loss as the cross-entropy of the correct start and end indices. In the case of multiple occurrences of the same answer, we only minimize the span of the lowest loss.","We train the network in two steps: First, the network is trained on SQuAD, following the procedure by weissenborn2017fastqa (pre-training phase). Second, we fine-tune the network parameters on BioASQ (fine-tuning phase). For both phases, we use the Adam optimizer BIBREF6 with an exponentially decaying learning rate. We start with learning rates of $10^{-3}$ and $10^{-4}$ for the pre-training and fine-tuning phases, respectively.","During fine-tuning, we extract answer spans from the BioASQ training data by looking for occurrences of the gold standard answer in the provided snippets. Note that this approach is not perfect as it can produce false positives (e.g., the answer is mentioned in a sentence which does not answer the question) and false negatives (e.g., a sentence answers the question, but the exact string used is not in the synonym list).","Because BioASQ usually contains multiple snippets for a given question, we process all snippets independently and then aggregate the answer spans, sorting globally according to their probability $p_{span}^{i, j}$ .","During the inference phase, we retrieve the top 20 answers span via beam search with beam size 20. From this sorted list of answer strings, we remove all duplicate strings. For factoid questions, we output the top five answer strings as our ranked list of answer candidates. For list questions, we use a probability cutoff threshold $t$ , such that $\\lbrace (i, j)|p_{span}^{i, j} \\ge t\\rbrace $ is the set of answers. We set $t$ to be the threshold for which the list F1 score on the development set is optimized.","In order to further tweak the performance of our systems, we built a model ensemble. For this, we trained five single models using 5-fold cross-validation on the entire training set. These models are combined by averaging their start and end scores before computing the span probabilities (Equations 8 - 10 ). As a result, we submit two systems to the challenge: The best single model (according to its development set) and the model ensemble.","We implemented our system using TensorFlow BIBREF7 . It was trained on an NVidia GForce Titan X GPU."]},{"idx":5,"heading":"Results & discussion","paragraphs":["We report the results for all five test batches of BioASQ 5 (Task 5b, Phase B) in Table 1 . Note that the performance numbers are not final, as the provided synonyms in the gold-standard answers will be updated as a manual step, in order to reflect valid responses by the participating systems. This has not been done by the time of writing. Note also that – in contrast to previous BioASQ challenges – systems are no longer allowed to provide an own list of synonyms in this year's challenge.","In general, the single and ensemble system are performing very similar relative to the rest of field: Their ranks are almost always right next to each other. Between the two, the ensemble model performed slightly better on average.","On factoid questions, our system has been very successful, winning three out of five batches. On list questions, however, the relative performance varies significantly. We expect our system to perform better on factoid questions than list questions, because our pre-training dataset (SQuAD) does not contain any list questions.","Starting with batch 3, we also submitted responses to yes/no questions by always answering yes. Because of a very skewed class distribution in the BioASQ dataset, this is a strong baseline. Because this is done merely to have baseline performance for this question type and because of the naivety of the method, we do not list or discuss the results here."]},{"idx":6,"heading":"Conclusion","paragraphs":["In this paper, we summarized the system design of our BioASQ 5B submission for factoid and list questions. We use a neural architecture which is trained end-to-end on the QA task. This approach has not been applied to BioASQ questions in previous challenges. Our results show that our approach achieves state-of-the art results on factoid questions and competitive results on list questions."]},{"idx":7,"heading":"Figures and Tables","paragraphs":["Figure 1: Neural architecture of our system. Question and context (i.e., the snippets) are mapped directly to start and end probabilities for each context token. We use FastQA (Weissenborn et al., 2017) with modified input vectors and an output layer that supports list answers in addition to factoid answers.","Table 1: Preliminary results for factoid and list questions for all five batches and for our single and ensemble systems. We report MRR and F1 scores for factoid and list questions, respectively. In parentheses, we report the rank of the respective systems relative to all other systems in the challenge. The last row averages the performance numbers of the respective system and question type across the five batches."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1707.02892","title":"A Generalized Recurrent Neural Architecture for Text Classification with Multi-Task Learning","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Multi-task learning leverages potential correlations among related tasks to extract common features and yield performance gains. However, most previous works only consider simple or weak interactions, thereby failing to model complex correlations among three or more tasks. In this paper, we propose a multi-task learning architecture with four types of recurrent neural layers to fuse information across multiple related tasks. The architecture is structurally flexible and considers various interactions among tasks, which can be regarded as a generalized case of many previous works. Extensive experiments on five benchmark datasets for text classification show that our model can significantly improve performances of related tasks with additional information from others."]},{"idx":1,"heading":"Introduction","paragraphs":["Neural network based models have been widely exploited with the prosperities of Deep Learning BIBREF0 and achieved inspiring performances on many NLP tasks, such as text classification BIBREF1 , BIBREF2 , semantic matching BIBREF3 , BIBREF4 and machine translation BIBREF5 . These models are robust at feature engineering and can represent words, sentences and documents as fix-length vectors, which contain rich semantic information and are ideal for subsequent NLP tasks.","One formidable constraint of deep neural networks (DNN) is their strong reliance on large amounts of annotated corpus due to substantial parameters to train. A DNN trained on limited data is prone to overfitting and incapable to generalize well. However, constructions of large-scale high-quality labeled datasets are extremely labor-intensive. To solve the problem, these models usually employ a pre-trained lookup table, also known as Word Embedding BIBREF6 , to map words into vectors with semantic implications. However, this method just introduces extra knowledge and does not directly optimize the targeted task. The problem of insufficient annotated resources is not solved either.","Multi-task learning leverages potential correlations among related tasks to extract common features, increase corpus size implicitly and yield classification improvements. Inspired by BIBREF7 , there are a large literature dedicated for multi-task learning with neural network based models BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 . These models basically share some lower layers to capture common features and further feed them to subsequent task-specific layers, which can be classified into three types:","In this paper, we propose a generalized multi-task learning architecture with four types of recurrent neural layers for text classification. The architecture focuses on Type-III, which involves more complicated interactions but has not been researched yet. All the related tasks are jointly integrated into a single system and samples from different tasks are trained in parallel. In our model, every two tasks can directly interact with each other and selectively absorb useful information, or communicate indirectly via a shared intermediate layer. We also design a global memory storage to share common features and collect interactions among all tasks.","We conduct extensive experiments on five benchmark datasets for text classification. Compared to learning separately, jointly learning multiple relative tasks in our model demonstrate significant performance gains for each task.","Our contributions are three-folds:"]},{"idx":2,"heading":"Single-Task Learning","paragraphs":["For a single supervised text classification task, the input is a word sequences denoted by INLINEFORM0 , and the output is the corresponding class label INLINEFORM1 or class distribution INLINEFORM2 . A lookup layer is used first to get the vector representation INLINEFORM3 of each word INLINEFORM4 . A classification model INLINEFORM5 is trained to transform each INLINEFORM6 into a predicted distribution INLINEFORM7 . DISPLAYFORM0","and the training objective is to minimize the total cross-entropy of the predicted and true distributions over all samples. DISPLAYFORM0","where INLINEFORM0 denotes the number of training samples and INLINEFORM1 is the class number."]},{"idx":3,"heading":"Multi-Task Learning","paragraphs":["Given INLINEFORM0 supervised text classification tasks, INLINEFORM1 , a jointly learning model INLINEFORM2 is trained to transform multiple inputs into a combination of predicted distributions in parallel. DISPLAYFORM0","where INLINEFORM0 are sequences from each tasks and INLINEFORM1 are the corresponding predictions.","The overall training objective of INLINEFORM0 is to minimize the weighted linear combination of costs for all tasks. DISPLAYFORM0","where INLINEFORM0 denotes the number of sample collections, INLINEFORM1 and INLINEFORM2 are class numbers and weights for each task INLINEFORM3 respectively."]},{"idx":4,"heading":"Three Perspectives of Multi-Task Learning","paragraphs":["Different tasks may differ in characteristics of the word sequences INLINEFORM0 or the labels INLINEFORM1 . We compare lots of benchmark tasks for text classification and conclude three different perspectives of multi-task learning.","Multi-Cardinality Tasks are similar except for cardinality parameters, for example, movie review datasets with different average sequence lengths and class numbers.","Multi-Domain Tasks involve contents of different domains, for example, product review datasets on books, DVDs, electronics and kitchen appliances.","Multi-Objective Tasks are designed for different objectives, for example, sentiment analysis, topics classification and question type judgment.","The simplest multi-task learning scenario is that all tasks share the same cardinality, domain and objective, while come from different sources, so it is intuitive that they can obtain useful information from each other. However, in the most complex scenario, tasks may vary in cardinality, domain and even objective, where the interactions among different tasks can be quite complicated and implicit. We will evaluate our model on different scenarios in the Experiment section."]},{"idx":5,"heading":"Methodology","paragraphs":["Recently neural network based models have obtained substantial interests in many natural language processing tasks for their capabilities to represent variable-length text sequences as fix-length vectors, for example, Neural Bag-of-Words (NBOW), Recurrent Neural Networks (RNN), Recursive Neural Networks (RecNN) and Convolutional Neural Network (CNN). Most of them first map sequences of words, n-grams or other semantic units into embedding representations with a pre-trained lookup table, then fuse these vectors with different architectures of neural networks, and finally utilize a softmax layer to predict categorical distribution for specific classification tasks. For recurrent neural network, input vectors are absorbed one by one in a recurrent way, which makes RNN particularly suitable for natural language processing tasks."]},{"idx":6,"heading":"Recurrent Neural Network","paragraphs":["A recurrent neural network maintains a internal hidden state vector INLINEFORM0 that is recurrently updated by a transition function INLINEFORM1 . At each time step INLINEFORM2 , the hidden state INLINEFORM3 is updated according to the current input vector INLINEFORM4 and the previous hidden state INLINEFORM5 . DISPLAYFORM0","where INLINEFORM0 is usually a composition of an element-wise nonlinearity with an affine transformation of both INLINEFORM1 and INLINEFORM2 .","In this way, recurrent neural networks can comprehend a sequence of arbitrary length into a fix-length vector and feed it to a softmax layer for text classification or other NLP tasks. However, gradient vector of INLINEFORM0 can grow or decay exponentially over long sequences during training, also known as the gradient exploding or vanishing problems, which makes it difficult to learn long-term dependencies and correlations for RNNs.","BIBREF12 proposed Long Short-Term Memory Network (LSTM) to tackle the above problems. Apart from the internal hidden state INLINEFORM0 , LSTM also maintains a internal hidden memory cell and three gating mechanisms. While there are numerous variants of the standard LSTM, here we follow the implementation of BIBREF13 . At each time step INLINEFORM1 , states of the LSTM can be fully represented by five vectors in INLINEFORM2 , an input gate INLINEFORM3 , a forget gate INLINEFORM4 , an output gate INLINEFORM5 , the hidden state INLINEFORM6 and the memory cell INLINEFORM7 , which adhere to the following transition functions. DISPLAYFORM0","where INLINEFORM0 is the current input, INLINEFORM1 denotes logistic sigmoid function and INLINEFORM2 denotes element-wise multiplication. By selectively controlling portions of the memory cell INLINEFORM3 to update, erase and forget at each time step, LSTM can better comprehend long-term dependencies with respect to labels of the whole sequences."]},{"idx":7,"heading":"A Generalized Architecture","paragraphs":["Based on the LSTM implementation of BIBREF13 , we propose a generalized multi-task learning architecture for text classification with four types of recurrent neural layers to convey information inside and among tasks. Figure FIGREF21 illustrates the structure design and information flows of our model, where three tasks are jointly learned in parallel.","As Figure FIGREF21 shows, each task owns a LSTM-based Single Layer for intra-task learning. Pair-wise Coupling Layer and Local Fusion Layer are designed for direct and indirect inter-task interactions. And we further utilize a Global Fusion Layer to maintain a global memory for information shared among all tasks.","Each task owns a LSTM-based Single Layer with a collection of parameters INLINEFORM0 , taking Eqs.() for example. DISPLAYFORM0","Input sequences of each task are transformed into vector representations INLINEFORM0 , which are later recurrently fed into the corresponding Single Layers. The hidden states at the last time step INLINEFORM1 of each Single Layer can be regarded as fix-length representations of the whole sequences, which are followed by a fully connected layer and a softmax non-linear layer to produce class distributions. DISPLAYFORM0","where INLINEFORM0 is the predicted class distribution for INLINEFORM1 .","Besides Single Layers, we design Coupling Layers to model direct pair-wise interactions between tasks. For each pair of tasks, hidden states and memory cells of the Single Layers can obtain extra information directly from each other, as shown in Figure FIGREF21 .","We re-define Eqs.( EQREF26 ) and utilize a gating mechanism to control the portion of information flows from one task to another. The memory content INLINEFORM0 of each Single Layer is updated on the leverage of pair-wise couplings. DISPLAYFORM0","where INLINEFORM0 controls the portion of information flow from INLINEFORM1 to INLINEFORM2 , based on the correlation strength between INLINEFORM3 and INLINEFORM4 at the current time step.","In this way, the hidden states and memory cells of each Single Layer can obtain extra information from other tasks and stronger relevance results in higher chances of reception.","Different from Coupling Layers, Local Fusion Layers introduce a shared bi-directional LSTM Layer to model indirect pair-wise interactions between tasks. For each pair of tasks, we feed the Local Fusion Layer with the concatenation of both inputs, INLINEFORM0 , as shown in Figure FIGREF21 . We denote the output of the Local Fusion Layer as INLINEFORM1 , a concatenation of hidden states from the forward and backward LSTM at each time step.","Similar to Coupling Layers, hidden states and memory cells of the Single Layers can selectively decide how much information to accept from the pair-wise Local Fusion Layers. We re-define Eqs.( EQREF29 ) by considering the interactions between the memory content INLINEFORM0 and outputs of the Local Fusion Layers as follows. DISPLAYFORM0","where INLINEFORM0 denotes the coupling term in Eqs.( EQREF29 ) and INLINEFORM1 represents the local fusion term. Again, we employ a gating mechanism INLINEFORM2 to control the portion of information flow from the Local Coupling Layers to INLINEFORM3 .","Indirect interactions between Single Layers can be pair-wise or global, so we further propose the Global Fusion Layer as a shared memory storage among all tasks. The Global Fusion Layer consists of a bi-directional LSTM Layer with the inputs INLINEFORM0 and the outputs INLINEFORM1 .","We denote the global fusion term as INLINEFORM0 and the memory content INLINEFORM1 is calculated as follows. DISPLAYFORM0","As a result, our architecture covers complicated interactions among different tasks. It is capable of mapping a collection of input sequences from different tasks into a combination of predicted class distributions in parallel, as shown in Eqs.( EQREF11 )."]},{"idx":8,"heading":"Sampling & Training","paragraphs":["Most previous multi-task learning models BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 belongs to Type-I or Type-II. The total number of input samples is INLINEFORM0 , where INLINEFORM1 are the sample numbers of each task.","However, our model focuses on Type-III and requires a 4-D tensor INLINEFORM0 as inputs, where INLINEFORM1 are total number of input collections, task number, sequence length and embedding size respectively. Samples from different tasks are jointly learned in parallel so the total number of all possible input collections is INLINEFORM2 . We propose a Task Oriented Sampling algorithm to generate sample collections for improvements of a specific task INLINEFORM3 .","[ht] Task Oriented Sampling [1] INLINEFORM0 samples from each task INLINEFORM1 ; INLINEFORM2 , the oriented task index; INLINEFORM3 , upsampling coefficient s.t. INLINEFORM4 sequence collections INLINEFORM5 and label combinations INLINEFORM6","each INLINEFORM0 generate a set INLINEFORM1 with INLINEFORM2 samples for each task: INLINEFORM3 repeat each sample for INLINEFORM4 times INLINEFORM5 randomly select INLINEFORM6 samples without replacements randomly select INLINEFORM7 samples with replacements each INLINEFORM8 randomly select a sample from each INLINEFORM9 without replacements combine their features and labels as INLINEFORM10 and INLINEFORM11 merge all INLINEFORM12 and INLINEFORM13 to produce the sequence collections INLINEFORM14 and label combinations INLINEFORM15","Given the generated sequence collections INLINEFORM0 and label combinations INLINEFORM1 , the overall loss function can be calculated based on Eqs.( EQREF12 ) and ( EQREF27 ). The training process is conducted in a stochastic manner until convergence. For each loop, we randomly select a collection from the INLINEFORM2 candidates and update the parameters by taking a gradient step."]},{"idx":9,"heading":"Experiment","paragraphs":["In this section, we design three different scenarios of multi-task learning based on five benchmark datasets for text classification. we investigate the empirical performances of our model and compare it to existing state-of-the-art models."]},{"idx":10,"heading":"Datasets","paragraphs":["As Table TABREF35 shows, we select five benchmark datasets for text classification and design three experiment scenarios to evaluate the performances of our model.","Multi-Cardinality Movie review datasets with different average lengths and class numbers, including SST-1 BIBREF14 , SST-2 and IMDB BIBREF15 .","Multi-Domain Product review datasets on different domains from Multi-Domain Sentiment Dataset BIBREF16 , including Books, DVDs, Electronics and Kitchen.","Multi-Objective Classification datasets with different objectives, including IMDB, RN BIBREF17 and QC BIBREF18 ."]},{"idx":11,"heading":"Hyperparameters and Training","paragraphs":["The whole network is trained through back propagation with stochastic gradient descent BIBREF19 . We obtain a pre-trained lookup table by applying Word2Vec BIBREF20 on the Google News corpus, which contains more than 100B words with a vocabulary size of about 3M. All involved parameters are randomly initialized from a truncated normal distribution with zero mean and standard deviation.","For each task INLINEFORM0 , we conduct TOS with INLINEFORM1 to improve its performance. After training our model on the generated sample collections, we evaluate the performance of task INLINEFORM2 by comparing INLINEFORM3 and INLINEFORM4 on the test set. We apply 10-fold cross-validation and different combinations of hyperparameters are investigated, of which the best one, as shown in Table TABREF41 , is reserved for comparisons with state-of-the-art models."]},{"idx":12,"heading":"Results","paragraphs":["We compare performances of our model with the implementation of BIBREF13 and the results are shown in Table TABREF43 . Our model obtains better performances in Multi-Domain scenario with an average improvement of 4.5%, where datasets are product reviews on different domains with similar sequence lengths and the same class number, thus producing stronger correlations. Multi-Cardinality scenario also achieves significant improvements of 2.77% on average, where datasets are movie reviews with different cardinalities.","However, Multi-Objective scenario benefits less from multi-task learning due to lacks of salient correlation among sentiment, topic and question type. The QC dataset aims to classify each question into six categories and its performance even gets worse, which may be caused by potential noises introduced by other tasks. In practice, the structure of our model is flexible, as couplings and fusions between some empirically unrelated tasks can be removed to alleviate computation costs.","We further explore the influence of INLINEFORM0 in TOS on our model, which can be any positive integer. A higher value means larger and more various samples combinations, while requires higher computation costs.","Figure FIGREF45 shows the performances of datasets in Multi-Domain scenario with different INLINEFORM0 . Compared to INLINEFORM1 , our model can achieve considerable improvements when INLINEFORM2 as more samples combinations are available. However, there are no more salient gains as INLINEFORM3 gets larger and potential noises from other tasks may lead to performance degradations. For a trade-off between efficiency and effectiveness, we determine INLINEFORM4 as the optimal value for our experiments.","In order to measure the correlation strength between two task INLINEFORM0 and INLINEFORM1 , we learn them jointly with our model and define Pair-wise Performance Gain as INLINEFORM2 , where INLINEFORM3 are the performances of tasks INLINEFORM4 and INLINEFORM5 when learned individually and jointly.","We calculate PPGs for every two tasks in Table TABREF35 and illustrate the results in Figure FIGREF47 , where darkness of colors indicate strength of correlation. It is intuitive that datasets of Multi-Domain scenario obtain relatively higher PPGs with each other as they share similar cardinalities and abundant low-level linguistic characteristics. Sentences of QC dataset are much shorter and convey unique characteristics from other tasks, thus resulting in quite lower PPGs."]},{"idx":13,"heading":"Comparisons with State-of-the-art Models","paragraphs":["We apply the optimal hyperparameter settings and compare our model against the following state-of-the-art models:","NBOW Neural Bag-of-Words that simply sums up embedding vectors of all words.","PV Paragraph Vectors followed by logistic regression BIBREF21 .","MT-RNN Multi-Task learning with Recurrent Neural Networks by a shared-layer architecture BIBREF11 .","MT-CNN Multi-Task learning with Convolutional Neural Networks BIBREF8 where lookup tables are partially shared.","MT-DNN Multi-Task learning with Deep Neural Networks BIBREF9 that utilizes bag-of-word representations and a hidden shared layer.","GRNN Gated Recursive Neural Network for sentence modeling BIBREF1 .","As Table TABREF48 shows, our model obtains competitive or better performances on all tasks except for the QC dataset, as it contains poor correlations with other tasks. MT-RNN slightly outperforms our model on SST, as sentences from this dataset are much shorter than those from IMDB and MDSD, and another possible reason may be that our model are more complex and requires larger data for training. Our model proposes the designs of various interactions including coupling, local and global fusion, which can be further implemented by other state-of-the-art models and produce better performances."]},{"idx":14,"heading":"Related Work","paragraphs":["There are a large body of literatures related to multi-task learning with neural networks in NLP BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 .","BIBREF8 belongs to Type-I and utilizes shared lookup tables for common features, followed by task-specific neural layers for several traditional NLP tasks such as part-of-speech tagging and semantic parsing. They use a fix-size window to solve the problem of variable-length texts, which can be better handled by recurrent neural networks.","BIBREF9 , BIBREF10 , BIBREF11 all belong to Type-II where samples from different tasks are learned sequentially. BIBREF9 applies bag-of-word representation and information of word orders are lost. BIBREF10 introduces an external memory for information sharing with a reading/writing mechanism for communicating, and BIBREF11 proposes three different models for multi-task learning with recurrent neural networks. However, models of these two papers only involve pair-wise interactions, which can be regarded as specific implementations of Coupling Layer and Fusion Layer in our model.","Different from the above models, our model focuses on Type-III and utilize recurrent neural networks to comprehensively capture various interactions among tasks, both direct and indirect, local and global. Three or more tasks are learned simultaneously and samples from different tasks are trained in parallel benefitting from each other, thus obtaining better sentence representations."]},{"idx":15,"heading":"Conclusion and Future Work","paragraphs":["In this paper, we propose a multi-task learning architecture for text classification with four types of recurrent neural layers. The architecture is structurally flexible and can be regarded as a generalized case of many previous works with deliberate designs. We explore three different scenarios of multi-task learning and our model can improve performances of most tasks with additional related information from others in all scenarios.","In future work, we would like to investigate further implementations of couplings and fusions, and conclude more multi-task learning perspectives."]},{"idx":16,"heading":"Figures and Tables","paragraphs":["Figure 1: A generalized recurrent neural architecture for modeling text with multi-task learning","Table 1: Five benchmark classification datasets: SST, IMDB, MDSD, RN, QC.","Figure 2: Influences of n0 in TOS on different datasets","Table 2: Hyperparameter settings","Table 4: Comparisons with state-of-the-art models","Table 3: Results of our model on different scenarios"]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1707.03569","title":"Multitask Learning for Fine-Grained Twitter Sentiment Analysis","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Traditional sentiment analysis approaches tackle problems like ternary (3-category) and fine-grained (5-category) classification by learning the tasks separately. We argue that such classification tasks are correlated and we propose a multitask approach based on a recurrent neural network that benefits by jointly learning them. Our study demonstrates the potential of multitask models on this type of problems and improves the state-of-the-art results in the fine-grained sentiment classification problem."]},{"idx":1,"heading":"Introduction","paragraphs":["Automatic classification of sentiment has mainly focused on categorizing tweets in either two (binary sentiment analysis) or three (ternary sentiment analysis) categories BIBREF0 . In this work we study the problem of fine-grained sentiment classification where tweets are classified according to a five-point scale ranging from VeryNegative to VeryPositive. To illustrate this, Table TABREF3 presents examples of tweets associated with each of these categories. Five-point scales are widely adopted in review sites like Amazon and TripAdvisor, where a user's sentiment is ordered with respect to its intensity. From a sentiment analysis perspective, this defines a classification problem with five categories. In particular, Sebastiani et al. BIBREF1 defined such classification problems whose categories are explicitly ordered to be ordinal classification problems. To account for the ordering of the categories, learners are penalized according to how far from the true class their predictions are.","Although considering different scales, the various settings of sentiment classification are related. First, one may use the same feature extraction and engineering approaches to represent the text spans such as word membership in lexicons, morpho-syntactic statistics like punctuation or elongated word counts BIBREF2 , BIBREF3 . Second, one would expect that knowledge from one task can be transfered to the others and this would benefit the performance. Knowing that a tweet is “Positive” in the ternary setting narrows the classification decision between the VeryPositive and Positive categories in the fine-grained setting. From a research perspective this raises the question of whether and how one may benefit when tackling such related tasks and how one can transfer knowledge from one task to another during the training phase.","Our focus in this work is to exploit the relation between the sentiment classification settings and demonstrate the benefits stemming from combining them. To this end, we propose to formulate the different classification problems as a multitask learning problem and jointly learn them. Multitask learning BIBREF4 has shown great potential in various domains and its benefits have been empirically validated BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 using different types of data and learning approaches. An important benefit of multitask learning is that it provides an elegant way to access resources developed for similar tasks. By jointly learning correlated tasks, the amount of usable data increases. For instance, while for ternary classification one can label data using distant supervision with emoticons BIBREF9 , there is no straightforward way to do so for the fine-grained problem. However, the latter can benefit indirectly, if the ternary and fine-grained tasks are learned jointly.","The research question that the paper attempts to answer is the following: Can twitter sentiment classification problems, and fine-grained sentiment classification in particular, benefit from multitask learning? To answer the question, the paper brings the following two main contributions: (i) we show how jointly learning the ternary and fine-grained sentiment classification problems in a multitask setting improves the state-of-the-art performance, and (ii) we demonstrate that recurrent neural networks outperform models previously proposed without access to huge corpora while being flexible to incorporate different sources of data."]},{"idx":2,"heading":"Multitask Learning for Twitter Sentiment Classification","paragraphs":["In his work, Caruana BIBREF4 proposed a multitask approach in which a learner takes advantage of the multiplicity of interdependent tasks while jointly learning them. The intuition is that if the tasks are correlated, the learner can learn a model jointly for them while taking into account the shared information which is expected to improve its generalization ability. People express their opinions online on various subjects (events, products..), on several languages and in several styles (tweets, paragraph-sized reviews..), and it is exactly this variety that motivates the multitask approaches. Specifically for Twitter for instance, the different settings of classification like binary, ternary and fine-grained are correlated since their difference lies in the sentiment granularity of the classes which increases while moving from binary to fine-grained problems.","There are two main decisions to be made in our approach: the learning algorithm, which learns a decision function, and the data representation. With respect to the former, neural networks are particularly suitable as one can design architectures with different properties and arbitrary complexity. Also, as training neural network usually relies on back-propagation of errors, one can have shared parts of the network trained by estimating errors on the joint tasks and others specialized for particular tasks. Concerning the data representation, it strongly depends on the data type available. For the task of sentiment classification of tweets with neural networks, distributed embeddings of words have shown great potential. Embeddings are defined as low-dimensional, dense representations of words that can be obtained in an unsupervised fashion by training on large quantities of text BIBREF10 .","Concerning the neural network architecture, we focus on Recurrent Neural Networks (RNNs) that are capable of modeling short-range and long-range dependencies like those exhibited in sequence data of arbitrary length like text. While in the traditional information retrieval paradigm such dependencies are captured using INLINEFORM0 -grams and skip-grams, RNNs learn to capture them automatically BIBREF11 . To circumvent the problems with capturing long-range dependencies and preventing gradients from vanishing, the long short-term memory network (LSTM) was proposed BIBREF12 . In this work, we use an extended version of LSTM called bidirectional LSTM (biLSTM). While standard LSTMs access information only from the past (previous words), biLSTMs capture both past and future information effectively BIBREF13 , BIBREF11 . They consist of two LSTM networks, for propagating text forward and backwards with the goal being to capture the dependencies better. Indeed, previous work on multitask learning showed the effectiveness of biLSTMs in a variety of problems: BIBREF14 tackled sequence prediction, while BIBREF6 and BIBREF15 used biLSTMs for Named Entity Recognition and dependency parsing respectively.","Figure FIGREF2 presents the architecture we use for multitask learning. In the top-left of the figure a biLSTM network (enclosed by the dashed line) is fed with embeddings INLINEFORM0 that correspond to the INLINEFORM1 words of a tokenized tweet. Notice, as discussed above, the biLSTM consists of two LSTMs that are fed with the word sequence forward and backwards. On top of the biLSTM network one (or more) hidden layers INLINEFORM2 transform its output. The output of INLINEFORM3 is led to the softmax layers for the prediction step. There are INLINEFORM4 softmax layers and each is used for one of the INLINEFORM5 tasks of the multitask setting. In tasks such as sentiment classification, additional features like membership of words in sentiment lexicons or counts of elongated/capitalized words can be used to enrich the representation of tweets before the classification step BIBREF3 . The lower part of the network illustrates how such sources of information can be incorporated to the process. A vector “Additional Features” for each tweet is transformed from the hidden layer(s) INLINEFORM6 and then is combined by concatenation with the transformed biLSTM output in the INLINEFORM7 layer."]},{"idx":3,"heading":"Experimental setup","paragraphs":["Our goal is to demonstrate how multitask learning can be successfully applied on the task of sentiment classification of tweets. The particularities of tweets are to be short and informal text spans. The common use of abbreviations, creative language etc., makes the sentiment classification problem challenging. To validate our hypothesis, that learning the tasks jointly can benefit the performance, we propose an experimental setting where there are data from two different twitter sentiment classification problems: a fine-grained and a ternary. We consider the fine-grained task to be our primary task as it is more challenging and obtaining bigger datasets, e.g. by distant supervision, is not straightforward and, hence we report the performance achieved for this task.","Ternary and fine-grained sentiment classification were part of the SemEval-2016 “Sentiment Analysis in Twitter” task BIBREF16 . We use the high-quality datasets the challenge organizers released. The dataset for fine-grained classification is split in training, development, development_test and test parts. In the rest, we refer to these splits as train, development and test, where train is composed by the training and the development instances. Table TABREF7 presents an overview of the data. As discussed in BIBREF16 and illustrated in the Table, the fine-grained dataset is highly unbalanced and skewed towards the positive sentiment: only INLINEFORM0 of the training examples are labeled with one of the negative classes.","Feature representation We report results using two different feature sets. The first one, dubbed nbow, is a neural bag-of-words that uses text embeddings to generate low-dimensional, dense representations of the tweets. To construct the nbow representation, given the word embeddings dictionary where each word is associated with a vector, we apply the average compositional function that averages the embeddings of the words that compose a tweet. Simple compositional functions like average were shown to be robust and efficient in previous work BIBREF17 . Instead of training embeddings from scratch, we use the pre-trained on tweets GloVe embeddings of BIBREF10 . In terms of resources required, using only nbow is efficient as it does not require any domain knowledge. However, previous research on sentiment analysis showed that using extra resources, like sentiment lexicons, can benefit significantly the performance BIBREF3 , BIBREF2 . To validate this and examine at which extent neural networks and multitask learning benefit from such features we evaluate the models using an augmented version of nbow, dubbed nbow+. The feature space of the latter, is augmented using 1,368 extra features consisting mostly of counts of punctuation symbols ('!?#@'), emoticons, elongated words and word membership features in several sentiment lexicons. Due to space limitations, for a complete presentation of these features, we refer the interested reader to BIBREF2 , whose open implementation we used to extract them.","Evaluation measure To reproduce the setting of the SemEval challenges BIBREF16 , we optimize our systems using as primary measure the macro-averaged Mean Absolute Error ( INLINEFORM0 ) given by: INLINEFORM1","where INLINEFORM0 is the number of categories, INLINEFORM1 is the set of instances whose true class is INLINEFORM2 , INLINEFORM3 is the true label of the instance INLINEFORM4 and INLINEFORM5 the predicted label. The measure penalizes decisions far from the true ones and is macro-averaged to account for the fact that the data are unbalanced. Complementary to INLINEFORM6 , we report the performance achieved on the micro-averaged INLINEFORM7 measure, which is a commonly used measure for classification.","The models To evaluate the multitask learning approach, we compared it with several other models. Support Vector Machines (SVMs) are maximum margin classification algorithms that have been shown to achieve competitive performance in several text classification problems BIBREF16 . SVM INLINEFORM0 stands for an SVM with linear kernel and an one-vs-rest approach for the multi-class problem. Also, SVM INLINEFORM1 is an SVM with linear kernel that employs the crammer-singer strategy BIBREF18 for the multi-class problem. Logistic regression (LR) is another type of linear classification method, with probabilistic motivation. Again, we use two types of Logistic Regression depending on the multi-class strategy: LR INLINEFORM2 that uses an one-vs-rest approach and multinomial Logistic Regression also known as the MaxEnt classifier that uses a multinomial criterion.","Both SVMs and LRs as discussed above treat the problem as a multi-class one, without considering the ordering of the classes. For these four models, we tuned the hyper-parameter INLINEFORM0 that controls the importance of the L INLINEFORM1 regularization part in the optimization problem with grid-search over INLINEFORM2 using 10-fold cross-validation in the union of the training and development data and then retrained the models with the selected values. Also, to account for the unbalanced classification problem we used class weights to penalize more the errors made on the rare classes. These weights were inversely proportional to the frequency of each class. For the four models we used the implementations of Scikit-learn BIBREF19 .","For multitask learning we use the architecture shown in Figure FIGREF2 , which we implemented with Keras BIBREF20 . The embeddings are initialized with the 50-dimensional GloVe embeddings while the output of the biLSTM network is set to dimension 50. The activation function of the hidden layers is the hyperbolic tangent. The weights of the layers were initialized from a uniform distribution, scaled as described in BIBREF21 . We used the Root Mean Square Propagation optimization method. We used dropout for regularizing the network. We trained the network using batches of 128 examples as follows: before selecting the batch, we perform a Bernoulli trial with probability INLINEFORM0 to select the task to train for. With probability INLINEFORM1 we pick a batch for the fine-grained sentiment classification problem, while with probability INLINEFORM2 we pick a batch for the ternary problem. As shown in Figure FIGREF2 , the error is backpropagated until the embeddings, that we fine-tune during the learning process. Notice also that the weights of the network until the layer INLINEFORM3 are shared and therefore affected by both tasks.","To tune the neural network hyper-parameters we used 5-fold cross validation. We tuned the probability INLINEFORM0 of dropout after the hidden layers INLINEFORM1 and for the biLSTM for INLINEFORM2 , the size of the hidden layer INLINEFORM3 and the probability INLINEFORM4 of the Bernoulli trials from INLINEFORM5 . During training, we monitor the network's performance on the development set and apply early stopping if the performance on the validation set does not improve for 5 consecutive epochs.","Experimental results Table TABREF9 illustrates the performance of the models for the different data representations. The upper part of the Table summarizes the performance of the baselines. The entry “Balikas et al.” stands for the winning system of the 2016 edition of the challenge BIBREF2 , which to the best of our knowledge holds the state-of-the-art. Due to the stochasticity of training the biLSTM models, we repeat the experiment 10 times and report the average and the standard deviation of the performance achieved.","Several observations can be made from the table. First notice that, overall, the best performance is achieved by the neural network architecture that uses multitask learning. This entails that the system makes use of the available resources efficiently and improves the state-of-the-art performance. In conjunction with the fact that we found the optimal probability INLINEFORM0 , this highlights the benefits of multitask learning over single task learning. Furthermore, as described above, the neural network-based models have only access to the training data as the development are hold for early stopping. On the other hand, the baseline systems were retrained on the union of the train and development sets. Hence, even with fewer resources available for training on the fine-grained problem, the neural networks outperform the baselines. We also highlight the positive effect of the additional features that previous research proposed. Adding the features both in the baselines and in the biLSTM-based architectures improves the INLINEFORM1 scores by several points.","Lastly, we compare the performance of the baseline systems with the performance of the state-of-the-art system of BIBREF2 . While BIBREF2 uses n-grams (and character-grams) with INLINEFORM0 , the baseline systems (SVMs, LRs) used in this work use the nbow+ representation, that relies on unigrams. Although they perform on par, the competitive performance of nbow highlights the potential of distributed representations for short-text classification. Further, incorporating structure and distributed representations leads to the gains of the biLSTM network, in the multitask and single task setting.","Similar observations can be drawn from Figure FIGREF10 that presents the INLINEFORM0 scores. Again, the biLSTM network with multitask learning achieves the best performance. It is also to be noted that although the two evaluation measures are correlated in the sense that the ranking of the models is the same, small differences in the INLINEFORM1 have large effect on the scores of the INLINEFORM2 measure."]},{"idx":4,"heading":"Conclusion","paragraphs":["In this paper, we showed that by jointly learning the tasks of ternary and fine-grained classification with a multitask learning model, one can greatly improve the performance on the second. This opens several avenues for future research. Since sentiment is expressed in different textual types like tweets and paragraph-sized reviews, in different languages (English, German, ..) and in different granularity levels (binary, ternary,..) one can imagine multitask approaches that could benefit from combining such resources. Also, while we opted for biLSTM networks here, one could use convolutional neural networks or even try to combine different types of networks and tasks to investigate the performance effect of multitask learning. Lastly, while our approach mainly relied on the foundations of BIBREF4 , the internal mechanisms and the theoretical guarantees of multitask learning remain to be better understood."]},{"idx":5,"heading":"Acknowledgements","paragraphs":["This work is partially supported by the CIFRE N 28/2015."]},{"idx":6,"heading":"Figures and Tables","paragraphs":["Table 1 The example demonstrates the different levels of sentiment a tweet may convey. Also, note the Twitter-specific use of language and symbols.","Figure 1. The neural network architecture for multitask learning. The biLSTM output is transformed by the hidden layers H1, HM and is led to N output layers, one for each of the tasks. The lower part of the network can be used to incorporate additional information.","Table 2 Cardinality and class distributions of the datasets.","Table 3 The scores on MAEM for the systems. The best (lowest) score is shown in bold and is achieved in the multitask setting with the biLSTM architecture of Figure 1.","Figure 2. F1 scores using the nbow+ representations. The best performance is achieved with the multitask setting."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1707.05589","title":"On the State of the Art of Evaluation in Neural Language Models","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Ongoing innovations in recurrent neural network architectures have provided a steady influx of apparently state-of-the-art results on language modelling benchmarks. However, these have been evaluated using differing code bases and limited computational resources, which represent uncontrolled sources of experimental variation. We reevaluate several popular architectures and regularisation methods with large-scale automatic black-box hyperparameter tuning and arrive at the somewhat surprising conclusion that standard LSTM architectures, when properly regularised, outperform more recent models. We establish a new state of the art on the Penn Treebank and Wikitext-2 corpora, as well as strong baselines on the Hutter Prize dataset."]},{"idx":1,"heading":"Introduction","paragraphs":["The scientific process by which the deep learning research community operates is guided by empirical studies that evaluate the relative quality of models. Complicating matters, the measured performance of a model depends not only on its architecture (and data), but it can strongly depend on hyperparameter values that affect learning, regularisation, and capacity. This hyperparameter dependence is an often inadequately controlled source of variation in experiments, which creates a risk that empirically unsound claims will be reported.","In this paper, we use a black-box hyperparameter optimisation technique to control for hyperparameter effects while comparing the relative performance of language modelling architectures based on LSTMs, Recurrent Highway Networks BIBREF0 and NAS BIBREF1 . We specify flexible, parameterised model families with the ability to adjust embedding and recurrent cell sizes for a given parameter budget and with fine grain control over regularisation and learning hyperparameters.","Once hyperparameters have been properly controlled for, we find that LSTMs outperform the more recent models, contra the published claims. Our result is therefore a demonstration that replication failures can happen due to poorly controlled hyperparameter variation, and this paper joins other recent papers in warning of the under-acknowledged existence of replication failure in deep learning BIBREF2 , BIBREF3 . However, we do show that careful controls are possible, albeit at considerable computational cost.","Several remarks can be made in light of these results. First, as (conditional) language models serve as the central building block of many tasks, including machine translation, there is little reason to expect that the problem of unreliable evaluation is unique to the tasks discussed here. However, in machine translation, carefully controlling for hyperparameter effects would be substantially more expensive because standard datasets are much larger. Second, the research community should strive for more consensus about appropriate experimental methodology that balances costs of careful experimentation with the risks associated with false claims. Finally, more attention should be paid to hyperparameter sensitivity. Models that introduce many new hyperparameters or which perform well only in narrow ranges of hyperparameter settings should be identified as such as part of standard publication practice."]},{"idx":2,"heading":"Models","paragraphs":["Our focus is on three recurrent architectures:","Our aim is strictly to do better model comparisons for these architectures and we thus refrain from including techniques that are known to push perplexities even lower, but which are believed to be largely orthogonal to the question of the relative merits of these recurrent cells. In parallel work with a remarkable overlap with ours, BIBREF5 demonstrate the utility of adding a Neural Cache BIBREF6 . Building on their work, BIBREF7 show that Dynamic Evaluation BIBREF8 contributes similarly to the final perplexity.","As pictured in Fig. FIGREF1 , our models with LSTM or NAS cells have all the standard components: an input embedding lookup table, recurrent cells stacked as layers with additive skip connections combining outputs of all layers to ease optimisation. There is an optional down-projection whose presence is governed by a hyperparameter from this combined output to a smaller space which reduces the number of output embedding parameters. Unless otherwise noted, input and output embeddings are shared, see BIBREF9 and BIBREF10 .","Dropout is applied to feedforward connections denoted by dashed arrows in the figure. From the bottom up: to embedded inputs (input dropout), to connections between layers (intra-layer dropout), to the combined and the down-projected outputs (output dropout). All these dropouts have random masks drawn independently per time step, in contrast to the dropout on recurrent states where the same mask is used for all time steps in the sequence.","RHN based models are typically conceived of as a single horizontal “highway” to emphasise how the recurrent state is processed through time. In Fig. FIGREF1 , we choose to draw their schema in a way that makes the differences from LSTMs immediately apparent. In a nutshell, the RHN state is passed from the topmost layer to the lowest layer of the next time step. In contrast, each LSTM layer has its own recurrent connection and state.","The same dropout variants are applied to all three model types, with the exception of intra-layer dropout which does not apply to RHNs since only the recurrent state is passed between the layers. For the recurrent states, all architectures use either variational dropout BIBREF11 or recurrent dropout BIBREF12 , unless explicitly noted otherwise."]},{"idx":3,"heading":"Datasets","paragraphs":["We compare models on three datasets. The smallest of them is the Penn Treebank corpus by BIBREF13 with preprocessing from BIBREF14 . We also include another word level corpus: Wikitext-2 by BIBREF15 . It is about twice the size of Penn Treebank with a larger vocabulary and much lighter preprocessing. The third corpus is Enwik8 from the Hutter Prize dataset BIBREF16 . Following common practice, we use the first 90 million characters for training, and the remaining 10 million evenly split between validation and test."]},{"idx":4,"heading":"Training details","paragraphs":["When training word level models we follow common practice and use a batch size of 64, truncated backpropagation with 35 time steps, and we feed the final states from the previous batch as the initial state of the subsequent one. At the beginning of training and test time, the model starts with a zero state. To bias the model towards being able to easily start from such a state at test time, during training, with probability 0.01 a constant zero state is provided as the initial state.","Optimisation is performed by Adam BIBREF17 with INLINEFORM0 but otherwise default parameters ( INLINEFORM1 , INLINEFORM2 ). Setting INLINEFORM3 so turns off the exponential moving average for the estimates of the means of the gradients and brings Adam very close to RMSProp without momentum, but due to Adam's bias correction, larger learning rates can be used.","Batch size is set to 64. The learning rate is multiplied by 0.1 whenever validation performance does not improve ever during 30 consecutive checkpoints. These checkpoints are performed after every 100 and 200 optimization steps for Penn Treebank and Wikitext-2, respectively.","For character level models (i.e. Enwik8), the differences are: truncated backpropagation is performed with 50 time steps. Adam's parameters are INLINEFORM0 , INLINEFORM1 . Batch size is 128. Checkpoints are only every 400 optimisation steps and embeddings are not shared."]},{"idx":5,"heading":"Evaluation","paragraphs":["For evaluation, the checkpoint with the best validation perplexity found by the tuner is loaded and the model is applied to the test set with a batch size of 1. For the word based datasets, using the training batch size makes results worse by 0.3 PPL while Enwik8 is practically unaffected due to its evaluation and training sets being much larger. Preliminary experiments indicate that MC averaging would bring a small improvement of about 0.4 in perplexity and 0.005 in bits per character, similar to the results of BIBREF11 , while being a 1000 times more expensive which is prohibitive on larger datasets. Therefore, throughout we use the mean-field approximation for dropout at test time."]},{"idx":6,"heading":"Hyperparameter Tuning","paragraphs":["Hyperparameters are optimised by Google Vizier BIBREF19 , a black-box hyperparameter tuner based on batched GP bandits using the expected improvement acquisition function BIBREF20 . Tuners of this nature are generally more efficient than grid search when the number of hyperparameters is small. To keep the problem tractable, we restrict the set of hyperparameters to learning rate, input embedding ratio, input dropout, state dropout, output dropout, weight decay. For deep LSTMs, there is an extra hyperparameter to tune: intra-layer dropout. Even with this small set, thousands of evaluations are required to reach convergence.","Motivated by recent results from BIBREF21 , we compare models on the basis of the total number of trainable parameters as opposed to the number of hidden units. The tuner is given control over the presence and size of the down-projection, and thus over the tradeoff between the number of embedding vs. recurrent cell parameters. Consequently, the cells' hidden size and the embedding size is determined by the actual parameter budget, depth and the input embedding ratio hyperparameter.","For Enwik8 there are relatively few parameters in the embeddings since the vocabulary size is only 205. Here we choose not to share embeddings and to omit the down-projection unconditionally."]},{"idx":7,"heading":"Penn Treebank","paragraphs":["We tested LSTMs of various depths and an RHN of depth 5 with parameter budgets of 10 and 24 million matching the sizes of the Medium and Large LSTMs by BIBREF18 . The results are summarised in Table TABREF9 .","Notably, in our experiments even the RHN with only 10M parameters has better perplexity than the 24M one in the original publication. Our 24M version improves on that further. However, a shallow LSTM-based model with only 10M parameters enjoys a very comfortable margin over that, with deeper models following near the estimated noise range. At 24M, all depths obtain very similar results, reaching exp(4.065) [fixed,zerofill,precision=1] at depth 4. Unsurprisingly, NAS whose architecture was chosen based on its performance on this dataset does almost equally well, even better than in BIBREF1 ."]},{"idx":8,"heading":"Wikitext-2","paragraphs":["Wikitext-2 is not much larger than Penn Treebank, so it is not surprising that even models tuned for Penn Treebank perform reasonably on this dataset, and this is in fact how results in previous works were produced. For a fairer comparison, we also tune hyperparameters on the same dataset. In Table TABREF14 , we report numbers for both approaches. All our results are well below the previous state of the are for models without dynamic evaluation or caching. That said, our best result, exp(4.188) [fixed,zerofill,precision=1] compares favourably even to the Neural Cache BIBREF6 whose innovations are fairly orthogonal to the base model.","Shallow LSTMs do especially well here. Deeper models have gradually degrading perplexity, with RHNs lagging all of them by a significant margin. NAS is not quite up there with the LSTM suggesting its architecture might have overfitted to Penn Treebank, but data for deeper variants would be necessary to draw this conclusion."]},{"idx":9,"heading":"Enwik8","paragraphs":["In contrast to the previous datasets, our numbers on this task (reported in BPC, following convetion) are slightly off the state of the art. This is most likely due to optimisation being limited to 14 epochs which is about a tenth of what the model of BIBREF0 was trained for. Nevertheless, we match their smaller RHN with our models which are very close to each other. NAS lags the other models by a surprising margin at this task."]},{"idx":10,"heading":"Analysis","paragraphs":["On two of the three datasets, we improved previous results substantially by careful model specification and hyperparameter optimisation, but the improvement for RHNs is much smaller compared to that for LSTMs. While it cannot be ruled out that our particular setup somehow favours LSTMs, we believe it is more likely that this effect arises due to the original RHN experimental condition having been tuned more extensively (this is nearly unavoidable during model development).","Naturally, NAS benefitted only to a limited degree from our tuning, since the numbers of BIBREF1 were already produced by employing similar regularisation methods and a grid search. The small edge can be attributed to the suboptimality of grid search (see Section SECREF23 ).","In summary, the three recurrent cell architectures are closely matched on all three datasets, with minuscule differences on Enwik8 where regularisation matters the least. These results support the claims of BIBREF21 , that capacities of various cells are very similar and their apparent differences result from trainability and regularisation. While comparing three similar architectures cannot prove this point, the inclusion of NAS certainly gives it more credence. This way we have two of the best human designed and one machine optimised cell that was the top performer among thousands of candidates."]},{"idx":11,"heading":"The Effect of Individual Features","paragraphs":["Down-projection was found to be very beneficial by the tuner for some depth/budget combinations. On Penn Treebank, it improved results by about 2–5 perplexity points at depths 1 and 2 at 10M, and depth 1 at 24M, possibly by equipping the recurrent cells with more capacity. The very same models benefited from down-projection on Wikitext-2, but even more so with gaps of about 10–18 points which is readily explained by the larger vocabulary size.","We further measured the contribution of other features of the models in a series of experiments. See Table TABREF22 . To limit the number of resource used, in these experiments only individual features were evaluated (not their combinations) on Penn Treebank at the best depth for each architecture (LSTM or RHN) and parameter budget (10M or 24M) as determined above.","First, we untied input and output embeddings which made perplexities worse by about 6 points across the board which is consistent with the results of BIBREF9 .","Second, without variational dropout the RHN models suffer quite a bit since there remains no dropout at all in between the layers. The deep LSTM also sees a similar loss of perplexity as having intra-layer dropout does not in itself provide enough regularisation.","Third, we were also interested in how recurrent dropout BIBREF12 would perform in lieu of variational dropout. Dropout masks were shared between time steps in both methods, and our results indicate no consistent advantage to either of them."]},{"idx":12,"heading":"Model Selection","paragraphs":["With a large number of hyperparameter combinations evaluated, the question of how much the tuner overfits arises. There are multiple sources of noise in play,","non-deterministic ordering of floating-point operations in optimised linear algebra routines,","different initialisation seeds,","the validation and test sets being finite samples from a infinite population.","To assess the severity of these issues, we conducted the following experiment: models with the best hyperparameter settings for Penn Treebank and Wikitext-2 were retrained from scratch with various initialisation seeds and the validation and test scores were recorded. If during tuning, a model just got a lucky run due to a combination of UID19 and UID20 , then retraining with the same hyperparameters but with different seeds would fail to reproduce the same good results.","There are a few notable things about the results. First, in our environment (Tensorflow with a single GPU) even with the same seed as the one used by the tuner, the effect of UID19 is almost as large as that of UID19 and UID20 combined. Second, the variance induced by UID19 and UID20 together is roughly equivalent to an absolute difference of 0.4 in perplexity on Penn Treebank and 0.5 on Wikitext-2. Third, the validation perplexities of the best checkpoints are about one standard deviation lower than the sample mean of the reruns, so the tuner could fit the noise only to a limited degree.","Because we treat our corpora as a single sequence, test set contents are not i.i.d., and we cannot apply techniques such as the bootstrap to assess UID21 . Instead, we looked at the gap between validation and test scores as a proxy and observed that it is very stable, contributing variance of 0.12–0.3 perplexity to the final results on Penn Treebank and Wikitext-2, respectively.","We have not explicitly dealt with the unknown uncertainty remaining in the Gaussian Process that may affect model comparisons, apart from running it until apparent convergence. All in all, our findings suggest that a gap in perplexity of 1.0 is a statistically robust difference between models trained in this way on these datasets. The distribution of results was approximately normal with roughly the same variance for all models, so we still report numbers in a tabular form instead of plotting the distribution of results, for example in a violin plot BIBREF26 ."]},{"idx":13,"heading":"Sensitivity","paragraphs":["To further verify that the best hyperparameter setting found by the tuner is not a fluke, we plotted the validation loss against the hyperparameter settings. Fig. FIGREF24 shows one such typical plot, for a 4-layer LSTM. We manually restricted the ranges around the best hyperparameter values to around 15–25% of the entire tuneable range, and observed that the vast majority of settings in that neighbourhood produced perplexities within 3.0 of the best value. Widening the ranges further leads to quickly deteriorating results.","Satisfied that the hyperparameter surface is well behaved, we considered whether the same results could have possibly been achieved with a simple grid search. Omitting input embedding ratio because the tuner found having a down-projection suboptimal almost non-conditionally for this model, there remain six hyperparameters to tune. If there were 5 possible values on the grid for each hyperparameter (with one value in every 20% interval), then we would need INLINEFORM0 , nearly 8000 trials to get within 3.0 of the best perplexity achieved by the tuner in about 1500 trials."]},{"idx":14,"heading":"Tying LSTM gates","paragraphs":["Normally, LSTMs have two independent gates controlling the retention of cell state and the admission of updates (Eq. EQREF26 ). A minor variant which reduces the number of parameters at the loss of some flexibility is to tie the input and forget gates as in Eq. . A possible middle ground that keeps the number of parameters the same but ensures that values of the cell state INLINEFORM0 remain in INLINEFORM1 is to cap the input gate as in Eq. . DISPLAYFORM0","Where the equations are based on the formulation of BIBREF27 . All LSTM models in this paper use the third variant, except those titled “Untied gates” and “Tied gates” in Table TABREF22 corresponding to Eq. EQREF26 and , respectively.","The results show that LSTMs are insensitive to these changes and the results vary only slightly even though more hidden units are allocated to the tied version to fill its parameter budget. Finally, the numbers suggest that deep LSTMs benefit from bounded cell states."]},{"idx":15,"heading":"Conclusion","paragraphs":["During the transitional period when deep neural language models began to supplant their shallower predecessors, effect sizes tended to be large, and robust conclusions about the value of the modelling innovations could be made, even in the presence of poorly controlled “hyperparameter noise.” However, now that the neural revolution is in full swing, researchers must often compare competing deep architectures. In this regime, effect sizes tend to be much smaller, and more methodological care is required to produce reliable results. Furthermore, with so much work carried out in parallel by a growing research community, the costs of faulty conclusions are increased.","Although we can draw attention to this problem, this paper does not offer a practical methodological solution beyond establishing reliable baselines that can be the benchmarks for subsequent work. Still, we demonstrate how, with a huge amount of computation, noise levels of various origins can be carefully estimated and models meaningfully compared. This apparent tradeoff between the amount of computation and the reliability of results seems to lie at the heart of the matter. Solutions to the methodological challenges must therefore make model evaluation cheaper by, for instance, reducing the number of hyperparameters and the sensitivity of models to them, employing better hyperparameter optimisation strategies, or by defining “leagues” with predefined computational budgets for a single model representing different points on the tradeoff curve."]},{"idx":16,"heading":"Figures and Tables","paragraphs":["Figure 1: Recurrent networks with optional down-projection, per-step and per-sequence dropout (dashed and solid lines).","Table 1: Validation and test set perplexities on Penn Treebank for models with different numbers of parameters and depths. All results except those from Zaremba are with shared input and output embeddings. VD stands for Variational Dropout from Gal & Ghahramani (2016). †: parallel work.","Table 2: Validation and test set perplexities on Wikitext-2. All results are with shared input and output embeddings. †: parallel work.","Table 3: Validation and test set BPCs on Enwik8 from the Hutter Prize dataset.","Table 4: Validation and test set perplexities on Penn Treebank for variants of our best LSTM and RHN models of two sizes.","Figure 2: Average per-word negative log-likelihoods of hyperparameter combinations in the neighbourhood of the best solution for a 4-layer LSTM with 24M weights on the Penn Treebank dataset."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1708.09025","title":"Unsupervised Terminological Ontology Learning based on Hierarchical Topic Modeling","sections":[{"idx":0,"heading":"Abstract","paragraphs":["In this paper, we present hierarchical relationbased latent Dirichlet allocation (hrLDA), a data-driven hierarchical topic model for extracting terminological ontologies from a large number of heterogeneous documents. In contrast to traditional topic models, hrLDA relies on noun phrases instead of unigrams, considers syntax and document structures, and enriches topic hierarchies with topic relations. Through a series of experiments, we demonstrate the superiority of hrLDA over existing topic models, especially for building hierarchies. Furthermore, we illustrate the robustness of hrLDA in the settings of noisy data sets, which are likely to occur in many practical scenarios. Our ontology evaluation results show that ontologies extracted from hrLDA are very competitive with the ontologies created by domain experts."]},{"idx":1,"heading":"Introduction","paragraphs":["Although researchers have made significant progress on knowledge acquisition and have proposed many ontologies, for instance, WordNet BIBREF0 , DBpedia BIBREF1 , YAGO BIBREF2 , Freebase, BIBREF3 Nell BIBREF4 , DeepDive BIBREF5 , Domain Cartridge BIBREF6 , Knowledge Vault BIBREF7 , INS-ES BIBREF8 , iDLER BIBREF9 , and TransE-NMM BIBREF10 , current ontology construction methods still rely heavily on manual parsing and existing knowledge bases. This raises challenges for learning ontologies in new domains. While a strong ontology parser is effective in small-scale corpora, an unsupervised model is beneficial for learning new entities and their relations from new data sources, and is likely to perform better on larger corpora.","In this paper, we focus on unsupervised terminological ontology learning and formalize a terminological ontology as a hierarchical structure of subject-verb-object triplets. We divide a terminological ontology into two components: topic hierarchies and topic relations. Topics are presented in a tree structure where each node is a topic label (noun phrase), the root node represents the most general topic, the leaf nodes represent the most specific topics, and every topic is composed of its topic label and its descendant topic labels. Topic hierarchies are preserved in topic paths, and a topic path connects a list of topics labels from the root to a leaf. Topic relations are semantic relationships between any two topics or properties used to describe one topic. Figure FIGREF1 depicts an example of a terminological ontology learned from a corpus about European cities. We extract terminological ontologies by applying unsupervised hierarchical topic modeling and relation extraction to plain text.","Topic modeling was originally used for topic extraction and document clustering. The classical topic model, latent Dirichlet allocation (LDA) BIBREF11 , simplifies a document as a bag of its words and describes a topic as a distribution of words. Prior research BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17 , BIBREF18 has shown that LDA-based approaches are adequate for (terminological) ontology learning. However, these models are deficient in that they still need human supervision to decide the number of topics, and to pick meaningful topic labels usually from a list of unigrams. Among models not using unigrams, LDA-based Global Similarity Hierarchy Learning (LDA+GSHL) BIBREF13 only extracts a subset of relations: “broader\" and “related\" relations. In addition, the topic hierarchies of KB-LDA BIBREF17 rely on hypernym-hyponym pairs capturing only a subset of hierarchies.","Considering the shortcomings of the existing methods, the main objectives of applying topic modeling to ontology learning are threefold.","To achieve the first objective, we extract noun phrases and then propose a sampling method to estimate the number of topics. For the second objective, we use language parsing and relation extraction to learn relations for the noun phrases. Regarding the third objective, we adapt and improve the hierarchical latent Dirichlet allocation (hLDA) model BIBREF19 , BIBREF20 . hLDA is not ideal for ontology learning because it builds topics from unigrams (which are not descriptive enough to serve as entities in ontologies) and the topics may contain words from multiple domains when input data have documents from many domains (see Section SECREF2 and Figure FIGREF55 ). Our model, hrLDA, overcomes these deficiencies. In particular, hrLDA represents topics with noun phrases, uses syntax and document structures such as paragraph indentations and item lists, assigns multiple topic paths for every document, and allows topic trees to grow vertically and horizontally.","The primary contributions of this work can be specified as follows.","The rest of this paper is organized into five parts. In Section 2, we provide a brief background of hLDA. In Section 3, we present our hrLDA model and the ontology generation method. In Section 4, we demonstrate empirical results regarding topic hierarchies and generated terminological ontologies. Finally, in Section 5, we present some concluding remarks and discuss avenues for future work and improvements."]},{"idx":2,"heading":"Background","paragraphs":["In this section, we introduce our main baseline model, hierarchical latent Dirichlet allocation (hLDA), and some of its extensions. We start from the components of hLDA - latent Dirichlet allocation (LDA) and the Chinese Restaurant Process (CRP)- and then explain why hLDA needs improvements in both building hierarchies and drawing topic paths.","LDA is a three-level Bayesian model in which each document is a composite of multiple topics, and every topic is a distribution over words. Due to the lack of determinative information, LDA is unable to distinguish different instances containing the same content words, (e.g. “I trimmed my polished nails\" and “I have just hammered many rusty nails\"). In addition, in LDA all words are probabilistically independent and equally important. This is problematic because different words and sentence elements should have different contributions to topic generation. For instance, articles contribute little compared to nouns, and sentence subjects normally contain the main topics of a document.","Introduced in hLDA, CRP partitions words into several topics by mimicking a process in which customers sit down in a Chinese restaurant with an infinite number of tables and an infinite number of seats per table. Customers enter one by one, with a new customer choosing to sit at an occupied table or a new table. The probability of a new customer sitting at the table with the largest number of customers is the highest. In reality, customers do not always join the largest table but prefer to dine with their acquaintances. The theory of distance-dependent CRP was formerly proposed by David Blei BIBREF21 . We provide later in Section SECREF15 an explicit formula for topic partition given that adjacent words and sentences tend to deal with the same topics.","hLDA combines LDA with CRP by setting one topic path with fixed depth INLINEFORM0 for each document. The hierarchical relationships among nodes in the same path depend on an INLINEFORM1 dimensional Dirichlet distribution that actually arranges the probabilities of topics being on different topic levels. Despite the fact that the single path was changed to multiple paths in some extensions of hLDA - the nested Chinese restaurant franchise processes BIBREF22 and the nested hierarchical Dirichlet Processes BIBREF23 , - this topic path drawing strategy puts words from different domains into one topic when input data are mixed with topics from multiple domains. This means that if a corpus contains documents in four different domains, hLDA is likely to include words from the four domains in every topic (see Figure FIGREF55 ). In light of the various inadequacies discussed above, we propose a relation-based model, hrLDA. hrLDA incorporates semantic topic modeling with relation extraction to integrate syntax and has the capacity to provide comprehensive hierarchies even in corpora containing mixed topics."]},{"idx":3,"heading":"Hierarchical Relation-based Latent Dirichlet Allocation","paragraphs":["The main problem we address in this section is generating terminological ontologies in an unsupervised fashion. The fundamental concept of hrLDA is as follows. When people construct a document, they start with selecting several topics. Then, they choose some noun phrases as subjects for each topic. Next, for each subject they come up with relation triplets to describe this subject or its relationships with other subjects. Finally, they connect the subject phrases and relation triplets to sentences via reasonable grammar. The main topic is normally described with the most important relation triplets. Sentences in one paragraph, especially adjacent sentences, are likely to express the same topic.","We begin by describing the process of reconstructing LDA. Subsequently, we explain relation extraction from heterogeneous documents. Next, we propose an improved topic partition method over CRP. Finally, we demonstrate how to build topic hierarchies that bind with extracted relation triplets."]},{"idx":4,"heading":"Relation-based Latent Dirichlet Allocation","paragraphs":["Documents are typically composed of chunks of texts, which may be referred to as sections in Word documents, paragraphs in PDF documents, slides in presentation documents, etc. Each chunk is composed of multiple sentences that are either atomic or complex in structure, which means a document is also a collection of atomic and/or complex sentences. An atomic sentence (see module INLINEFORM0 in Figure FIGREF10 ) is a sentence that contains only one subject ( INLINEFORM1 ), one object ( INLINEFORM2 ) and one verb ( INLINEFORM3 ) between the subject and the object. For every atomic sentence whose object is also a noun phrase, there are at least two relation triplets (e.g., “The tiger that gave the excellent speech is handsome\" has relation triplets: (tiger, give, speech), (speech, be given by, tiger), and (tiger, be, handsome)). By contrast, a complex sentence can be subdivided into multiple atomic sentences. Given that the syntactic verb in a relation triplet is determined by the subject and the object, a document INLINEFORM4 in a corpus INLINEFORM5 can be ultimately reduced to INLINEFORM6 subject phrases (we convert objects to subjects using passive voice) associated with INLINEFORM7 relation triplets INLINEFORM8 . Number INLINEFORM9 is usually larger than the actual number of noun phrases in document INLINEFORM10 . By replacing the unigrams in LDA with relation triplets, we retain definitive information and assign salient noun phrases high weights.","We define INLINEFORM0 as a Dirichlet distribution parameterized by hyperparameters INLINEFORM1 , INLINEFORM2 as a multinomial distribution parameterized by hyperparameters INLINEFORM3 , INLINEFORM4 as a Dirichlet distribution parameterized by INLINEFORM5 , and INLINEFORM6 as a multinomial distribution parameterized by INLINEFORM7 . We assume the corpus has INLINEFORM8 topics. Assigning INLINEFORM9 topics to the INLINEFORM10 relation triplets of document INLINEFORM11 follows a multinomial distribution INLINEFORM12 with prior INLINEFORM13 . Selecting the INLINEFORM14 relation triplets for document INLINEFORM15 given the INLINEFORM16 topics follows a multinomial distribution INLINEFORM17 with prior INLINEFORM18 . We denote INLINEFORM19 as the list of relation triplet lists extracted from all documents in the corpus, and INLINEFORM20 as the list of topic assignments of INLINEFORM21 . We denote the relation triplet counts of documents in the corpus by INLINEFORM22 . The graphical representation of the relation-based latent Dirichlet allocation (rLDA) model is illustrated in Figure FIGREF10 .","The plate notation can be decomposed into two types of Dirichlet-multinomial conjugated structures: document-topic distribution INLINEFORM0 and topic-relation distribution INLINEFORM1 . Hence, the joint distribution of INLINEFORM2 and INLINEFORM3 can be represented as DISPLAYFORM0","where INLINEFORM0 is the number of unique relations in all documents, INLINEFORM1 is the number of occurrences of the relation triplet INLINEFORM2 generated by topic INLINEFORM3 in all documents, and INLINEFORM4 is the number of relation triplets generated by topic INLINEFORM5 in document INLINEFORM6 . INLINEFORM7 is a conjugate prior for INLINEFORM8 and thus the posterior distribution is a new Dirichlet distribution parameterized by INLINEFORM9 . The same rule applies to INLINEFORM10 ."]},{"idx":5,"heading":"Relation Triplet Extraction","paragraphs":["Extracting relation triplets is the essential step of hrLDA, and it is also the key process for converting a hierarchical topic tree to an ontology structure. The idea is to find all syntactically related noun phrases and their connections using a language parser such as the Stanford NLP parser BIBREF24 and Ollie BIBREF25 . Generally, there are two types of relation triplets:","Subject-predicate-object-based relations,","e.g., New York is the largest city in the United States INLINEFORM0 (New York, be the largest city in, the United States);","Noun-based/hidden relations,","e.g., Queen Elizabeth INLINEFORM0 (Elizabeth, be, queen).","A special type of relation triplets can be extracted from presentation documents such as those written in PowerPoint using document structures. Normally lines in a slide are not complete sentences, which means language parsing does not work. However, indentations and bullet types usually express inclusion relationships between adjacent lines. Starting with the first line in an itemized section, our algorithm scans the content in a slide line by line, and creates relations based on the current item and the item that is one level higher."]},{"idx":6,"heading":"Acquaintance Chinese Restaurant Process","paragraphs":["As mentioned in Section 2, CRP always assigns the highest probability to the largest table, which assumes customers are more likely to sit at the table that has the largest number of customers. This ignores the social reality that a person is more willing to choose the table where his/her closest friend is sitting even though the table also seats unknown people who are actually friends of friends. Similarly with human-written documents, adjacent sentences usually describe the same topics. We consider a restaurant table as a topic, and a person sitting at any of the tables as a noun phrase. In order to penalize the largest topic and assign high probabilities to adjacent noun phrases being in the same topics, we introduce an improved partition method, Acquaintance Chinese Restaurant Process (ACRP).","The ultimate purposes of ACRP are to estimate INLINEFORM0 , the number of topics for rLDA, and to set the initial topic distribution states for rLDA. Suppose a document is read from top to bottom and left to right. As each noun phrase belongs to one sentence and one text chunk (e.g., section, paragraph and slide), the locations of all noun phrases in a document can be mapped to a two-dimensional space where sentence location is the x axis and text chunk location is the y axis (the first noun phrase of a document holds value (0, 0)). More specifically, every noun phrase has four attributes: content, location, one-to-many relation triplets, and document ID. Noun phrases in the same text chunk are more likely to be “acquaintances;\" they are even closer to each other if they are in the same sentence. In contrast to CRP, ACRP assigns probabilities based on closeness, which is specified in the following procedure.","Let INLINEFORM0 be the integer-valued random variable corresponding to the index of a topic assigned to the INLINEFORM1 phrase. Draw a probability INLINEFORM2 from Equations EQREF18 to EQREF25 below for the INLINEFORM3 noun phrase INLINEFORM4 , joining each of the existing INLINEFORM5 topics and the new INLINEFORM6 topic given the topic assignments of previous INLINEFORM7 noun phrases, INLINEFORM8 . If a noun phrase joins any of the existing k topics, we denote the corresponding topic index by INLINEFORM9 .","The probability of choosing the INLINEFORM0 topic: DISPLAYFORM0","The probability of selecting any of the INLINEFORM0 topics:","if the content of INLINEFORM0 is synonymous with or an acronym of a previously analyzed noun phrase INLINEFORM1 INLINEFORM2 in the INLINEFORM3 topic, DISPLAYFORM0","else if the document ID of INLINEFORM0 is different from all document IDs belonging to the INLINEFORM1 topic, DISPLAYFORM0","otherwise, DISPLAYFORM0","where INLINEFORM0 refers to the current number of noun phrases in the INLINEFORM1 topic, INLINEFORM2 represents the vector of chunk location differences of the INLINEFORM3 noun phrase and all members in the INLINEFORM4 topic, INLINEFORM5 stands for the vector of sentence location differences, and INLINEFORM6 is a penalty factor.","Normalize the ( INLINEFORM0 ) probabilities to guarantee they are each in the range of [0, 1] and their sum is equal to 1.","Based on the probabilities EQREF18 to EQREF25 , we sample a topic index INLINEFORM0 from INLINEFORM1 for every noun phrase, and we count the number of unique topics INLINEFORM2 in the end. We shuffle the order of documents and iterate ACRP until INLINEFORM3 is unchanged."]},{"idx":7,"heading":"Nested Acquaintance Chinese Restaurant Process","paragraphs":["The procedure for extending ACRP to hierarchies is essential to why hrLDA outperforms hLDA. Instead of a predefined tree depth INLINEFORM0 , the tree depth for hrLDA is optional and data-driven. More importantly, clustering decisions are made given a global distribution of all current non-partitioned phrases (leaves) in our algorithm. This means there can be multiple paths traversed down a topic tree for each document. With reference to the topic tree, every node has a noun phrase as its label and represents a topic that may have multiple sub-topics. The root node is visited by all phrases. In practice, we do not link any phrases to the root node, as it contains the entire vocabulary. An inner node of a topic tree contains a selected topic label. A leaf node contains an unprocessed noun phrase. We define a hashmap INLINEFORM1 with a document ID as the key and the current leaf nodes of the document as the value. We denote the current tree level by INLINEFORM2 . We next outline the overall algorithm.","We start with the root node ( INLINEFORM0 ) and apply rLDA to all the documents in a corpus.","Collect the current leaf nodes of every document. INLINEFORM0 initially contains all noun phrases in the corpus. Assign a cluster partition to the leaf nodes in each document based on ACRP and sample the cluster partition until the number of topics of all noun phrases in INLINEFORM1 is stable or the iteration reaches the predefined number of iteration times (whichever occurs first).","Mark the number of topics (child nodes) of parent node INLINEFORM0 at level INLINEFORM1 as INLINEFORM2 . Build a INLINEFORM3 - dimensional topic proportion vector INLINEFORM4 based on INLINEFORM5 .","For every noun phrase INLINEFORM0 in document INLINEFORM1 , form the topic assignments INLINEFORM2 based on INLINEFORM3 .","Generate relation triplets from INLINEFORM0 given INLINEFORM1 and the associated topic vector INLINEFORM2 .","Eliminate partitioned leaf nodes from INLINEFORM0 . Update the current level INLINEFORM1 by 1.","If phrases in INLINEFORM0 are not yet completely partitioned to the next level and INLINEFORM1 is less than INLINEFORM2 , continue the following steps. For each leaf node, we set the top phrase (i.e., the phrase having the highest probability) as the topic label of this leaf node and the leaf node becomes an inner node. We next update INLINEFORM3 and repeat procedures INLINEFORM4 .","To summarize this process more succinctly: we build the topic hierarchies with rLDA in a divisive way (see Figure FIGREF35 ). We start with the collection of extracted noun phrases and split them using rLDA and ACRP. Then, we apply the procedure recursively until each noun phrase is selected as a topic label. After every rLDA assignment, each inner node only contains the topic label (top phrase), and the rest of the phrases are divided into nodes at the next level using ACRP and rLDA. Hence, we build a topic tree with each node as a topic label (noun phrase), and each topic is composed of its topic labels and the topic labels of the topic's descendants. In the end, we finalize our terminological ontology by linking the extracted relation triplets with the topic labels as subjects.","We use collapsed Gibbs sampling BIBREF26 for inference from posterior distribution INLINEFORM0 based on Equation EQREF11 . Assume the INLINEFORM1 noun phrase INLINEFORM2 in parent node INLINEFORM3 comes from document INLINEFORM4 . We denote unassigned noun phrases from document INLINEFORM5 in parent node INLINEFORM6 by INLINEFORM7 , and unique noun phrases in parent node INLINEFORM8 by INLINEFORM9 . We simplify the probability of assigning the INLINEFORM10 noun phrase in parent node INLINEFORM11 to topic INLINEFORM12 among INLINEFORM13 topics as DISPLAYFORM0","where INLINEFORM0 refers to all topic assignments other than INLINEFORM1 , INLINEFORM2 is multinational document-topic distribution for unassigned noun phrases INLINEFORM3 , INLINEFORM4 is the multinational topic-relation distribution for topic INLINEFORM5 , INLINEFORM6 is the number of occurrences of noun phrase INLINEFORM7 in topic INLINEFORM8 except the INLINEFORM9 noun phrase in INLINEFORM10 , INLINEFORM11 stands for the number of times that topic INLINEFORM12 occurs in INLINEFORM13 excluding the INLINEFORM14 noun phrase in INLINEFORM15 . The time complexity of hrLDA is INLINEFORM16 , where INLINEFORM17 is the number of topics at level INLINEFORM18 . The space complexity is INLINEFORM19 .","In order to build a hierarchical topic tree of a specific domain, we must generate a subset of the relation triplets using external constraints or semantic seeds via a pruning process BIBREF27 . As mentioned above, in a relation triplet, each relation connects one subject and one object. By assembling all subject and object pairs, we can build an undirected graph with the objects and the subjects constituting the nodes of the graph BIBREF28 . Given one or multiple semantic seeds as input, we first collect a set of nodes that are connected to the seed(s), and then take the relations from the set of nodes as input to retrieve associated subject and object pairs. This process constitutes one recursive step. The subject and object pairs become the input of the subsequent recursive step."]},{"idx":8,"heading":"Implementation","paragraphs":["We utilized the Apache poi library to parse texts from pdfs, word documents and presentation files; the MALLET toolbox BIBREF29 for the implementations of LDA, optimized_LDA BIBREF30 and hLDA; the Apache Jena library to add relations, properties and members to hierarchical topic trees; and Stanford Protege for illustrating extracted ontologies. We make our code and data available . We used the same empirical hyper-parameter setting (i.e., INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 ) across all our experiments. We then demonstrate the evaluation results from two aspects: topic hierarchy and ontology rule."]},{"idx":9,"heading":"Hierarchy Evaluation","paragraphs":["In this section, we present the evaluation results of hrLDA tested against optimized_LDA, hLDA, and phrase_hLDA (i.e., hLDA based on noun phrases) as well as ontology examples that hrLDA extracted from real-world text data. The entire corpus we generated contains 349,362 tokens (after removing stop words and cleaning) and is built from articles on INLINEFORM0 INLINEFORM1 . It includes 84 presentation files, articles from 1,782 Wikipedia pages and 3,000 research papers that were published in IEEE manufacturing conference proceedings within the last decade. In order to see the performance in data sets of different scales, we also used a smaller corpus Wiki that holds the articles collected from the Wikipedia pages only.","We extract a single level topic tree using each of the four models; hrLDA becomes rLDA, and phrase_hLDA becomes phrase-based LDA. We have tested the average perplexity and running time performance of ten independent runs on each of the four models BIBREF31 , BIBREF32 . Equation EQREF41 defines the perplexity, which we employed as an empirical measure. DISPLAYFORM0","where INLINEFORM0 is a vector containing the INLINEFORM1 relation triplets in document INLINEFORM2 , and INLINEFORM3 is the topic assignment for INLINEFORM4 .","The comparison results on our Wiki corpus are shown in Figure FIGREF42 . hrLDA yields the lowest perplexity and reasonable running time. As the running time spent on parameter optimization is extremely long (the optimized_LDA requires 19.90 hours to complete one run), for efficiency, we adhere to the fixed parameter settings for hrLDA.","Superiority","Figures FIGREF43 to FIGREF49 illustrates the perplexity trends of the three hierarchical topic models (i.e., hrLDA, phrase_hLDA and hLDA) applied to both the Wiki corpus and the entire corpus with INLINEFORM0 “chip\" given different level settings. From left to right, hrLDA retains the lowest perplexities compared with other models as the corpus size grows. Furthermore, from top to bottom, hrLDA remains stable as the topic level increases, whereas the perplexity of phrase_hLDA and especially the perplexity of hLDA become rapidly high. Figure FIGREF52 highlights the perplexity values of the three models with confidence intervals in the final state. As shown in the two types of experiments, hrLDA has the lowest average perplexities and smallest confidence intervals, followed by phrase_hLDA, and then hLDA.","Our interpretation is that hLDA and phrase_hLDA tend to assign terms to the largest topic and thus do not guarantee that each topic path contains terms with similar meaning.","Robustness","Figure FIGREF55 shows exhaustive hierarchical topic trees extracted from a small text sample with topics from four domains: INLINEFORM0 , INLINEFORM1 INLINEFORM2 , INLINEFORM3 , and INLINEFORM4 . hLDA tends to mix words from different domains into one topic. For instance, words on the first level of the topic tree come from all four domains. This is because the topic path drawing method in existing hLDA-based models takes words in the most important topic of every document and labels them as the main topic of the corpus. In contrast, hrLDA is able to create four big branches for the four domains from the root. Hence, it generates clean topic hierarchies from the corpus."]},{"idx":10,"heading":"Gold Standard-based Ontology Evaluation","paragraphs":["The visualization of one concrete ontology on the INLINEFORM0 INLINEFORM1 domain is presented in Figure FIGREF60 . For instance, Topic packaging contains topic integrated circuit packaging, and topic label jedec is associated with relation triplet (jedec, be short for, joint electron device engineering council).","We use KB-LDA, phrase_hLDA, and LDA+GSHL as our baseline methods, and compare ontologies extracted from hrLDA, KB-LDA, phrase_hLDA, and LDA+GSHL with DBpedia ontologies. We use precision, recall and F-measure for this ontology evaluation. A true positive case is an ontology rule that can be found in an extracted ontology and the associated ontology of DBpedia. A false positive case is an incorrectly identified ontology rule. A false negative case is a missed ontology rule. Table TABREF61 shows the evaluation results of ontologies extracted from Wikipedia articles pertaining to European Capital Cities (Corpus E), Office Buildings in Chicago (Corpus O) and Birds of the United States (Corpus B) using hrLDA, KB-LDA, phrase_hLDA (tree depth INLINEFORM0 = 3), and LDA+GSHL in contrast to these gold ontologies belonging to DBpedia. The three corpora used in this evaluation were collected from Wikipedia abstracts, the same text source of DBpedia. The seeds of hrLDA and the root concepts of LDA+GSHL are capital, building, and bird. For both KB-LDA and phrase_hLDA we kept the top five tokens in each topic as each node of their topic trees is a distribution/list of phrases. hrLDA achieves the highest precision and F-measure scores in the three experiments compared to the other models. KB-LDA performs better than phrase_hLDA and LDA+GSHL, and phrase_hLDA performs similarly to LDA+GSHL. In general, hrLDA works well especially when the pre-knowledge already exists inside the corpora. Consider the following two statements taken from the corpus on Birds of the United States as an example. In order to use two short documents “The Acadian flycatcher is a small insect-eating bird.\" and “The Pacific loon is a medium-sized member of the loon.\" to infer that the Acadian flycatcher and the Pacific loon are both related to topic bird, the pre-knowledge that “the loon is a species of bird\" is required for hrLDA. This example explains why the accuracy of extracting ontologies from this kind of corpus is low."]},{"idx":11,"heading":"Concluding Remarks","paragraphs":["In this paper, we have proposed a completely unsupervised model, hrLDA, for terminological ontology learning. hrLDA is a domain-independent and self-learning model, which means it is very promising for learning ontologies in new domains and thus can save significant time and effort in ontology acquisition.","We have compared hrLDA with popular topic models to interpret how our algorithm learns meaningful hierarchies. By taking syntax and document structures into consideration, hrLDA is able to extract more descriptive topics. In addition, hrLDA eliminates the restrictions on the fixed topic tree depth and the limited number of topic paths. Furthermore, ACRP allows hrLDA to create more reasonable topics and to converge faster in Gibbs sampling.","We have also compared hrLDA to several unsupervised ontology learning models and shown that hrLDA can learn applicable terminological ontologies from real world data. Although hrLDA cannot be applied directly in formal reasoning, it is efficient for building knowledge bases for information retrieval and simple question answering. Also, hrLDA is sensitive to the quality of extracted relation triplets. In order to give optimal answers, hrLDA should be embedded in more complex probabilistic modules to identify true facts from extracted ontology rules. Finally, one issue we have not addressed in our current study is capturing pre-knowledge. Although a direct solution would be adding the missing information to the data set, a more advanced approach would be to train topic embeddings to extract hidden semantics."]},{"idx":12,"heading":"Acknowledgments","paragraphs":["This work was supported in part by Intel Corporation, Semiconductor Research Corporation (SRC). We are obliged to Professor Goce Trajcevski from Northwestern University for his insightful suggestions and discussions. This work was partly conducted using the Protege resource, which is supported by grant GM10331601 from the National Institute of General Medical Sciences of the United States National Institutes of Health."]},{"idx":13,"heading":"Figures and Tables","paragraphs":["Figure 1: A representation of a terminological ontology. (Left: topic hierarchies) Topic city is composed of most populous city, capital, London, Berlin, etc. City → capital → London and city → capital → Berlin are two topic paths. (Right: topic relations) Every topic label has relations to itself and/or with other labels. Be the capital city of Germany is one relation/property of topic Berlin. Be on the north of is one relation of topic Berlin to London.","Figure 2: Plate notation of rLDA","Figure 3: Graphical representation of hrLDA","Figure 4: Comparison results of hrLDA, phrase hLDA, hLDA and optimized LDA on perplexity and running time","Figure 5: Perplexity trends within 2000 iterations with level = 2","Figure 7: Perplexity trends within 2000 iterations with level = 10","Figure 6: Perplexity trends within 2000 iterations with level = 6","Figure 8: Average perplexities with confidence intervals of the three models in the final 2000th iteration with level = 10","Figure 10: A 10-level semiconductor ontology that contains 2063 topics and 6084 relation triplets","Figure 9: Performance of hLDA and hrLDA on a toy corpus of diversified topics","Table I: Precision, recall and F-measure (%)"]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1709.06136","title":"Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Models","sections":[{"idx":0,"heading":"Abstract","paragraphs":["In this paper, we present a deep reinforcement learning (RL) framework for iterative dialog policy optimization in end-to-end task-oriented dialog systems. Popular approaches in learning dialog policy with RL include letting a dialog agent to learn against a user simulator. Building a reliable user simulator, however, is not trivial, often as difficult as building a good dialog agent. We address this challenge by jointly optimizing the dialog agent and the user simulator with deep RL by simulating dialogs between the two agents. We first bootstrap a basic dialog agent and a basic user simulator by learning directly from dialog corpora with supervised training. We then improve them further by letting the two agents to conduct task-oriented dialogs and iteratively optimizing their policies with deep RL. Both the dialog agent and the user simulator are designed with neural network models that can be trained end-to-end. Our experiment results show that the proposed method leads to promising improvements on task success rate and total task reward comparing to supervised training and single-agent RL training baseline models."]},{"idx":1,"heading":"Introduction","paragraphs":["Task-oriented dialog system is playing an increasingly important role in enabling human-computer interactions via natural spoken language. Different from chatbot type of conversational agents BIBREF0 , BIBREF1 , BIBREF2 , task-oriented dialog systems assist users to complete everyday tasks, which usually involves aggregating information from external resources and planning over multiple dialog turns. Conventional task-oriented dialog systems are designed with complex pipelines BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , and there are usually separately developed modules for natural language understanding (NLU) BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , dialog state tracking (DST) BIBREF11 , BIBREF12 , and dialog management (DM) BIBREF13 , BIBREF14 , BIBREF15 . Such pipeline approach inherently make it hard to scale a dialog system to new domains, as each of these modules has to be redesigned separately with domain expertise. Moreover, credit assignment in such pipeline systems can be challenging, as errors made in upper stream modules may propagate and be amplified in downstream components.","Recent efforts have been made in designing end-to-end frameworks for task-oriented dialogs. Wen et al. BIBREF16 and Liu et al. BIBREF17 proposed supervised learning (SL) based end-to-end trainable neural network models. Zhao and Eskenazi BIBREF18 and Li et al. BIBREF19 introduced end-to-end trainable systems using deep reinforcement learning (RL) for dialog policy optimization. Comparing to SL based models, systems trained with RL by exploring the space of possible strategies showed improved model robustness against diverse dialog situations.","In RL based dialog policy learning, ideally the agent should be deployed to interact with users and get rewards from real user feedback. However, many dialog samples may be required for RL policy shaping, making it impractical to learn from real users directly from the beginning. Therefore, a user simulator BIBREF20 , BIBREF21 , BIBREF22 is usually used to train the dialog agent to a sufficient starting level before deploying it to the real environment. Quality of such user simulator has a direct impact on the effectiveness of dialog policy learning. Designing a reliable user simulator, however, is not trivial, often as difficult as building a good dialog agent. User simulators used in most of the recent RL based dialog models BIBREF18 , BIBREF22 , BIBREF19 are designed with expert knowledge and complex rules.","To address the challenge of lacking a reliable user simulator for dialog agent policy learning, we propose a method in jointly optimizing the dialog agent policy and the user simulator policy with deep RL. We first bootstrap a basic neural dialog agent and a basic neural user simulator by learning directly from dialog corpora with supervised training. We then improve them further by simulating task-oriented dialogs between the two agents and iteratively optimizing their dialog policies with deep RL. The intuition is that we model task-oriented dialog as a goal fulfilling task, in which we let the dialog agent and the user simulator to positively collaborate to achieve the goal. The user simulator is given a goal to complete, and it is expected to demonstrate coherent but diverse user behavior. The dialog agent attempts to estimate the user's goal and fulfill his request by conducting meaningful conversations. Both the two agents aim to learn to collaborate with each other to complete the task but without exploiting the game.","Our contribution in this work is two-fold. Firstly, we propose an iterative dialog policy learning method that jointly optimizes the dialog agent and the user simulator in end-to-end trainable neural dialog systems. Secondly, we design a novel neural network based user simulator for task-oriented dialogs that can be trained in a data-driven manner without requiring the design of complex rules.","The remainder of the paper is organized as follows. In section 2, we discuss related work on end-to-end trainable task-oriented dialog systems and RL policy learning methods. In section 3, we describe the proposed framework and learning methods in detail. In Section 4, we discuss the experiment setup and analyze the results. Section 5 gives the conclusions."]},{"idx":2,"heading":"Related Work","paragraphs":["Popular approaches for developing task-oriented dialog systems include treating the problem as a partially observable Markov Decision Process (POMDP) BIBREF6 . RL methods BIBREF23 can be applied to optimize the dialog policy online with the feedback collected via interacting with users. In order to make the RL policy learning tractable, dialog state and system actions have to be carefully designed.","Recently, people have proposed neural network based methods for task-oriented dialogs, motivated by their superior performance in modeling chit-chat type of conversations BIBREF24 , BIBREF1 , BIBREF2 , BIBREF25 . Bordes and Weston BIBREF26 proposed modeling task-oriented dialogs with a reasoning approach using end-to-end memory networks. Their model skips the belief tracking stage and selects the final system response directly from a list of response candidates. Comparing to this approach, our model explicitly tracks dialog belief state over the sequence of turns, as robust dialog state tracking has been shown BIBREF27 to boost the success rate in task completion. Wen et al. BIBREF16 proposed an end-to-end trainable neural network model with modularity connected system components. This system is trained in supervised manner, and thus may not be robust enough to handle diverse dialog situations due to the limited varieties in dialog corpus. Our system is trained by a combination of SL and deep RL methods, as it is shown that RL training may effectively improved the system robustness and dialog success rate BIBREF28 , BIBREF19 , BIBREF29 . Moreover, other than having separated dialog components as in BIBREF16 , we use a unified network for belief tracking, knowledge base (KB) operation, and dialog management, to fully explore the knowledge that can be shared among different tasks.","In many of the recent work on using RL for dialog policy learning BIBREF18 , BIBREF30 , BIBREF19 , hand-designed user simulators are used to interact with the dialog agent. Designing a good performing user simulator is not easy. A too basic user simulator as in BIBREF18 may only be able to produce short and simple utterances with limited variety, making the final system lack of robustness against noise in real world user inputs. Advanced user simulators BIBREF31 , BIBREF22 may demonstrate coherent user behavior, but they typically require designing complex rules with domain expertise. We address this challenge using a hybrid learning method, where we firstly bootstrapping a basic functioning user simulator with SL on human annotated corpora, and continuously improving it together with the dialog agent during dialog simulations with deep RL.","Jointly optimizing policies for dialog agent and user simulator with RL has also been studied in literature. Chandramohan et al. BIBREF32 proposed a co-adaptation framework for dialog systems by jointly optimizing the policies for multiple agents. Georgila et al. BIBREF33 discussed applying multi-agent RL for policy learning in a resource allocation negotiation scenario. Barlier et al. BIBREF34 modeled non-cooperative task dialog as a stochastic game and learned jointly the strategies of both agents. Comparing to these previous work, our proposed framework focuses on task-oriented dialogs where the user and the agent positively collaborate to achieve the user's goal. More importantly, we work towards building end-to-end models for task-oriented dialogs that can handle noises and ambiguities in natural language understanding and belief tracking, which is not taken into account in previous work."]},{"idx":3,"heading":"Proposed Framework","paragraphs":["In this section, we first provide a high level description of our proposed framework. We then discuss each module component and the training methods in detail.","In the supervised pre-training stage, we train the dialog agent and the user simulator separately using task-oriented dialog corpora. In the RL training stage, we simulate dialogs between the two agents. The user simulator starts the conversation based on a sampled user goal. The dialog agent attempts to estimate the user's goal and complete the task with the user simulator by conducting multi-turn conversation. At the end of each simulated dialog, a reward is generated based on the level of task completion. This reward is used to further optimize the dialog policies of the two agents with RL."]},{"idx":4,"heading":"Dialog Agent","paragraphs":["Figure 1 illustrates the design of the dialog agent. The dialog agent is capable of tracking dialog state, issuing API calls to knowledge bases (KB), and producing corresponding system actions and responses by incorporating the query results, which are key skill sets BIBREF26 in conducting task-oriented dialogs. State of the dialog agent is maintained in the LSTM BIBREF35 state and being updated after the processing of each turn. At the INLINEFORM0 th turn of a dialog, the dialog agent takes in (1) the previous agent output encoding INLINEFORM1 , (2) the user input encoding INLINEFORM2 , (3) the retrieved KB result encoding INLINEFORM3 , and updates its internal state conditioning on the previous agent state INLINEFORM4 . With the updated agent state INLINEFORM5 , the dialog agent emits (1) a system action INLINEFORM6 , (2) an estimation of the belief state, and (3) a pointer to an entity in the retrieved query results. These outputs are then passed to an NLG module to generate the final agent response.","Utterance Encoding For natural language format inputs at turn INLINEFORM0 , we use bi-directional LSTM to encode the utterance to a continuous vector INLINEFORM1 . With INLINEFORM2 representing the utterance at the INLINEFORM3 th turn with INLINEFORM4 words, the utterance vector INLINEFORM5 is produced by concatenating the last forward and backward LSTM states: INLINEFORM6 .","Action Modeling We use dialog acts as system actions, which can be seen as shallow representations of the utterance semantics. We treat system action modeling as a multi-class classification problem, where the agent select an appropriate action from a predefined list of system actions based on current dialog state INLINEFORM0 : DISPLAYFORM0","where INLINEFORM0 in the agent's network is a multilayer perceptron (MLP) with a single hidden layer and a INLINEFORM1 activation function over all possible system actions.","Belief Tracking Belief tracker, or dialog state tracker BIBREF36 , BIBREF37 , continuously tracks the user's goal by accumulating evidence in the conversation. We represent the user's goal using a list of slot values. The belief tracker maintains and updates a probability distribution INLINEFORM0 over candidate values for each slot type INLINEFORM1 at each turn INLINEFORM2 : DISPLAYFORM0","where INLINEFORM0 is an MLP with a single hidden layer and a INLINEFORM1 activation function for slot type INLINEFORM2 .","KB Operation The proposed dialog agent is able to access external information by interfacing with a KB or a database by issuing API calls. Making API call is one of the dialog acts that can be emitted by the agent, conditioning on the state of the conversation. An API call command template is firstly generated with slot type tokens. The final API call command is produced by replacing slot type tokens with the corresponding slot values from the belief tracker outputs.","At the INLINEFORM0 th turn of a dialog, the KB input encoding INLINEFORM1 is a binary value informing the availability of the entities that match the KB query. Corresponding output is the probability distribution of the entity pointer. We treat the KB results as a list of structured entities and let the model to maintain an entity pointer. The agent learns to adjust the entity pointer when user requests for alternative options.","Response Generation We use a template-based NLG module to convert the agent outputs (system action, slot values, and KB entity values) to natural language format."]},{"idx":5,"heading":"User Simulator","paragraphs":["Figure 2 shows the design of the user simulator. User simulator is given a randomly sampled goal at the beginning of the conversation. Similar to the design of the dialog agent, state of the user simulator is maintained in the state of an LSTM. At the INLINEFORM0 th turn of a dialog, the user simulator takes in (1) the goal encoding INLINEFORM1 , (2) the previous user output encoding INLINEFORM2 , (3) the current turn agent input encoding INLINEFORM3 , and updates its internal state conditioning on the previous user state INLINEFORM4 . On the output side, the user simulator firstly emits a user action INLINEFORM5 based on the updated state INLINEFORM6 . Conditioning on this emitted user action and the user dialog state INLINEFORM7 , a set of slot values are emitted. The user action and slot values are then passed to an NLG module to generate the final user utterance.","User Goal We define a user's goal INLINEFORM0 using a list of informable and requestable slots BIBREF38 . Informable slots are the slots that users can provide a value for to describe their goal (e.g. slots for food type, area, etc.). Requestable slots are the slots that users want to request the value for, such as requesting the address of a restaurant. We treat informable slots as discrete type of inputs that can take multiple values, and treat requestable slots as inputs that take binary values (i.e. a slot is either requested or not). In this work, once the a goal is sampled at the beginning of the conversation, we fix the user goal and do not change it during the conversation.","Action Selection Similar to the action modeling in dialog agent, we treat user action modeling as a multi-class classification problem conditioning on the dialog context encoded in the dialog-level LSTM state INLINEFORM0 on the user simulator side: DISPLAYFORM0","Once user action is generated at turn INLINEFORM0 , it is used together with the current user dialog state INLINEFORM1 to generate value for each informable slot: DISPLAYFORM0","Similar to the design of the dialog agent, INLINEFORM0 and INLINEFORM1 are MLPs with a single hidden layer and use INLINEFORM2 activation over their corresponding outputs.","Utterance Generation We use a template-based NLG module to convert the user simulator's outputs (action and slot values) to the natural language surface form."]},{"idx":6,"heading":"Deep RL Policy Optimization","paragraphs":["RL policy optimization is performed on top of the supervised pre-trained networks. The system architecture is shown in Figure FIGREF10 . We defines the state, action, and reward in our RL training setting and present the training details.","State For RL policy learning, states of the dialog agent and the user simulator at the INLINEFORM0 th turn are the dialog-level LSTM state INLINEFORM1 and INLINEFORM2 respectively. Both INLINEFORM3 and INLINEFORM4 captures the dialog history up till current turn. The user state INLINEFORM5 also encodes the user's goal.","Action Actions of the dialog agent and user simulator are the system action outputs INLINEFORM0 and INLINEFORM1 . An action is sampled by the agent based on a stochastic representation of the policy, which produces a probability distribution over actions given a dialog state. Action space is finite and discrete for both the dialog agent and the user simulator.","Reward Reward is calculated based on the level of task completion. A turn level reward INLINEFORM0 is applied based on the progress that the agent and user made in completing the predefined task over the past turn. At the end of each turn, a score INLINEFORM1 is calculated indicating to what extend the agent has fulfilled the user's request so far. The turn level reward INLINEFORM2 is then calculated by the difference of the scores received in two consecutive turns: DISPLAYFORM0","where INLINEFORM0 is the scoring function. INLINEFORM1 and INLINEFORM2 are the true user's goal and the agent's estimation of the user's goal, both are represented by slot-value pairs. Alternatively, the turn level reward INLINEFORM3 can be obtained by using the discounted reward received at the end of the dialog (positive reward for task success, and negative reward for task failure).","Policy Gradient RL For policy optimization with RL, policy gradient method is preferred over Q-learning in our system as the policy network parameters can be initialized with the INLINEFORM0 parameters learnied during supervised pre-training stage. With REINFORCE BIBREF39 , the objective function can be written as INLINEFORM1 , with INLINEFORM2 being the discount factor. We optimize parameter sets INLINEFORM3 and INLINEFORM4 for the dialog agent and the user simulator to maximize INLINEFORM5 . For the dialog agent, with likelihood ratio gradient estimator, gradient of INLINEFORM6 can be derived as: DISPLAYFORM0","This last expression above gives us an unbiased gradient estimator. We sample agent action and user action at each dialog turn and compute the policy gradient. Similarly, gradient on the user simulator side can be derived as: DISPLAYFORM0","A potential drawback of using REINFORCE is that the policy gradient might have high variance, since the agent may take many steps over the course of a dialog episode. We also explore using Advantage Actor-Critic (A2C) in our study, in which we approximate a state-value function using a feed forward neural network.","During model training, we use softmax policy for both the dialog agent and the user simulator to encourage exploration. Softmax policy samples action from the action probability distribution calculated by the INLINEFORM0 in the system action output. During evaluation, we apply greedy policy to the dialog agent, and still apply softmax policy to the user simulator. This is to increase randomness and diversity in the user simulator behavior, which is closer to the realistic dialog system evaluation settings with human users. This also prevents the two agents from fully cooperating with each other and exploiting the game."]},{"idx":7,"heading":"Dataset","paragraphs":["We prepare the data in our study based the corpus from the second Dialog State Tracking Challenge (DSTC2) BIBREF40 . We converte this corpus to our required format by adding API call commands and the corresponding KB query results. The dialogs simulation is based on real KB search results, which makes the dialog agent evaluation closer to real cases. Different from DSTC2, agent and user actions in our system are generated by concatenating the act and slot names in the original dialog act output (e.g. “ INLINEFORM0 ” maps to “ INLINEFORM1 ”). Slot values are captured in the belief tracking outputs. Table 1 shows the statistics of the dataset used in our experiments."]},{"idx":8,"heading":"Training Procedure","paragraphs":["In supervised pre-training, the dialog agent and the user simulator are trained separately against dialog corpus. We use the same set of neural network model configurations for both agents. Hidden layer sizes of the dialog-level LSTM for dialog modeling and utterance-level LSTM for utterance encoding are both set as 150. We perform mini-batch training using Adam optimization method BIBREF41 . Initial learning rate is set as 1e-3. Dropout BIBREF42 ( INLINEFORM0 ) is applied during model training to prevent to model from over-fitting.","In deep RL training stage, the policy network parameters are initialized with INLINEFORM0 parameters from the SL training. State-value function network parameters in A2C are initialized randomly. To ameliorate the non-stationarity problem when jointly training the two agents, we update the two agents iteratively during RL training. We take 100 episodes as a RL training cycle, in which we fix one agent and only update the other, and switch the training agent in the next cycle until convergence. In dialog simulation, we end the dialog if the dialog length exceeds the maximum turn size (20 in our experiment) or the user simulator emits the end of dialog action."]},{"idx":9,"heading":"Results and Analysis","paragraphs":["We evaluate the system on task success rate, average task reward, and average dialog length on simulated dialogs. A dialog is considered successful if the agent's belief tracking outputs match the informable user goal slot values completely, and all user requested slots are fulfilled. Note that the results on task success rate in this work should not be directly compared to the numbers in BIBREF16 , BIBREF43 , as both the dialog agent and the user simulator in our study are end-to-end models that take noisy natural language utterance as input and directly generate the final dialog act output. Moreover, instead of using greedy policy on user simulator, we sample user actions based on the action probability distribution from the user policy network to encourage diversity and variety in user behavior.","Table 2 shows the evaluation results. The baseline model uses the SL trained agents. REINFORCE-agent and A2C-agent apply RL training on the dialog agent only, without updating the user simulator. REINFORCE-joint and A2C-joint apply RL on both the dialog agent and user simulator over the SL pre-trained models. Figure 4, 5, and 6 show the learning curves of these five models during RL training on dialog success rate, average reward, and average success dialog length.","Success Rate As shown in Table 2, the SL model achieves the lowest task success rate. Model trained with SL on dialog corpus has limited capabilities in capturing the change in state, and thus may not be able to generalize well to unseen dialog situations during simulation. RL efficiently improves the dialog task success rate, as it enables the dialog agent to explore strategies that are not in the training corpus. The agent-update-only models using REINFORCE and A2C achieve similar results, outperforming the baseline model by 14.9% and 15.3% respectively. The jointly optimized models improved the performance further over the agent-update-only models. Model using A2C for joint policy optimization achieves the best task success rate.","Average Reward RL curves on average dialog reward show similar trends as above. One difference is that the joint training model using REINFORCE achieves the highest average reward, outperforming that using A2C by a small margin. This is likely due to the better performance of our REINFORCE models in earning reward in the failed dialogs. We find that our user simulator trained with A2C tends to have sharper action distribution from the softmax policy, making it easier to get stuck when it falls into an unfavorable state. We are interested in exploring fine-grained control strategies in joint RL policy optimization framework in our future work.","Average Success Turn Size The average turn size of success dialogs tends to decrease along the episode of RL policy learning. This is in line with our expectation as both the dialog agent and the user simulator improve their policies for more efficient and coherent strategies with the RL training."]},{"idx":10,"heading":"Conclusions","paragraphs":["In this work, we propose a reinforcement learning framework for dialog policy optimization in end-to-end task-oriented dialog systems. The proposed method addresses the challenge of lacking a reliable user simulator for policy learning in task-oriented dialog systems. We present an iterative policy learning method that jointly optimizes the dialog agent and the user simulator with deep RL by simulating dialogs between the two agents. Both the dialog agent and the user simulator are designed with neural network models that can be trained end-to-end. Experiment results show that our proposed method leads to promising improvements on task success rate and task reward over the supervised training and single-agent RL training baseline models."]},{"idx":11,"heading":"Figures and Tables","paragraphs":["Fig. 1. Dialog agent network architecture.","Fig. 2. User simulator network architecture.","Fig. 3. System architecture for joint dialog agent and user simulator policy optimization with deep RL","Table 1. Statistics of the dataset","Fig. 5. Learning curve for average reward","Table 2. Evaluation results on the converted DSTC2 dataset.","Fig. 4. Learning curve for task success rate.","Fig. 6. Learning curve for average turn size"]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1710.01789","title":"Enhanced Neural Machine Translation by Learning from Draft","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Neural machine translation (NMT) has recently achieved impressive results. A potential problem of the existing NMT algorithm, however, is that the decoding is conducted from left to right, without considering the right context. This paper proposes an two-stage approach to solve the problem. In the first stage, a conventional attention-based NMT system is used to produce a draft translation, and in the second stage, a novel double-attention NMT system is used to refine the translation, by looking at the original input as well as the draft translation. This drafting-and-refinement can obtain the right-context information from the draft, hence producing more consistent translations. We evaluated this approach using two Chinese-English translation tasks, one with 44k pairs and 1M pairs respectively. The experiments showed that our approach achieved positive improvements over the conventional NMT system: the improvements are 2.4 and 0.9 BLEU points on the small-scale and large-scale tasks, respectively."]},{"idx":1,"heading":"Introduction","paragraphs":["Neural machine translation (NMT) BIBREF0 BIBREF1 BIBREF2 has continuously gained attention from both academia and industry, and has obtained the state-of-the-art performance on many translation tasks, e.g., English-to-French, English-to-German, Turkish-to-English, and Chinese-to-English BIBREF2 BIBREF3 BIBREF4 BIBREF5 BIBREF6 .","The basic NMT model employs a sequence-to-sequence architecture BIBREF2 , where the meaning and intention of the source sentence is encoded into a representation vector with fixed dimensions, by which the translation (target sentence) is produced word by word. This architecture was later extended to an attention-based model BIBREF4 BIBREF5 , which allows the decoder being aware of the location that it should focus on at each decoding step. In a typical implementation of the attention-based NMT architecture, the encoder and decoder are both recurrent neural networks (RNN), where the hidden units are often some kinds of gated memory, e.g., long short-term memory units (LSTM) and gated recurrent units (GRU). The encoder turns the source sentence into a sequence of semantic representations, or hidden states. During decoding, the attention mechanism aligns the state of the decoder to all the hidden states generated by the encoder, and decides which part of the input should be paid more attention. By this information, the decoder can translate the semantic meaning of the input piece by piece.","A feature of this attention-based NMT model is that at each decoding step, the information of the decoding history, e.g., the words that have been produced so far, is utilized to obtain a smooth translation. This is essentially a kind of language model. A potential problem here is that we only use the left context (decoding history), but ignores the right context (future words), although the right context could be valuable. This shortage can be partially alleviated by beam search, where the decision of the target word is delayed by a few steps, so the `future' information can be employed to impact the `past' decision. However, the potential of this beam search is rather limited, and we have found that most of the sequences in the buffer share the same prefix BIBREF7 . Thus a better solution is desired.","In this paper, we propose a two-stage translation approach to tackle this problem. This approach is based on the idea of drafting-and-refinement, by which a draft translation is produced at the first stage, and at the second stage, the draft is refined by referring to the draft translation. Since the draft has given a rough idea what the translation would be, the right context can be obtained and utilized to make the refinement. In our implementation, the first stage (drafting) uses a typical attention-based NMT system, and the second stage (refinement) uses a double-attention NMT model that we will present shortly after.","The remaining of the paper is structured as follows: the next section reviews some related work, and Section 3 briefly describes the attention-based NMT model. Section 4 introduces the double-attention NMT model, and Section 5 presents the experiments. Finally, the paper ends up with a conclusion."]},{"idx":2,"heading":"Related Work","paragraphs":["The idea of using the right context to aid translation has been used in several studies. Sutskever et al. found that his sequence-to-sequence model achieved a promising improvement when reversing the source sentence “ INLINEFORM0 ” to “ INLINEFORM1 ” BIBREF8 . They argued that reversing the input may result in better memory usage during decoding, but it could also be possible that the right context is more informative when encoding the source input. The importance of the right context is also demonstrated by the fact that a significant improvement could be obtained when using a bi-directional RNN rather than using a uni-directional RNN BIBREF9 .","Recently, Novak et al. proposed an iterative translation approaches BIBREF10 . Similar to our two-stage approach, they got a draft translation using NMT, and then designed a `word correction' model that can correct the potential errors in the draft translation. The author raised the a similar argument that the right context is important to regularize the translation; the difference is that they focused on error correction but we perform a complete new translation. Our approach may avoid the co-correction problem, i.e., correcting one word may impact the correctness of other words.","The drafting-and-refinement idea was also used in other tasks. For example in automatic Chinese poetry composition BIBREF11 , a draft poem was firstly produced, and then the output was used as the input of the next iteration, to produce a new poem with better quality. The same approach was used in image generation BIBREF12 , by which an image was drawn step-by-step, and the residual error was minimized at each step."]},{"idx":3,"heading":"Background: Attention-based NMT","paragraphs":["Our study is based on the attention-based NMT model BIBREF4 , so we give a brief introduction for the sake of completeness. For simplicity, our introduction is just the basic architecture presented in BIBREF4 . Recent development of the attention-based NMT using different architectures can be found in BIBREF13 , BIBREF14 .","This typical attention-based model is shown in Fig. 1, where the encoder and decoders are implemented as two RNNs. Put it in brief, a source sentence INLINEFORM0 is encoded by the encoder RNN into a sequence of annotations INLINEFORM1 . Then the decoder RNN initiates a decoding process from a `start' symbol. At each decoding step INLINEFORM2 , the decoder computes the relevance between the decoder state INLINEFORM3 and each annotation INLINEFORM4 , resulting in the attention weight INLINEFORM5 . The target word is generated by maximizing the conditional probability INLINEFORM6 ."]},{"idx":4,"heading":"Encoder","paragraphs":["The encoder adopts the form of a bidirectional RNN (BiRNN) BIBREF9 , in which the hidden units can be either GRUs BIBREF15 or LSTMs BIBREF16 . In this paper, we used the GRU units. This BiRNN decoder consists of a forward RNN INLINEFORM0 and a backward RNN INLINEFORM1 . The forward RNN reads the source sentence from left to right and generates a sequence of forward annotations: INLINEFORM2","in which DISPLAYFORM0","Similarly, the backward RNN reads the input sequence from right to left and generates a sequence of backward annotations: INLINEFORM0","The final annotation INLINEFORM0 is then obtained by a concatenation of INLINEFORM1 and INLINEFORM2 , i.e., INLINEFORM3","The double-attention model involves two encoders: the first encoder INLINEFORM0 serves to encode the source sentence INLINEFORM1 , and the second one INLINEFORM2 encodes the draft sentence INLINEFORM3 . Both encoders are BiRNNs and generate annotations. The formulations for the encoding are the same as ( EQREF2 ). At each encoding step INLINEFORM4 , the annotations INLINEFORM5 and INLINEFORM6 are calculated as: DISPLAYFORM0 DISPLAYFORM1","Note that INLINEFORM0 and INLINEFORM1 both concatenate the forward and backward annotations. The two sequences of annotations are correspondingly written as INLINEFORM2 and INLINEFORM3 ."]},{"idx":5,"heading":"Attention","paragraphs":["When decoding the INLINEFORM0 target word, the attention mechanism computes the attention weights: INLINEFORM1","where INLINEFORM0 is the softmax function, and INLINEFORM1","where INLINEFORM0 is the hidden state of the decoder at step INLINEFORM1 , and INLINEFORM2 is the attention function that can be implemented by a neural network. The context vector INLINEFORM3 is calculated as a weighted sum of annotations INLINEFORM4 , given by: DISPLAYFORM0","In this way, the decoder will pay attention to the annotations that are most relevant to the present decoding status, where the target-relatedness is represented by the attention weight INLINEFORM0 .","The double-attention model involves two attention mechanisms, one for the original source input and the other for the draft translation. The final context vector is the concatenation of the context vectors on the two sequences: INLINEFORM0","where INLINEFORM0 is the decoding step, INLINEFORM1 is the context vector produced by the attention mechanism on the original input, and the INLINEFORM2 is the context vector produced by the attention mechanism on the draft translation. These two context vectors are computed exactly as the attention mechanism of the conventional attention-based NMT model, as presented in the previous section."]},{"idx":6,"heading":"Decoder","paragraphs":["As soon as we get the context vector INLINEFORM0 from INLINEFORM1 at decoding step INLINEFORM2 , the conditional probability of selecting a word INLINEFORM3 is calculated as: DISPLAYFORM0","where INLINEFORM0 is the the hidden state of the decoder at the INLINEFORM1 step, and it is updated according to the previous hidden state INLINEFORM2 , the previous output INLINEFORM3 , and the context vector INLINEFORM4 : DISPLAYFORM0","Using the concatenated context vector INLINEFORM0 , the decoder performs the translation as the conventional attention-based NMT: DISPLAYFORM0","where INLINEFORM0 . The hidden state of the decoder is computed the same as ( EQREF8 ), and the initial value of the hidden state is calculated as an average sum of the first backward annotations of the two input sequences: DISPLAYFORM0"]},{"idx":7,"heading":"Training","paragraphs":["All the parameters in the attention-based NMT model are optimized by maximizing the following conditional log-likelihood on the training dataset: DISPLAYFORM0","where INLINEFORM0 denotes the INLINEFORM1 training sample, i.e., a bi-lingual sentence pair, and INLINEFORM2 represents the model parameters that we need to optimize. This optimization can be conducted by any numerical optimization approach, but stochastic gradient descend (SGD) is the most often used.","The training of the double-attention NMT model is similar to the conventional attention-based NMT model, though the log likelihood function now depends on two input sequences INLINEFORM0 and INLINEFORM1 . This is written as follows: DISPLAYFORM0","Note that to simplify the training, the architecture and the parameters of the first-stage NMT model can be inherited and re-used in the double-attention model. In our study, all the word embeddings (both on the source and target sides) are inherited from the first-stage NMT model and are fixed during the double-attention model training.","There are two reasons to keep these embeddings fixed. First of all, the embeddings have been well learned in the first stage, and re-using them in the second stage will significantly simplify the model training. The second and more important, the double-attention model consists of a large amount of model parameters, which makes it prone to over-fitting, especially when the training data is limited. We have observed the over-fitting problem on the small-scale task in our experiments, and re-using the word embeddings indeed reduced the over-fitting risk."]},{"idx":8,"heading":"Translation by Learning from Draft","paragraphs":["For the attention-based NMT, the posterior probability for the target word prediction is in the form INLINEFORM0 . Notice that it is conditioned on the entire source sentence INLINEFORM1 and the decoding history INLINEFORM2 , which is the left context. However, it does not involve any right context, although that information might be useful.","One may argue that the backward information has been involved in the annotations by the BiRNN encoding, therefore the right context information has been already taken into account. But this is not the case. The right context we refer to is nothing to do with the semantic content that have been encoded; instead, it is a regulation imposed by the target words that would be decoded.","We designed a two-state translation approach to solve the problem. By this approach, the source sentence INLINEFORM0 is firstly translated into a draft INLINEFORM1 by an conventional attention-based NMT system, like the one in BIBREF4 . Then a second-stage translation system will refine or `re-translate' this draft. In this pipeline, the right context, although not very accurate, can be roughly obtained from the draft. This information will offer valuable regularization at the second-stage decoding, thus delivering a refined translation. In practice, we design a double-attention NMT model to utilize the right context information. This model accepts both the target draft INLINEFORM2 and the original source sentence INLINEFORM3 , and pays attention to both the sequences during decoding. The main architecture is shown in Fig. 2."]},{"idx":9,"heading":"Datasets and evaluation metric","paragraphs":["The experiments were conducted on two Chinese-English translation tasks, one using the large-scale NIST dataset and the other using the small-scale IWSLT dataset. The NIST training data consisted of 1M sentence pairs, which involved 19M source tokens and 24M target tokens. We used the NIST 2005 test set as the development set and the NIST 2003 test set as the test set. The IWSLT training data consisted of 44K sentences sampled from the tourism and travel domain. The development set was composed of the ASR devset 1 and devset 2 from IWSLT 2005, and the test set was the IWSLT 2005 test set. As for the evaluation metric, we used the case-insensitive 4-gram NIST BLEU score BIBREF17 ."]},{"idx":10,"heading":"Comparison systems","paragraphs":["We compared our two-stage system with two baseline systems: one is a conventional SMT system and the other is an attention-based NMT system (which is actually the first stage of our two-stage system).","Moses BIBREF18 is a widely-used SMT system and a state-of-the-art open-source toolkit. Although NMT has developed very quickly and outperforms SMT in some large-scale tasks, SMT is still a strong baseline for small-scale tasks. In our experiments, the following features were enabled for the SMT system: relative translation frequencies and lexical translation probabilities on both directions, distortion distance, language model and word penalty. For the language model, the KenLM toolkit BIBREF19 was employed to build a 5-gram language model (with the Keneser-Ney smoothing) on the target side of the training data.","We reproduced the attention-based NMT system proposed by Bahdanau et al. BIBREF4 . The implementation was based on Tensorflow. We compared our implementation with a public implementation using Theano, and got a comparable performance on the same data sets with the same parameter settings."]},{"idx":11,"heading":"Settings","paragraphs":["For a fair comparison, the configurations of the attention-based NMT system and the two-stage NMT system were intentionally set to be identical. The dimensionality of word embeddings, the number of hidden units and the vocabulary size were empirically set to 620, 1000, 30000 respectively for the large-scale task and were halved for the small-scale task. In the training process, we used the minibatch SGD algorithm together with the Adam algorithm BIBREF20 to change the learning rate. The batch size was set to be 80. The initial learning rate was set to be 0.0001 for the large-scale task and 0.001 for the small-scale task. The decoding was implemented as a beam search, where the beam size was set to be 5."]},{"idx":12,"heading":"Results","paragraphs":["The BLEU results are given in Table I. It can be seen that our two-stage NMT system delivers notable performance improvement compared to the NMT baseline. On the large-scale task (NIST), the two-stage system outperforms the NMT baseline by 0.9 BLEU points, and it also outperforms the SMT baseline by 1.1 points. On the small-scale task (IWSLT), the two-stage approach outperforms the NMT baseline by 2.4 BLEU points, though it is still worse than the SMT baseline (mainly because the SMT model is able to capture most details in the language pairs while the NMT model tends to seize the generalities and treats rare details as noise, which is common when dataset is small). These results demonstrated that after the refinement with the double-attention model, the quality of the translation has been clearly improved."]},{"idx":13,"heading":"Conclusions","paragraphs":["The attention-based NMT model performs the decoding from left to right, which can not fully utilize the right context. In this paper, we propose a two-stage translation approach that obtains a draft translation by a conventional NMT system, and then refines the translation by considering both the original input and the draft translation. By this way, the right context can be obtained from the draft and utilized to regularize the second-stage translation. Our experiments demonstrated that the two-stage approach indeed performs better than the conventional attention-based NMT system. In the future work, we will investigate a better architecture to integrate the draft translation. Moreover, the memory usage of the double-attention model needs to be reduced."]},{"idx":14,"heading":"Acknowledgment","paragraphs":["This work was supported by the National Natural Science Foundation of China under Grant No. 61371136 / 61633013 and the National Basic Research Program (973 Program) of China under Grant No. 2013CB329302."]},{"idx":15,"heading":"Figures and Tables","paragraphs":["Fig. 1. The encoder-decoder NMT with attention.","Fig. 2. The double-attention NMT model.","TABLE I BLEU SCORES ON CHINESE-ENGLISH TRANSLATION"]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1710.03348","title":"What does Attention in Neural Machine Translation Pay Attention to?","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Attention in neural machine translation provides the possibility to encode relevant parts of the source sentence at each translation step. As a result, attention is considered to be an alignment model as well. However, there is no work that specifically studies attention and provides analysis of what is being learned by attention models. Thus, the question still remains that how attention is similar or different from the traditional alignment. In this paper, we provide detailed analysis of attention and compare it to traditional alignment. We answer the question of whether attention is only capable of modelling translational equivalent or it captures more information. We show that attention is different from alignment in some cases and is capturing useful information other than alignments."]},{"idx":1,"heading":"Introduction","paragraphs":["Neural machine translation (NMT) has gained a lot of attention recently due to its substantial improvements in machine translation quality achieving state-of-the-art performance for several languages BIBREF0 , BIBREF1 , BIBREF2 . The core architecture of neural machine translation models is based on the general encoder-decoder approach BIBREF3 . Neural machine translation is an end-to-end approach that learns to encode source sentences into distributed representations and decode these representations into sentences in the target language. Among the different neural MT models, attentional NMT BIBREF4 , BIBREF5 has become popular due to its capability to use the most relevant parts of the source sentence at each translation step. This capability also makes the attentional model superior in translating longer sentences BIBREF4 , BIBREF5 .","Figure FIGREF1 shows an example of how attention uses the most relevant source words to generate a target word at each step of the translation. In this paper we focus on studying the relevance of the attended parts, especially cases where attention is `smeared out' over multiple source words where their relevance is not entirely obvious, see, e.g., “would\" and “like\" in Figure FIGREF1 . Here, we ask whether these are due to errors of the attention mechanism or are a desired behavior of the model.","Since the introduction of attention models in neural machine translation BIBREF4 various modifications have been proposed BIBREF5 , BIBREF6 , BIBREF7 . However, to the best of our knowledge there is no study that provides an analysis of what kind of phenomena is being captured by attention. There are some works that have looked to attention as being similar to traditional word alignment BIBREF8 , BIBREF6 , BIBREF7 , BIBREF9 . Some of these approaches also experimented with training the attention model using traditional alignments BIBREF8 , BIBREF7 , BIBREF9 . liu-EtAl:2016:COLING have shown that attention could be seen as a reordering model as well as an alignment model.","In this paper, we focus on investigating the differences between attention and alignment and what is being captured by the attention mechanism in general. The questions that we are aiming to answer include: Is the attention model only capable of modelling alignment? And how similar is attention to alignment in different syntactic phenomena?","Our analysis shows that attention models traditional alignment in some cases more closely while it captures information beyond alignment in others. For instance, attention agrees with traditional alignments to a high degree in the case of nouns. However, it captures other information rather than only the translational equivalent in the case of verbs.","This paper makes the following contributions: 1) We provide a detailed comparison of attention in NMT and word alignment. 2) We show that while different attention mechanisms can lead to different degrees of compliance with respect to word alignments, global compliance is not always helpful for word prediction. 3) We show that attention follows different patterns depending on the type of the word being generated. 4) We demonstrate that attention does not always comply with alignment. We provide evidence showing that the difference between attention and alignment is due to attention model capability to attend the context words influencing the current word translation."]},{"idx":2,"heading":"Related Work","paragraphs":["liu-EtAl:2016:COLING investigate how training the attention model in a supervised manner can benefit machine translation quality. To this end they use traditional alignments obtained by running automatic alignment tools (GIZA++ BIBREF10 and fast_align BIBREF11 ) on the training data and feed it as ground truth to the attention network. They report some improvements in translation quality arguing that the attention model has learned to better align source and target words. The approach of training attention using traditional alignments has also been proposed by others BIBREF9 , BIBREF8 . chen2016guided show that guided attention with traditional alignment helps in the domain of e-commerce data which includes lots of out of vocabulary (OOV) product names and placeholders, but not much in the other domains. alkhouli-EtAl:2016:WMT have separated the alignment model and translation model, reasoning that this avoids propagation of errors from one model to the other as well as providing more flexibility in the model types and training of the models. They use a feed-forward neural network as their alignment model that learns to model jumps in the source side using HMM/IBM alignments obtained by using GIZA++.","shi-padhi-knight:2016:EMNLP2016 show that various kinds of syntactic information are being learned and encoded in the output hidden states of the encoder. The neural system for their experimental analysis is not an attentional model and they argue that attention does not have any impact for learning syntactic information. However, performing the same analysis for morphological information, belinkov2017neural show that attention has also some effect on the information that the encoder of neural machine translation system encodes in its output hidden states. As part of their analysis they show that a neural machine translation system that has an attention model can learn the POS tags of the source side more efficiently than a system without attention.","Recently, koehn2017six carried out a brief analysis of how much attention and alignment match in different languages by measuring the probability mass that attention gives to alignments obtained from an automatic alignment tool. They also report differences based on the most attended words.","The mixed results reported by chen2016guided, alkhouli-EtAl:2016:WMT, liu-EtAl:2016:COLING on optimizing attention with respect to alignments motivates a more thorough analysis of attention models in NMT."]},{"idx":3,"heading":"Attention Models","paragraphs":["This section provides a short background on attention and discusses two most popular attention models which are also used in this paper. The first model is a non-recurrent attention model which is equivalent to the “global attention\" method proposed by DBLPjournalscorrLuongPM15. The second attention model that we use in our investigation is an input-feeding model similar to the attention model first proposed by bahdanau-EtAl:2015:ICLR and turned to a more general one and called input-feeding by DBLPjournalscorrLuongPM15. Below we describe the details of both models.","Both non-recurrent and input-feeding models compute a context vector INLINEFORM0 at each time step. Subsequently, they concatenate the context vector to the hidden state of decoder and pass it through a non-linearity before it is fed into the softmax output layer of the translation network. DISPLAYFORM0","The difference of the two models lays in the way they compute the context vector. In the non-recurrent model, the hidden state of the decoder is compared to each hidden state of the encoder. Often, this comparison is realized as the dot product of vectors. Then the comparison result is fed to a softmax layer to compute the attention weight. DISPLAYFORM0 DISPLAYFORM1","Here INLINEFORM0 is the hidden state of the decoder at time INLINEFORM1 , INLINEFORM2 is INLINEFORM3 th hidden state of the encoder and INLINEFORM4 is the length of the source sentence. Then the computed alignment weights are used to compute a weighted sum over the encoder hidden states which results in the context vector mentioned above: DISPLAYFORM0","The input-feeding model changes the context vector computation in a way that at each step INLINEFORM0 the context vector is aware of the previously computed context INLINEFORM1 . To this end, the input-feeding model feeds back its own INLINEFORM2 to the network and uses the resulting hidden state instead of the context-independent INLINEFORM3 , to compare to the hidden states of the encoder. This is defined in the following equations: DISPLAYFORM0 DISPLAYFORM1","Here, INLINEFORM0 is the function that the stacked LSTM applies to the input, INLINEFORM1 is the last generated target word, and INLINEFORM2 is the output of previous time step of the input-feeding network itself, meaning the output of Equation EQREF2 in the case that context vector has been computed using INLINEFORM3 from Equation EQREF7 ."]},{"idx":4,"heading":"Comparing Attention with Alignment","paragraphs":["As mentioned above, it is a commonly held assumption that attention corresponds to word alignments. To verify this, we investigate whether higher consistency between attention and alignment leads to better translations."]},{"idx":5,"heading":"Measuring Attention-Alignment Accuracy","paragraphs":["In order to compare attentions of multiple systems as well as to measure the difference between attention and word alignment, we convert the hard word alignments into soft ones and use cross entropy between attention and soft alignment as a loss function. For this purpose, we use manual alignments provided by RWTH German-English dataset as the hard alignments. The statistics of the data are given in Table TABREF8 . We convert the hard alignments to soft alignments using Equation EQREF10 . For unaligned words, we first assume that they have been aligned to all the words in the source side and then do the conversion. DISPLAYFORM0","Here INLINEFORM0 is the set of source words aligned to target word INLINEFORM1 and INLINEFORM2 is the number of source words in the set.","After conversion of the hard alignments to soft ones, we compute the attention loss as follows: DISPLAYFORM0","Here INLINEFORM0 is the source sentence and INLINEFORM1 is the weight of the alignment link between source word INLINEFORM2 and the target word (see Equation EQREF10 ). INLINEFORM3 is the attention weight INLINEFORM4 (see Equation EQREF4 ) of the source word INLINEFORM5 , when generating the target word INLINEFORM6 .","In our analysis, we also look into the relation between translation quality and the quality of the attention with respect to the alignments. For measuring the quality of attention, we use the attention loss defined in Equation EQREF11 . As a measure of translation quality, we choose the loss between the output of our NMT system and the reference translation at each translation step, which we call word prediction loss. The word prediction loss for word INLINEFORM0 is logarithm of the probability given in Equation EQREF12 . DISPLAYFORM0","Here INLINEFORM0 is the source sentence, INLINEFORM1 is target word at time step INLINEFORM2 , INLINEFORM3 is the target history given by the reference translation and INLINEFORM4 is given by Equation EQREF2 for either non-recurrent or input-feeding attention models.","Spearman's rank correlation is used to compute the correlation between attention loss and word prediction loss: DISPLAYFORM0","where INLINEFORM0 and INLINEFORM1 are the ranks of the attention losses and word prediction losses, respectively, INLINEFORM2 is the covariance between two input variables, and INLINEFORM3 and INLINEFORM4 are the standard deviations of INLINEFORM5 and INLINEFORM6 .","If there is a close relationship between word prediction quality and consistency of attention versus alignment, then there should be high correlation between word prediction loss and attention loss. Figure FIGREF13 shows an example with different levels of consistency between attention and word alignments. For the target words “will\" and “come\" the attention is not focused on the manually aligned word but distributed between the aligned word and other words. The focus of this paper is examining cases where attention does not follow alignment, answering the questions whether those cases represent errors or desirable behavior of the attention model."]},{"idx":6,"heading":"Measuring Attention Concentration","paragraphs":["As another informative variable in our analysis, we look into the attention concentration. While most word alignments only involve one or a few words, attention can be distributed more freely. We measure the concentration of attention by computing the entropy of the attention distribution: DISPLAYFORM0"]},{"idx":7,"heading":"Empirical Analysis of Attention Behaviour","paragraphs":["We conduct our analysis using the two different attention models described in Section SECREF3 . Our first attention model is the global model without input-feeding as introduced by DBLPjournalscorrLuongPM15. The second model is the input-feeding model BIBREF5 , which uses recurrent attention. Our NMT system is a unidirectional encoder-decoder system as described in BIBREF5 , using 4 recurrent layers.","We trained the systems with dimension size of 1,000 and batch size of 80 for 20 epochs. The vocabulary for both source and target side is set to be the 30K most common words. The learning rate is set to be 1 and a maximum gradient norm of 5 has been used. We also use a dropout rate of 0.3 to avoid overfitting."]},{"idx":8,"heading":"Impact of Attention Mechanism","paragraphs":["We train both of the systems on the WMT15 German-to-English training data, see Table TABREF18 for some statistics. Table TABREF17 shows the BLEU scores BIBREF12 for both systems on different test sets.","Since we use POS tags and dependency roles in our analysis, both of which are based on words, we chose not to use BPE BIBREF13 which operates at the sub-word level.","We report alignment error rate (AER) BIBREF14 , which is commonly used to measure alignment quality, in Table TABREF20 to show the difference between attentions and human alignments provided by RWTH German-English dataset. To compute AER over attentions, we follow DBLPjournalscorrLuongPM15 to produce hard alignments from attentions by choosing the most attended source word for each target word. We also use GIZA++ BIBREF10 to produce automatic alignments over the data set to allow for a comparison between automatically generated alignments and the attentions generated by our systems. GIZA++ is run in both directions and alignments are symmetrized using the grow-diag-final-and refined alignment heuristic.","As shown in Table TABREF20 , the input-feeding system not only achieves a higher BLEU score, but also uses attentions that are closer to the human alignments.","Table TABREF21 compares input-feeding and non-recurrent attention in terms of attention loss computed using Equation EQREF11 . Here the losses between the attention produced by each system and the human alignments is reported. As expected, the difference in attention losses are in line with AER.","The difference between these comparisons is that AER only takes the most attended word into account while attention loss considers the entire attention distribution."]},{"idx":9,"heading":"Alignment Quality Impact on Translation","paragraphs":["Based on the results in Section SECREF19 , one might be inclined to conclude that the closer the attention is to the word alignments the better the translation. However, chen2016guided, liu-EtAl:2016:COLING, alkhouli-EtAl:2016:WMT report mixed results by optimizing their NMT system with respect to word prediction and alignment quality. These findings warrant a more fine-grained analysis of attention. To this end, we include POS tags in our analysis and study the patterns of attention based on POS tags of the target words. We choose POS tags because they exhibit some simple syntactic characteristics. We use the coarse grained universal POS tags BIBREF15 given in Table TABREF25 .","To better understand how attention accuracy affects translation quality, we analyse the relationship between attention loss and word prediction loss for individual part-of-speech classes. Figure FIGREF22 shows how attention loss differs when generating different POS tags. One can see that attention loss varies substantially across different POS tags. In particular, we focus on the cases of NOUN and VERB which are the most frequent POS tags in the dataset. As shown, the attention of NOUN is the closest to alignments on average. But the average attention loss for VERB is almost two times larger than the loss for NOUN.","Considering this difference and the observations in Section SECREF19 , a natural follow-up would be to focus on getting the attention of verbs to be closer to alignments. However, Figure FIGREF22 shows that the average word prediction loss for verbs is actually smaller compared to the loss for nouns. In other words, although the attention for verbs is substantially more inconsistent with the word alignments than for nouns, the NMT system translates verbs more accurately than nouns on average.","To formalize this relationship we compute Spearman's rank correlation between word prediction loss and attention loss, based on the POS tags of the target side, for the input-feeding model, see Figure FIGREF27 .","The low correlation for verbs confirms that attention to other parts of source sentence rather than the aligned word is necessary for translating verbs and that attention does not necessarily have to follow alignments. However, the higher correlation for nouns means that consistency of attention with alignments is more desirable. This could, in a way, explain the mixed result reported for training attention using alignments BIBREF9 , BIBREF7 , BIBREF8 . Especially the results by chen2016guided in which large improvements are achieved for the e-commerce domain which contains many OOV product names and placeholders, but no or very weak improvements were achieved over common domains."]},{"idx":10,"heading":"Attention Concentration","paragraphs":["In word alignment, most target words are aligned to one source word. The average number of source words aligned to nouns and verbs is 1.1 and 1.2 respectively. To investigate to what extent this also holds for attention we measure the attention concentration by computing the entropy of the attention distribution, see Equation EQREF16 .","Figure FIGREF28 shows the average entropy of attention based on POS tags. As shown, nouns have one of the lowest entropies meaning that on average the attention for nouns tends to be concentrated. This also explains the closeness of the attention to alignments for nouns. In addition, the correlation between attention entropy and attention loss in case of nouns is high as shown in Figure FIGREF28 . This means that attention entropy can be used as a measure of closeness of attention to alignment in the case of nouns.","The higher attention entropy for verbs, in Figure FIGREF28 , shows that the attention is more distributed compared to nouns. The low correlation between attention entropy and word prediction loss (see Figure FIGREF32 ) shows that attention concentration is not required when translating into verbs. This also confirms that the correct translation of verbs requires the systems to pay attention to different parts of the source sentence.","Another interesting observation here is the low correlation for pronouns (PRON) and particles (PRT), see Figure FIGREF32 . As can be seen in Figure FIGREF28 , these tags have more distributed attention comparing to nouns, for example. This could either mean that the attention model does not know where to focus or it deliberately pays attention to multiple, somehow relevant, places to be able to produce a better translation. The latter is supported by the relatively low word prediction losses, shown in the Figure FIGREF22 ."]},{"idx":11,"heading":"Attention Distribution","paragraphs":["To further understand under which conditions attention is paid to words other than the aligned words, we study the distribution of attention over the source words. First, we measure how much attention is paid to the aligned words for each POS tag, on average. To this end, we compute the percentage of the probability mass that the attention model has assigned to aligned words for each POS tag, see Table TABREF35 .","One can notice that less than half of the attention is paid to alignment points for most of the POS tags. To examine how the rest of attention in each case has been distributed over the source sentence we measure the attention distribution over dependency roles in the source side. We first parse the source side of RWTH data using the ParZu parser BIBREF16 . Then we compute how the attention probability mass given to the words other than the alignment points, is distributed over dependency roles. Table TABREF33 gives the most attended roles for each POS tag. Here, we focus on POS tags discussed earlier. One can see that the most attended roles when translating to nouns include adjectives and determiners and in the case of translating to verbs, it includes auxiliary verbs, adverbs (including negation), subjects, and objects."]},{"idx":12,"heading":"Conclusion","paragraphs":["In this paper, we have studied attention in neural machine translation and provided an analysis of the relation between attention and word alignment. We have shown that attention agrees with traditional alignment to a certain extent. However, this differs substantially by attention mechanism and the type of the word being generated. We have shown that attention has different patterns based on the POS tag of the target word. The concentrated pattern of attention and the relatively high correlations for nouns show that training the attention with explicit alignment labels is useful for generating nouns. However, this is not the case for verbs, since the large portion of attention being paid to words other than alignment points, is already capturing other relevant information. Training attention with alignments in this case will force the attention model to forget these useful information. This explains the mixed results reported when guiding attention to comply with alignments BIBREF9 , BIBREF7 , BIBREF8 ."]},{"idx":13,"heading":"Acknowledgments","paragraphs":["This research was funded in part by the Netherlands Organization for Scientific Research (NWO) under project numbers 639.022.213 and 612.001.218."]},{"idx":14,"heading":"Figures and Tables","paragraphs":["Figure 1: Visualization of the attention paid to the relevant parts of the source sentence for each generated word of a translation example. See how the attention is ‘smeared out’ over multiple source words in the case of “would” and “like”.","Table 1: Statistics of manual alignments provided by RWTH German-English data.","Figure 2: An example of inconsistent attention and alignment. The outlined cells show the manual alignments from the RWTH dataset (see Table 1). See how attention is deviated from alignment points in the case of “will” and “come”.","Table 2: Performance of our experimental system in BLEU on different standard WMT test sets.","Table 5: Average loss between attention generated by input-feeding and non-recurrent systems and the manual alignment over RWTH GermanEnglish data.","Figure 3: Average attention losses and word prediction losses from the input-feeding system.","Table 6: List of the universal POS tags used in our analysis.","Figure 4: Correlation between word prediction loss and attention loss for the input-feeding model.","Figure 5: Attention entropy and its correlation with attention loss for the input-feeding system.","Figure 6: Correlation of attention entropy and word prediction loss for the input-feeding system.","Table 7: The most attended dependency roles with their received attention percentage from the attention probability mass paid to the words other than the alignment points. Here, we focus on the POS tags discussed earlier.","Table 8: Distribution of attention probability mass (in %) over alignment points and the rest of the words for each POS tag."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1710.10380","title":"Speeding up Context-based Sentence Representation Learning with Non-autoregressive Convolutional Decoding","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Context plays an important role in human language understanding, thus it may also be useful for machines learning vector representations of language. In this paper, we explore an asymmetric encoder-decoder structure for unsupervised context-based sentence representation learning. We carefully designed experiments to show that neither an autoregressive decoder nor an RNN decoder is required. After that, we designed a model which still keeps an RNN as the encoder, while using a non-autoregressive convolutional decoder. We further combine a suite of effective designs to significantly improve model efficiency while also achieving better performance. Our model is trained on two different large unlabelled corpora, and in both cases the transferability is evaluated on a set of downstream NLP tasks. We empirically show that our model is simple and fast while producing rich sentence representations that excel in downstream tasks."]},{"idx":1,"heading":"Introduction","paragraphs":["Learning distributed representations of sentences is an important and hard topic in both the deep learning and natural language processing communities, since it requires machines to encode a sentence with rich language content into a fixed-dimension vector filled with real numbers. Our goal is to build a distributed sentence encoder learnt in an unsupervised fashion by exploiting the structure and relationships in a large unlabelled corpus.","Numerous studies in human language processing have supported that rich semantics of a word or sentence can be inferred from its context BIBREF0 , BIBREF1 . The idea of learning from the co-occurrence BIBREF2 was recently successfully applied to vector representation learning for words in BIBREF3 and BIBREF4 .","A very recent successful application of the distributional hypothesis BIBREF0 at the sentence-level is the skip-thoughts model BIBREF5 . The skip-thoughts model learns to encode the current sentence and decode the surrounding two sentences instead of the input sentence itself, which achieves overall good performance on all tested downstream NLP tasks that cover various topics. The major issue is that the training takes too long since there are two RNN decoders to reconstruct the previous sentence and the next one independently. Intuitively, given the current sentence, inferring the previous sentence and inferring the next one should be different, which supports the usage of two independent decoders in the skip-thoughts model. However, BIBREF6 proposed the skip-thought neighbour model, which only decodes the next sentence based on the current one, and has similar performance on downstream tasks compared to that of their implementation of the skip-thoughts model.","In the encoder-decoder models for learning sentence representations, only the encoder will be used to map sentences to vectors after training, which implies that the quality of the generated language is not our main concern. This leads to our two-step experiment to check the necessity of applying an autoregressive model as the decoder. In other words, since the decoder's performance on language modelling is not our main concern, it is preferred to reduce the complexity of the decoder to speed up the training process. In our experiments, the first step is to check whether “teacher-forcing” is required during training if we stick to using an autoregressive model as the decoder, and the second step is to check whether an autoregressive decoder is necessary to learn a good sentence encoder. Briefly, the experimental results show that an autoregressive decoder is indeed not essential in learning a good sentence encoder; thus the two findings of our experiments lead to our final model design.","Our proposed model has an asymmetric encoder-decoder structure, which keeps an RNN as the encoder and has a CNN as the decoder, and the model explores using only the subsequent context information as the supervision. The asymmetry in both model architecture and training pair reduces a large amount of the training time.","The contribution of our work is summarised as:","The following sections will introduce the components in our “RNN-CNN” model, and discuss our experimental design."]},{"idx":2,"heading":"RNN-CNN Model","paragraphs":["Our model is highly asymmetric in terms of both the training pairs and the model structure. Specifically, our model has an RNN as the encoder, and a CNN as the decoder. During training, the encoder takes the INLINEFORM0 -th sentence INLINEFORM1 as the input, and then produces a fixed-dimension vector INLINEFORM2 as the sentence representation; the decoder is applied to reconstruct the paired target sequence INLINEFORM3 that contains the subsequent contiguous words. The distance between the generated sequence and the target one is measured by the cross-entropy loss at each position in INLINEFORM4 . An illustration is in Figure FIGREF4 . (For simplicity, we omit the subscript INLINEFORM5 in this section.)","1. Encoder: The encoder is a bi-directional Gated Recurrent Unit (GRU, BIBREF7 ). Suppose that an input sentence INLINEFORM0 contains INLINEFORM1 words, which are INLINEFORM2 , and they are transformed by an embedding matrix INLINEFORM3 to word vectors. The bi-directional GRU takes one word vector at a time, and processes the input sentence in both the forward and backward directions; both sets of hidden states are concatenated to form the hidden state matrix INLINEFORM7 , where INLINEFORM8 is the dimension of the hidden states INLINEFORM9 ( INLINEFORM10 ).","2. Representation: We aim to provide a model with faster training speed and better transferability than existing algorithms; thus we choose to apply a parameter-free composition function, which is a concatenation of the outputs from a global mean pooling over time and a global max pooling over time, on the computed sequence of hidden states INLINEFORM0 . The composition function is represented as DISPLAYFORM0","where INLINEFORM0 is the max operation on each row of the matrix INLINEFORM1 , which outputs a vector with dimension INLINEFORM2 . Thus the representation INLINEFORM3 . 3. Decoder: The decoder is a 3-layer CNN to reconstruct the paired target sequence INLINEFORM4 , which needs to expand INLINEFORM5 , which can be considered as a sequence with only one element, to a sequence with INLINEFORM6 elements. Intuitively, the decoder could be a stack of deconvolution layers. For fast training speed, we optimised the architecture to make it possible to use fully-connected layers and convolution layers in the decoder, since generally, convolution layers run faster than deconvolution layers in modern deep learning frameworks.","Suppose that the target sequence INLINEFORM0 has INLINEFORM1 words, which are INLINEFORM2 , the first layer of deconvolution will expand INLINEFORM3 , into a feature map with INLINEFORM4 elements. It can be easily implemented as a concatenation of outputs from INLINEFORM5 linear transformations in parallel. Then the second and third layer are 1D-convolution layers. The output feature map is INLINEFORM6 , where INLINEFORM7 is the dimension of the word vectors.","Note that our decoder is not an autoregressive model and has high training efficiency. We will discuss the reason for choosing this decoder which we call a predict-all-words CNN decoder.","4. Objective: The training objective is to maximise the likelihood of the target sequence being generated from the decoder. Since in our model, each word is predicted independently, a softmax layer is applied after the decoder to produce a probability distribution over words in INLINEFORM0 at each position, thus the probability of generating a word INLINEFORM1 in the target sequence is defined as: DISPLAYFORM0","where, INLINEFORM0 is the vector representation of INLINEFORM1 in the embedding matrix INLINEFORM2 , and INLINEFORM3 is the dot-product between the word vector and the feature vector produced by the decoder at position INLINEFORM4 . The training objective is to minimise the sum of the negative log-likelihood over all positions in the target sequence INLINEFORM5 : DISPLAYFORM0","where INLINEFORM0 and INLINEFORM1 contain the parameters in the encoder and the decoder, respectively. The training objective INLINEFORM2 is summed over all sentences in the training corpus."]},{"idx":3,"heading":"Architecture Design","paragraphs":["We use an encoder-decoder model and use context for learning sentence representations in an unsupervised fashion. Since the decoder won't be used after training, and the quality of the generated sequences is not our main focus, it is important to study the design of the decoder. Generally, a fast training algorithm is preferred; thus proposing a new decoder with high training efficiency and also strong transferability is crucial for an encoder-decoder model."]},{"idx":4,"heading":"CNN as the decoder","paragraphs":["Our design of the decoder is basically a 3-layer ConvNet that predicts all words in the target sequence at once. In contrast, existing work, such as skip-thoughts BIBREF5 , and CNN-LSTM BIBREF9 , use autoregressive RNNs as the decoders. As known, an autoregressive model is good at generating sequences with high quality, such as language and speech. However, an autoregressive decoder seems to be unnecessary in an encoder-decoder model for learning sentence representations, since it won't be used after training, and it takes up a large portion of the training time to compute the output and the gradient. Therefore, we conducted experiments to test the necessity of using an autoregressive decoder in learning sentence representations, and we had two findings.","Finding I: It is not necessary to input the correct words into an autoregressive decoder for learning sentence representations.","The experimental design was inspired by BIBREF10 . The model we designed for the experiment has a bi-directional GRU as the encoder, and an autoregressive decoder, including both RNN and CNN. We started by analysing the effect of different sampling strategies of the input words on learning an auto-regressive decoder.","We compared three sampling strategies of input words in decoding the target sequence with an autoregressive decoder: (1) Teacher-Forcing: the decoder always gets the ground-truth words; (2) Always Sampling: at time step INLINEFORM0 , a word is sampled from the multinomial distribution predicted at time step INLINEFORM1 ; (3) Uniform Sampling: a word is uniformly sampled from the dictionary INLINEFORM2 , then fed to the decoder at every time step.","The results are presented in Table TABREF10 (top two subparts). As we can see, the three decoding settings do not differ significantly in terms of the performance on selected downstream tasks, with RNN or CNN as the decoder. The results show that, in terms of learning good sentence representations, the autoregressive decoder doesn't require the correct ground-truth words as the inputs.","Finding II: The model with an autoregressive decoder performs similarly to the model with a predict-all-words decoder.","With Finding I, we conducted an experiment to test whether the model needs an autoregressive decoder at all. In this experiment, the goal is to compare the performance of the predict-all-words decoders and that of the autoregressive decoders separate from the RNN/CNN distinction, thus we designed a predict-all-words CNN decoder and RNN decoder. The predict-all-words CNN decoder is described in Section SECREF2 , which is a stack of three convolutional layers, and all words are predicted once at the output layer of the decoder. The predict-all-words RNN decoder is built based on our CNN decoder. To keep the number of parameters of the two predict-all-words decoder roughly the same, we replaced the last two convolutional layers with a bidirectional GRU.","The results are also presented in Table TABREF10 (3rd and 4th subparts). The performance of the predict-all-words RNN decoder does not significantly differ from that of any one of the autoregressive RNN decoders, and the same situation can be also observed in CNN decoders.","These two findings indeed support our choice of using a predict-all-words CNN as the decoder, as it brings the model high training efficiency while maintaining strong transferability."]},{"idx":5,"heading":"Mean+Max Pooling","paragraphs":["Since the encoder is a bi-directional RNN in our model, we have multiple ways to select/compute on the generated hidden states to produce a sentence representation. Instead of using the last hidden state as the sentence representation as done in skip-thoughts BIBREF5 and SDAE BIBREF11 , we followed the idea proposed in BIBREF12 . They built a model for supervised training on the SNLI dataset BIBREF13 that concatenates the outputs from a global mean pooling over time and a global max pooling over time to serve as the sentence representation, and showed a performance boost on the SNLI task. BIBREF14 found that the model with global max pooling function provides stronger transferability than the model with a global mean pooling function does.","In our proposed RNN-CNN model, we empirically show that the mean+max pooling provides stronger transferability than the max pooling alone does, and the results are presented in the last two sections of Table TABREF10 . The concatenation of a mean-pooling and a max pooling function is actually a parameter-free composition function, and the computation load is negligible compared to all the heavy matrix multiplications in the model. Also, the non-linearity of the max pooling function augments the mean pooling function for constructing a representation that captures a more complex composition of the syntactic information."]},{"idx":6,"heading":"Tying Word Embeddings and Word Prediction Layer","paragraphs":["We choose to share the parameters in the word embedding layer of the RNN encoder and the word prediction layer of the CNN decoder. Tying was shown in both BIBREF15 and BIBREF16 , and it generally helped to learn a better language model. In our model, tying also drastically reduces the number of parameters, which could potentially prevent overfitting.","Furthermore, we initialise the word embeddings with pretrained word vectors, such as word2vec BIBREF3 and GloVe BIBREF4 , since it has been shown that these pretrained word vectors can serve as a good initialisation for deep learning models, and more likely lead to better results than a random initialisation."]},{"idx":7,"heading":"Study of the Hyperparameters in Our Model Design","paragraphs":["We studied hyperparameters in our model design based on three out of 10 downstream tasks, which are SICK-R, SICK-E BIBREF17 , and STS14 BIBREF18 . The first model we created, which is reported in Section SECREF2 , is a decent design, and the following variations didn't give us much performance change except improvements brought by increasing the dimensionality of the encoder. However, we think it is worth mentioning the effect of hyperparameters in our model design. We present the Table TABREF21 in the supplementary material and we summarise it as follows:","1. Decoding the next sentence performed similarly to decoding the subsequent contiguous words.","2. Decoding the subsequent 30 words, which was adopted from the skip-thought training code, gave reasonably good performance. More words for decoding didn't give us a significant performance gain, and took longer to train.","3. Adding more layers into the decoder and enlarging the dimension of the convolutional layers indeed sightly improved the performance on the three downstream tasks, but as training efficiency is one of our main concerns, it wasn't worth sacrificing training efficiency for the minor performance gain.","4. Increasing the dimensionality of the RNN encoder improved the model performance, and the additional training time required was less than needed for increasing the complexity in the CNN decoder. We report results from both smallest and largest models in Table TABREF16 ."]},{"idx":8,"heading":"Experiment Settings","paragraphs":["The vocabulary for unsupervised training contains the 20k most frequent words in BookCorpus. In order to generalise the model trained with a relatively small, fixed vocabulary to the much larger set of all possible English words, we followed the vocabulary expansion method proposed in BIBREF5 , which learns a linear mapping from the pretrained word vectors to the learnt RNN word vectors. Thus, the model benefits from the generalisation ability of the pretrained word embeddings.","The downstream tasks for evaluation include semantic relatedness (SICK, BIBREF17 ), paraphrase detection (MSRP, BIBREF19 ), question-type classification (TREC, BIBREF20 ), and five benchmark sentiment and subjective datasets, which include movie review sentiment (MR, BIBREF21 , SST, BIBREF22 ), customer product reviews (CR, BIBREF23 ), subjectivity/objectivity classification (SUBJ, BIBREF24 ), opinion polarity (MPQA, BIBREF25 ), semantic textual similarity (STS14, BIBREF18 ), and SNLI BIBREF13 . After unsupervised training, the encoder is fixed, and applied as a representation extractor on the 10 tasks.","To compare the effect of different corpora, we also trained two models on Amazon Book Review dataset (without ratings) which is the largest subset of the Amazon Review dataset BIBREF26 with 142 million sentences, about twice as large as BookCorpus.","Both training and evaluation of our models were conducted in PyTorch, and we used SentEval provided by BIBREF14 to evaluate the transferability of our models. All the models were trained for the same number of iterations with the same batch size, and the performance was measured at the end of training for each of the models."]},{"idx":9,"heading":"Related work and Comparison","paragraphs":["Table TABREF16 presents the results on 9 evaluation tasks of our proposed RNN-CNN models, and related work. The “small RNN-CNN” refers to the model with the dimension of representation as 1200, and the “large RNN-CNN” refers to that as 4800. The results of our “large RNN-CNN” model on SNLI is presented in Table TABREF19 .","Our work was inspired by analysing the skip-thoughts model BIBREF5 . The skip-thoughts model successfully applied this form of learning from the context information into unsupervised representation learning for sentences, and then, BIBREF29 augmented the LSTM with proposed layer-normalisation (Skip-thought+LN), which improved the skip-thoughts model generally on downstream tasks. In contrast, BIBREF11 proposed the FastSent model which only learns source and target word embeddings and is an adaptation of Skip-gram BIBREF3 to sentence-level learning without word order information. BIBREF9 applied a CNN as the encoder, but still applied LSTMs for decoding the adjacent sentences, which is called CNN-LSTM.","Our RNN-CNN model falls in the same category as it is an encoder-decoder model. Instead of decoding the surrounding two sentences as in skip-thoughts, FastSent and the compositional CNN-LSTM, our model only decodes the subsequent sequence with a fixed length. Compared with the hierarchical CNN-LSTM, our model showed that, with a proper model design, the context information from the subsequent words is sufficient for learning sentence representations. Particularly, our proposed small RNN-CNN model runs roughly three times faster than our implemented skip-thoughts model on the same GPU machine during training.","Proposed by BIBREF30 , BYTE m-LSTM model uses a multiplicative LSTM unit BIBREF31 to learn a language model. This model is simple, providing next-byte prediction, but achieves good results likely due to the extremely large training corpus (Amazon Review data, BIBREF26 ) that is also highly related to many of the sentiment analysis downstream tasks (domain matching).","We experimented with the Amazon Book review dataset, the largest subset of the Amazon Review. This subset is significantly smaller than the full Amazon Review dataset but twice as large as BookCorpus. Our RNN-CNN model trained on the Amazon Book review dataset resulted in performance improvement on all single-sentence classification tasks relative to that achieved with training under BookCorpus.","Unordered sentences are also useful for learning representations of sentences. ParagraphVec BIBREF32 learns a fixed-dimension vector for each sentence by predicting the words within the given sentence. However, after training, the representation for a new sentence is hard to derive, since it requires optimising the sentence representation towards an objective. SDAE BIBREF11 learns the sentence representations with a denoising auto-encoder model. Our proposed RNN-CNN model trains faster than SDAE does, and also because we utilised the sentence-level continuity as a supervision which SDAE doesn't, our model largely performs better than SDAE.","Another transfer approach is to learn a supervised discriminative classifier by distinguishing whether the sentence pair or triple comes from the same context. BIBREF33 proposed a model that learns to classify whether the input sentence triplet contains three contiguous sentences. DiscSent BIBREF34 and DisSent BIBREF35 both utilise the annotated explicit discourse relations, which is also good for learning sentence representations. It is a very promising research direction since the proposed models are generally computational efficient and have clear intuition, yet more investigations need to be done to augment the performance.","Supervised training for transfer learning is also promising when a large amount of human-annotated data is accessible. BIBREF14 proposed the InferSent model, which applies a bi-directional LSTM as the sentence encoder with multiple fully-connected layers to classify whether the hypothesis sentence entails the premise sentence in SNLI BIBREF13 , and MultiNLI BIBREF36 . The trained model demonstrates a very impressive transferability on downstream tasks, including both supervised and unsupervised. Our RNN-CNN model trained on Amazon Book Review data in an unsupervised way has better results on supervised tasks than InferSent but slightly inferior results on semantic relatedness tasks. We argue that labelling a large amount of training data is time-consuming and costly, while unsupervised learning provides great performance at a fraction of the cost. It could potentially be leveraged to initialise or more generally augment the costly human labelling, and make the overall system less costly and more efficient."]},{"idx":10,"heading":"Discussion","paragraphs":["In BIBREF11 , internal consistency is measured on five single sentence classification tasks (MR, CR, SUBJ, MPQA, TREC), MSRP and STS-14, and was found to be only above the “acceptable” threshold. They empirically showed that models that worked well on supervised evaluation tasks generally didn't perform well on unsupervised ones. This implies that we should consider supervised and unsupervised evaluations separately, since each group has higher internal consistency.","As presented in Table TABREF16 , the encoders that only sum over pretrained word vectors perform better overall than those with RNNs on unsupervised evaluation tasks, including STS14. In recent proposed log-bilinear models, such as FastSent BIBREF11 and SiameseBOW BIBREF37 , the sentence representation is composed by summing over all word representations, and the only tunable parameters in the models are word vectors. These resulting models perform very well on unsupervised tasks. By augmenting the pretrained word vectors with a weighted averaging process, and removing the top few principal components, which mainly encode frequently-used words, as proposed in BIBREF38 and BIBREF39 , the performance on the unsupervised evaluation tasks gets even better. Prior work suggests that incorporating word-level information helps the model to perform better on cosine distance based semantic textual similarity tasks.","Our model predicts all words in the target sequence at once, without an autoregressive process, and ties the word embedding layer in the encoder with the prediction layer in the decoder, which explicitly uses the word vectors in the target sequence as the supervision in training. Thus, our model incorporates the word-level information by using word vectors as the targets, and it improves the model performance on STS14 compared to other RNN-based encoders.","BIBREF38 conducted an experiment to show that the word order information is crucial in getting better results on supervised tasks. In our model, the encoder is still an RNN, which explicitly utilises the word order information. We believe that the combination of encoding a sentence with its word order information and decoding all words in a sentence independently inherently leverages the benefits from both log-linear models and RNN-based models."]},{"idx":11,"heading":"Conclusion","paragraphs":["Inspired by learning to exploit the contextual information present in adjacent sentences, we proposed an asymmetric encoder-decoder model with a suite of techniques for improving context-based unsupervised sentence representation learning. Since we believe that a simple model will be faster in training and easier to analyse, we opt to use simple techniques in our proposed model, including 1) an RNN as the encoder, and a predict-all-words CNN as the decoder, 2) learning by inferring subsequent contiguous words, 3) mean+max pooling, and 4) tying word vectors with word prediction. With thorough discussion and extensive evaluation, we justify our decision making for each component in our RNN-CNN model. In terms of the performance and the efficiency of training, we justify that our model is a fast and simple algorithm for learning generic sentence representations from unlabelled corpora. Further research will focus on how to maximise the utility of the context information, and how to design simple architectures to best make use of it."]},{"idx":12,"heading":"Supplemental Material","paragraphs":["Table TABREF21 presents the effect of hyperparameters."]},{"idx":13,"heading":"Decoding Sentences vs. Decoding Sequences","paragraphs":["Given that the encoder takes a sentence as input, decoding the next sentence versus decoding the next fixed length window of contiguous words is conceptually different. This is because decoding the subsequent fixed-length sequence might not reach or might go beyond the boundary of the next sentence. Since the CNN decoder in our model takes a fixed-length sequence as the target, when it comes to decoding sentences, we would need to zero-pad or chop the sentences into a fixed length. As the transferability of the models trained in both cases perform similarly on the evaluation tasks (see rows 1 and 2 in Table TABREF21 ), we focus on the simpler predict-all-words CNN decoder that learns to reconstruct the next window of contiguous words."]},{"idx":14,"heading":"Length of the Target Sequence TT","paragraphs":["We varied the length of target sequences in three cases, which are 10, 30 and 50, and measured the performance of three models on all tasks. As stated in rows 1, 3, and 4 in Table TABREF21 , decoding short target sequences results in a slightly lower Pearson score on SICK, and decoding longer target sequences lead to a longer training time. In our understanding, decoding longer target sequences leads to a harder optimisation task, and decoding shorter ones leads to a problem that not enough context information is included for every input sentence. A proper length of target sequences is able to balance these two issues. The following experiments set subsequent 30 contiguous words as the target sequence."]},{"idx":15,"heading":"RNN Encoder vs. CNN Encoder","paragraphs":["The CNN encoder we built followed the idea of AdaSent BIBREF41 , and we adopted the architecture proposed in BIBREF14 . The CNN encoder has four layers of convolution, each followed by a non-linear activation function. At every layer, a vector is calculated by a global max-pooling function over time, and four vectors from four layers are concatenated to serve as the sentence representation. We tweaked the CNN encoder, including different kernel size and activation function, and we report the best results of CNN-CNN model at row 6 in Table TABREF21 .","Even searching over many hyperparameters and selecting the best performance on the evaluation tasks (overfitting), the CNN-CNN model performs poorly on the evaluation tasks, although the model trains much faster than any other models with RNNs (which were not similarly searched). The RNN and CNN are both non-linear systems, and they both are capable of learning complex composition functions on words in a sentence. We hypothesised that the explicit usage of the word order information will augment the transferability of the encoder, and constrain the search space of the parameters in the encoder. The results support our hypothesis.","The future predictor in BIBREF9 also applies a CNN as the encoder, but the decoder is still an RNN, listed at row 11 in Table TABREF21 . Compared to our designed CNN-CNN model, their CNN-LSTM model contains more parameters than our model does, but they have similar performance on the evaluation tasks, which is also worse than our RNN-CNN model."]},{"idx":16,"heading":"Dimensionality","paragraphs":["Clearly, we can tell from the comparison between rows 1, 9 and 12 in Table TABREF21 , increasing the dimensionality of the RNN encoder leads to better transferability of the model.","Compared with RNN-RNN model, even with double-sized encoder, the model with CNN decoder still runs faster than that with RNN decoder, and it slightly outperforms the model with RNN decoder on the evaluation tasks.","At the same dimensionality of representation with Skip-thought and Skip-thought+LN, our proposed RNN-CNN model performs better on all tasks but TREC, on which our model gets similar results as other models do.","Compared with the model with larger-size CNN decoder, apparently, we can see that larger encoder size helps more than larger decoder size does (rows 7,8, and 9 in Table TABREF21 ).","In other words, an encoder with larger size will result in a representation with higher dimensionality, and generally, it will augment the expressiveness of the vector representation, and the transferability of the model."]},{"idx":17,"heading":"Experimental Details","paragraphs":["Our small RNN-CNN model has a bi-directional GRU as the encoder, with 300 dimension each direction, and the large one has 1200 dimension GRU in each direction. The batch size we used for training our model is 512, and the sequence length for both encoding and decoding are 30. The initial learning rate is INLINEFORM0 , and the Adam optimiser BIBREF40 is applied to tune the parameters in our model."]},{"idx":18,"heading":"Results including supervised task-dependent models","paragraphs":["Table TABREF26 contains all supervised task-dependent models for comparison."]},{"idx":19,"heading":"Figures and Tables","paragraphs":["Figure 1: Our proposed model is composed of an RNN encoder, and a CNN decoder. During training, a batch of sentences are sent to the model, and the RNN encoder computes a vector representation for each of sentences; then the CNN decoder needs to reconstruct the paired target sequence, which contains 30 contiguous words right after the input sentence, given the vector representation. 300 is the dimension of word vectors. 2dh is the dimension of the sentence representation which varies with the RNN encoder size. (Better view in colour.)","Table 1: The models here all have a bi-directional GRU as the encoder (dimensionality 300 in each direction). The default way of producing the representation is a concatenation of outputs from a global mean-pooling and a global max-pooling, while “·-MaxOnly” refers to the model with only global maxpooling. Bold numbers are the best results among all presented models. We found that 1) inputting correct words to an autoregressive decoder is not necessary; 2) predict-all-words decoders work roughly the same as autoregressive decoders; 3) mean+max pooling provides stronger transferability than the max-pooling alone does. The table supports our choice of the predict-all-words CNN decoder and the way of producing vector representations from the bi-directional RNN encoder.","Table 2: Related Work and Comparison. As presented, our designed asymmetric RNN-CNN model has strong transferability, and is overall better than existing unsupervised models in terms of fast training speed and good performance on evaluation tasks. “†”s refer to our models, and “small/large” refers to the dimension of representation as 1200/4800. Bold numbers are the best ones among the models with same training and transferring setting, and underlined numbers are best results among all transfer learning models. The training time of each model was collected from the paper that proposed it.","Table 3: We implemented the same classifier as mentioned in Vendrov et al. (2015) on top of the features computed by our model. Our proposed RNN-CNN model gets similar result on SNLI as skip-thoughts, but with much less training time.","Table 4: Architecture Comparison. As shown in the table, our designed asymmetric RNN-CNN model (row 1,9, and 12) works better than other asymmetric models (CNN-LSTM, row 11), and models with symmetric structure (RNN-RNN, row 5 and 10). In addition, with larger encoder size, our model demonstrates stronger transferability. The default setting for our CNN decoder is that it learns to reconstruct 30 words right next to every input sentence. “CNN(10)” represents a CNN decoder with the length of outputs as 10, and “CNN(50)” represents it with the length of outputs as 50. “†” indicates that the CNN decoder learns to reconstruct next sentence. “‡” indicates the results reported in Gan et al. as future predictor. The CNN encoder in our experiment, noted as “§”, was based on AdaSent in Zhao et al. and Conneau et al.. Bold numbers are best results among models at same dimension, and underlined numbers are best results among all models. For STS14, the performance measures are Pearson’s and Spearman’s score. For MSRP, the performance measures are accuracy and F1 score.","Table 5: Related Work and Comparison. As presented, our designed asymmetric RNN-CNN model has strong transferability, and is overall better than existing unsupervised models in terms of fast training speed and good performance on evaluation tasks. “†”s refer to our models, and “small/large” refers to the dimension of representation as 1200/4800. Bold numbers are the best ones among the models with same training and transferring setting, and underlined numbers are best results among all transfer learning models. The training time of each model was collected from the paper that proposed it."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1710.11027","title":"Named Entity Recognition in Twitter using Images and Text","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Named Entity Recognition (NER) is an important subtask of information extraction that seeks to locate and recognise named entities. Despite recent achievements, we still face limitations with correctly detecting and classifying entities, prominently in short and noisy text, such as Twitter. An important negative aspect in most of NER approaches is the high dependency on hand-crafted features and domain-specific knowledge, necessary to achieve state-of-the-art results. Thus, devising models to deal with such linguistically complex contexts is still challenging. In this paper, we propose a novel multi-level architecture that does not rely on any specific linguistic resource or encoded rule. Unlike traditional approaches, we use features extracted from images and text to classify named entities. Experimental tests against state-of-the-art NER for Twitter on the Ritter dataset present competitive results (0.59 F-measure), indicating that this approach may lead towards better NER models."]},{"idx":1,"heading":"Introduction","paragraphs":["Named Entity Recognition (NER) is an important step in most of the natural language processing (NLP) pipelines. It is designed to robustly handle proper names, which is essential for many applications. Although a seemingly simple task, it faces a number of challenges in noisy datasets and it is still considered an emerging research area BIBREF0 , BIBREF1 . Despite recent efforts, we still face limitations at identifying entities and (consequently) correctly classifying them. Current state-of-the-art NER systems typically have about 85-90% accuracy on news text - such as articles (CoNLL03 shared task data set) - but they still perform poorly (about 30-50% accuracy) on short texts, which do not have implicit linguistic formalism (e.g. punctuation, spelling, spacing, formatting, unorthodox capitalisation, emoticons, abbreviations and hashtags) BIBREF2 , BIBREF3 , BIBREF4 , BIBREF1 . Furthermore, the lack of external knowledge resources is an important gap in the process regardless of writing style BIBREF5 . To face these problems, research has been focusing on microblog-specific information extraction techniques BIBREF2 , BIBREF6 .","In this paper, we propose a joint clustering architecture that aims at minimizing the current gap between world knowledge and knowledge available in open domain knowledge bases (e.g., Freebase) for NER systems, by extracting features from unstructured data sources. To this aim, we use images and text from the web as input data. Thus, instead of relying on encoded information and manually annotated resources (the major limitation in NER architectures) we focus on a multi-level approach for discovering named entities, combining text and image features with a final classifier based on a decision tree model. We follow an intuitive and simple idea: some types of images are more related to people (e.g. faces) whereas some others are more related to organisations (e.g. logos), for instance. This principle is applied similarly to the text retrieved from websites: keywords for search engines representing names and surnames of people will often return similarly related texts, for instance. Thus, we derive some indicators (detailed in sec:finalclassifier which are then used as input features in a final classifier.","To the best of our knowledge, this is the first report of a NER architecture which aims to provide a priori information based on clusters of images and text features."]},{"idx":2,"heading":"Related Work","paragraphs":["Over the past few years, the problem of recognizing named entities in natural language texts has been addressed by several approaches and frameworks BIBREF7 , BIBREF8 . Existing approaches basically adopt look-up strategies and use standard local features, such as part-of-speech tags, previous and next words, substrings, shapes and regex expressions, for instance. The main drawback is the performance of those models with noisy data, such as Tweets. A major reason is that they rely heavily on hand-crafted features and domain-specific knowledge. In terms of architecture, NER algorithms may also be designed based on generative (e.g., Naive Bayes) or discriminative (e.g., MaxEnt) models. Furthermore, sequence models (HMMs, CMM, MEMM and CRF) are a natural choice to design such systems. A more recent study proposed by Lample et al., 2016 BIBREF9 used neural architectures to solve this problem. Similarly in terms of architecture, Al-Rfou et al., 2015 BIBREF10 had also proposed a model (without dependency) that learns distributed word representations (word embeddings) which encode semantic and syntactic features of words in each language. Chiu and Nichols, 2015 BIBREF11 proposed a neural network architecture that automatically detects word and character-level features using a hybrid bidirectional LSTM and CNN. Thus, these models work without resorting to any language-specific knowledge or resources such as gazetteers. They, however, focused on newswire to improve current state-of-the-art systems and not on the microblogs context, in which they are naturally harder to outperform due to the aforementioned issues. According to Derczynski et al., 2015 BIBREF1 some approaches have been proposed for Twitter, but they are mostly still in development and often not freely available."]},{"idx":3,"heading":"Conceptual Architecture","paragraphs":["The main insight underlying this work is that we can produce a NER model which performs similarly to state-of-the-art approaches but without relying on any specific resource or encoded rule. To this aim, we propose a multi-level architecture which intends to produce biased indicators to a certain class (LOC, PER or ORG). These outcomes are then used as input features for our final classifier. We perform clustering on images and texts associated to a given term INLINEFORM0 existing in complete or partial sentences INLINEFORM1 (e.g., “new york” or “einstein”), leveraging the global context obtained from the Web providing valuable insights apart from standard local features and hand-coded information. fig:architecture gives an overview of the proposed architecture.","In the first step (A), we simply apply POS Tagging and Shallow Parsing to filter out tokens except for those tagged as INLINEFORM0 or INLINEFORM1 and their INLINEFORM2 (local context). Afterwards, we use the search engine (B) to query and cache (C) the top INLINEFORM3 texts and images associated to each term INLINEFORM4 , where INLINEFORM5 is the set resulting of the pre-processing step (A) for each (partial or complete) sentence INLINEFORM6 . This resulting data (composed of excerpts of texts and images from web pages) is used to predict a possible class for a given term. These outcomes are then used in the first two levels (D.1 and D.2) of our approach: the Computer Vision and Text Analytics components, respectively, which we introduce as follows:","Computer Vision (CV): Detecting Objects: Function Description (D.1): given a set of images INLINEFORM0 , the basic idea behind this component is to detect a specific object (denoted by a class INLINEFORM1 ) in each image. Thus, we query the web for a given term INLINEFORM2 and then extract the features from each image and try to detect a specific object (e.g., logos for ORG) for the top INLINEFORM3 images retrieved as source candidates. The mapping between objects and NER classes is detailed in tab:tbempirical.","Training (D.1): we used SIFT (Scale Invariant Feature Transform) features BIBREF12 for extracting image descriptors and BoF (Bag of Features) BIBREF13 , BIBREF14 for clustering the histograms of extracted features. The clustering is possible by constructing a large vocabulary of many visual words and representing each image as a histogram of the frequency words that are in the image. We use k-means BIBREF15 to cluster the set of descriptors to INLINEFORM0 clusters. The resulting clusters are compact and separated by similar characteristics. An empirical analysis shows that some image groups are often related to certain named entities (NE) classes when using search engines, as described in tab:tbempirical. For training purposes, we used the Scene 13 dataset BIBREF16 to train our classifiers for location (LOC), “faces” from Caltech 101 Object Categories BIBREF17 to train our person (PER) and logos from METU dataset BIBREF18 for organisation ORG object detection. These datasets produces the training data for our set of supervised classifiers (1 for ORG, 1 for PER and 10 for LOC). We trained our classifiers using Support Vector Machines BIBREF19 once they generalize reasonably enough for the task.","Text Analytics (TA): Text Classification - Function Description (D.2): analogously to (D.1), we perform clustering to group texts together that are “distributively” similar. Thus, for each retrieved web page (title and excerpt of its content), we perform the classification based on the main NER classes. We extracted features using a classical sparse vectorizer (Term frequency-Inverse document frequency - TF-IDF. In experiments, we did not find a significant performance gain using HashingVectorizer) - Training (D.2): with this objective in mind, we trained classifiers that rely on a bag-of-words technique. We collected data using DBpedia instances to create our training dataset ( INLINEFORM0 ) and annotated each instance with the respective MUC classes, i.e. PER, ORG and LOC. Listing shows an example of a query to obtain documents of organizations (ORG class). Thereafter, we used this annotated dataset to train our model.","where INLINEFORM0 and INLINEFORM1 represent the INLINEFORM2 and INLINEFORM3 position of INLINEFORM4 and INLINEFORM5 , respectively. INLINEFORM6 represents the n-gram of POS tag. INLINEFORM7 and INLINEFORM8 ( INLINEFORM9 ) represent the total objects found by a classifier INLINEFORM10 for a given class INLINEFORM11 ( INLINEFORM12 ) (where N is the total of retrieved images INLINEFORM15 ). INLINEFORM16 and INLINEFORM17 represent the distance between the two higher predictions ( INLINEFORM18 ), i.e. INLINEFORM19 . Finally, INLINEFORM20 represents the sum of all predictions made by all INLINEFORM21 classifiers INLINEFORM22 ( INLINEFORM23 ). - Training (E): the outcomes of D.1 and D.2 ( INLINEFORM26 ) are used as input features to our final classifier. We implemented a simple Decision Tree (non-parametric supervised learning method) algorithm for learning simple decision rules inferred from the data features (since it does not require any assumptions of linearity in the data and also works well with outliers, which are expected to be found more often in a noisy environment, such as the Web of Documents)."]},{"idx":4,"heading":"Experiments","paragraphs":["In order to check the overall performance of the proposed technique, we ran our algorithm without any further rule or apriori knowledge using a gold standard for NER in microblogs (Ritter dataset BIBREF2 ), achieving INLINEFORM0 F1. tab:performance details the performance measures per class. tab:relatedwork presents current state-of-the-art results for the same dataset. The best model achieves INLINEFORM1 F1-measure, but uses encoded rules. Models which are not rule-based, achieve INLINEFORM2 and INLINEFORM3 . We argue that in combination with existing techniques (such as linguistic patterns), we can potentially achieve even better results.","As an example, the sentence “paris hilton was once the toast of the town” can show the potential of the proposed approach. The token “paris” with a LOC bias (0.6) and “hilton” (global brand of hotels and resorts) with indicators leading to LOC (0.7) or ORG (0.1, less likely though). Furthermore, “town” being correctly biased to LOC (0.7). The algorithm also suggests that the compound “paris hilton” is more likely to be a PER instead (0.7) and updates (correctly) the previous predictions. As a downside in this example, the algorithm misclassified “toast” as LOC. However, in this same example, Stanford NER annotates (mistakenly) only “paris” as LOC. It is worth noting also the ability of the algorithm to take advantage of search engine capabilities. When searching for “miCRs0ft”, the returned values strongly indicate a bias for ORG, as expected ( INLINEFORM0 = 0.2, INLINEFORM1 = 0.8, INLINEFORM2 = 0.0, INLINEFORM3 = 6, INLINEFORM4 = -56, INLINEFORM5 = 0.0, INLINEFORM6 = 0.5, INLINEFORM7 = 0.0, INLINEFORM8 = 5). More local organisations are also recognized correctly, such as “kaufland” (German supermarket), which returns the following metadata: INLINEFORM9 = 0.2, INLINEFORM10 = 0.4, INLINEFORM11 = 0.0, INLINEFORM12 = 2, INLINEFORM13 = -50, INLINEFORM14 = 0.1, INLINEFORM15 = 0.4, INLINEFORM16 = 0.0, INLINEFORM17 = 3."]},{"idx":5,"heading":"Discussion","paragraphs":["A disadvantage when using web search engines is that they are not open and free. This can be circumvented by indexing and searching on other large sources of information, such as Common Crawl and Flickr. However, maintaining a large source of images would be an issue, e.g. the Flickr dataset may not be comprehensive enough (i.e. tokens may not return results). This will be a subject of future work. Besides, an important step in the pre-processing is the classification of part-of-speech tags. In the Ritter dataset our current error propagation is 0.09 (107 tokens which should be classified as NOUN) using NLTK 3.0. Despite good performance (91% accuracy), we plan to benchmark this component. In terms of processing time, the bottleneck of the current implementation is the time required to extract features from images, as expected. Currently we achieve a performance of 3~5 seconds per sentence and plan to also optimize this component. The major advantages of this approach are: 1) the fact that there are no hand-crafted rules encoded; 2) the ability to handle misspelled words (because the search engine alleviates that and returns relevant or related information) and incomplete sentences; 3) the generic design of its components, allowing multilingual processing with little effort (the only dependency is the POS tagger) and straightforward extension to support more NER classes (requiring a corpus of images and text associated to each desired NER class, which can be obtained from a Knowledge Base, such as DBpedia, and an image dataset, such as METU dataset). While initial results in a gold standard dataset showed the potential of the approach, we also plan to integrate these outcomes into a Sequence Labeling (SL) system, including neural architectures such as LSTM, which are more suitable for such tasks as NER or POS. We argue that this can potentially reduce the existing (significant) gap in NER performance on microblogs."]},{"idx":6,"heading":"Conclusions","paragraphs":["In this paper we presented a novel architecture for NER that expands the feature set space based on feature clustering of images and texts, focused on microblogs. Due to their terse nature, such noisy data often lack enough context, which poses a challenge to the correct identification of named entities. To address this issue we have presented and evaluated a novel approach using the Ritter dataset, showing consistent results over state-of-the-art models without using any external resource or encoded rule, achieving an average of 0.59 F1. The results slightly outperformed state-of-the-art models which do not rely on encoded rules (0.49 and 0.54 F1), suggesting the viability of using the produced metadata to also boost existing NER approaches. A further important contribution is the ability to handle single tokens and misspelled words successfully, which is of utmost importance in order to better understand short texts. Finally, the architecture of the approach and its indicators introduce potential to transparently support multilingual data, which is the subject of ongoing investigation."]},{"idx":7,"heading":"Acknowledgments","paragraphs":["This research was supported in part by an EU H2020 grant provided for the HOBBIT project (GA no. 688227) and CAPES Foundation (BEX 10179135)."]},{"idx":8,"heading":"Figures and Tables","paragraphs":["Fig. 1: Overview of the approach: combining computer vision and machine learning in a generic NER architecture","Table 1: NER classes and respective objects to be detected in a given image. For LOC we trained more models due to the diversity of the object candidates.","Table 2: Performancemeasure for our approach in Ritter dataset: 4-fold cross validation","Table 3: Performance measures (PER, ORG and LOC classes) of state-of-the-art NER for short texts (Ritter dataset). Approaches which do not rely on hand-crafted rules and Gazetteers are highlighted in gray. Etter et al., 2013 trained using 10 classes."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1711.04964","title":"Dynamic Fusion Networks for Machine Reading Comprehension","sections":[{"idx":0,"heading":"Abstract","paragraphs":["This paper presents a novel neural model - Dynamic Fusion Network (DFN), for machine reading comprehension (MRC). DFNs differ from most state-of-the-art models in their use of a dynamic multi-strategy attention process, in which passages, questions and answer candidates are jointly fused into attention vectors, along with a dynamic multi-step reasoning module for generating answers. With the use of reinforcement learning, for each input sample that consists of a question, a passage and a list of candidate answers, an instance of DFN with a sample-specific network architecture can be dynamically constructed by determining what attention strategy to apply and how many reasoning steps to take. Experiments show that DFNs achieve the best result reported on RACE, a challenging MRC dataset that contains real human reading questions in a wide variety of types. A detailed empirical analysis also demonstrates that DFNs can produce attention vectors that summarize information from questions, passages and answer candidates more effectively than other popular MRC models."]},{"idx":1,"heading":"Introduction","paragraphs":["The goal of Machine Reading Comprehension (MRC) is to have machines read a text passage and then generate an answer (or select an answer from a list of given candidates) for any question about the passage. There has been a growing interest in the research community in exploring neural MRC models in an end-to-end fashion, thanks to the availability of large-scale datasets, such as CNN/DM BIBREF0 and SQuAD BIBREF1 .","Despite the variation in model structures, most state-of-the-art models perform reading comprehension in two stages. First, the symbolic representations of passages and questions are mapped into vectors in a neural space. This is commonly achieved via embedding and attention BIBREF2 , BIBREF3 or fusion BIBREF4 . Then, reasoning is performed on the vectors to generate the right answer.","Ideally, the best attention and reasoning strategies should adapt organically in order to answer different questions. However, most MRC models use a static attention and reasoning strategy indiscriminately, regardless of various question types. One hypothesis is because these models are optimized on those datasets whose passages and questions are domain-specific (or of a single type). For example, in CNN/DM, all the passages are news articles, and the answer to each question is an entity in the passage. In SQuAD, the passages came from Wikipedia articles and the answer to each question is a text span in the article. Such a fixed-strategy MRC model does not adapt well to other datasets. For example, the exact-match score of BiDAF BIBREF2 , one of the best models on SQuAD, drops from 81.5 to 55.8 when applied to TriviaQA BIBREF5 , whereas human performance is 82.3 and 79.7 on SQuAD and TriviaQA, respectively.","In real-world MRC tasks, we must deal with questions and passages of different types and complexities, which calls for models that can dynamically determine what attention and reasoning strategy to use for any input question-passage pair on the fly. In a recent paper, BIBREF6 proposed dynamic multi-step reasoning, where the number of reasoning steps is determined spontaneously (using reinforcement learning) based on the complexity of the input question and passage. With a similar intuition, in this paper we propose a novel MRC model which is dynamic not only on the number of reasoning steps it takes, but also on the way it performs attention. To the best of our knowledge, this is the first MRC model with this dual-dynamic capability.","The proposed model is called a Dynamic Fusion Network (DFN). In this paper, we describe the version of DFN developed on the RACE dataset BIBREF7 . In RACE, a list of candidate answers is provided for each passage-question pair. So DFN for RACE is a scoring model - the answer candidate with the highest score will be selected as the final answer.","Like other MRC models, DFNs also perform machine reading in two stages: attention and reasoning. DFN is unique in its use of a dynamic multi-strategy attention process in the attention stage. Here “attention” refers to the process that texts from different sources (passage, question, answers) are combined in the network. In literature, a fixed attention mechanism is usually employed in MRC models. In DFN, the attention strategy is not static; instead, the actual strategy for drawing attention among the three text sources are chosen on the fly for each sample. This lends flexibility to adapt to various question types that require different comprehension skills. The output of the attention stage is then fed into the reasoning module to generate the answer score. The reasoning module in DFN uses dynamic multi-step reasoning, where the number of steps depends on the complexity of the question-passage pair and varies from sample to sample.","Inspired by ReasoNet BIBREF6 and dynamic neural module networks BIBREF8 , we use deep reinforcement learning methods BIBREF9 , BIBREF10 to dynamically choose the optimal attention strategy and the optimal number of reasoning steps for a given sample. We use RL in favor of other simpler methods (like cascading, pooling or weighted averaging) mainly because we intend to learn a policy that constructs an instance of DFN of a sample-specific structure. Given an input sample consisting of a question, a passage and a list of candidate answers in RACE, an instance of DFN can be constructed via RL step by step on the fly. Such a policy is particularly appealing as it also provides insights on how the model performs on different types of questions. At each decision step, the policy maps its “state”, which represents an input sample, and DFN's partial knowledge of the right answer, to the action of assembling proper attention and reasoning modules for DFN.","Experiments conducted on the RACE dataset show that DFN significantly outperforms previous state-of-the-art MRC models and has achieved the best result reported on RACE. A thorough empirical analysis also demonstrates that DFN is highly effective in understanding passages of a wide variety of styles and answering questions of different complexities."]},{"idx":2,"heading":"Related Work","paragraphs":["The recent progress in MRC is largely due to the introduction of large-scale datasets. CNN/Daily Mail BIBREF0 and SQuAD BIBREF1 are two popular and widely-used datasets. More recently, other datasets using different collection methodologies have been introduced, such as MS MARCO BIBREF11 , NewsQA BIBREF12 and RACE BIBREF7 . For example, MS MARCO collects data from search engine queries and user-clicked results, thus contains a broader topic coverage than Wikipedia and news articles in SQuAD and CNN/Daily Mail. Among the large number of MRC datasets, RACE focuses primarily on developing MRC models with near-human capability. Questions in RACE come from real English exams designed specifically to test human comprehension. This makes RACE an appealing testbed for DFN; we will further illustrate this in Section \" RACE - The MRC Task\" .","The word “fusion” for MRC was first used by FusionNet BIBREF4 to refer to the process of updating the representation of passage (or question) using information from the question (or passage) representation. A typical way of fusion is through attention: for example, BiDAF BIBREF2 uses a bi-directional attention, where the representation of passage (or question) vectors are re-weighted by their similarities to the question (or passage) vectors. We will use “fusion” and “attention” interchangeably throughout the paper.","In the attention process of state-of-the-art MRC models, a pre-defined attention strategy is often applied. BIBREF13 proposed a Bi-directional Multi-Perspective Matching (BiMPM) model, which uses attention with multiple perspectives characterized by different parameters. Although multi-perspective attention might be able to handle different types of questions, all perspectives are used for all the questions. DFN is inspired by BiMPM, but our dynamic attention process is more adaptive to variations of questions.","Another important component of MRC systems is the answer module, which performs reasoning to generate the final prediction. The reasoning methods in existing literature can be grouped into three categories: 1) single-step reasoning BIBREF14 , BIBREF15 , BIBREF2 , BIBREF16 ; 2) multi-step reasoning with a fixed number of steps BIBREF17 , BIBREF18 , BIBREF19 ; and 3) dynamic multi-step reasoning (ReasoNet BIBREF6 ). In particular, BIBREF19 proposed handling the variations in passages and questions using Maxout units and iterative reasoning. However, this model still applies static attention and reasoning (with fixed multiple steps), where the same attention strategy is applied to all questions. DFN can be seen as an extension of ReasoNet, in the sense that the dynamic strategy is applied not only in the reasoning process but also in the attention process.","The idea of dynamic attention has been applied to article recommendations BIBREF20 . For MRC, Andreas et al. (2016) proposed a dynamic decision process for reading comprehension task BIBREF8 . In their dynamic neural module networks, the MRC task is divided into several predefined steps (e.g., finding, lookup, relating), and a neural network is dynamically composed via RL based on parsing information. In DFN, we also incorporate dynamic decisions, but instead of using fixed steps, we apply dynamic decisions to various attention strategies and flexible reasoning steps."]},{"idx":3,"heading":"RACE - The MRC Task","paragraphs":["In this section, we first give a brief introduction to the RACE dataset, and then explain the rationale behind choosing RACE as the testbed in our study."]},{"idx":4,"heading":"The Dataset","paragraphs":["RACE (Reading Comprehension Dataset From Examinations) is a recently released MRC dataset consisting of 27,933 passages and 97,867 questions from English exams, targeting Chinese students aged 12-18. RACE consists of two subsets, RACE-M and RACE-H, from middle school and high school exams, respectively. RACE-M has 28,293 questions and RACE-H has 69,574. Each question is associated with 4 candidate answers, one of which is correct. The data generation process of RACE differs from most MRC datasets - instead of generating questions and answers by heuristics or crowd-sourcing, questions in RACE are specifically designed for testing human reading skills, and are created by domain experts."]},{"idx":5,"heading":"Distinctive Characteristics in RACE","paragraphs":["The RACE dataset has some distinctive characteristics compared to other datasets, making it an ideal testbed for developing generic MRC systems for real-world human reading tasks.","Variety in Comprehension Skills. RACE requires a much broader spectrum of comprehension skills than other MRC datasets. Figure 1 shows some example questions from RACE and SQuAD: most SQuAD questions lead to direct answers that can be found in the original passage, while questions in RACE require more sophisticated reading comprehension skills such as summarizing (1st question), inference (2nd question) and deduction (3rd question). For humans, various tactics and skills are required to answer different questions. Similarly, it is important for MRC systems to adapt to different question types.","Complexity of Answers. As shown in Figure 2 , the answers in CNN/DM dataset are entities only. In SQuAD-like datasets, answers are often constrained to spans in the passage. Different from these datasets, answer candidates in RACE are natural language sentences generated by human experts, which increases the difficulty of the task. Real-world machine reading tasks are less about span exact matching, and more about summarizing the content and extending the obtained knowledge through reasoning.","Multi-step reasoning. Reasoning is an important skill in human reading comprehension. It refers to the skill of making connection between sentences and summarizing information throughout the passage. Table 1 shows a comparison on the requirement of reasoning level among different datasets. The low numbers on SQuAD and CNN/DM show that reasoning skills are less critical in getting the correct answers in these datasets, whereas such skills are essential for answering RACE questions."]},{"idx":6,"heading":"Dynamic Fusion Networks","paragraphs":["In this section, we present the model details of DFN. Section \"Conclusion\" describes the overall architecture, and each component is explained in detail in subsequent subsections. Section \" Training Details\" describes the reinforcement learning methods used to train DFN."]},{"idx":7,"heading":"Model Architecture","paragraphs":["The overall architecture of DFN is depicted by Figure 3 . The input is a question $Q$ in length $l_q$ , a passage $P$ in length $l_p$ , and a list of $r$ answer candidates $\\mathcal {A}=\\lbrace A_1,...,A_r\\rbrace $ in length $l_{a}^1,...,l_a^r$ . The model produces scores $c_1, c_2, ..., c_r$ for each answer candidate $A_1, A_2, ..., A_r$ respectively. The final prediction module selects the answer with the highest score.","The architecture consists of a standard Lexicon Encoding Layer and a Context Encoding Layer, on top of which are a Dynamic Fusion Layer and a Memory Generation Layer. The Dynamic Fusion Layer applies different attention strategies to different question types, and the Memory Generation Layer encodes question-related information in the passage for answer prediction. Multi-step reasoning is conducted over the output from the Dynamic Fusion and Memory Generation layers, in the Answer Scoring Module. The final output of the model is an answer choice $C\\in \\lbrace 1,2,...,r\\rbrace $ from the Answer Prediction Module.","In the following subsections, we will describe the details of each component in DFN (bold letters represent trainable parameters)."]},{"idx":8,"heading":"Lexicon Encoding Layer","paragraphs":["The first layer of DFN transforms each word in the passage, question and answer candidates independently into a fixed-dimension vector. This vector is the concatenation of two parts. The first part is the pre-trained GloVe embedding BIBREF21 of each word. For each out-of-vocabulary word, we map it to an all-zero vector. The second part is the character encodings. This is carried out by mapping each character to a trainable embedding, and then feeding all characters into an LSTM BIBREF22 . The last state of this LSTM is used as the character encodings. The output of the Lexicon Encoding layer is a set of vectors for $Q,P$ and each answer candidate in $\\mathcal {A}$ , respectively: $Q^\\text{embed}=\\lbrace q^\\text{embed}_i\\rbrace _{i=1}^{l_q}, P^\\text{embed}=\\lbrace p^\\text{embed}_i\\rbrace _{i=1}^{l_p}$ , and $A^\\text{embed}_j=\\lbrace a^\\text{embed}_{i,j}\\rbrace _{i=1}^{l_a^j}, j=1,2,...,r$ ."]},{"idx":9,"heading":"Context Encoding Layer","paragraphs":["The Context Encoding Layer passes $Q^{\\text{embed}}, p^{\\text{embed}}$ and $A^{\\text{embed}}$ into a bi-directional LSTM (BiLSTM) to obtain context representations. Since answer candidates $A_1,...,A_r$ are not always complete sentences, we append the question before each answer candidate and feed the concatenated sentence into BiLSTM. We use the same BiLSTM to encode the information in $P,Q$ and $\\mathcal {A}$ . The obtained context vectors are represented as: $ &Q^\\text{c}=\\textbf {BiL}\\textbf {STM}_1(Q^\\text{embed})=\\lbrace \\overrightarrow{q_i^\\text{c}},\\overleftarrow{q_i^\\text{c}}\\rbrace _{i=1}^{l_q},\\\\ &P^\\text{c}=\\textbf {BiL}\\textbf {STM}_1(P^\\text{embed})=\\lbrace \\overrightarrow{p_i^\\text{c}},\\overleftarrow{p_i^\\text{c}}\\rbrace _{i=1}^{l_p},\\\\ &(Q+A)^\\text{c}_j=\\textbf {BiLSTM}_1(Q^\\text{embed}+A^\\text{embed}_j)\\\\ &=\\lbrace \\overrightarrow{a^\\text{c}_{i,j}},\\overleftarrow{a^\\text{c}_{i,j}}\\rbrace _{i=1}^{l_p+l_a^j}, j=1,2,...,r. $"]},{"idx":10,"heading":"Dynamic Fusion Layer","paragraphs":["This layer is the core of DFN. For each given question-passage pair, one of $n$ different attention strategies is selected to perform attention across the passage, question and answer candidates.","The dynamic fusion is conducted in two steps: in the first step, an attention strategy $G\\in \\lbrace 1,2,...,n\\rbrace $ is randomly sampled from the output of the strategy gate $f^\\text{sg}(Q^c)$ . The strategy gate takes input from the last-word representation of the question $Q^\\text{c}$ , and outputs a softmax over $\\lbrace 1,2,...n\\rbrace $ . In the second step, the $G$ -th attention strategy is activated, and computes the attention results according to $G$ -th strategy. Each strategy, denoted by Attention $_k,k=1,2,...,n$ , is essentially a function of $Q^\\text{c},P^\\text{c}$ and one answer candidate $(Q+A)^\\text{c}_j$ that performs attention in different directions. The output of each strategy is a fixed-dimension representation, as the attention result. $ f^\\text{sg}(Q^c) \\leftarrow & \\text{softmax}(\\mathbf {W_1}(\\overrightarrow{q_{l_q}^\\text{c}};\\overleftarrow{q_{1}^\\text{c}})\\\\ G\\sim & \\text{Category}\\left(f^\\text{sg}(Q^c)\\right),\\\\ s_j\\leftarrow & \\textbf {Attention}_{G}(Q^\\text{c},P^\\text{c},(Q+A)^\\text{c}_j),\\\\ & j=1,2,...,r. $","Attention Strategies. For experiment on RACE, we choose $n=3$ and use the following strategies:","Integral Attention: We treat the question and answer as a whole, and attend each word in $(Q+A)_j^\\text{c}$ to the passage $P^\\text{c}$ (Figure 4 ). This handles questions with short answers (e.g., the last question in upper box of Figure 1 ).","Formally, $Q^{\\text{int}}_{j},A^{\\text{int}}_{j}\\leftarrow \\text{Split}\\left((Q+A)^\\text{c}_j\\operatornamewithlimits{{\\color {blue} \\triangleright }}P^\\text{c}\\right). $","The operator $\\operatornamewithlimits{{\\color {blue} \\triangleright }}$ represents any one-sided attention function. For DFN, we use the single direction version of multi-perspective matching in BiMPM BIBREF13 ; For two text segments $X,X^{\\prime }\\in \\lbrace P,Q,A_j,(Q+A)_j\\rbrace $ , $X \\operatornamewithlimits{{\\color {blue} \\triangleright }}X^{\\prime }$ matches each word $w\\in X$ with respect to the whole sentence $X^{\\prime }$ , and has the same length as $X$ . We defer details of the $\\operatornamewithlimits{{\\color {blue} \\triangleright }}$ operator to Section \"Memory Generation Layer\" when we introduce our memory generation.","The Split $()$ function splits a vector representation in length $l_q+l_a^j$ into two vector representations in length $l_q$ and $l_a^j$ , to be consistent with other strategies.","Answer-only Attention: This strategy only attends each word in the answer candidate to the passage (Figure 4 ), without taking the question into consideration. This is to handle questions with full-sentence answer candidates (e.g., the first and the third questions in the upper box of Figure 1 ). $ M_a\\leftarrow & A_j^\\text{c}\\operatornamewithlimits{{\\color {blue} \\triangleright }}P^\\text{c},\\\\ Q_j^{\\text{aso}},A_j^{\\text{aso}}\\leftarrow &Q^\\text{c},M_a. $","Entangled Attention: As shown in Figure 4 , each word in question and answer is attended to the passage, denoted by $M_q$ and $M_a$ . Then, we entangle the results by attending each word in $M_q$ to $M_a$ , and also $M_a$ to $M_q$ . This attention is more complicated than the other two mentioned above, and targets questions that require reasoning (e.g., the second question in the upper box of Figure 1 ). $ M_q\\leftarrow & Q^\\text{c}\\operatornamewithlimits{{\\color {blue} \\triangleright }}P^\\text{c}\\\\ M_a\\leftarrow & A_j^\\text{c}\\operatornamewithlimits{{\\color {blue} \\triangleright }}P^\\text{c},\\\\ Q_j^{\\text{ent}},A_j^{\\text{ent}}\\leftarrow &M_q \\operatornamewithlimits{{\\color {blue} \\triangleright }}M_a, M_a \\operatornamewithlimits{{\\color {blue} \\triangleright }}M_q. $","We can incorporate a large number of strategies into the framework depending on the question types we need to deal with. In this paper, we use three example strategies to demonstrate the effectiveness of DFN.","Attention Aggregation. Following previous work, we aggregate the result of each attention strategy through a BiLSTM. The first and the last states of these BiLSTMs are used as the output of the attention strategies. We use different BiLSTM for different strategies, which proved to slightly improve the model performance. $ Q_j^x,A_j^x\\leftarrow &\\textbf {BiLSTM}^x(Q_j^x), \\textbf {BiLSTM}^x(A_j^x),\\\\ \\textbf {Attention}_k\\leftarrow & \\text{FinalState}(Q_j^x,A_j^x),\\\\ &\\text{ for } (k,x)\\in \\lbrace \\text{(1,int),(2,aso),(3,ent)}\\rbrace . $","The main advantages of dynamic multi-strategy fusion are three-fold: 1) It provides adaptivity for different types of questions. This addresses the challenge in the rich variety of comprehension skills aforementioned in Section \"Distinctive Characteristics in RACE\" . The key to adaptivity is the strategy gate $G$ . Our observation is that the model performance degrades when trained using simpler methods such as max-pooling or model averaging. 2) The dynamic fusion takes all three elements (question, passage and answer candidates) into account in the attention process. This way, answer candidates are fused together with the question and the passage to get a complete understanding of the full context. 3) There is no restriction on the attention strategy used in this layer, which allows flexibility for incorporating existing attention mechanisms.","Although some of the attention strategies appear to be straightforward (e.g., long/short answers), it is difficult to use simple heuristic rules for strategy selection. For example, questions with a placeholder “_” might be incomplete question sentences that require integral attention; but in some questions (e.g., “we can infer from the passage that _ .”), the choices are full sentences and the answer-only attention should be applied here instead. Therefore, we turn to reinforcement learning methods (see Section \" Training Details\" ) to optimize the choice of attention strategies, which leads to a policy that give important insights on our model behavior."]},{"idx":11,"heading":"Memory Generation Layer","paragraphs":["A memory is generated for the answer module in this layer. The memory $M$ has the same length as $P$ , and is the result of attending each word in $P^\\text{c}$ to the question $Q^\\text{c}$ (Figure 4 ). We use the same attention function for $M$ as that for attention strategies, and then aggregate the results. The memory is computed as $M\\leftarrow \\textbf {BiLSTM}_2(Q^\\text{c}\\operatornamewithlimits{{\\color {blue} \\triangleright }}P^\\text{c})$ , where $\\operatornamewithlimits{{\\color {blue} \\triangleright }}$ is the attention operator specified as below.","Our attention operator takes the same form as BiMPM BIBREF13 . For simplicity, we use $P,Q,(Q+A)_j$ to denote $P^\\text{c},Q^\\text{c}$ and $(Q+A)^\\text{c}_j$ in this section. Recall that for $X,X^{\\prime }\\in \\lbrace P,Q,A_j,(Q+A)_j\\rbrace $ , and $X \\operatornamewithlimits{{\\color {blue} \\triangleright }}X^{\\prime }$ computes the relevance of each word $w\\in X$ with respect to the whole sentence $X^{\\prime }$ . $X \\operatornamewithlimits{{\\color {blue} \\triangleright }}X^{\\prime }$ has the same length as $X^{\\prime }$ . Each operation $\\operatornamewithlimits{{\\color {blue} \\triangleright }}$ is associated with a set of trainable weights denoted by $P^\\text{c},Q^\\text{c}$0 . For $P^\\text{c},Q^\\text{c}$1 in different strategies, we use different sets of trainable weights; the only exception is for $P^\\text{c},Q^\\text{c}$2 computed both in Answer-only Attention and Entangled Attention: These two operations have the same weights since they are exactly the same. We find untying weights in different $P^\\text{c},Q^\\text{c}$3 operations can slightly improve our model performance.","We use a multi-perspective function to describe $\\operatornamewithlimits{{\\color {blue} \\triangleright }}$ . For any two vectors $v_1,v_2\\in \\mathbb {R}^d$ , define the multi-perspective function $g(v_1,v_2;\\mathbf {W})=\\left\\lbrace \\cos (\\mathbf {W}^{(k)} \\circ v_1, \\mathbf {W}^{(k)}\\circ v_2 )\\right\\rbrace _{k=1}^N, $","where $\\mathbf {W}\\in \\mathbb {R}^{N\\times d}$ is a trainable parameter, $N$ is a hyper-parameter (the number of perspectives), and $\\mathbf {W}^{(k)}$ denotes the $k$ -th row of $\\mathbf {W}$ . In our experiments, we set $N=10$ .","Now we define $X \\operatornamewithlimits{{\\color {blue} \\triangleright }}X^{\\prime }$ using $g$ and four different ways to combine vectors in text $X,X^{\\prime }$ . Denote by $x_i,x_i^{\\prime }\\in \\mathbb {R}^d$ the $i$ -th vector in $X,X^{\\prime }$ respectively. The function work concurrently for the forward and backward LSTM activations (generated by BiLSTM in the Context Encoding layer) in $X$ and $X^{\\prime }$ ; denoted by $\\overrightarrow{x}_i$ and $\\overleftarrow{x}_i$ , the forward and backward activations respectively (and similarly for $g$0 ). The output of $g$1 also has activations in two directions for further attention operation (e.g., in Entangled Attention). The two directions are concatenated before feeding into the aggregation BiLSTM.","Let $l_x,l^{\\prime }_x$ be the length of $X,X^{\\prime }$ respectively. $X\\operatornamewithlimits{{\\color {blue} \\triangleright }}X^{\\prime }$ outputs two groups of vectors $\\lbrace \\overrightarrow{u}_i,\\overleftarrow{u}_i \\rbrace _{i=1}^{l_x}$ by concatenating the following four parts below:","Full Matching: $\\overrightarrow{u}_i^{\\text{full}}=g(\\overrightarrow{x}_i,\\overrightarrow{x}_{l_x^{\\prime }},\\mathbf {W}_{o1}),$ $\\overleftarrow{u}_i^{\\text{full}}=g(\\overleftarrow{x}_i,\\overleftarrow{x}^{\\prime }_{1},\\mathbf {W}_{o2}). $","Maxpooling Matching: $\\overrightarrow{u}_i^{\\text{max}}=\\max _{j\\in \\lbrace 1,...,l_x\\rbrace }g(\\overrightarrow{x}_i,\\overrightarrow{x}_j^{\\prime },\\mathbf {W}_{o3}),$ $\\overleftarrow{u}_i^{\\text{max}}=\\max _{j\\in \\lbrace 1,...,l_x\\rbrace }g(\\overleftarrow{x}_i,\\overleftarrow{x}_j^{\\prime },\\mathbf {W}_{o4}),$","here $\\max $ means element-wise maximum.","Attentive Matching: for $j=1,2,...,N$ compute $\\overrightarrow{\\alpha }_{i,j}=\\cos (\\overrightarrow{x}_i,\\overrightarrow{x}_j^{\\prime }),\\overleftarrow{\\alpha }_{i,j}=\\cos (\\overleftarrow{x}_i,\\overleftarrow{x}_j^{\\prime }). $","Take weighted mean according to $\\overrightarrow{\\alpha }_{i,j},\\overleftarrow{\\alpha }_{i,j}$ : $\\overrightarrow{x}_i^{\\text{mean}}=\\frac{\\sum _{j=1}^{l_x^{\\prime }} \\overrightarrow{\\alpha }_{i,j} \\cdot \\overrightarrow{x}_j^{\\prime }}{\\sum _{j=1}^{l_x^{\\prime }} \\overrightarrow{\\alpha }_{i,j}},$ $\\overleftarrow{x}_i^{\\text{mean}}=\\frac{\\sum _{j=1}^{l_x^{\\prime }} \\overleftarrow{\\alpha }_{i,j} \\cdot \\overleftarrow{x}_j^{\\prime }}{\\sum _{j=1}^{l_x^{\\prime }} \\overleftarrow{\\alpha }_{i,j}}.$","Use multi-perspective function to obtain attentive matching: $\\overrightarrow{u}_i^{\\text{att}}=g(\\overrightarrow{x}_i,\\overrightarrow{x}_i^{\\text{mean}},\\mathbf {W}_{o5}),$ $\\overleftarrow{u}_i^{\\text{att}}=g(\\overleftarrow{x}_i,\\overleftarrow{x}_i^{\\text{mean}},\\mathbf {W}_{o6}).$","Max-Attentive Matching: The same as attentive matching, but taking the maximum over $\\overrightarrow{\\alpha }_{i,j},\\overleftarrow{\\alpha }_{i,j}, j=1,2,...,l_x^{\\prime }$ instead of using the weighted mean."]},{"idx":12,"heading":"Answer Scoring Module","paragraphs":["This module performs multi-step reasoning in the neural space to generate the right answer. This unit adopts the architecture of ReasoNet BIBREF6 . We simulate multi-step reasoning with a GRU cell BIBREF23 to skim through the memory several times, changing its internal state as the skimming progresses. The initial state $s_j^{(0)}=s_j$ is generated from the Dynamic Fusion Layer for each answer candidate $j=1,2,...,r$ . We skim through the passage for at most $\\mathcal {T}_{\\max }$ times. In every step $t\\in \\lbrace 1,2,...,\\mathcal {T}_{\\max }\\rbrace $ , an attention vector $f^{(t)}_{\\text{att}}$ is generated from the previous state $s_j^{t-1}$ and the memory $M$ . To compute $f_{\\text{att}}$ , an attention score $a_{t,i}$ is computed based on each word $m_i$ in memory $j=1,2,...,r$0 and state $j=1,2,...,r$1 as $j=1,2,...,r$2","where $l_m=l_p$ is the memory length, and $\\mathbf {W_2},\\mathbf {W_3}$ are trainable weights. We set $\\lambda =10$ in our experiments. The attention vector is then computed as a weighted sum of memory vectors using attention scores, i.e., $f^{(t)}_{\\text{att}}\\leftarrow \\sum _{i=1}^{l_m} a_{i,j}^{(t)}m_i.$ Then, the GRU cell takes the attention vector $f_{\\text{att}}^{(t)}$ as input and changes its internal state. $ s_j^{(0)}\\leftarrow s_j, \\;s_j^{(t)}\\leftarrow \\textbf {GRU}\\left(f_{\\text{att}}^{(t)},s_j^{(t-1)}\\right). $","To decide when to stop skimming, a termination gate (specified below) takes $s_j^{(t)}, j=1,...,r$ at step $t$ as the input, and outputs a probability $p_t$ of whether to stop reading. The number of reading steps is decided by sampling a Bernoulli variable $T_t$ with parameter $p_t$ . If $T_t$ is 1, the Answer Scoring Module stops skimming, and score $c_j\\leftarrow \\mathbf {W}_5\\text{ReLU}(\\mathbf {W}_4s_j^{(t)}) $ is generated for each answer candidate $j$ . The input to the termination gate in step $t$ is the state representation of all possible answers, $s_j^{(t)}, j=1,2,...,r$ . We do not use separate termination gates for each answer candidate. This is to restrain the size of the action space and variance in training. Since answers are mutable, the input weights for each answer candidate fed into the gate softmax are the same. $t$0","Answer Prediction. Finally, an answer prediction is drawn from the softmax distribution over the scores of each answer candidate: $C\\sim \\text{Softmax}\\left(c_1,c_2,...,c_r\\right). $"]},{"idx":13,"heading":"Training Details","paragraphs":["Since the strategy choice and termination steps are discrete random variables, DFN cannot be optimized by backpropagation directly. Instead, we see strategy choice $G$ , termination decision $T_t$ and final prediction $C$ as policies, and use the REINFORCE algorithm BIBREF24 to train the network. Let $T$ be the actual skimming steps taken, i.e., $T=\\min \\lbrace t:T_t=1\\rbrace $ . We define the reward $r$ to be 1 if $C$ (final answer) is correct, and 0 otherwise. Each possible value pair of $(C,G,T)$ corresponds to a possible episode, which leads to $r\\cdot n \\cdot \\mathcal {T}$ possible episodes. Let $\\pi (c,g,t;\\theta )$ be any policy parameterized by DFN parameter $T_t$0 , and $T_t$1 be the expected reward. Then:","$$&\\nabla _\\theta J(\\theta )\\nonumber \\\\ =&E_{\\pi (g,c,t;\\theta )}\\left[\\nabla _\\theta \\log \\pi (c,g,t;\\theta )(r-b)\\right]\\nonumber \\\\ =&\\sum _{g,c,t}\\pi (g,c,t;\\theta )\\left[\\nabla _\\theta \\log \\pi (c,g,t;\\theta )(r-b)\\right].$$ (Eq. 29)","where $b$ is a critic value function. Following BIBREF6 , we set $b=\\sum _{g,c,t}\\pi (g,c,t;\\theta )r$ and replace the $(r-b)$ term above by $(r/b-1)$ to achieve better performance and stability."]},{"idx":14,"heading":"Experiments","paragraphs":["To evaluate the proposed DFN model, we conducted experiments on the RACE dataset. Statistics of the training/dev/test data are provided in Table 2 . In this section, we present the experimental results, with a detailed analysis on the dynamic selection of strategies and multi-step reasoning. An ablation study is also provided to demonstrate the effectiveness of dynamic fusion and reasoning in DFN."]},{"idx":15,"heading":"Parameter Setup","paragraphs":["Most of our parameter settings follow BIBREF13 and BIBREF6 . We use ( 29 ) to update the model, and use ADAM BIBREF25 with a learning rate of 0.001 and batch size of 64 for optimization. A small dropout rate of 0.1 is applied to each layer. For word embedding, we use 300-dimension GloVe BIBREF21 embedding from the 840B Common Crawl corpus. The word embeddings are not updated during training. The character embedding has 20 dimensions and the character LSTM has 50 hidden units. All other LSTMs have a hidden dimension of 100. The maximum reasoning step $\\mathcal {T}$ is set to 5. We limit the length of passage/question/answer to a maximum of 500/100/100 for efficient computation. We also train an ensemble model of 9 DFNs using randomly initialized parameters. Training usually converges within 20 epochs. The model is implemented with Tensorflow BIBREF26 and the source code will be released upon paper acceptance."]},{"idx":16,"heading":"Model Performance","paragraphs":["Table 3 shows a comparison between DFN and a few previously proposed models. All models were trained with the full RACE dataset, and tested on RACE-M and RACE-H, respectively. As shown in the table, on RACE-M, DFN leads to a 7.8% and 7.3% performance boost over GA and Stanford AR, respectively. On RACE-H, the outperformance is 1.5% and 2.7%. The ensemble models also gained a performance boost of 4-5% comparing to previous methods. We suspect that the lower gain on RACE-H might result from the higher level of difficulty in those questions in RACE-H, as well as ambiguity in the dataset. Human performance drops from 85.1 on RACE-M to 69.4 on RACE-H, which indicates RACE-H is very challenging even for human.","Figure 5 shows six randomly-selected questions from the dataset that DFN answered correctly, grouped by their attention strategies. Recall that the three attention strategies proposed for this task are: 1) Integral Attention for short answers; 2) Answer-only Attention for long answers; and 3) Entangled Attention for deeper reasoning. Question 1 and 2 in Figure 5 present two examples that used Integral Attention. In both of the questions, the question and answer candidates are partial sentences. So the system chose Integral Attention in this case. In the first question, DFN used 3 steps of reasoning, which indicates the question requires some level of reasoning (e.g., resolving coreference of “the third way”). In the second question, the combined sentence comes directly from the passage, so DFN only used 1 step of reasoning.","Question 3 and 4 in Figure 5 provide two instances that use answer-only attentions. As shown in these examples, Answer-only attention usually deals with long and natural language answer candidates. Such answers cannot be derived without the model reading through multiple sentences in the passage, and this requires multi-step reasoning. So in both examples, the system went through 5 steps of reasoning.","Question 5 and 6 in Figure 5 show two examples that used the Entangled Attention. Both questions require a certain level of reasoning. Question 5 asks for the causes of a scenario, which is not explicitly mentioned in the passage. And question 6 asks for a counting of concepts, which is implicit and has to be derived from the text as well. For both cases, the entangled attention was selected by the model. As for the reasoning steps, we find that for the majority of questions that use Entangled Attention, DFN only uses one reasoning step. This is probably because entangled attention is powerful enough to derive the answer.","We also examined the strategy choices with respect to certain keywords. For each word $w$ in vocabulary, we computed the distribution $\\Pr [G,T|w\\in Q]$ , i.e., the conditional distribution of strategy and step when $w$ appeared in the question. Table 4 provides some keywords and their associated dominant strategies and step choices. The results validate the assumption that DFN dynamically selects specific attention strategy based on different question types. For example, the underline “_” indicates that the question and choice should be concatenated to form a sentence. This led to Integral Attention being most favorable when “_” is present. In another example, “not” and “except” usually appear in questions like “Which of the following is not TRUE”. Such questions usually have long answer candidates that require more reasoning. So Answer-only Attention with Reasoning Step#5 became dominant."]},{"idx":17,"heading":"Ablation Studies","paragraphs":["For ablation studies, we conducted experiments with 4 different model configurations:","The full DFN model with all the components aforementioned.","DFN without dynamic fusion (DF). We dropped the Strategy Gate $G$ , and used only one attention strategy in the Dynamic Fusion Layer.","DFN without multi-step reasoning (MR). Here we dropped the Answer Scoring Module, and used the output of Dynamic Fusion Layer to generate a score for each answer.","DFN without DF and MR.","To select the best strategy for each configuration, we trained 3 different models for ii) and iv), and chose the best model based on their performance on the dev set. This explains the smaller performance gap between the full model and ablation models on the dev set than that on the test set. Experimental results show that for both ii) and iv), the Answer-Only Attention gave the best performance.","To avoid variance in training and provide a fair comparison, 3 ensembles of each model were trained and evaluated on both dev and test sets. As shown in Table 5 , the DFN model has a 1.6% performance gain over the basic model (without DF and MR). This performance boost was contributed by both multi-step reasoning and dynamic fusion. When omitting DF or MR alone, the performance of DFN model dropped by 1.1% and 1.2%, respectively.","To validate the effectiveness of the DFN model, we also performed a significance test and compared the full model with each ablation model. The null hypothesis is: the full DFN model has the same performance as the ablation model. As shown in Table 5 , the combination of DF and MR leads to an improvement with a statistically significant margin in our experiments, although neither DF or MR can, individually."]},{"idx":18,"heading":"Conclusion","paragraphs":["In this work, we propose a novel neural model - Dynamic Fusion Network (DFN), for MRC. For a given input sample, DFN can dynamically construct an model instance with a sample-specific network structure by picking an optimal attention strategy and an optimal number of reasoning steps on the fly. The capability allows DFN to adapt effectively to handling questions of different types. By training the policy of model construction with reinforcement learning, our DFN model can substantially outperform previous state-of-the-art MRC models on the challenging RACE dataset. Experiments show that by marrying dynamic fusion (DF) with multi-step reasoning (MR), the performance boost of DFN over baseline models is statistically significant. For future directions, we plan to incorporate more comprehensive attention strategies into the DFN model, and to apply the model to other challenging MRC tasks with more complex questions that need DF and MR jointly. Future extension also includes constructing a “composable” structure on the fly - by making the Dynamic Fusion Layer more flexible than it is now."]},{"idx":19,"heading":"Figures and Tables","paragraphs":["Figure 1: Above: Questions from English exams. Below: Questions from SQuAD.","Table 1: Percentage of questions in each dataset that requires Single-sentence Inference(SI) and Multi-sentence Inference(MI), from (Lai et al., 2017).","Figure 2: Left: Examples from RACE. Right: Examples from CNN dataset.","Figure 3: Architecture of MUSIC. i) Passage, question and answer choices are mapped through word and character embeddings in the Embedding Layer. ii) The embeddings are then fed into a Bi-LSTM in the Context Layer. iii) The Gated Matching Layer makes customized attention matchings across the three representations from passage, question and answer choices. iv) The Multi-step Reasoning Unit reads through the memory for a dynamic number of steps. v) Answer prediction gives the final answer.","Figure 4: Examples of choosing Matching Gate and number of inference steps in different questions types. Left: The question and the answer choice are part of a full sentence, a case handled by Gate 1, with one-step inference. Middle: A complex question with natural language answer choices, a case where Gate 2 was applied and 5 inteference steps were required. Right: A question that requires deep inference, a case handled by Gate 3, with one-step inference.","Table 2: Accuracy% of MUSIC compared to baseline methods on RACE test sets. Results of baseline models come from (Lai et al., 2017) and unpublished (Anonymous, 2018). Note that ElimiNetEnsemble ensembles 6 equivalent models.","Table 3: Ablation studies of MUSIC for multistrategy matching (MM) and multi-step reasoning (MR)."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1712.00733","title":"Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Visual Question Answering (VQA) has attracted much attention since it offers insight into the relationships between the multi-modal analysis of images and natural language. Most of the current algorithms are incapable of answering open-domain questions that require to perform reasoning beyond the image contents. To address this issue, we propose a novel framework which endows the model capabilities in answering more complex questions by leveraging massive external knowledge with dynamic memory networks. Specifically, the questions along with the corresponding images trigger a process to retrieve the relevant information in external knowledge bases, which are embedded into a continuous vector space by preserving the entity-relation structures. Afterwards, we employ dynamic memory networks to attend to the large body of facts in the knowledge graph and images, and then perform reasoning over these facts to generate corresponding answers. Extensive experiments demonstrate that our model not only achieves the state-of-the-art performance in the visual question answering task, but can also answer open-domain questions effectively by leveraging the external knowledge."]},{"idx":1,"heading":"Introduction","paragraphs":["Visual Question Answering (VQA) is a ladder towards a better understanding of the visual world, which pushes forward the boundaries of both computer vision and natural language processing. A system in VQA tasks is given a text-based question about an image, which is expected to generate a correct answer corresponding to the question. In general, VQA is a kind of Visual Turing Test, which rigorously assesses whether a system is able to achieve human-level semantic analysis of images BIBREF0 , BIBREF1 . A system could solve most of the tasks in computer vision if it performs as well as or better than humans in VQA. In this case, it has garnered increasing attentions due to its numerous potential applications BIBREF2 , such as providing a more natural way to improve human-computer interaction, enabling the visually impaired individuals to get information about images, etc.","To fulfill VQA tasks, it requires to endow the responder to understand intention of the question, reason over visual elements of the image, and sometimes have general knowledge about the world. Most of the present methods solve VQA by jointly learning interactions and performing inference over the question and image contents based on the recent success of deep learning BIBREF3 , BIBREF2 , BIBREF4 , BIBREF5 , BIBREF6 , which can be further improved by introducing the attention mechanisms BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 . However, most of questions in the current VQA dataset are quite simple, which are answerable by analyzing the question and image alone BIBREF2 , BIBREF13 . It can be debated whether the system can answer questions that require prior knowledge ranging common sense to subject-specific and even expert-level knowledge. It is attractive to develop methods that are capable of deeper image understanding by answering open-domain questions BIBREF13 , which requires the system to have the mechanisms in connecting VQA with structured knowledge, as is shown in Fig. 1 . Some efforts have been made in this direction, but most of them can only handle a limited number of predefined types of questions BIBREF14 , BIBREF15 .","Different from the text-based QA problem, it is unfavourable to conduct the open-domain VQA based on the knowledge-based reasoning, since it is inevitably incomplete to describe an image with structured forms BIBREF16 . The recent availability of large training datasets BIBREF13 makes it feasible to train a complex model in an end-to-end fashion by leveraging the recent advances in deep neural networks (DNN) BIBREF2 , BIBREF5 , BIBREF7 , BIBREF10 , BIBREF12 . Nevertheless, it is non-trivial to integrate knowledge into DNN-based methods, since the knowledge are usually represented in a symbol-based or graph-based manner (e.g., Freebase BIBREF17 , DBPedia BIBREF18 ), which is intrinsically different from the DNN-based features. A few attempts are made in this direction BIBREF19 , but it may involve much irrelevant information and fail to implement multi-hop reasoning over several facts.","The memory networks BIBREF20 , BIBREF21 , BIBREF22 offer an opportunity to address these challenges by reading from and writing to the external memory module, which is modeled by the actions of neural networks. Recently, it has demonstrated the state-of-the-art performance in numerous NLP applications, including the reading comprehension BIBREF23 and textual question answering BIBREF24 , BIBREF22 . Some seminal efforts are also made to implement VQA based on dynamic memory networks BIBREF25 , but it does not involve the mechanism to incorporate the external knowledge, making it incapable of answering open-domain visual questions. Nevertheless, the attractive characteristics motivate us to leverage the memory structures to encode the large-scale structured knowledge and fuse it with the image features, which offers an approach to answer open domain visual questions."]},{"idx":2,"heading":"Our Proposal","paragraphs":["To address the aforementioned issues, we propose a novel Knowledge-incorporated Dynamic Memory Network framework (KDMN), which allows to introduce the massive external knowledge to answer open-domain visual questions by exploiting the dynamic memory network. It endows a system with an capability to answer a broad class of open-domain questions by reasoning over the image content incorporating the massive knowledge, which is conducted by the memory structures.","Different from most of existing techniques that focus on answering visual questions solely on the image content, we propose to address a more challenging scenario which requires to implement reasoning beyond the image content. The DNN-based approaches BIBREF2 , BIBREF5 , BIBREF7 are therefore not sufficient, since they can only capture information present in the training images. Recent advances witness several attempts to link the knowledge to VQA methods BIBREF14 , BIBREF15 , which make use of structured knowledge graphs and reason about an image on the supporting facts. Most of these algorithms first extract the visual concepts from a given image, and implement reasoning over the structured knowledge bases explicitly. However, it is non-trivial to extract sufficient visual attributes, since an image lacks the structure and grammatical rules as language. To address this issue, we propose to retrieve a bath of candidate knowledge corresponding to the given image and related questions, and feed them to the deep neural network implicitly. The proposed approach provides a general pipeline that simultaneously preserves the advantages of DNN-based approaches BIBREF2 , BIBREF5 , BIBREF7 and knowledge-based techniques BIBREF14 , BIBREF15 .","In general, the underlying symbolic nature of a Knowledge Graph (KG) makes it difficult to integrate with DNNs. The usual knowledge graph embedding models such as TransE BIBREF26 focus on link prediction, which is different from VQA task aiming to fuse knowledge. To tackle this issue, we propose to embed the entities and relations of a KG into a continuous vector space, such that the factual knowledge can be used in a more simple manner. Each knowledge triple is treated as a three-word SVO $(subject, verb, object)$ phrase, and embedded into a feature space by feeding its word-embedding through an RNN architecture. In this case, the proposed knowledge embedding feature shares a common space with other textual elements (questions and answers), which provides an additional advantage to integrate them more easily.","Once the massive external knowledge is integrated into the model, it is imperative to provide a flexible mechanism to store a richer representation. The memory network, which contains scalable memory with a learning component to read from and write to it, allows complex reasoning by modeling interaction between multiple parts of the data BIBREF20 , BIBREF25 . In this paper, we adopt the most recent advance of Improved Dynamic Memory Networks (DMN+) BIBREF25 to implement the complex reasoning over several facts. Our model provides a mechanism to attend to candidate knowledge embedding in an iterative manner, and fuse it with the multi-modal data including image, text and knowledge triples in the memory component. The memory vector therefore memorizes useful knowledge to facilitate the prediction of the final answer. Compared with the DMN+ BIBREF25 , we introduce the external knowledge into the memory network, and endows the system an ability to answer open-domain question accordingly.","To summarize, our framework is capable of conducting the multi-modal data reasoning including the image content and external knowledge, such that the system is endowed with a more general capability of image interpretation. Our main contributions are as follows:","To our best knowledge, this is the first attempt to integrating the external knowledge and image representation with a memory mechanism, such that the open-domain visual question answering can be conducted effectively with the massive knowledge appropriately harnessed;","We propose a novel structure-preserved method to embed the knowledge triples into a common space with other textual data, making it flexible to integrate different modalities of data in an implicit manner such as image, text and knowledge triples;","We propose to exploit the dynamic memory network to implement multi-hop reasonings, which has a capability to automatically retrieve the relevant information in the knowledge bases and infer the most probable answers accordingly."]},{"idx":3,"heading":"Overview","paragraphs":["In this section, we outline our model to implement the open-domain visual question answering. In order to conduct the task, we propose to incorporate the image content and external knowledge by exploiting the most recent advance of dynamic memory network BIBREF22 , BIBREF25 , yielding three main modules in Fig. 2 . The system is therefore endowed with an ability to answer arbitrary questions corresponding to a specific image.","Considering of the fact that most of existing VQA datasets include a minority of questions that require prior knowledge, the performance therefore cannot reflect the particular capabilities. We automatically produce a collection of more challenging question-answer pairs, which require complex reasoning beyond the image contents by incorporating the external knowledge. We hope that it can serve as a benchmark for evaluating the capability of various VQA models on the open-domain scenarios .","Given an image, we apply the Fast-RCNN BIBREF27 to detect the visual objects of the input image, and extract keywords of the corresponding questions with syntax analysis. Based on these information, we propose to learn a mechanism to retrieve the candidate knowledge by querying the large-scale knowledge graph, yielding a subgraph of relevant knowledge to facilitate the question answering. During the past years, a substantial amount of large-scale knowledge bases have been developed, which store common sense and factual knowledge in a machine readable fashion. In general, each piece of structured knowledge is represented as a triple $(subject, rel, object)$ with $subject$ and $object$ being two entities or concepts, and $rel$ corresponding to the specific relationship between them. In this paper, we adopt external knowledge mined from ConceptNet BIBREF28 , an open multilingual knowledge graph containing common-sense relationships between daily words, to aid the reasoning of open-domain VQA.","Our VQA model provides a novel mechanism to integrate image information with that extracted from the ConceptNet within a dynamic memory network. In general, it is non-trivial to integrate the structured knowledge with the DNN features due to their different modalities. To address this issue, we embed the entities and relations of the subgraph into a continuous vector space, which preserves the inherent structure of the KGs. The feature embedding provides a convenience to fuse with the image representation in a dynamic memory network, which builds on the attention mechanism and the memory update mechanism. The attention mechanism is responsible to produce the contextual vector with relevance inferred by the question and previous memory status. The memory update mechanism then renews the memory status based on the contextual vector, which can memorize useful information for predicting the final answer. The novelty lies the fact that these disparate forms of information are embedded into a common space based on memory network, which facilities the subsequent answer reasoning.","Finally, we generate a predicted answer by reasoning over the facts in the memory along with the image contents. In this paper, we focus on the task of multi-choice setting, where several multi-choice candidate answers are provided along with a question and a corresponding image. For each question, we treat every multi-choice answer as input, and predict whether the image-question-answer triplet is correct. The proposed model tries to choose one candidate answer with the highest probability by inferring the cross entropy error on the answers through the entire network."]},{"idx":4,"heading":"Answer Open-Domain Visual Questions","paragraphs":["In this section, we elaborate on the details and formulations of our proposed model for answering open-domain visual questions. We first retrieve an appropriate amount of candidate knowledge from the large-scale ConceptNet by analyzing the image content and the corresponding questions; afterward, we propose a novel framework based on dynamic memory network to embed these symbolic knowledge triples into a continuous vector space and store it in a memory bank; finally, we exploit these information to implement the open-domain VQA by fusing the knowledge with image representation."]},{"idx":5,"heading":"Candidate Knowledge Retrieval","paragraphs":["In order to answer the open-domain visual questions, we should sometime access information not present in the image by retrieving the candidate knowledge in the KBs. A desirable knowledge retrieval should include most of the useful information while ignore the irrelevant ones, which is essential to avoid model misleading and reduce the computation cost. To this end, we take the following three principles in consideration as (1) entities appeared in images and questions (key entities) are critical; (2) the importance of entities that have direct or indirect links to key entities decays as the number of link hops increases; (3) edges between these entities are potentially useful knowledge.","Following these principles, we propose a three-step procedure to retrieve that candidate knowledge that are relevant to the context of images and questions. The retrieval procedure pays more attention on graph nodes that are linked to semantic entities, which also takes account of graph structure for measuring edge importance.","In order to retrieve the most informative knowledge, we first extract the candidate nodes in the ConceptNet by analyzing the prominent visual objects in images with Fast-RCNN BIBREF27 , and textual keywords with the Natural Language Toolkit BIBREF29 . Both of them are then associated with the corresponding semantic entities in ConceptNet BIBREF28 by matching all possible n-grams of words. Afterwards, we retrieve the first-order subgraph using these selected nodes from ConceptNet BIBREF28 , which includes all edges connecting with at least one candidate node. It is assumed that the resultant subgraph contains the most relevant information, which is sufficient to answer questions by reducing the redundancy. The resultant first-order knowledge subgraph is denoted as $G$ .","Finally, we compress the subgraph $G$ by evaluating and ranking the importance of edges in $G$ using a designed score function, and carefully select the top- $N$ edges along with the nodes for subsequent task. Specifically, we first assign initial weights $w_{i}$ for each subgraph node, e.g., the initial weights for visual object can be proportional to their corresponding bounding-box area such that the dominant objects receive more attention, the textual keywords are treated equally. Then, we calculate the importance score of each node in $G$ by traversing each edge and propagating node weights to their neighbors with a decay factor $r\\in (0,1)$ as","$$score(i)=w_{i}+\\sum _{j \\in G \\backslash i} r ^n w_{j},$$ (Eq. 8)","where $n$ is the number of link hops between the entity $i$ and $j$ . For simplicity, we ignore the edge direction and edge type (relation type), and define the importance of edge $w_{i,j}$ as the weights sum of two connected nodes as","$$w_{i,j}=score(i)+score(j), \\quad \\forall (i,j) \\in G.$$ (Eq. 9)","In this paper, we take the top- $N$ edges ranked by $w_{i,j}$ as the final candidate knowledge for the given context, denoted as $G^\\ast $ ."]},{"idx":6,"heading":"Knowledge Embedding in Memories","paragraphs":["The candidate knowledge that we have extracted is represented in a symbolic triplet format, which is intrinsically incompatible with DNNs. This fact urges us to embed the entities and relation of knowledge triples into a continuous vector space. Moreover, we regard each entity-relation-entity triple as one knowledge unit, since each triple naturally represents one piece of fact. The knowledge units can be stored in memory slots for reading and writing, and distilled through an attention mechanism for the subsequent tasks.","In order to embed the symbolic knowledge triples into memory vector slots, we treat the entities and relations as words, and map them into a continuous vector space using word embedding BIBREF30 . Afterwards, the embedded knowledge is encoded into a fixed-size vector by feeding it to a recurrent neural network (RNN). Specifically, we initialize the word-embedding matrix with a pre-trained GloVe word-embedding BIBREF30 , and refine it simultaneously with the rest of procedure of question and candidate answer embedding. In this case, the entities and relations share a common embedding space with other textual elements (questions and answers), which makes them much more flexible to fuse later.","Afterwards, the knowledge triples are treated as SVO phrases of $(subject, verb, object)$ , and fed to to a standard two-layer stacked LSTM as","$$&C^{(t)}_{i} = \\text{LSTM}\\left(\\mathbf {L}[w^{t}_{i}], C^{(t-1)}_{i}\\right), \\\\ & t=\\lbrace 1,2,3\\rbrace , \\text{ and } i=1, \\cdots , N,\\nonumber $$ (Eq. 11)","where $w^{t}_{i}$ is the $t_{\\text{th}}$ word of the $i_{\\text{th}}$ SVO phrase, $(w^{1}_{i},w^{2}_{i},w^{3}_{i}) \\in G^\\ast $ , $\\mathbf {L}$ is the word embedding matrix BIBREF30 , and $C_{i}$ is the internal state of LSTM cell when forwarding the $i_{\\text{th}}$ SVO phrase. The rationale lies in the fact that the LSTM can capture the semantic meanings effectively when the knowledge triples are treated as SVO phrases.","For each question-answering context, we take the LSTM internal states of the relevant knowledge triples as memory vectors, yielding the embedded knowledge stored in memory slots as","$$\\mathbf {M}=\\left[C^{(3)}_{i}\\right],$$ (Eq. 12)","where $\\mathbf {M}(i)$ is the $i_{\\text{th}}$ memory slot corresponding to the $i_{\\text{th}}$ knowledge triples, which can be used for further answer inference. Note that the method is different from the usual knowledge graph embedding models, since our model aims to fuse knowledge with the latent features of images and text, whereas the alternative models such as TransE BIBREF26 focus on link prediction task."]},{"idx":7,"heading":"Attention-based Knowledge Fusion with DNNs","paragraphs":["We have stored $N$ relevant knowledge embeddings in memory slots for a given question-answer context, which allows to incorporate massive knowledge when $N$ is large. The external knowledge overwhelms other contextual information in quantity, making it imperative to distill the useful information from the candidate knowledge. The Dynamic Memory Network (DMN) BIBREF22 , BIBREF25 provides a mechanism to address the problem by modeling interactions among multiple data channels. In the DMN module, an episodic memory vector is formed and updated during an iterative attention process, which memorizes the most useful information for question answering. Moreover, the iterative process brings a potential capability of multi-hop reasoning.","This DMN consists of an attention component which generates a contextual vector using the previous memory vector, and an episodic memory updating component which updates itself based on the contextual vector. Specifically, we propose a novel method to generate the query vector $\\mathbf {q}$ by feeding visual and textual features to a non-linear fully-connected layer to capture question-answer context information as","$$\\mathbf {q} = \\tanh \\left(\\mathbf {W}_{1} \\left[\\mathbf {f}^{(I)};\\mathbf {f}^{(Q)};\\mathbf {f}^{(A)}\\right]+\\mathbf {b}_{1}\\right),$$ (Eq. 14)","where $\\mathbf {W}_1$ and $\\mathbf {b}_{1}$ are the weight matrix and bias vector, respectively; and, $\\mathbf {f}^{(I)}$ , $\\mathbf {f}^{(Q)}$ and $\\mathbf {f}^{(A)}$ are denoted as DNN features corresponding to the images, questions and multi-choice answers, respectively. The query vector $\\mathbf {q}$ captures information from question-answer context. During the training process, the query vector $\\mathbf {q}$ initializes an episodic memory vector $\\mathbf {m}^{(0)}$ as $\\mathbf {m}^{(0)}=\\mathbf {q}$ . A iterative attention process is then triggered, which gradually refines the episodic memory $\\mathbf {m}$ until the maximum number of iterations steps $\\mathbf {b}_{1}$0 is reached. By the $\\mathbf {b}_{1}$1 iteration, the episodic memory $\\mathbf {b}_{1}$2 will memorize useful visual and external information to answer the question. Attention component. At the $\\mathbf {b}_{1}$3 iteration, we concatenate each knowledge embedding $\\mathbf {b}_{1}$4 with last iteration episodic memory $\\mathbf {b}_{1}$5 and query vector $\\mathbf {b}_{1}$6 , then apply the basic soft attention procedure to obtain the $\\mathbf {b}_{1}$7 context vector $\\mathbf {b}_{1}$8 as","$$\\mathbf {z}_{i}^{(t)} &= \\left[\\mathbf {M}_{i};\\mathbf {m}^{(t-1)};\\mathbf {q}\\right] \\\\ \\alpha ^{(t)} &= softmax\\left(\\mathbf {w}\\tanh \\left(\\mathbf {W}_{2}\\mathbf {z}_{i}^{(t)}+\\mathbf {b}_{2}\\right) \\right) \\\\ \\mathbf {c}^{(t)}&=\\sum _{i=1}^{N}\\alpha _{i}^{(t)}\\mathbf {M}_{i} \\quad t=1, \\cdots , T,$$ (Eq. 15)","where $\\mathbf {z}_{i}^{(t)}$ is the concatenated vector for the $i_{\\text{th}}$ candidate memory at the $t_{\\text{th}}$ iteration; $\\alpha _{i}^{(t)}$ is the $i_{\\text{th}}$ element of $\\alpha ^{(t)}$ representing the normalized attention weight for $\\mathbf {M}_{i}$ at the $t_{\\text{th}}$ iteration; and, $\\mathbf {w}$ , $\\mathbf {W}_{2}$ and $i_{\\text{th}}$0 are parameters to be optimized in deep neural networks.","Hereby, we obtain the contextual vector $\\mathbf {c}^{(t)}$ , which captures useful external knowledge for updating episodic memory $\\mathbf {m}^{(t-1)}$ and providing the supporting facts to answer the open-domain questions.","Episodic memory updating component. We apply the memory update mechanism BIBREF21 , BIBREF25 as","$$\\mathbf {m}^{(t)}=ReLU\\left(\\mathbf {W}_{3} \\left[\\mathbf {m}^{(t-1)};\\mathbf {c}^{(t)};\\mathbf {q}\\right]+\\mathbf {b}_{3}\\right),$$ (Eq. 16)","where $\\mathbf {W}_{3}$ and $\\mathbf {b}_{3}$ are parameters to be optimized. After the iteration, the episodic memory $\\mathbf {m}^{(T)}$ memorizes useful knowledge information to answer the open-domain question.","Compared with the DMN+ model implemented in BIBREF25 , we allows the dynamic memory network to incorporate the massive external knowledge into procedure of VQA reasoning. It endows the system with the capability to answer more general visual questions relevant but beyond the image contents, which is more attractive in practical applications.","Fusion with episodic memory and inference. Finally, we embed visual features $\\mathbf {f}^{(I)}$ along with the textual features $\\mathbf {f}^{(Q)}$ and $\\mathbf {f}^{(A)}$ to a common space, and fuse them together using Hadamard product (element-wise multiplication) as","$$&\\mathbf {e}^{(k)}=\\tanh \\left(\\mathbf {W}^{(k)}\\mathbf {f}^{(k)}+\\mathbf {b}^{(k)}\\right), k \\in \\lbrace I, Q, A\\rbrace \\\\ &\\mathbf {h} =\\mathbf {e}^{(I)} \\odot \\mathbf {e}^{(Q)} \\odot \\mathbf {e}^{(A)},$$ (Eq. 17)","where $\\mathbf {e}^{(I)}$ , $\\mathbf {e}^{(Q)}$ and $\\mathbf {e}^{(A)}$ are embedded features for image, question and answer, respectively; $\\mathbf {h}$ is the fused feature in this common space; and, $\\mathbf {W}^{(I)}$ , $\\mathbf {W}^{(Q)}$ and $\\mathbf {W}^{(A)}$ are corresponding to the parameters in neural networks.","The final episodic memory $\\mathbf {m}^{(T)}$ are concatenated with the fused feature $\\mathbf {h}$ to predict the probability of whether the multi-choice candidate answer is correct as","$$ans^* = \\operatornamewithlimits{arg\\,max}_{ans \\in \\lbrace 1,2,3,4\\rbrace } softmax\\left(\\mathbf {W}_{4}\\left[\\mathbf {h}_{ans};\\mathbf {m}^{(T)}_{ans}\\right]+\\mathbf {b}_{4}\\right),$$ (Eq. 18)","where $ans$ represents the index of multi-choice candidate answers; the supported knowledge triples are stored in $\\mathbf {m}^{(T)}_{ans}$ ; and, $\\mathbf {W}_{4}$ and $\\mathbf {b}_{4}$ are the parameters to be optimized in the DNNs. The final choice are consequentially obtained once we have $ans^\\ast $ .","Our training objective is to learn parameters based on a cross-entropy loss function as","$$\\mathcal {L} = -\\frac{1}{D}\\sum _{i}^{D}\\big (y_{i}\\log \\hat{y_{i}}+(1-y_{i})\\log (1-\\hat{y_{i}})\\big ),$$ (Eq. 19)","where $\\hat{y_{i}}=p_{i}(A^{(i)}|I^{(i)},Q^{(i)},K^{(i)};\\theta )$ represents the probability of predicting the answer $A^{(i)}$ , given the $i_{\\text{th}}$ image $I^{(i)}$ , question $Q^{(i)}$ and external knowledge $K^{(i)}$ ; $\\theta $ represents the model parameters; $D$ is the number of training samples; and $y_{i}$ is the label for the $i_{\\text{th}}$ sample. The model can be trained in an end-to-end manner once we have the candidate knowledge triples are retrieved from the original knowledge graph."]},{"idx":8,"heading":"Experiments","paragraphs":["In this section, we conduct extensive experiments to evaluate performance of our proposed model, and compare it with its variants and the alternative methods. We specifically implement the evaluation on a public benchmark dataset (Visual7W) BIBREF7 for the close-domain VQA task, and also generate numerous arbitrary question-answers pairs automatically to evaluate the performance on open-domain VQA. In this section, we first briefly review the dataset and the implementation details, and then report the performance of our proposed method comparing with several baseline models on both close-domain and open-domain VQA tasks."]},{"idx":9,"heading":"Datasets","paragraphs":["We train and evaluate our model on a public available large-scale visual question answering datasets, the Visual7W dataset BIBREF7 , due to the diversity of question types. Besides, since there is no public available open-domain VQA dataset for evaluation now, we automatically build a collection of open-domain visual question-answer pairs to examine the potentiality of our model for answering open-domain visual questions.","The Visual7W dataset BIBREF7 is built based on a subset of images from Visual Genome BIBREF31 , which includes questions in terms of (what, where, when, who, why, which and how) along with the corresponding answers in a multi-choice format. Similar as BIBREF7 , we divide the dataset into training, validation and test subsets, with totally 327,939 question-answer pairs on 47,300 images. Compared with the alternative dataset, Visual7W has a diverse type of question-answer and image content BIBREF13 , which provides more opportunities to assess the human-level capability of a system on the open-domain VQA.","In this paper, we automatically generate numerous question-answer pairs by considering the image content and relevant background knowledge, which provides a test bed for the evaluation of a more realistic VQA task. Specifically, we generate a collection automatically based on the test image in the Visual7W by filling a set of question-answer templates, which means that the information is not present during the training stage. To make the task more challenging, we selectively sample the question-answer pairs that need to reasoning on both visual concept in the image and the external knowledge, making it resemble the scenario of the open-domain visual question answering. In this paper, we generate 16,850 open-domain question-answer pairs on images in Visual7W test split. More details on the QA generation and relevant information can be found in the supplementary material."]},{"idx":10,"heading":"Implementation Details","paragraphs":["In our experiments, we fix the joint-embedding common space dimension as 1024, word-embedding dimension as 300 and the dimension of LSTM internal states as 512. We use a pre-trained ResNet-101 BIBREF32 model to extract image feature, and select 20 candidate knowledge triples for each QA pair through the experiments. Empirical study demonstrates it is sufficient in our task although more knowledge triples are also allowed. The iteration number of a dynamic memory network update is set to 2, and the dimension of episodic memory is set to 2048, which is equal to the dimension of memory slots.","In this paper, we combine the candidate Question-Answer pair to generate a hypothesis, and formulate the multi-choice VQA problem as a classification task. The correct answer can be determined by choosing the one with the largest probability. In each iteration, we randomly sample a batch of 500 QA pairs, and apply stochastic gradient descent algorithm with a base learning rate of 0.0001 to tune the model parameters. The candidate knowledge is first retrieved, and other modules are trained in an end-to-end manner.","In order to analyze the contributions of each component in our knowledge-enhanced, memory-based model, we ablate our full model as follows:","KDMN-NoKG: baseline version of our model. No external knowledge involved in this model. Other parameters are set the same as full model.","KDMN-NoMem: a version without memory network. External knowledge triples are used by one-pass soft attention.","KDMN: our full model. External knowledge triples are incorporated in Dynamic Memory Network.","We also compare our method with several alternative VQA methods including (1) LSTM-Att BIBREF7 , a LSTM model with spatial attention; (2) MemAUG BIBREF33 : a memory-augmented model for VQA; (3) MCB+Att BIBREF6 : a model combining multi-modal features by Multimodal Compact Bilinear pooling; (4) MLAN BIBREF11 : an advanced multi-level attention model."]},{"idx":11,"heading":"Results and Analysis","paragraphs":["In this section, we report the quantitative evaluation along with representative samples of our method, compared with our ablative models and the state-of-the-art method for both the conventional (close-domain) VQA task and open-domain VQA.","In this section, we report the quantitative accuracy in Table 1 along with the sample results in 3 . The overall results demonstrate that our algorithm obtains different boosts compared with the competitors on various kinds of questions, e.g., significant improvements on the questions of Who ( $5.9\\%$ ), and What ( $4.9\\%$ ) questions, and slightly boost on the questions of When ( $1.4\\%$ ) and How ( $2.0\\%$ ). After inspecting the success and failure cases, we found that the Who and What questions have larger diversity in questions and multi-choice answers compared to other types, therefore benefit more from external background knowledge. Note that compared with the method of MemAUG BIBREF33 in which a memory mechanism is also adopted, our algorithm still gain significant improvement, which further confirms our belief that the background knowledge provides critical supports.","We further make comprehensive comparisons among our ablative models. To make it fair, all the experiments are implemented on the same basic network structure and share the same hyper-parameters. In general, our KDMN model on average gains $1.6\\%$ over the KDMN-NoMem model and $4.0\\%$ over the KDMN-NoKG model, which further implies the effectiveness of dynamic memory networks in exploiting external knowledge. Through iterative attention processes, the episodic memory vector captures background knowledge distilled from external knowledge embeddings. The KDMN-NoMem model gains $2.4\\%$ over the KDMN-NoKG model, which implies that the incorporated external knowledge brings additional advantage, and act as a supplementary information for predicting the final answer. The indicative examples in Fig. 3 also demonstrate the impact of external knowledge, such as the 4th example of “Why is the light red?”. It would be helpful if we could retrieve the function of the traffic lights from the external knowledge effectively.","In this section, we report the quantitative performance of open-domain VQA in Table 2 along with the sample results in Fig. 4 . Since most of the alternative methods do not provide the results in the open-domain scenario, we make comprehensive comparison with our ablative models. As expected, we observe that a significant improvement ( $12.7\\%$ ) of our full KDMN model over the KDMN-NoKG model, where $6.8\\%$ attributes to the involvement of external knowledge and $5.9\\%$ attributes to the usage of memory network. Examples in Fig. 4 further provide some intuitive understanding of our algorithm. It is difficult or even impossible for a system to answer the open domain question when comprehensive reasoning beyond image content is required, e.g., the background knowledge for prices of stuff is essential for a machine when inferring the expensive ones. The larger performance improvement on open-domain dataset supports our belief that background knowledge is essential to answer general visual questions. Note that the performance can be further improved if the technique of ensemble is allowed. We fused the results of several KDMN models which are trained from different initializations. Experiments demonstrate that we can further obtain an improvement about $3.1\\%$ ."]},{"idx":12,"heading":"Conclusion","paragraphs":["In this paper, we proposed a novel framework named knowledge-incorporate dynamic memory network (KDMN) to answer open-domain visual questions by harnessing massive external knowledge in dynamic memory network. Context-relevant external knowledge triples are retrieved and embedded into memory slots, then distilled through a dynamic memory network to jointly inference final answer with visual features. The proposed pipeline not only maintains the superiority of DNN-based methods, but also acquires the ability to exploit external knowledge for answering open-domain visual questions. Extensive experiments demonstrate that our method achieves competitive results on public large-scale dataset, and gain huge improvement on our generated open-domain dataset."]},{"idx":13,"heading":"Details of our Open-domain Dataset Generation","paragraphs":["We obey several principles when building the open-domain VQA dataset for evaluation: (1) The question-answer pairs should be generated automatically; (2) Both of visual information and external knowledge should be required when answering these generated open-domain visual questions; (3) The dataset should in multi-choices setting, in accordance with the Visual7W dataset for fair comparison.","The open-domain question-answer pairs are generated based on a subset of images in Visual7W BIBREF7 standard test split, so that the test images are not present during the training stage. For one particular image that we need to generate open-domain question-answer pairs about, we firstly extract several prominent visual objects and randomly select one visual object. After linked to a semantic entity in ConceptNet BIBREF28 , the visual object connects other entities in ConceptNet through various relations, e.g. UsedFor, CapableOf, and forms amount of knowledge triples $(head, relation, tail)$ , where either $head$ or $tail$ is the visual object. Again, we randomly select one knowledge triple, and fill into a $relation$ -related question-answer template to obtain the question-answer pair. These templates assume that the correct answer satisfies knowledge requirement as well as appear in the image, as shown in table 3 .","For each open-domain question-answer pair, we generate three additional confusing items as candidate answers. These candidate answers are randomly sampled from a collection of answers, which is composed of answers from other question-answer pairs belonging to the same $relation$ type. In order to make the open-domain dataset more challenging, we selectively sample confusing answers, which either satisfy knowledge requirement or appear in the image, but not satisfy both of them as the ground-truth answers do. Specifically, one of the confusing answers satisfies knowledge requirement but not appears in image, so that the model must attend to visual objects in image; another one of the confusing answers appears in the image but not satisfies knowledge requirement, so that the model must reason on external knowledge to answer these open-domain questions. Please see examples in Figure 5 .","In total, we generate 16,850 open-domain question-answer pairs based on 8,425 images in Visual7W test split."]},{"idx":14,"heading":"Figures and Tables","paragraphs":["Figure 1: A real case of open-domain visual question answering based on internal representation of an image and external knowledge. Recent success of deep learning provides a good opportunity to implement the closed-domain VQAs, but it is incapable of answering open-domain questions when external knowledge is needed. In this example, the system should recognize the giraffes and then query the knowledge bases for the main diet of giraffes. In this paper, we propose to explore the external knowledge along with the image representation based on a dynamic memory network, which allows a multi-hop reasoning over several facts.","Figure 2: Overall architecture of our proposed KDMN network. Given an image and the corresponding questions, the visual objects of the input image and key words of the corresponding questions are extracted using the Fast-RCNN and syntax analysis, respectively. Afterwards, we propose to assess the importance of entities in the knowledge graph and retrieve the most informative context-relevant knowledge triples, which are fed to the memory network after embedding the candidate knowledge into a continuous feature space. Consequentially, we integrate the representations of images and extracted knowledge into a common space, and store the features in a dynamic memory module. The open-domain VQA can be implemented by interpreting the joint representation under attention mechanism.","Figure 3: Example results on the Visual7W dataset for (close-domain) VQA tasks. Given an image and the corresponding question, we report the corresponding answers obtained via our algorithm. Specifically, pr denotes the predicted probability generated by our model, and pr-NoKG is the predicted probability by the ablative model of KDMN-NoKG. We make the predicted choices bold accordingly. The external knowledge triples are also provided if they are retrieved to support the joint reasoning by our method automatically. As is observed, the external knowledge is essential even for the conventional VQA tasks, e.g., in the 5th example, it is much easier to infer the place accordingly by incorporating external knowledge when a giraffe is recognized.","Figure 4: Example results of open-domain visual question answering based on our proposed knowledge-incorporate dynamic memory network. Given an images, we automatically generate the open-domain question-answer pair by considering of the image content and the relevant background knowledge. We report the corresponding answers obtained via our algorithm. Specifically, pr denotes the predicted probability generated by our model, and pr-NoKG is the predicted probability by the ablative model of KDMN-NoKG. The results demonstrate that external knowledge plays an essential role in answer open-questions. A system is incapable of inferring the food in the 1st example and the stuff prices in the 3rd example.","Table 1: Accuracy on Visual7W dataset","Table 2: Accuracy on our generated open-domain dataset.","Table 3: Templates for generate open-domain question-answer pairs. {visual} is the KG entity representing visual object. {other} is the KG entity that has a connection with {visual}. We take {visual} as the generated ground-truth answer.","Figure 5: Examples from our generated open-domain dataset. We mark ground-truth answers green. The bottom KG triples just provide insights into the generation process, and will not be included in the dataset. The candidate answers can be quite confusing in some questions, e.g., in the 1st example, the ground-truth “candle” appearing in the image can be used for light, while “cake” also appears in the image but cannot be used for light, “sun” can also be used for light but not appear in the image."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1801.02073","title":"Analysis of Wikipedia-based Corpora for Question Answering","sections":[{"idx":0,"heading":"Abstract","paragraphs":["This paper gives comprehensive analyses of corpora based on Wikipedia for several tasks in question answering. Four recent corpora are collected,WikiQA, SelQA, SQuAD, and InfoQA, and first analyzed intrinsically by contextual similarities, question types, and answer categories. These corpora are then analyzed extrinsically by three question answering tasks, answer retrieval, selection, and triggering. An indexing-based method for the creation of a silver-standard dataset for answer retrieval using the entire Wikipedia is also presented. Our analysis shows the uniqueness of these corpora and suggests a better use of them for statistical question answering learning."]},{"idx":1,"heading":"Introduction","paragraphs":["Question answering (QA) has been a blooming research field for the last decade. Selection-based QA implies a family of tasks that find answer contexts from large data given questions in natural language. Three tasks have been proposed for selection-based QA. Given a document, answer extraction BIBREF0 , BIBREF1 finds answer phrases whereas answer selection BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 and answer triggering BIBREF6 , BIBREF7 find answer sentences instead, although the presence of the answer context is not assumed within the provided document for answer triggering but it is for the other two tasks. Recently, various QA tasks that are not selection-based have been proposed BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 ; however, selection-based QA remains still important because of its practical value to real applications (e.g., IBM Watson, MIT Start).","Several datasets have been released for selection-based QA. wang:07a created the QASent dataset consisting of 277 questions, which has been widely used for benchmarking the answer selection task. feng:15a presented InsuranceQA comprising 16K+ questions on insurance contexts. yang:15a introduced WikiQA for answer selection and triggering. jurczyk:16 created SelQA for large real-scale answer triggering. rajpurkar2016squad presented SQuAD for answer extraction and selection as well as for reading comprehension. Finally, morales-EtAl:2016:EMNLP2016 provided InfoboxQA for answer selection.","These corpora make it possible to evaluate the robustness of statistical question answering learning. Although all of these corpora target on selection-based QA, they are designed for different purposes such that it is important to understand the nature of these corpora so a better use of them can be made. In this paper, we make both intrinsic and extrinsic analyses of four latest corpora based on Wikipedia, WikiQA, SelQA, SQuAD, and InfoboxQA. We first give a thorough intrinsic analysis regarding contextual similarities, question types, and answer categories (Section SECREF2 ). We then map questions in all corpora to the current version of English Wikipedia and benchmark another selection-based QA task, answer retrieval (Section SECREF3 ). Finally, we present an extrinsic analysis through a set of experiments cross-testing these corpora using a convolutional neural network architecture (Section SECREF4 )."]},{"idx":2,"heading":"Intrinsic Analysis","paragraphs":["Four publicly available corpora are selected for our analysis. These corpora are based on Wikipedia, so more comparable than the others, and have already been used for the evaluation of several QA systems.","WikiQA BIBREF6 comprises questions selected from the Bing search queries, where user click data give the questions and their corresponding Wikipedia articles. The abstracts of these articles are then extracted to create answer candidates. The assumption is made that if many queries lead to the same article, it must contain the answer context; however, this assumption fails for some occasions, which makes this dataset more challenging. Since the existence of answer contexts is not guaranteed in this task, it is called answer triggering instead of answer selection.","SelQA BIBREF7 is a product of five annotation tasks through crowdsourcing. It consists of about 8K questions where a half of the questions are paraphrased from the other half, aiming to reduce contextual similarities between questions and answers. Each question is associated with a section in Wikipedia where the answer context is guaranteed, and also with five sections selected from the entire Wikipedia where the selection is made by the Lucene search engine. This second dataset does not assume the existence of the answer context, so can be used for the evaluation of answer triggering.","SQuAD BIBREF12 presents 107K+ crowdsourced questions on 536 Wikipedia articles, where the answer contexts are guaranteed to exist within the provided paragraph. It contains annotation of answer phrases as well as the pointers to the sentences including the answer phrases; thus, it can be used for both answer extraction and selection. This corpus also provides human accuracy on those questions, setting up a reasonable upper bound for machines. To avoid overfitting, the evaluation set is not publicly available although system outputs can be evaluated by their provided script.","InfoboxQA BIBREF13 gives 15K+ questions based on the infoboxes from 150 articles in Wikipedia. Each question is crowdsourced and associated with an infobox, where each line of the infobox is considered an answer candidate. This corpus emphasizes the gravity of infoboxes, which summary arguably the most commonly asked information about those articles. Although the nature of this corpus is different from the others, it can also be used to evaluate answer selection."]},{"idx":3,"heading":"Analysis","paragraphs":["All corpora provide datasets/splits for answer selection, whereas only (WikiQA, SQuAD) and (WikiQA, SelQA) provide datasets for answer extraction and answer triggering, respectively. SQuAD is much larger in size although questions in this corpus are often paraphrased multiple times. On the contrary, SQuAD's average candidates per question ( INLINEFORM0 ) is the smallest because SQuAD extracts answer candidates from paragraphs whereas the others extract them from sections or infoboxes that consist of bigger contexts. Although InfoboxQA is larger than WikiQA or SelQA, the number of token types ( INLINEFORM1 ) in InfoboxQA is smaller than those two, due to the repetitive nature of infoboxes.","All corpora show similar average answer candidate lengths ( INLINEFORM0 ), except for InfoboxQA where each line in the infobox is considered a candidate. SelQA and SQuAD show similar average question lengths ( INLINEFORM1 ) because of the similarity between their annotation schemes. It is not surprising that WikiQA's average question length is the smallest, considering their questions are taken from search queries. InfoboxQA's average question length is relatively small, due to the restricted information that can be asked from the infoboxes. InfoboxQA and WikiQA show the least question-answer word overlaps over questions and answers ( INLINEFORM2 and INLINEFORM3 in Table TABREF2 ), respectively. In terms of the F1-score for overlapping words ( INLINEFORM4 ), SQuAD gives the least portion of overlaps between question-answer pairs although WikiQA comes very close.","Fig. FIGREF4 shows the distributions of seven question types grouped deterministically from the lexicons. Although these corpora have been independently developed, a general trend is found, where the what question type dominates, followed by how and who, followed by when and where, and so on.","Fig. FIGREF6 shows the distributions of answer categories automatically classified by our Convolutional Neural Network model trained on the data distributed by li:02a. Interestingly, each corpus focuses on different categories, Numeric for WikiQA and SelQA, Entity for SQuAD, and Person for InfoboxQA, which gives enough diversities for statistical learning to build robust models."]},{"idx":4,"heading":"Answer Retrieval","paragraphs":["This section describes another selection-based QA task, called answer retrieval, that finds the answer context from a larger dataset, the entire Wikipedia. SQuAD provides no mapping of the answer contexts to Wikipedia, whereas WikiQA and SelQA provide mappings; however, their data do not come from the same version of Wikipedia. We propose an automatic way of mapping the answer contexts from all corpora to the same version of Wikipeda so they can be coherently used for answer retrieval.","Each paragraph in Wikipedia is first indexed by Lucene using {1,2,3}-grams, where the paragraphs are separated by WikiExtractor and segmented by NLP4J (28.7M+ paragraphs are indexed). Each answer sentence from the corpora in Table TABREF3 is then queried to Lucene, and the top-5 ranked paragraphs are retrieved. The cosine similarity between each sentence in these paragraphs and the answer sentence is measured for INLINEFORM0 -grams, say INLINEFORM1 . A weight is assigned to each INLINEFORM2 -gram score, say INLINEFORM3 , and the weighted sum is measured: INLINEFORM4 . The fixed weights of INLINEFORM5 are used for our experiments, which can be improved.","If there exists a sentence whose INLINEFORM0 , the paragraph consisting of that sentence is considered the silver-standard answer passage. Table TABREF3 shows how robust these silver-standard passages are based on human judgement ( INLINEFORM1 ) and how many passages are collected ( INLINEFORM2 ) for INLINEFORM3 , where the human judgement is performed on 50 random samples for each case. For answer retrieval, a dataset is created by INLINEFORM4 , which gives INLINEFORM5 accuracy and INLINEFORM6 coverage, respectively. Finally, each question is queried to Lucene and the top- INLINEFORM7 paragraphs are retrieved from the entire Wikipedia. If the answer sentence exists within those retrieved paragraphs according to the silver-standard, it is considered correct.","Finding a paragraph that includes the answer context out of the entire Wikipedia is an extremely difficult task (128.7M). The last row of Table TABREF3 shows results from answer retrieval. Given INLINEFORM0 , SelQA and SQuAD show about 34% and 35% accuracy, which are reasonable. However, WikiQA shows a significantly lower accuracy of 12.47%; this is because the questions in WikiQA is about twice shorter than the questions in the other corpora such that not enough lexicons can be extracted from these questions for the Lucene search."]},{"idx":5,"heading":"Answer Selection","paragraphs":["Answer selection is evaluated by two metrics, mean average precision (MAP) and mean reciprocal rank (MRR). The bigram CNN introduced by yu:14a is used to generate all the results in Table TABREF11 , where models are trained on either single or combined datasets. Clearly, the questions in WikiQA are the most challenging, and adding more training data from the other corpora hurts accuracy due to the uniqueness of query-based questions in this corpus. The best model is achieved by training on W+S+Q for SelQA; adding InfoboxQA hurts accuracy for SelQA although it gives a marginal gain for SQuAD. Just like WikiQA, InfoboxQA performs the best when it is trained on only itself. From our analysis, we suggest that to use models trained on WikiQA and InfoboxQA for short query-like questions, whereas to use ones trained on SelQA and SQuAD for long natural questions."]},{"idx":6,"heading":"Answer Triggering","paragraphs":["The results of INLINEFORM0 from the answer retrieval task in Section SECREF13 are used to create the datasets for answer triggering, where about 65% of the questions are not expected to find their answer contexts from the provided paragraphs for SelQA and SQuAD and 87.5% are not expected for WikiQA. Answer triggering is evaluated by the F1 scores as presented in Table TABREF11 , where three corpora are cross validated. The results on WikiQA are pretty low as expected from the poor accuracy on the answer retrieval task. Training on SelQA gives the best models for both WikiQA and SelQA. Training on SQuAD gives the best model for SQuAD although the model trained on SelQA is comparable. Since the answer triggering datasets are about 5 times larger than the answer selection datasets, it is computationally too expensive to combine all data for training. We plan to find a strong machine to perform this experiment in near future."]},{"idx":7,"heading":"Related work","paragraphs":["Lately, several deep learning approaches have been proposed for question answering. yu:14a presented a CNN model that recognizes the semantic similarity between two sentences. wang-nyberg:2015:ACL-IJCNLP presented a stacked bidirectional LSTM approach to read words in sequence, then outputs their similarity scores. feng:15a applied a general deep learning framework to non-factoid question answering. santos:16a introduced an attentive pooling mechanism that led to further improvements in selection-based QA."]},{"idx":8,"heading":"Conclusion","paragraphs":["We present a comprehensive comparison study of the existing corpora for selection-based question answering. Our intrinsic analysis provides a better understanding of the uniqueness or similarity between these corpora. Our extrinsic analysis shows the strength or weakness of combining these corpora together for statistical learning. Additionally, we create a silver-standard dataset for answer retrieval and triggering, which will be publicly available. In the future, we will explore different ways of improving the quality of our silver-standard datasets by fine-tuning the hyper-parameters."]},{"idx":9,"heading":"Figures and Tables","paragraphs":["Table 1: Comparisons between the four corpora for answer selection. Note that both WIKIQA and SELQA provide separate annotation for answer triggering, which is not shown in this table. The SQUAD column shows statistics excluding the evaluation set, which is not publicly available. AE/AS/AT: annotation for answer extraction/selection/triggering, q/c: # of questions/answer candidates, w/t: # of tokens/token types, µq/c: average length of questions/answer candidates, Ωq/a: macro average in % of overlapping words between question-answer pairs normalized by the questions/answers lengths, Ωf : (2·Ωq·Ωa)/(Ωq+Ωa).","Table 2: Statistics of the silver-standard dataset (first three rows) and the accuracies of answer retrieval in % (last row). ρ: robustness of the silver-standard in %, γc/p: #/% of retrieved silver-standard passages (coverage).","Figure 1: Distributions of question types in %.","Figure 2: Distributions of answer categories in %.","Table 3: Results for answer selection and triggering in % trained and evaluated across all corpora splits. The first column shows the training source, and the other columns show the evaluation sources. W: WIKIQA, S: SELQA, Q: SQUAD, I: INFOBOXQA."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1803.02914","title":"Translating Questions into Answers using DBPedia n-triples","sections":[{"idx":0,"heading":"Abstract","paragraphs":["In this paper we present a question answering system using a neural network to interpret questions learned from the DBpedia repository. We train a sequence-to-sequence neural network model with n-triples extracted from the DBpedia Infobox Properties. Since these properties do not represent the natural language, we further used question-answer dialogues from movie subtitles. Although the automatic evaluation shows a low overlap of the generated answers compared to the gold standard set, a manual inspection of the showed promising outcomes from the experiment for further work."]},{"idx":1,"heading":"Introduction","paragraphs":["Recent work on sequence-to-sequence neural networks have shown remarkable progress in many domains such as speech recognition, computer vision and language processing. These neural models can be used to map a sequence to another sequence, which has direct applications in natural language understanding BIBREF0 . One of the major advantages of this framework is that it requires a limited amount of feature engineering while still improving state-of-the-art results. This allows researchers to work on tasks for which domain knowledge may not be available. Question answering (QA) systems can directly benefit from this novel approaches, because it only requires mapping between questions and their answers. Due to the complexity of this mapping, most of the previous research in this domain used a complex pipeline of conventional linguistically-based NLP techniques, such as parsing, part-of-speech tagging and co-reference resolution BIBREF1 .","In this work, we build an open-domain QA system with sequence-to-sequence neural network models. The system was trained with question-answer pairs on the world knowledge represented in the DBpedia repository. Although building an open-domain system allows us to use an extensive amount of data stored in DBpedia, one of the challenges is to store this large amount of knowledge in the neural network architecture. Although these systems generally involve a smaller learning pipeline, they require a significant amount of training data.","Figure FIGREF1 illustrates how a sequence-to-sequence neural network can be trained on question-answering pairs. First, an sequence-to-sequence framework reads the source sentence, i.e. question, using an encoder to build a dense vector, a sequence of non-zero values that represents the meaning of a question. A decoder, processes this vector to predict an answer. In this manner, these encoder-decoder models can capture long-range dependencies in languages, e.g., gender agreements or syntax structures."]},{"idx":2,"heading":"Related Work","paragraphs":["Recent approaches on building QA systems are dominated by the usage of neural networks. BIBREF2 present an approach for conversational modelling, which uses a sequence-to-sequence neural model. Their model predicts the next sentence given the previous sentences for an IT helpdesk domain, as well as for an open-domain trained on a subtitles dataset. For an open-domain dialogue generation, BIBREF3 propose using adversarial training. Therefore, reinforcement learning is used to train the system that produces sequences that are indistinguishable from human-generated dialogue utterances. For this, they jointly train two systems, a generative model, which produces response sequences and a discriminator to distinguish between the human-generated dialogues and the machine-generated ones. The outputs from the discriminator are then used as rewards for the generative model, guiding the system to generate dialogues that mostly resemble human dialogues. BIBREF4 demonstrate an end-to-end neural network model for generative QA. Their model is built on the encoder-decoder framework for sequence-to-sequence learning, while equipped with the ability to query a knowledge-base, which they demonstrate on the Chinese encyclopedia web site. The authors show that the proposed model is capable of generating natural and right answers by referring to the facts in the knowledge base. Additionally to the previous approaches, BIBREF5 extend the hierarchical recurrent encoder-decoder neural network to the open domain dialogue system and demonstrate that this model is competitive with state-of-the-art neural language models and back-off n-gram models. They illustrate limitations of similar approaches and show how the performance can be improved by bootstrapping the learning from a larger question-answer pair corpus and from pre-trained word embeddings. A heuristic that guides the development of neural baseline systems for the extractive QA task is described in BIBREF6 , which serves as guideline for the development of two neural baseline systems. Their RNN-based system, called FastQA, demonstrates good performance for extractive question answering due to the awareness of question words while processing the context. Additionally they introduce a composition function that goes beyond simple bag-of-words modelling. BIBREF7 demonstrate an approach to non-factoid answer generation with a separate component, which bases on BiLSTM to determine the importance of segments in the input. In contrast to other attention-based models, they determine the importance while assuming the independence of questions and candidate answers. BIBREF8 present their Gated-Attention reader for answering cloze-style questions over documents. The reader features a novel multiplicative gating mechanism in combination with a multi-hop architecture, which is based on multiplicative interactions between the query embedding and the intermediate states of a recurrent neural network document reader. This enables the reader to build query-specific representations of tokens in the document for accurate answer selection.","The work mentioned above employ neural networks, but incorporate only a small amount of question-answering pairs to build their QA systems. Work on QA, which uses the DBpedia knowledge on the other hand, use much more complex linguistically-based NLP techniques to generate an answer for a given question. BIBREF9 propose a general purpose a QA system using Wikipedia data as its knowledge source, which can answer wh-interrogated questions. Their QA system includes tasks of named entity tagging, question classification, information retrieval and answer extraction. BIBREF10 present a QA approach on the DBpedia dataset. With predefined templates and dependency parsing on questions they obtain the entity and property mention from the question. They split the QA process into two steps; first they build an entity-centric index to search the target entity, whereby in the second step they expand the property mention with WordNet and ConceptNet to match the target entity with the searched entity property. Similarly, BIBREF11 present a QA system over Linked Data (DBpedia), which focuses on construct a bridge between the users and the Linked Data. Based on the consisting of subject-property-object (SPO) triples, each natural language question firstly is transformed into a triple-based representation (query triple). Then, the corresponding resources in DBpedia, including class, entity, property, are mapped for the phrases in the query triples. Finally, the optimal SPARQL query is generated as the output result. With this approach, their system can not only deal with the single-relation questions but also complex questions containing multi-relations. Differentially to the approaches mentioned above, we combine the neural network approaches with the large number of facts (> 20M n-triples) extracted from the DBpedia repository."]},{"idx":3,"heading":"Experimental Setting","paragraphs":["In this section, we give an overview of the sequence-to-sequence framework used in this experiment as well as information on the datasets used in our experiment. Furthermore, we give insights into the techniques to evaluate the correctness of the provided answers."]},{"idx":4,"heading":"Sequence-to-sequence Neural Network Toolkit","paragraphs":["OpenNMT BIBREF12 is a generic deep learning framework mainly specialised in sequence-to-sequence (seq2seq) models covering a variety of tasks such as machine translation, summarisation, speech processing and question answering. We used the default neural network training parameters, i.e. 2 hidden layers, 500 hidden LSTM units, input feeding enabled, batch size of 64, 0.3 dropout probability and a dynamic learning rate decay. We train the network for 13 epochs and report the results in Section SECREF5 .","A common problem in training a neural network is the computational complexity, which cause that the vocabulary has to be limited to a specific threshold. Because of this the neural network can not learn expressions of rare and unknown words, e.g. domain-specific expressions. Therefore, if the training method does not see a specific word or phrase multiple times during training, it will not learn the interpretation of the word. This challenge is even more evident in sequence-to-sequence models used for summarisation, question answering or machine translation. Therefore the vocabulary is often limited only to 50,000 or 100,000 words (in comparison to 300,000 or more unique words in our training data sets, see Table TABREF5 ). To overcome this limitation, different methods were suggested, i.e. character based neural model BIBREF13 , BIBREF14 or using subword units, e.g. Byte Pair Encoding (BPE). The latter one was successfully adapted for word segmentation specifically for the NMT scenario BIBREF15 . BPE BIBREF16 is a form of data compression that iteratively replaces the most frequent pair of bytes in a sequence with a single, unused byte. Instead of merging frequent pairs of bytes as shown in the original algorithm, characters or character sequences are merged for the purposes of natural language generation. To achieve this, the symbol vocabulary is initialised with the character vocabulary, and each word is represented as a sequence of characters—plus a special end-of-word symbol, which allows to restore the original tokenisation after the generating the answer based on the given question. This process is repeated as many times as new symbols are created."]},{"idx":5,"heading":"Training Datasets for the QA system","paragraphs":["For our QA system we used the DBpedia repository and the Subtitles corpus. Due to the nature of training a sequence-to-sequence neural model, questions and answers need to be aligned. The statistics on the used data are shown in Table TABREF5 .","In our work, we use the DBpedia BIBREF17 repository (version 2016-04). The DBpedia project aims to extract structured knowledge from the knowledge added to the Wikipedia repository. DBpedia allows users to semantically query relationships and properties of Wikipedia resources, including links to other related datasets.","For our experiment on training a QA system, we extracted (see Section SECREF4 ) the knowledge stored in Dbpedia Infobox Properties, which with its n-triple structure represents answers to particular questions. From this dataset we extracted more than 21M question-answer entries.","Since the extracted information form the DBpedia n-triples does not represent a natural language, we added to the extracted n-triples answers with dialogues from the OpenSubtitles dataset BIBREF18 . Since this dataset is stored in an XML structure with time codes, only sentences were extracted, where the first sentence ends with a question mark and the second sentence does not end with a question mark. Additionally, to ensure a better consistency between an question and the answer, the second sentence has to follow the first sentence by less than 20 seconds. From the 14M sentence corpus of question-answer pairs provided by the OpenNMT project, we used 5M dialogue entries to modify the language generation part. Table TABREF10 shows some examples from the OpenSubtitles dataset."]},{"idx":6,"heading":"Evaluation Dataset","paragraphs":["For the automatic evaluation of the QA system, we used 10,000 entries extracted from the DBpedia Infobox Property repository (Table TABREF5 )."]},{"idx":7,"heading":"Evaluation Metrics","paragraphs":["The automatic evaluation of the proposed QA system is based on the correspondence between the generated answer and gold standard. For the automatic evaluation we used the BLEU BIBREF19 , METEOR BIBREF20 and chrF BIBREF21 metrics. BLEU (Bilingual Evaluation Understudy) is calculated for individual generated segments (n-grams) by comparing them with a dataset of of the gold standard. Considering the shortness of the questions, we report besides the four-gram overlap (BLEU) also scores based on the unigram overlap (BLEU-1). Those scores, between 0 and 100 (perfect overlap), are then averaged over the whole evaluation dataset to reach an estimate of the overall quality of the generated answers.","INLINEFORM0","METEOR (Metric for Evaluation of Translation with Explicit ORdering) is based on the harmonic mean of precision and recall, whereby recall is weighted higher than precision. Along with exact word (or phrase) matching it has additional features, i.e. stemming, paraphrasing and synonymy matching. In contrast to BLEU, the metric produces good correlation with human judgement at the sentence or segment level. Differently to the BLEU metric, which penalises the hypothesis, if words were generated with a different inflection, chrF3 is a character n-gram metric, which showed very good correlations with human judgements for morphologically rich languages.","INLINEFORM0","chrP represents the percentage of n-grams in the generated answer which has a counterpart in the gold standard answer, whereby chrR represents the percentage of character n-grams in the gold standard, which are also present in the predicted answer. INLINEFORM0 is a parameter which assigns INLINEFORM1 times more importance to recall than to precision."]},{"idx":8,"heading":"Methodology","paragraphs":["The training approach to the sequence-to-sequence neural model requires an aligned dataset of questions and answers, which are aligned on a sentence level. To train our QA system on generic knowledge, we extracted this required information from the DBpedia Infobox Property repository."]},{"idx":9,"heading":"Evaluation","paragraphs":["In this section we report the results of the automatic and a manual evaluation of the answers generated by the neural models."]},{"idx":10,"heading":"Automatic Evaluation","paragraphs":["The results of the automatic evaluation using the BLEU, METEOR and chrF metric are shown in Table TABREF17 . We observer BLEU and Meteor scores, which evaluate the generated answers on based on a n-gram overlap, of 16.48 and 13.97, respectively. BLEU-1 and the chrF score, which perform on a single word or character level, are slightly higher, i.e. 56.0 and 26.84."]},{"idx":11,"heading":"Manual Evaluation","paragraphs":["Besides the automatic evaluation of answer generation over the DBpedia Infobox properties, we performed a detailed manual evaluation of the answers produced by the neural networks.","We first evaluated, which answers, based on the DBpedia properties (of the property-subject pair) were correctly generated based on the METEOR score. Table TABREF18 illustrates that answers, based on the property next single of or basin countries of were mostly generated correctly. On the other hand, answers, based on question containing the properties operating system of or intercommunality of were mostly generated wrongly.","Finally, Table TABREF19 presents different examples of answers, based on the Dbpedia Infobox Properties. Although the properties domain of or capital of appear only once in the evaluation set, they were answered correctly. Furthermore, the neural networks often provide answers, which are synonyms or closely related words to the gold standard answers. As an example, the generated answer lawyer is an synonym word of attorney, and usa can also be referred to as united states only. Similarly, a closely related word to french is france, although this cannot be counted as a correctly generated answer to the question nationality of jean michel diot. At last, the proposed approach of using neural networks also demonstrated various wrongly generated answered. As an example, clark duke is not an football player and edith roger was born in vestby, norway and not in vienna , austria as answered by the proposed neural network approach."]},{"idx":12,"heading":"Conclusions","paragraphs":["We presented the work on a QA system trained with a sequence-to-sequence neural model with the DBpedia knowledge and movie dialogues. Although the automatic evaluation shows a low overlap of generated answers compared to the gold standard, a manual inspection of the showed promising outcomes from the experiment. Due to the nature of the training dataset, short answers are preferred, since they are more likely to have a lower log-likelihood score than the longer ones. Nevertheless, we observed several correct answers, which shows the availability of storing the entire DBpedia knowledge into neural networks. Our future work will focus on providing longer answers, as well as focusing on answering more complex questions."]},{"idx":13,"heading":"Acknowledgement","paragraphs":["This publication has emanated from research conducted with the financial support of Science Foundation Ireland (SFI) under Grant Number SFI/12/RC/2289 (Insight)."]},{"idx":14,"heading":"Figures and Tables","paragraphs":["Figure 1: Neural network with the encoderdecoder architecture.","Table 1: Statistics on the datasets used to train, validate and evaluate the sequence-to-sequence models.","Table 2: Examples of the OpenSubtitles questionanswering sentence pairs.","Table 3: Example of DBpedia n-triples Aristotle and it’s transformation into aligned data format.","Table 4: Automatic evaluation of the neural networks based on the gold standard","Table 5: Statistics on DBpedia Properties, which contributed to the quality of the generated answers based on the METEOR evaluation metric.","Table 6: Examples of generated answers compared to the Dbpedia Infobox Property gold standard (GS)."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1804.00720","title":"Simple and Effective Semi-Supervised Question Answering","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Recent success of deep learning models for the task of extractive Question Answering (QA) is hinged on the availability of large annotated corpora. However, large domain specific annotated corpora are limited and expensive to construct. In this work, we envision a system where the end user specifies a set of base documents and only a few labelled examples. Our system exploits the document structure to create cloze-style questions from these base documents; pre-trains a powerful neural network on the cloze style questions; and further fine-tunes the model on the labeled examples. We evaluate our proposed system across three diverse datasets from different domains, and find it to be highly effective with very little labeled data. We attain more than 50% F1 score on SQuAD and TriviaQA with less than a thousand labelled examples. We are also releasing a set of 3.2M cloze-style questions for practitioners to use while building QA systems."]},{"idx":1,"heading":"Introduction","paragraphs":["Deep learning systems have shown a lot of promise for extractive Question Answering (QA), with performance comparable to humans when large scale data is available. However, practitioners looking to build QA systems for specific applications may not have the resources to collect tens of thousands of questions on corpora of their choice. At the same time, state-of-the-art machine reading systems do not lend well to low-resource QA settings where the number of labeled question-answer pairs are limited (c.f. Table 2 ). Semi-supervised QA methods like BIBREF0 aim to improve this performance by leveraging unlabeled data which is easier to collect.","In this work, we present a semi-supervised QA system which requires the end user to specify a set of base documents and only a small set of question-answer pairs over a subset of these documents. Our proposed system consists of three stages. First, we construct cloze-style questions (predicting missing spans of text) from the unlabeled corpus; next, we use the generated clozes to pre-train a powerful neural network model for extractive QA BIBREF1 , BIBREF2 ; and finally, we fine-tune the model on the small set of provided QA pairs.","Our cloze construction process builds on a typical writing phenomenon and document structure: an introduction precedes and summarizes the main body of the article. Many large corpora follow such a structure, including Wikipedia, academic papers, and news articles. We hypothesize that we can benefit from the un-annotated corpora to better answer various questions – at least ones that are lexically similar to the content in base documents and directly require factual information.","We apply the proposed system on three datasets from different domains – SQuAD BIBREF3 , TriviaQA-Web BIBREF4 and the BioASQ challenge BIBREF5 . We observe significant improvements in a low-resource setting across all three datasets. For SQuAD and TriviaQA, we attain an F1 score of more than 50% by merely using 1% of the training data. Our system outperforms the approaches for semi-supervised QA presented in BIBREF0 , and a baseline which uses the same unlabeled data but with a language modeling objective for pretraining. In the BioASQ challenge, we outperform the best performing system from previous year's challenge, improving over a baseline which does transfer learning from the SQuAD dataset. Our analysis reveals that questions which ask for factual information and match to specific parts of the context documents benefit the most from pretraining on automatically constructed clozes."]},{"idx":2,"heading":"Related Work","paragraphs":["Semi-supervised learning augments the labeled dataset $L$ with a potentially larger unlabeled dataset $U$ . BIBREF0 presented a model, GDAN, which trained an auxiliary neural network to generate questions from passages by reinforcement learning, and augment the labeled dataset with the generated questions to train the QA model. Here we use a much simpler heuristic to generate the auxiliary questions, which also turns out to be more effective as we show superior performance compared to GDAN. Several approaches have been suggested for generating natural questions BIBREF6 , BIBREF7 , BIBREF8 , however none of them show a significant improvement of using the generated questions in a semi-supervised setting. Recent papers also use unlabeled data for QA by training large language models and extracting contextual word vectors from them to input to the QA model BIBREF9 , BIBREF10 , BIBREF11 . The applicability of this method in the low-resource setting is unclear as the extra inputs increase the number of parameters in the QA model, however, our pretraining can be easily applied to these models as well.","Domain adaptation (and Transfer learning) leverage existing large scale datasets from a source domain (or task) to improve performance on a target domain (or task). For deep learning and QA, a common approach is to pretrain on the source dataset and then fine-tune on the target dataset BIBREF12 , BIBREF13 . BIBREF14 used SQuAD as a source for the target BioASQ dataset, and BIBREF15 used Book Test BIBREF16 as source for the target SQuAD dataset. BIBREF17 transfer learned model layers from the tasks of sequence labeling, text classification and relation classification to show small improvements on SQuAD. All these works use manually curated source datatset, which in themselves are expensive to collect. Instead, we show that it is possible to automatically construct the source dataset from the same domain as the target, which turns out to be more beneficial in terms of performance as well (c.f. Section \"Experiments & Results\" ). Several cloze datasets have been proposed in the literature which use heuristics for construction BIBREF18 , BIBREF19 , BIBREF20 . We further see the usability of such a dataset in a semi-supervised setting."]},{"idx":3,"heading":"Methodology","paragraphs":["Our system comprises of following three steps:","Cloze generation: Most of the documents typically follow a template, they begin with an introduction that provides an overview and a brief summary for what is to follow. We assume such a structure while constructing our cloze style questions. When there is no clear demarcation, we treat the first $K\\%$ (hyperparameter, in our case 20%) of the document as the introduction. While noisy, this heuristic generates a large number of clozes given any corpus, which we found to be beneficial for semi-supervised learning despite the noise.","We use a standard NLP pipeline based on Stanford CoreNLP (for SQuAD, TrivaQA and PubMed) and the BANNER Named Entity Recognizer (only for PubMed articles) to identify entities and phrases. Assume that a document comprises of introduction sentences $\\lbrace q_1, q_2, ... q_n\\rbrace $ , and the remaining passages $\\lbrace p_1, p_2, .. p_m\\rbrace $ . Additionally, let's say that each sentence $q_i$ in introduction is composed of words $\\lbrace w_1, w_2, ... w_{l_{q_i}}\\rbrace $ , where $l_{q_i}$ is the length of $q_i$ . We consider a $\\text{match} (q_i, p_j)$ , if there is an exact string match of a sequence of words $\\lbrace w_k, w_{k+1}, .. w_{l_{q_i}}\\rbrace $ between the sentence $q_i$ and passage $p_j$ . If this sequence is either a noun phrase, verb phrase, adjective phrase or a named entity in $\\lbrace p_1, p_2, .. p_m\\rbrace $0 , as recognized by CoreNLP or BANNER, we select it as an answer span $\\lbrace p_1, p_2, .. p_m\\rbrace $1 . Additionally, we use $\\lbrace p_1, p_2, .. p_m\\rbrace $2 as the passage $\\lbrace p_1, p_2, .. p_m\\rbrace $3 and form a cloze question $\\lbrace p_1, p_2, .. p_m\\rbrace $4 from the answer bearing sentence $\\lbrace p_1, p_2, .. p_m\\rbrace $5 by replacing $\\lbrace p_1, p_2, .. p_m\\rbrace $6 with a placeholder. As a result, we obtain passage-question-answer ( $\\lbrace p_1, p_2, .. p_m\\rbrace $7 ) triples (Table 1 shows an example). As a post-processing step, we prune out $\\lbrace p_1, p_2, .. p_m\\rbrace $8 triples where the word overlap between the question (Q) and passage (P) is less than 2 words (after excluding the stop words).","The process relies on the fact that answer candidates from the introduction are likely to be discussed in detail in the remainder of the article. In effect, the cloze question from the introduction and the matching paragraph in the body forms a question and context passage pair. We create two cloze datasets, one each from Wikipedia corpus (for SQuAD and TriviaQA) and PUBMed academic papers (for the BioASQ challenge), consisting of 2.2M and 1M clozes respectively. From analyzing the cloze data manually, we were able to answer 76% times for the Wikipedia set and 80% times for the PUBMed set using the information in the passage. In most cases the cloze paraphrased the information in the passage, which we hypothesized to be a useful signal for the downstream QA task.","We also investigate the utility of forming subsets of the large cloze corpus, where we select the top passage-question-answer triples, based on the different criteria, like i) jaccard similarity of answer bearing sentence in introduction and the passage ii) the tf-idf scores of answer candidates and iii) the length of answer candidates. However, we empirically find that we were better off using the entire set rather than these subsets.","Pre-training: We make use of the generated cloze dataset to pre-train an expressive neural network designed for the task of reading comprehension. We work with two publicly available neural network models – the GA Reader BIBREF2 (to enable comparison with prior work) and BiDAF + Self-Attention (SA) model from BIBREF1 (which is among the best performing models on SQuAD and TriviaQA). After pretraining, the performance of BiDAF+SA on a dev set of the (Wikipedia) cloze questions is 0.58 F1 score and 0.55 Exact Match (EM) score. This implies that the cloze corpus is neither too easy, nor too difficult to answer.","Fine Tuning: We fine tune the pre-trained model, from the previous step, over a small set of labelled question-answer pairs. As we shall later see, this step is crucial, and it only requires a handful of labelled questions to achieve a significant proportion of the performance typically attained by training on tens of thousands of questions."]},{"idx":4,"heading":"Datasets","paragraphs":["We apply our system to three datasets from different domains. SQuAD BIBREF3 consists of questions whose answers are free form spans of text from passages in Wikipedia articles. We follow the same setting as in BIBREF0 , and split $10\\%$ of training questions as the test set, and report performance when training on subsets of the remaining data ranging from $1\\%$ to $90\\%$ of the full set. We also report the performance on the dev set when trained on the full training set ( $1^\\ast $ in Table 2 ). We use the same hyperparameter settings as in prior work. We compare and study four different settings: 1) the Supervised Learning (SL) setting, which is only trained on the supervised data, 2) the best performing GDAN model from BIBREF0 , 3) pretraining on a Language Modeling (LM) objective and fine-tuning on the supervised data, and 4) pretraining on the Cloze dataset and fine-tuning on the supervised data. The LM and Cloze methods use exactly the same data for pretraining, but differ in the loss functions used. We report F1 and EM scores on our test set using the official evaluation scripts provided by the authors of the dataset.","TriviaQA BIBREF4 comprises of over 95K web question-answer-evidence triples. Like SQuAD, the answers are spans of text. Similar to the setting in SQuAD, we create multiple smaller subsets of the entire set. For our semi-supervised QA system, we use the BiDAF+SA model BIBREF1 – the highest performing publicly available system for TrivaQA. Here again, we compare the supervised learning (SL) settings against the pretraining on Cloze set and fine tuning on the supervised set. We report F1 and EM scores on the dev set.","We also test on the BioASQ 5b dataset, which consists of question-answer pairs from PubMed abstracts. We use the publicly available system from BIBREF14 , and follow the exact same setup as theirs, focusing only on factoid and list questions. For this setting, there are only 899 questions for training. Since this is already a low-resource problem we only report results using 5-fold cross-validation on all the available data. We report Mean Reciprocal Rank (MRR) on the factoid questions, and F1 score for the list questions."]},{"idx":5,"heading":"Main Results","paragraphs":["Table 2 shows a comparison of the discussed settings on both SQuAD and TriviaQA. Without any fine-tuning (column 0) the performance is low, probably because the model never saw a real question, but we see significant gains with Cloze pretraining even with very little labeled data. The BiDAF+SA model, exceeds an F1 score of $50\\%$ with only $1\\%$ of the training data (454 questions for SQuAD, and 746 questions for TriviaQA), and approaches $90\\%$ of the best performance with only $10\\%$ labeled data. The gains over the SL setting, however, diminish as the size of the labeled set increases and are small when the full dataset is available.","Cloze pretraining outperforms the GDAN baseline from BIBREF0 using the same SQuAD dataset splits. Additionally, we show improvements in the $90\\%$ data case unlike GDAN. Our approach is also applicable in the extremely low-resource setting of $1\\%$ data, which we suspect GDAN might have trouble with since it uses the labeled data to do reinforcement learning. Furthermore, we are able to use the same cloze dataset to improve performance on both SQuAD and TriviaQA datasets. When we use the same unlabeled data to pre-train with a language modeling objective, the performance is worse, showing the bias we introduce by constructing clozes is important.","On the BioASQ dataset (Table 3 ) we again see a significant improvement when pretraining with the cloze questions over the supervised baseline. The improvement is smaller than what we observe with SQuAD and TriviaQA datasets – we believe this is because questions are generally more difficult in BioASQ. BIBREF14 showed that pretraining on SQuAD dataset improves the downstream performance on BioASQ. Here, we show a much larger improvement by pretraining on cloze questions constructed in an unsupervised manner from the same domain."]},{"idx":6,"heading":"Analysis","paragraphs":["Regression Analysis: To understand which types of questions benefit from pre-training, we pre-specified certain features (see Figure 1 right) for each of the dev set questions in SQuAD, and then performed linear regression to predict the F1 score for that question from these features. We predict the F1 scores from the cloze pretrained model ( $y^{\\text{cloze}}$ ), the supervised model ( $y^{\\text{sl}}$ ), and the difference of the two ( $y^{\\text{cloze}}-y^{\\text{sl}}$ ), when using $10\\%$ of labeled data. The coefficients of the fitted model are shown in Figure 1 (left) along with their std errors. Positive coefficients indicate that a high value of that feature is predictive of a high F1 score, and a negative coefficient indicates that a small value of that feature is predictive of a high F1 score (or a high difference of F1 scores from the two models in the case of $y^{\\text{cloze}}-y^{\\text{sl}}$ ).","The two strongest effects we observe are that a high lexical overlap between the question and the sentence containing the answer is indicative of high boost with pretraining, and that a high lexical overlap between the question and the whole passage is indicative of the opposite. This is hardly surprising, since our cloze construction process is biased towards questions which have a similar phrasing to the answer sentences in context. Hence, test questions with a similar property are answered correctly after pretraining, whereas those with a high overlap with the whole passage tend to have lower performance. The pretraining also favors questions with short answers because the cloze construction process produces short answer spans. Also passages and questions which consist of tokens infrequent in the SQuAD training corpus receive a large boost after pretraining, since the unlabeled data covers a larger domain.","Performance on question types: Figure 2 shows the average gain in F1 score for different types of questions, when we pretrain on the clozes compared to the supervised case. This analysis is done on the $10\\%$ split of the SQuAD training set. We consider two classifications of each question – one determined on the first word (usually a wh-word) of the question (Figure 2 (bottom)) and one based on the output of a separate question type classifier adapted from BIBREF21 . We use the coarse grain labels namely Abbreviation (ABBR), Entity (ENTY), Description (DESC), Human (HUM), Location (LOC), Numeric (NUM) trained on a Logistic Regression classification system . While there is an improvement across the board, we find that abbreviation questions in particular receive a large boost. Also, \"why\" questions show the least improvement, which is in line with our expectation, since these usually require reasoning or world knowledge which cloze questions rarely require."]},{"idx":7,"heading":"Conclusion","paragraphs":["In this paper, we show that pre-training QA models with automatically constructed cloze questions improves the performance of the models significantly, especially when there are few labeled examples. The performance of the model trained only on the cloze questions is poor, validating the need for fine-tuning. Through regression analysis, we find that pretraining helps with questions which ask for factual information located in a specific part of the context. For future work, we plan to explore the active learning setup for this task – specifically, which passages and / or types of questions can we select to annotate, such that there is a maximum performance gain from fine-tuning. We also want to explore how to adapt cloze style pre-training to NLP tasks other than QA."]},{"idx":8,"heading":"Acknowledgments","paragraphs":["Bhuwan Dhingra is supported by NSF under grants CCF-1414030 and IIS-1250956 and by grants from Google. Danish Pruthi and Dheeraj Rajagopal are supported by the DARPA Big Mechanism program under ARO contract W911NF-14-1-0436."]},{"idx":9,"heading":"Figures and Tables","paragraphs":["Table 1: An example constructed cloze.","Table 2: A holistic view of the performance of our system compared against baseline systems on SQuAD and TriviaQA. Column groups represent different fractions of the training set used for training.","Table 3: 5-fold cross-validation results on BioASQ Task 5b. ∗Our SL experiments showed better performance than what was reported in (Wiese et al., 2017).","Figure 1: Left: Regression coefficients, along with std-errors, when predicting F1 score of cloze model, or sl model, or the difference of the two, from features computed from SQuAD dev set questions. Right: Descriptions of the features.","Figure 2: Performance gain with pretraining for different subsets of question types."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1804.09692","title":"Factors Influencing the Surprising Instability of Word Embeddings","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Despite the recent popularity of word embedding methods, there is only a small body of work exploring the limitations of these representations. In this paper, we consider one aspect of embedding spaces, namely their stability. We show that even relatively high frequency words (100-200 occurrences) are often unstable. We provide empirical evidence for how various factors contribute to the stability of word embeddings, and we analyze the effects of stability on downstream tasks."]},{"idx":1,"heading":"Introduction","paragraphs":["Word embeddings are low-dimensional, dense vector representations that capture semantic properties of words. Recently, they have gained tremendous popularity in Natural Language Processing (NLP) and have been used in tasks as diverse as text similarity BIBREF0 , part-of-speech tagging BIBREF1 , sentiment analysis BIBREF2 , and machine translation BIBREF3 . Although word embeddings are widely used across NLP, their stability has not yet been fully evaluated and understood. In this paper, we explore the factors that play a role in the stability of word embeddings, including properties of the data, properties of the algorithm, and properties of the words. We find that word embeddings exhibit substantial instabilities, which can have implications for downstream tasks.","Using the overlap between nearest neighbors in an embedding space as a measure of stability (see sec:definingStability below for more information), we observe that many common embedding spaces have large amounts of instability. For example, Figure FIGREF1 shows the instability of the embeddings obtained by training word2vec on the Penn Treebank (PTB) BIBREF4 . As expected, lower frequency words have lower stability and higher frequency words have higher stability. What is surprising however about this graph is the medium-frequency words, which show huge variance in stability. This cannot be explained by frequency, so there must be other factors contributing to their instability.","In the following experiments, we explore which factors affect stability, as well as how this stability affects downstream tasks that word embeddings are commonly used for. To our knowledge, this is the first study comprehensively examining the factors behind instability."]},{"idx":2,"heading":"Related Work","paragraphs":["There has been much recent interest in the applications of word embeddings, as well as a small, but growing, amount of work analyzing the properties of word embeddings.","Here, we explore three different embedding methods: PPMI BIBREF6 , word2vec BIBREF7 , and GloVe BIBREF8 . Various aspects of the embedding spaces produced by these algorithms have been previously studied. Particularly, the effect of parameter choices has a large impact on how all three of these algorithms behave BIBREF9 . Further work shows that the parameters of the embedding algorithm word2vec influence the geometry of word vectors and their context vectors BIBREF10 . These parameters can be optimized; Hellrich and Hahn ( BIBREF11 ) posit optimal parameters for negative sampling and the number of epochs to train for. They also demonstrate that in addition to parameter settings, word properties, such as word ambiguity, affect embedding quality.","In addition to exploring word and algorithmic parameters, concurrent work by Antoniak and Mimno ( BIBREF12 ) evaluates how document properties affect the stability of word embeddings. We also explore the stability of embeddings, but focus on a broader range of factors, and consider the effect of stability on downstream tasks. In contrast, Antoniak and Mimno focus on using word embeddings to analyze language BIBREF13 , rather than to perform tasks.","At a higher level of granularity, Tan et al. ( BIBREF14 ) analyze word embedding spaces by comparing two spaces. They do this by linearly transforming one space into another space, and they show that words have different usage properties in different domains (in their case, Twitter and Wikipedia).","Finally, embeddings can be analyzed using second-order properties of embeddings (e.g., how a word relates to the words around it). Newman-Griffis and Fosler-Lussier ( BIBREF15 ) validate the usefulness of second-order properties, by demonstrating that embeddings based on second-order properties perform as well as the typical first-order embeddings. Here, we use second-order properties of embeddings to quantify stability."]},{"idx":3,"heading":"Defining Stability","paragraphs":["We define stability as the percent overlap between nearest neighbors in an embedding space. Given a word INLINEFORM0 and two embedding spaces INLINEFORM1 and INLINEFORM2 , take the ten nearest neighbors of INLINEFORM3 in both INLINEFORM4 and INLINEFORM5 . Let the stability of INLINEFORM6 be the percent overlap between these two lists of nearest neighbors. 100% stability indicates perfect agreement between the two embedding spaces, while 0% stability indicates complete disagreement. In order to find the ten nearest neighbors of a word INLINEFORM7 in an embedding space INLINEFORM8 , we measure distance between words using cosine similarity. This definition of stability can be generalized to more than two embedding spaces by considering the average overlap between two sets of embedding spaces. Let INLINEFORM12 and INLINEFORM13 be two sets of embedding spaces. Then, for every pair of embedding spaces INLINEFORM14 , where INLINEFORM15 and INLINEFORM16 , take the ten nearest neighbors of INLINEFORM17 in both INLINEFORM18 and INLINEFORM19 and calculate percent overlap. Let the stability be the average percent overlap over every pair of embedding spaces INLINEFORM20 .","Consider an example using this metric. Table TABREF4 shows the top ten nearest neighbors for the word international in three randomly initialized word2vec embedding spaces trained on the NYT Arts domain (see Section SECREF11 for a description of this corpus). These models share some similar words, such as metropolitan and national, but there are also many differences. On average, each pair of models has four out of ten words in common, so the stability of international across these three models is 40%.","The idea of evaluating ten best options is also found in other tasks, like lexical substitution BIBREF16 and word association BIBREF17 , where the top ten results are considered in the final evaluation metric. To give some intuition for how changing the number of nearest neighbors affects our stability metric, consider Figure FIGREF5 . This graph shows how the stability of GloVe changes with the frequency of the word and the number of neighbors used to calculate stability; please see the figure caption for a more detailed explanation of how this graph is structured. Within each frequency bucket, the stability is consistent across varying numbers of neighbors. Ten nearest neighbors performs approximately as well as a higher number of nearest neighbors (e.g., 100). We see this pattern for low frequency words as well as for high frequency words. Because the performance does not change substantially by increasing the number of nearest neighbors, it is computationally less intensive to use a small number of nearest neighbors. We choose ten nearest neighbors as our metric throughout the rest of the paper."]},{"idx":4,"heading":"Factors Influencing Stability","paragraphs":["As we saw in Figure FIGREF1 , embeddings are sometimes surprisingly unstable. To understand the factors behind the (in)stability of word embeddings, we build a regression model that aims to predict the stability of a word given: (1) properties related to the word itself; (2) properties of the data used to train the embeddings; and (3) properties of the algorithm used to construct these embeddings. Using this regression model, we draw observations about factors that play a role in the stability of word embeddings."]},{"idx":5,"heading":"Methodology","paragraphs":["We use ridge regression to model these various factors BIBREF18 . Ridge regression regularizes the magnitude of the model weights, producing a more interpretable model than non-regularized linear regression. This regularization mitigates the effects of multicollinearity (when two features are highly correlated). Specifically, given INLINEFORM0 ground-truth data points with INLINEFORM1 extracted features per data point, let INLINEFORM2 be the features for sample INLINEFORM3 and let INLINEFORM4 be the set of labels. Then, ridge regression learns a set of weights INLINEFORM5 by minimizing the least squares function with INLINEFORM6 regularization, where INLINEFORM7 is a regularization constant: INLINEFORM8","We set INLINEFORM0 . In addition to ridge regression, we tried non-regularized linear regression. We obtained comparable results, but many of the weights were very large or very small, making them hard to interpret.","The goodness of fit of a regression model is measured using the coefficient of determination INLINEFORM0 . This measures how much variance in the dependent variable INLINEFORM1 is captured by the independent variables INLINEFORM2 . A model that always predicts the expected value of INLINEFORM3 , regardless of the input features, will receive an INLINEFORM4 score of 0. The highest possible INLINEFORM5 score is 1, and the INLINEFORM6 score can be negative.","Given this model, we create training instances by observing the stability of a large number of words across various combinations of two embedding spaces. Specifically, given a word INLINEFORM0 and two embedding spaces INLINEFORM1 and INLINEFORM2 , we encode properties of the word INLINEFORM3 , as well as properties of the datasets and the algorithms used to train the embedding spaces INLINEFORM4 and INLINEFORM5 . The target value associated with this features is the stability of the word INLINEFORM6 across embedding spaces INLINEFORM7 and INLINEFORM8 . We repeat this process for more than 2,500 words, several datasets, and three embedding algorithms.","Specifically, we consider all the words present in all seven of the data domains we are using (see Section SECREF11 ), 2,521 words in total. Using the feature categories described below, we generate a feature vector for each unique word, dataset, algorithm, and dimension size, resulting in a total of 27,794,025 training instances. To get good average estimates for each embedding algorithm, we train each embedding space five times, randomized differently each time (this does not apply to PPMI, which has no random component). We then train a ridge regression model on these instances. The model is trained to predict the stability of word INLINEFORM0 across embedding spaces INLINEFORM1 and INLINEFORM2 (where INLINEFORM3 and INLINEFORM4 are not necessarily trained using the same algorithm, parameters, or training data). Because we are using this model to learn associations between certain features and stability, no test data is necessary. The emphasis is on the model itself, not on the model's performance on a specific task.","We describe next each of the three main categories of factors examined in the model. An example of these features is given in Table TABREF7 ."]},{"idx":6,"heading":"Word Properties","paragraphs":["We encode several features that capture attributes of the word INLINEFORM0 . First, we use the primary and secondary part-of-speech (POS) of the word. Both of these are represented as bags-of-words of all possible POS, and are determined by looking at the primary (most frequent) and secondary (second most frequent) POS of the word in the Brown corpus BIBREF20 . If the word is not present in the Brown corpus, then all of these POS features are set to zero.","To get a coarse-grained representation of the polysemy of the word, we consider the number of different POS present. For a finer-grained representation, we use the number of different WordNet senses associated with the word BIBREF21 , BIBREF22 .","We also consider the number of syllables in a word, determined using the CMU Pronuncing Dictionary BIBREF23 . If the word is not present in the dictionary, then this is set to zero."]},{"idx":7,"heading":"Data Properties","paragraphs":["Data features capture properties of the training data (and the word in relation to the training data). For this model, we gather data from two sources: New York Times (NYT) BIBREF24 and Europarl BIBREF25 . Overall, we consider seven domains of data: (1) NYT - U.S., (2) NYT - New York and Region, (3) NYT - Business, (4) NYT - Arts, (5) NYT - Sports, (6) All of the data from domains 1-5 (denoted “All NYT\"), and (7) All of English Europarl. Table TABREF10 shows statistics about these datasets. The first five domains are chosen because they are the top five most common categories of news articles present in the NYT corpus. They are smaller than “All NYT\" and Europarl, and they have a narrow topical focus. The “All NYT\" domain is more diverse topically and larger than the first five domains. Finally, the Europarl domain is the largest domain, and it is focused on a single topic (European Parliamentary politics). These varying datasets allow us to consider how data-dependent properties affect stability.","We use several features related to domain. First, we consider the raw frequency of word INLINEFORM0 in both the domain of data used for embedding space INLINEFORM1 and the domain of data for space INLINEFORM2 . To make our regression model symmetric, we effectively encode three features: the higher raw frequency (between the two), the lower raw frequency, and the absolute difference in raw frequency.","We also consider the vocabulary size of each corpus (again, symmetrically) and the percent overlap between corpora vocabulary, as well as the domain of each of the two corpora, represented as a bag-of-words of domains. Finally, we consider whether the two corpora are from the same domain.","Our final data-level features explore the role of curriculum learning in stability. It has been posited that the order of the training data affects the performance of certain algorithms, and previous work has shown that for some neural network-based tasks, a good training data order (curriculum learning strategy) can improve performance BIBREF26 . Curriculum learning has been previously explored for word2vec, where it has been found that optimizing training data order can lead to small improvements on common NLP tasks BIBREF1 . Of the embedding algorithms we consider, curriculum learning only affects word2vec. Because GloVe and PPMI use the data to learn a complete matrix before building embeddings, the order of the training data will not affect their performance. To measure the effects of training data order, we include as features the first appearance of word INLINEFORM0 in the dataset for embedding space INLINEFORM1 and the first appearance of INLINEFORM2 in the dataset for embedding space INLINEFORM3 (represented as percentages of the total number of training sentences). We further include the absolute difference between these percentages."]},{"idx":8,"heading":"Algorithm Properties","paragraphs":["In addition to word and data properties, we encode features about the embedding algorithms. These include the different algorithms being used, as well as the different parameter settings of these algorithms. Here, we consider three embedding algorithms, word2vec, GloVe, and PPMI. The choice of algorithm is represented in our feature vector as a bag-of-words.","PPMI creates embeddings by first building a positive pointwise mutual information word-context matrix, and then reducing the dimensionality of this matrix using SVD BIBREF6 . A more recent word embedding algorithm, word2vec (skip-gram model) BIBREF7 uses a shallow neural network to learn word embeddings by predicting context words. Another recent method for creating word embeddings, GloVe, is based on factoring a matrix of ratios of co-occurrence probabilities BIBREF8 .","For each algorithm, we choose common parameter settings. For word2vec, two of the parameters that need to be chosen are window size and minimum count. Window size refers to the maximum distance between the current word and the predicted word (e.g., how many neighboring words to consider for each target word). Any word appearing less than the minimum count number of times in the corpus is discarded and not considered in the word2vec algorithm. For both of these features, we choose standard parameter settings, namely, a window size of 5 and a minimum count of 5. For GloVe, we also choose standard parameters. We use 50 iterations of the algorithm for embedding dimensions less than 300, and 100 iterations for higher dimensions.","We also add a feature reflecting the embedding dimension, namely one of five embedding dimensions: 50, 100, 200, 400, or 800."]},{"idx":9,"heading":"Lessons Learned: What Contributes to the Stability of an Embedding","paragraphs":["Overall, the regression model achieves a coefficient of determination ( INLINEFORM0 ) score of 0.301 on the training data, which indicates that the regression has learned a linear model that reasonably fits the training data given. Using the regression model, we can analyze the weights corresponding to each of the features being considered, shown in Table TABREF14 . These weights are difficult to interpret, because features have different distributions and ranges. However, we make several general observations about the stability of word embeddings.","Observation 1. Curriculum learning is important. This is evident because the top two features (by magnitude) of the regression model capture where the word first appears in the training data. Figure FIGREF15 shows trends between training data position and stability in the PTB. This figure contrasts word2vec with GloVe (which is order invariant).","To further understand the effect of curriculum learning on the model, we train a regression model with all of the features except the curriculum learning features. This model achieves an INLINEFORM0 score of 0.291 (compared to the full model's score of 0.301). This indicates that curriculum learning is a factor in stability.","Observation 2. POS is one of the biggest factors in stability. Table TABREF14 shows that many of the top weights belong to POS-related features (both primary and secondary POS). Table TABREF18 compares average stabilities for each primary POS. Here we see that the most stable POS are numerals, verbs, and determiners, while the least stable POS are punctuation marks, adpositions, and particles.","Observation 3. Stability within domains is greater than stability across domains. Table TABREF14 shows that many of the top factors are domain-related. Figure FIGREF19 shows the results of the regression model broken down by domain. This figure shows the highest stabilities appearing on the diagonal of the matrix, where the two embedding spaces both belong to the same domain. The stabilities are substantially lower off the diagonal.","Figure FIGREF19 also shows that “All NYT\" generalizes across the other NYT domains better than Europarl, but not as well as in-domain data (“All NYT\" encompasses data from US, NY, Business, Arts, and Sports). This is true even though Europarl is much larger than “All NYT\".","Observation 4. Overall, GloVe is the most stable embedding algorithm. This is particularly apparent when only in-domain data is considered, as in Figure FIGREF19 . PPMI achieves similar stability, while word2vec lags considerably behind.","To further compare word2vec and GloVe, we look at how the stability of word2vec changes with the frequency of the word and the number of neighbors used to calculate stability. This is shown in Figure FIGREF20 and is directly comparable to Figure FIGREF5 . Surprisingly, the stability of word2vec varies substantially with the frequency of the word. For lower-frequency words, as the number of nearest neighbors increases, the stability increases approximately exponentially. For high-frequency words, the lowest and highest number of nearest neighbors show the greatest stability. This is different than GloVe, where stability remains reasonably constant across word frequencies, as shown in Figure FIGREF5 . The behavior we see here agrees with the conclusion of BIBREF10 , who find that GloVe exhibits more well-behaved geometry than word2vec.","Observation 5. Frequency is not a major factor in stability. To better understand the role that frequency plays in stability, we run separate ablation experiments comparing regression models with frequency features to regression models without frequency features. Our current model (using raw frequency) achieves an INLINEFORM0 score of 0.301. Comparably, a model using the same features, but with normalized instead of raw frequency, achieves a score of 0.303. Removing frequency from either regression model gives a score of 0.301. This indicates that frequency is not a major factor in stability, though normalized frequency is a larger factor than raw frequency.","Finally, we look at regression models using only frequency features. A model using only raw frequency features has an INLINEFORM0 score of 0.008, while a model with only normalized frequency features has an INLINEFORM1 score of 0.0059. This indicates that while frequency is not a major factor in stability, it is also not negligible. As we pointed out in the introduction, frequency does correlate with stability (Figure FIGREF1 ). However, in the presence of all of these other features, frequency becomes a minor factor."]},{"idx":10,"heading":"Impact of Stability on Downstream Tasks","paragraphs":["Word embeddings are used extensively as the first stage of neural networks throughout NLP. Typically, embeddings are initalized based on a vector trained with word2vec or GloVe and then further modified as part of training for the target task. We study two downstream tasks to see whether stability impacts performance.","Since we are interested in seeing the impact of word vector stability, we choose tasks that have an intuitive evaluation at the word level: word similarity and POS tagging."]},{"idx":11,"heading":"Word Similarity","paragraphs":["To model word similarity, we use 300-dimensional word2vec embedding spaces trained on the PTB. For each pair of words, we take the cosine similarity between those words averaged over ten randomly initialized embedding spaces. We consider three datasets for evaluating word similarity: WS353 (353 pairs) BIBREF27 , MTurk287 (287 pairs) BIBREF28 , and MTurk771 (771 pairs) BIBREF29 . For each dataset, we normalize the similarity to be in the range INLINEFORM0 , and we take the absolute difference between our predicted value and the ground-truth value. Figure FIGREF22 shows the results broken down by stability of the two words (we always consider Word 1 to be the more stable word in the pair). Word similarity pairs where one of the words is not present in the PTB are omitted.","We find that these word similarity datasets do not contain a balanced distribution of words with respect to stability; there are substantially more unstable words than there are stable words. However, we still see a slight trend: As the combined stability of the two words increases, the average absolute error decreases, as reflected by the lighter color of the cells in Figure FIGREF22 while moving away from the (0,0) data point."]},{"idx":12,"heading":"Part-of-Speech Tagging","paragraphs":["Part-of-speech (POS) tagging is a substantially more complicated task than word similarity. We use a bidirectional LSTM implemented using DyNet BIBREF30 . We train nine sets of 128-dimensional word embeddings with word2vec using different random seeds. The LSTM has a single layer and 50-dimensional hidden vectors. Outputs are passed through a tanh layer before classification. To train, we use SGD with a learning rate of 0.1, an input noise rate of 0.1, and recurrent dropout of 0.4.","This simple model is not state-of-the-art, scoring 95.5% on the development set, but the word vectors are a central part of the model, providing a clear signal of their impact. For each word, we group tokens based on stability and frequency. Figure FIGREF24 shows the results. Fixing the word vectors provides a clearer pattern in the results, but also leads to much worse performance: 85.0% on the development set. Based on these results, it seems that training appears to compensate for stability. This hypothesis is supported by Figure FIGREF24 , which shows the similarity between the original word vectors and the shifted word vectors produced by the training. In general, lower stability words are shifted more during training.","Understanding how the LSTM is changing the input embeddings is useful information for tasks with limited data, and it could allow us to improve embeddings and LSTM training for these low-resource tasks."]},{"idx":13,"heading":"Conclusion and Recommendations","paragraphs":["Word embeddings are surprisingly variable, even for relatively high frequency words. Using a regression model, we show that domain and part-of-speech are key factors of instability. Downstream experiments show that stability impacts tasks using embedding-based features, though allowing embeddings to shift during training can reduce this effect. In order to use the most stable embedding spaces for future tasks, we recommend either using GloVe or learning a good curriculum for word2vec training data. We also recommend using in-domain embeddings whenever possible.","The code used in the experiments described in this paper is publicly available from http://lit.eecs.umich.edu/downloads.html."]},{"idx":14,"heading":"Acknowledgments","paragraphs":["We would like to thank Ben King and David Jurgens for helpful discussions about this paper, as well as our anonymous reviewers for useful feedback. This material is based in part upon work supported by the National Science Foundation (NSF #1344257) and the Michigan Institute for Data Science (MIDAS). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF or MIDAS."]},{"idx":15,"heading":"Figures and Tables","paragraphs":["Figure 1: Stability of word2vec as a property of frequency in the PTB. Stability is measured across ten randomized embedding spaces trained on the training portion of the PTB (determined using language modeling splits (Mikolov et al., 2010)). Each word is placed in a frequency bucket (x-axis), and each column (frequency bucket) is normalized.","Table 1: Top ten most similar words for the word international in three randomly intialized word2vec models trained on the NYT Arts Domain. Words in all three lists are in bold; words in only two of the lists are italicized.","Figure 2: Stability of GloVe on the PTB. Stability is measured across ten randomized embedding spaces trained on the training data of the PTB (determined using language modeling splits (Mikolov et al., 2010)). Each word is placed in a frequency bucket (left y-axis) and stability is determined using a varying number of nearest neighbors for each frequency bucket (right yaxis). Each row is normalized, and boxes with more than 0.01 of the row’s mass are outlined.","Table 2: Consider the word international in two embedding spaces. Suppose embedding spaceA is trained using word2vec (embedding dimension 100) on the NYT Arts domain, and embedding space B is trained using PPMI (embedding dimension 100) on Europarl. This table summarizes the resulting features for this word across the two embedding spaces.","Table 3: Dataset statistics.","Table 4: Regression weights with a magnitude greater than 0.1, sorted by magnitude.","Figure 3: Stability of both word2vec and GloVe as properties of the starting word position in the training data of the PTB. Stability is measured across ten randomized embedding spaces trained on the training data of the PTB (determined using language modeling splits (Mikolov et al., 2010)). Boxes with more than 0.02% of the total vocabulary mass are outlined.","Table 5: Percent stability broken down by part-ofspeech, ordered by decreasing stability.","Figure 4: Percent stability broken down by domain.","Figure 5: Percent stability broken down between algorithm (in-domain data only).","Figure 7: Absolute error for word similarity.","Figure 6: Stability of word2vec on the PTB. Stability is measured across ten randomized embedding spaces trained on the training data of the PTB (determined using language modeling splits (Mikolov et al., 2010)). Each word is placed in a frequency bucket (left y-axis) and stability is determined using a varying number of nearest neighbors for each frequency bucket (right yaxis). Each row is normalized, and boxes with more than 0.01 of the row’s mass are outlined.","Figure 8: Results for POS tagging. (a) and (b) show average POS tagging error divided by the number of tokens (darker is more errors) while either keeping word vectors fixed or not during training. (c) shows word vector shift, measured as cosine similarity between initial and final vectors. In all graphs, words are bucketed by frequency and stability."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1805.07882","title":"Sentence Modeling via Multiple Word Embeddings and Multi-level Comparison for Semantic Textual Similarity","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Different word embedding models capture different aspects of linguistic properties. This inspired us to propose a model (M-MaxLSTM-CNN) for employing multiple sets of word embeddings for evaluating sentence similarity/relation. Representing each word by multiple word embeddings, the MaxLSTM-CNN encoder generates a novel sentence embedding. We then learn the similarity/relation between our sentence embeddings via Multi-level comparison. Our method M-MaxLSTM-CNN consistently shows strong performances in several tasks (i.e., measure textual similarity, identify paraphrase, recognize textual entailment). According to the experimental results on STS Benchmark dataset and SICK dataset from SemEval, M-MaxLSTM-CNN outperforms the state-of-the-art methods for textual similarity tasks. Our model does not use hand-crafted features (e.g., alignment features, Ngram overlaps, dependency features) as well as does not require pre-trained word embeddings to have the same dimension."]},{"idx":1,"heading":"Introduction","paragraphs":["Measuring the semantic similarity/relation of two pieces of short text plays a fundamental role in a variety of language processing tasks (i.e., plagiarism detection, question answering, and machine translation). Semantic textual similarity (STS) task is challenging because of the diversity of linguistic expression. For example, two sentences with different lexicons could have a similar meaning. Moreover, the task requires to measure similarity at several levels (e.g., word level, phrase level, sentence level). These challenges give difficulties to conventional approaches using hand-crafted features.","Recently, the emergence of word embedding techniques, which encode the semantic properties of a word into a low dimension vector, leads to the successes of many learning models in natural language processing (NLP). For example, BIBREF0 randomly initialize word vectors, then tunes them during the training phase of a sentence classification task. By contrast, BIBREF1 initialize word vectors via the pre-train word2vec model trained on Google News BIBREF2 . BIBREF3 train a word embedding model on the paraphrase dataset PPDB, then apply the word representation for word and bi-gram similarity tasks.","Several pre-trained word embeddings are available, which are trained on various corpora under different models. BIBREF4 observed that different word embedding models capture different aspects of linguistic properties: a Bag-of-Words contexts based model tends to reflect the domain aspect (e.g., scientist and research) while a paraphrase-relationship based model captures semantic similarities of words (e.g., boy and kid). From experiments, we also observed that the performance of a word embedding model is usually inconsistent over different datasets. This inspired us to develop a model taking advantages of various pre-trained word embeddings for measuring textual similarity/relation.","In this paper, we propose a convolutional neural network (CNN) to learn a multi-aspect word embedding from various pre-trained word embeddings. We then apply the max-pooling scheme and Long Short Term Memory (LSTM) on this embedding to form a sentence representation. In STS tasks, BIBREF5 shows the efficiency of the max-pooling scheme in modeling sentences from word embedding representations refined via CNN. However, the max-pooling scheme lacks the property of word order (e.g., sentence(“Bob likes Marry”) = sentence(“Marry likes Bob”)). To address this weakness, we use LSTM as an additional scheme for modeling sentences with word order characteristics. For measuring the similarity/relation between two sentence representations, we propose Multi-level comparison which consists of word-word level, sentence-sentence level, and word-sentence level. Through these levels, our model comprehensively evaluates the similarity/relation between two sentences.","We evaluate our M-MaxLSTM-CNN model on three tasks: STS, textual entailment recognition, paraphrase identification. The advantages of M-MaxLSTM-CNN are: i) simple but efficient for combining various pre-trained word embeddings with different dimensions; ii) using Multi-level comparison shows better performances compared to using only sentence-sentence comparison; iii) does not require hand-crafted features (e.g., alignment features, Ngram overlaps, syntactic features, dependency features) compared to the state-of-the-art ECNU BIBREF6 on STS Benchmark dataset.","Our main contributions are as follows:","The remainder of this paper is organized as follows: Section 2 reviews the previous research, Section 3 introduces the architecture of our model, Section 4 describes the three tasks and datasets, Section 5 describes the experiment setting, Section 6 reports and discusses the results of the experiments, and Section 7 concludes our work."]},{"idx":2,"heading":"Related work","paragraphs":["Most prior research on modeling textual similarity relied on feature engineering. BIBREF7 extract INLINEFORM0 -gram overlap features and dependency-based features, while BIBREF8 employ features based on machine translation metrics. BIBREF9 propose a method using corpus-based and knowledge-based measures of similarity. BIBREF10 design a model which incorporates both syntax and lexical semantics using dependency grammars. BIBREF11 combine the fine-grained n-gram overlap features with the latent representation from matrix factorization. BIBREF12 develop a latent variable model which jointly learns paraphrase relations between word and sentence pairs. Using Dependency trees, BIBREF13 propose a robust monolingual aligner and successfully applied it for STS tasks.","The recent emergence of deep learning models has provided an efficient way to learn continuous vectors representing words/sentences. By using a neural network in the context of a word prediction task, BIBREF14 and BIBREF15 generate word embedding vectors carrying semantic meanings. The embedding vectors of words which share similar meanings are close to each other. To capture the morphology of words, BIBREF16 enrich the word embedding with character n-grams information. Closest to this approach, BIBREF17 also propose to represent a word or sentence using a character n-gram count vector. However, the objective function for learning these embeddings is based on paraphrase pairs.","For modeling sentences, composition approach attracted many studies. BIBREF18 model each word as a matrix and used iterated matrix multiplication to present a phrase. BIBREF19 design a Dependency Tree-Structured LSTM for modeling sentences. This model outperforms the linear chain LSTM in STS tasks. Convolutional neural network (CNN) has recently been applied efficiently for semantic composition BIBREF0 , BIBREF20 , BIBREF5 . This technique uses convolutional filters to capture local dependencies in term of context windows and applies a pooling layer to extract global features. BIBREF21 use CNN to extract features at multiple level of granularity. The authors then compare their sentence representations via multiple similarity metrics at several granularities. BIBREF22 propose a hierarchical CNN-LSTM architecture for modeling sentences. In this approach, CNN is used as an encoder to encode an sentence into a continuous representation, and LSTM is used as a decoder. BIBREF23 train a sentence encoder on a textual entailment recognition database using a BiLSTM-Maxpooling network. This encoder achieves competitive results on a wide range of transfer tasks.","At SemEval-2017 STS task, hybrid approaches obtain strong performances. BIBREF24 train a linear regression model with WordNet, alignment features and the word embedding word2vec. BIBREF6 develop an ensemble model with multiple boosting techniques (i.e., Random Forest, Gradient Boosting, and XGBoost). This model incorporates traditional features (i.e., n-gram overlaps, syntactic features, alignment features, bag-of-words) and sentence modeling methods (i.e., Averaging Word Vectors, Projecting Averaging Word Vectors, LSTM).","MVCNN model BIBREF25 and MGNC-CNN model BIBREF26 are close to our approach. In MVCNN, the authors use variable-size convolution filters on various pre-trained word embeddings for extracting features. However, MVCNN requires word embeddings to have the same size. In MGNC-CNN, the authors apply independently CNN on each pre-trained word embedding for extracting features and then concatenate these features for sentence classification. By contrast, our M-MaxLSTM-CNN model jointly applies CNN on all pre-trained word embeddings to learn a multi-aspect word embedding. From this word representation, we encode sentences via the max-pooling and LSTM. To learn the similarity/relation between two sentences, we employ Multi-level comparison."]},{"idx":3,"heading":"Model description","paragraphs":["Our model (shown in Figure FIGREF4 ) consists of three main components: i) learning a multi-aspect word embedding (Section 3.1); ii) modeling sentences from this embedding (Section 3.2); iii) measuring the similarity/relation between two sentences via Multi-level comparison (section 3.3)."]},{"idx":4,"heading":"Multi-aspect word embedding","paragraphs":["Given a word INLINEFORM0 , we transfer it into a word vector INLINEFORM1 via INLINEFORM2 pre-trained word embeddings as follows: DISPLAYFORM0","where INLINEFORM0 is concatenation operator, INLINEFORM1 is the word embedding vector of INLINEFORM2 in the INLINEFORM3 th pre-trained embedding.","To learn a multi-aspect word embedding INLINEFORM0 from the representation INLINEFORM1 , we design INLINEFORM2 convolutional filters. Each filter INLINEFORM3 is denoted as a weight vector with the same dimension as INLINEFORM4 and a bias value INLINEFORM5 . The INLINEFORM6 is obtained by applying these filters on the INLINEFORM7 as follows: DISPLAYFORM0","where INLINEFORM0 denotes a logistic sigmoid function.","The next section explains how to model a sentence from its multiple-aspect word embeddings."]},{"idx":5,"heading":"Sentence modeling","paragraphs":["Given an input sentence INLINEFORM0 , we obtain a sequence of multiple-aspect word embeddings INLINEFORM1 using Eq. (1-3). For modeling the sentence from the representation INLINEFORM2 , we use two schemes: max-pooling and LSTM.","Max-pooling scheme: To construct a max-pooling sentence embedding INLINEFORM0 , the most potential features are extracted from the representation INLINEFORM1 as follows: DISPLAYFORM0","where INLINEFORM0 is the INLINEFORM1 th element of INLINEFORM2 .","LSTM scheme: From Eq. (4), we find that the max-pooling scheme ignores the property of word order. Therefore, we construct a LSTM sentence embedding INLINEFORM0 to support the sentence embedding INLINEFORM1 . The representation INLINEFORM2 is transformed to a fix-length vector by recursively applying a LSTM unit to each input INLINEFORM3 and the previous step INLINEFORM4 . At each time step INLINEFORM5 , the LSTM unit with INLINEFORM6 -memory dimension defines six vectors in INLINEFORM7 : input gate INLINEFORM8 , forget gate INLINEFORM9 , output gate INLINEFORM10 , tanh layer INLINEFORM11 , memory cell INLINEFORM12 and hidden state INLINEFORM13 as follows (from BIBREF19 ): DISPLAYFORM0 DISPLAYFORM1","where INLINEFORM0 respectively denote a logistic sigmoid function and element-wise multiplication; INLINEFORM1 are respectively two weights matrices and a bias vector for input gate INLINEFORM2 . The denotation is similar to forget gate INLINEFORM3 , output gate INLINEFORM4 , tanh layer INLINEFORM5 , memory cell INLINEFORM6 and hidden state INLINEFORM7 .","Finally, the sentence embedding INLINEFORM0 is obtained by concatenating the two sentence embeddings INLINEFORM1 and INLINEFORM2 : DISPLAYFORM0"]},{"idx":6,"heading":"Multi-level comparison","paragraphs":["In this section, we describe the process for evaluating the similarity/relation between two sentences. We compare two sentences via three levels: word-word, sentence-sentence and word-sentence.","Given two input sentences INLINEFORM0 and INLINEFORM1 , we encode them into two sequences of multi-aspect word embeddings INLINEFORM2 and INLINEFORM3 (Section 3.2). We then compute a word-word similarity vector INLINEFORM4 as follows: DISPLAYFORM0","where INLINEFORM0 is the INLINEFORM1 th multi-aspect word embedding of sentence INLINEFORM2 ; INLINEFORM3 is a function to flatten a matrix into a vector; and INLINEFORM4 and INLINEFORM5 are respectively a weight matrix and a bias parameter.","Given two input sentences INLINEFORM0 and INLINEFORM1 , we encode them into two sentence embeddings INLINEFORM2 and INLINEFORM3 (Section 3.1 and 3.2). To compute the similarity/relation between the two embeddings, we introduce four comparison metrics:","Cosine similarity: DISPLAYFORM0","Multiplication vector & Absolute difference: DISPLAYFORM0","where INLINEFORM0 is element-wise multiplication.","Neural difference: DISPLAYFORM0","where INLINEFORM0 and INLINEFORM1 are respectively a weight matrix and a bias parameter.","As a result, we have a sentence-sentence similarity vector INLINEFORM0 as follows: DISPLAYFORM0","where INLINEFORM0 and INLINEFORM1 are respectively a weight matrix and a bias parameter.","Given a sentence embedding INLINEFORM0 and a sequence of multi-aspect word embeddings INLINEFORM1 , we compute a word-sentence similarity matrix INLINEFORM2 as follows: DISPLAYFORM0","where INLINEFORM0 is the multi-aspect word embedding of the INLINEFORM1 th word in sentence INLINEFORM2 ; INLINEFORM3 and INLINEFORM4 are respectively a weight matrix and a bias parameter.","As a result, we have a word-sentence similarity vector INLINEFORM0 for the two sentences as follows: DISPLAYFORM0","where INLINEFORM0 is a function to flatten a matrix into a vector; INLINEFORM1 and INLINEFORM2 are respectively a weight matrix and a bias parameter.","Finally, we compute a target score/label of a sentence pair as follows: DISPLAYFORM0","where INLINEFORM0 , INLINEFORM1 , INLINEFORM2 and INLINEFORM3 are model parameters; INLINEFORM4 is a predicted target score/label."]},{"idx":7,"heading":"Tasks & Datasets","paragraphs":["We evaluate our model on three tasks:","Table TABREF30 shows the statistic of the three datasets. Because of not dealing with name entities and multi-word idioms, the vocabulary size of SICK is quite small compared to the others."]},{"idx":8,"heading":"Pre-trained word embeddings","paragraphs":["We study five pre-trained word embeddings for our model:","word2vec is trained on Google News dataset (100 billion tokens). The model contains 300-dimensional vectors for 3 million words and phrases.","fastText is learned via skip-gram with subword information on Wikipedia text. The embedding representations in fastText are 300-dimensional vectors.","GloVe is a 300-dimensional word embedding model learned on aggregated global word-word co-occurrence statistics from Common Crawl (840 billion tokens).","Baroni uses a context-predict approach to learn a 400-dimensional semantic embedding model. It is trained on 2.8 billion tokens constructed from ukWaC, the English Wikipedia and the British National Corpus.","SL999 is trained under the skip-gram objective with negative sampling on word pairs from the paraphrase database PPDB. This 300-dimensional embedding model is tuned on SimLex-999 dataset BIBREF27 ."]},{"idx":9,"heading":"Model configuration","paragraphs":["In all of the tasks, we used the same model configuration as follows:","Convolutional filters: we used 1600 filters. It is also the dimension of the word embedding concatenated from the five pre-trained word embeddings.","LSTM dimension: we also selected 1600 for LSTM dimension.","Neural similarity layers: the dimension of INLINEFORM0 , INLINEFORM1 , INLINEFORM2 and INLINEFORM3 are respectively 50, 5, 5 and 100.","Penultimate fully-connected layer: has the dimension of 250 and is followed by a drop-out layer ( INLINEFORM0 ).","We conducted a grid search on 30% of STSB dataset to select these optimal hyper-parameters."]},{"idx":10,"heading":"Training Setting","paragraphs":["In these tasks, we use the cross-entropy objective function and employ AdaDelta as the stochastic gradient descent (SGD) update rule with mini-batch size as 30. Details of Adadelta method can be found in BIBREF28 . During the training phase, the pre-trained word embeddings are fixed.","To compute a similarity score of a sentence pair in the range INLINEFORM0 , where INLINEFORM1 is an integer, we replace Eq. (27) with the equations in BIBREF19 as follows: DISPLAYFORM0","where INLINEFORM0 , INLINEFORM1 , INLINEFORM2 and INLINEFORM3 are parameters; INLINEFORM4 ; INLINEFORM5 is a predicted similarity score.","A sparse target distribution INLINEFORM0 which satisfies INLINEFORM1 is computed as: DISPLAYFORM0","for INLINEFORM0 , and INLINEFORM1 is the similarity score.","To train the model, we minimize the regularized KL-divergence between INLINEFORM0 and INLINEFORM1 : DISPLAYFORM0","where INLINEFORM0 is the number of training pairs and INLINEFORM1 denotes the model parameters. The gradient descent optimization Adadelta is used to learn the model parameters. We also use mini-batch size as 30 and keep the pre-trained word embeddings fixed during the training phase. We evaluate our models through Pearson correlation INLINEFORM2 ."]},{"idx":11,"heading":"Experiments and Discussion","paragraphs":["This section describes two experiments: i) compare our model against recent systems; ii) evaluate the efficiency of using multiple pre-trained word embeddings."]},{"idx":12,"heading":"Overall evaluation","paragraphs":["Besides existing methods, we also compare our model with several sentence modeling approaches using multiple pre-trained word embeddings:","Word Average: DISPLAYFORM0","where INLINEFORM0 is the sentence embedding of a INLINEFORM1 -words sentence, and INLINEFORM2 is from Eq. (1)","Project Average: DISPLAYFORM0","where INLINEFORM0 is a INLINEFORM1 weight matrix, and INLINEFORM2 is a 1600 bias vector.","LSTM: apply Eq. (5-11) on INLINEFORM0 to construct the 1600-dimension INLINEFORM1 sentence embedding.","Max-CNN: apply Eq. (2-4) on INLINEFORM0 to construct the 1600-dimension INLINEFORM1 sentence embedding.","We report the results of these methods in Table TABREF49 . Overall, our M-MaxLSTM-CNN shows competitive performances in these tasks. Especially in the STS task, M-MaxLSTM-CNN outperforms the state-of-the-art methods on the two datasets. Because STSB includes complicated samples compared to SICK, the performances of methods on STSB are quite lower. In STSB, the prior top performance methods use ensemble approaches mixing hand-crafted features (word alignment, syntactic features, N-gram overlaps) and neural sentence representations, while our approach is only based on a neural sentence modeling architecture. In addition, we observed that InferSent shows the strong performance on SICK-R but quite low on STSB while our model consistently obtains the strong performances on both of the datasets. InferSent uses transfer knowledge on textual entailment data, consequently it obtains the strong performance on this entailment task.","According to BIBREF31 , using Word Average as the compositional architecture outperforms the other architectures (e.g., Project Average, LSTM) for STS tasks. In a multiple word embeddings setting, however, Word Average does not show its efficiency. Each word embedding model has its own architecture as well as objective function. These factors makes the vector spaces of word embeddings are different. Therefore, we intuitively need a step to learn or refine a representation from a set of pre-trained word embeddings rather than only averaging them. Because Project Average model, LSTM model and Max-CNN model have their parameters for learning sentence embeddings, they significantly outperform Word Average model.","We observed that MaxLSTM-CNN outperforms Max-CNN in both of the settings (i.e., sentence-sentence comparison, Multi-level comparison). As mentioned in Section 1, Max-CNN ignores the property of word order. Therefore, our model achieves improvement compared to Max-CNN by additionally employing LSTM for capturing this property.","We only applied Multi-level comparison on Max-CNN and MaxLSTM-CNN because these encoders generate multi-aspect word embeddings. The experimental results prove the efficiency of using Multi-level comparison. In the textual entailment dataset SICK-E, the task mainly focuses on interpreting the meaning of a whole sentence pair rather than comparing word by word. Therefore, the performance of Multi-level comparison is quite similar to sentence-sentence comparison in the SICK-E task. This is also the reason why LSTM, which captures global relationships in sentences, has the strong performance in this task."]},{"idx":13,"heading":"Evaluation of exploiting multiple pre-trained word embeddings","paragraphs":["In this section, we evaluate the efficiency of using multiple pre-trained word embeddings. We compare our multiple pre-trained word embeddings model against models using only one pre-trained word embedding. The same objective function and Multi-level comparison are applied for these models. In case of using one pre-trained word embedding, the dimension of LSTM and the number of convolutional filters are set to the length of the corresponding word embedding. Table TABREF57 shows the experimental results of this comparison. Because the approach using five word embeddings outperforms the approaches using two, three, or four word embeddings, we only report the performance of using five word embeddings. We also report INLINEFORM0 which is the proportion of vocabulary available in a pre-trained word embedding. SICK dataset ignores idiomatic multi-word expressions, and named entities, consequently the INLINEFORM1 of SICK is quite high.","We observed that no word embedding has strong results on all the tasks. Although trained on the paraphrase database and having the highest INLINEFORM0 , the SL999 embedding could not outperform the Glove embedding in SICK-R. HCTI BIBREF5 , which is the current state-of-the-art in the group of neural representation models on STSB, also used the Glove embedding. However, the performance of HTCI in STSB ( INLINEFORM1 ) is lower than our model using the Glove embedding. In SICK-R, InferSent BIBREF23 achieves a strong performance ( INLINEFORM2 ) using the Glove embedding with transfer knowledge, while our model with only the Glove embedding achieves a performance close to the performance of InferSent. These results confirm the efficiency of Multi-level comparison.","In STSB and MRPC, as employing the five pre-trained embeddings, the INLINEFORM0 is increased. This factor limits the number of random values when initializing word embedding representations because a word out of a pre-trained word embedding is assigned a random word embedding representation. In other words, a word out of a pre-trained word embedding is assigned a random semantic meaning. Therefore, the increase of the INLINEFORM1 improves the performance of measuring textual similarity. In STSB and MRPC, our multiple pre-trained word embedding achieves a significant improvement in performance compared against using one word embedding. In SICK-R and SICK-E, although the INLINEFORM2 is not increased when employing five pre-trained embeddings, the performance of our model is improved. This fact shows that our model learned an efficient word embedding via these pre-trained word embeddings."]},{"idx":14,"heading":"Conclusion","paragraphs":["In this work, we study an approach employing multiple pre-trained word embeddings and Multi-level comparison for measuring semantic textual relation. The proposed M-MaxLSTM-CNN architecture consistently obtains strong performances on several tasks. Compared to the state-of-the art methods in STS tasks, our model does not require handcrafted features (e.g., word alignment, syntactic features) as well as transfer learning knowledge. In addition, it allows using several pre-trained word embeddings with different dimensions.","Future work could apply our multiple word embeddings approach for transfer learning tasks. This strategy allows making use of pre-trained word embeddings as well as available resources."]},{"idx":15,"heading":"Acknowledgments","paragraphs":["This work was done while Nguyen Tien Huy was an intern at Toshiba Research Center."]},{"idx":16,"heading":"Figures and Tables","paragraphs":["Figure 1: The proposed M-MaxLSTM-CNN model: (a) MaxLSTM-CNN encoder; (b) Multi-level comparison.","Table 1: Statistic of datasets. |V |, l denote the vocabulary size, and the average length of sentences respectively.","Table 2: Test set results with Pearson’s r score×100 for STS tasks, and accuracy for other tasks. Boldface values show the highest scores in each dataset. SICK-R and SICK-E denote the STS task and the entailment task in SICK dataset respectively.","Table 3: Evaluation of exploiting multiple pre-trained word embeddings. |V |avai is the proportion of vocabulary available in a word embedding. In case of using all word embeddings, |V |avai denotes the proportion of vocabulary where each word is available in at least one word embedding."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1806.04330","title":"Neural Network Models for Paraphrase Identification, Semantic Textual Similarity, Natural Language Inference, and Question Answering","sections":[{"idx":0,"heading":"Abstract","paragraphs":["In this paper, we analyze several neural network designs (and their variations) for sentence pair modeling and compare their performance extensively across eight datasets, including paraphrase identification, semantic textual similarity, natural language inference, and question answering tasks. Although most of these models have claimed state-of-the-art performance, the original papers often reported on only one or two selected datasets. We provide a systematic study and show that (i) encoding contextual information by LSTM and inter-sentence interactions are critical, (ii) Tree-LSTM does not help as much as previously claimed but surprisingly improves performance on Twitter datasets, (iii) the Enhanced Sequential Inference Model is the best so far for larger datasets, while the Pairwise Word Interaction Model achieves the best performance when less data is available. We release our implementations as an open-source toolkit."]},{"idx":1,"heading":"Introduction","paragraphs":["This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http://creativecommons.org/licenses/by/4.0/.","Sentence pair modeling is a fundamental technique underlying many NLP tasks, including the following:","Traditionally, researchers had to develop different methods specific for each task. Now neural networks can perform all the above tasks with the same architecture by training end to end. Various neural models BIBREF1 , BIBREF0 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15 have declared state-of-the-art results for sentence pair modeling tasks; however, they were carefully designed and evaluated on selected (often one or two) datasets that can demonstrate the superiority of the model. The research questions are as follows: Do they perform well on other tasks and datasets? How much performance gain is due to certain system design choices and hyperparameter optimizations?","To answer these questions and better understand different network designs, we systematically analyze and compare the state-of-the-art neural models across multiple tasks and multiple domains. Namely, we implement five models and their variations on the same PyTorch platform: InferSent model BIBREF16 , Shortcut-stacked Sentence Encoder Model BIBREF17 , Pairwise Word Interaction Model BIBREF1 , Decomposable Attention Model BIBREF10 , and Enhanced Sequential Inference Model BIBREF0 . They are representative of the two most common approaches: sentence encoding models that learn vector representations of individual sentences and then calculate the semantic relationship between sentences based on vector distance and sentence pair interaction models that use some sorts of word alignment mechanisms (e.g., attention) then aggregate inter-sentence interactions. We focus on identifying important network designs and present a series of findings with quantitative measurements and in-depth analyses, including (i) incorporating inter-sentence interactions is critical; (ii) Tree-LSTM does not help as much as previously claimed but surprisingly improves performance on Twitter data; (iii) Enhanced Sequential Inference Model has the most consistent high performance for larger datasets, while Pairwise Word Interaction Model performs better on smaller datasets and Shortcut-Stacked Sentence Encoder Model is the best performaning model on the Quora corpus. We release our implementations as a toolkit to the research community."]},{"idx":2,"heading":"General Framework for Sentence Pair Modeling","paragraphs":["Various neural networks have been proposed for sentence pair modeling, all of which fall into two types of approaches. The sentence encoding approach encodes each sentence into a fixed-length vector and then computes sentence similarity directly. The model of this type has advantages in the simplicity of the network design and generalization to other NLP tasks. The sentence pair interaction approach takes word alignment and interactions between the sentence pair into account and often show better performance when trained on in-domain data. Here we outline the two types of neural networks under the same general framework:"]},{"idx":3,"heading":"Representative Models for Sentence Pair Modeling","paragraphs":["Table 1 gives a summary of typical models for sentence pair modeling in recent years. In particular, we investigate five models in depth: two are representative of the sentence encoding type of model, and three are representative of the interaction-aggregation type of model. These models have reported state-or-the-art results with varied architecture design (this section) and implementation details (Section \"Implementation Details\" )."]},{"idx":4,"heading":"The Bi-LSTM Max-pooling Network (InferSent)","paragraphs":["We choose the simple Bi-LSTM max-pooling network from InferSent BIBREF16 :","$$&\\overleftrightarrow{\\mathbf {h}}_{i} = BiLSTM(\\mathbf {x}_{i}, \\overleftrightarrow{\\mathbf {h}}_{i-1}) \\\\ &\\mathbf {v}=max(\\overleftrightarrow{\\mathbf {h}}_{1}, \\overleftrightarrow{\\mathbf {h}}_{2}, ..., \\overleftrightarrow{\\mathbf {h}}_{n})$$ (Eq. 17)","where $\\overleftrightarrow{\\mathbf {h}}_{i}$ represents the concatenation of hidden states in both directons. It has shown better transfer learning capabilities than several other sentence embedding models, including SkipThought BIBREF31 and FastSent BIBREF32 , when trained on the natural language inference datasets."]},{"idx":5,"heading":"The Shortcut-Stacked Sentence Encoder Model (SSE)","paragraphs":["The Shortcut-Stacked Sentence Encoder model BIBREF17 is a sentence-based embedding model, which enhances multi-layer Bi-LSTM with skip connection to avoid training error accumulation, and calculates each layer as follows:","$$&\\overleftrightarrow{\\mathbf {h}}_{i}^{k} = BiLSTM(\\mathbf {x}_{i}^{k}, \\overleftrightarrow{\\mathbf {h}}_{i-1}^{k}) \\\\ &\\mathbf {x}_{i}^{1}=\\mathbf {w}_{i} \\quad (k=1), \\qquad \\mathbf {x}_{i}^{k}=[\\mathbf {w}_{i}, \\overleftrightarrow{\\mathbf {h}}_{i}^{k-1}, \\overleftrightarrow{\\mathbf {h}}_{i}^{k-2}, ..., \\overleftrightarrow{\\mathbf {h}}_{i}^{1}] \\quad (k>1) \\\\ &\\mathbf {v}=max(\\overleftrightarrow{\\mathbf {h}}_{1}^{m}, \\overleftrightarrow{\\mathbf {h}}_{2}^{m}, ..., \\overleftrightarrow{\\mathbf {h}}_{n}^{m})$$ (Eq. 19)","where $\\mathbf {x}_{i}^{k}$ is the input of the $k$ th Bi-LSTM layer at time step $i$ , which is the combination of outputs from all previous layers, $\\overleftrightarrow{\\mathbf {h}}_{i}^{k}$ represents the hidden state of the $k$ th Bi-LSTM layer in both directions. The final sentence embedding $\\mathbf {v}$ is the row-based max pooling over the output of the last Bi-LSTM layer, where $n$ denotes the number of words within a sentence and $m$ is the number of Bi-LSTM layers ( $m = 3$ in SSE)."]},{"idx":6,"heading":"The Pairwise Word Interaction Model (PWIM)","paragraphs":["In the Pairwise Word Interaction model BIBREF1 , each word vector $\\mathbf {w}_{i}$ is encoded with context through forward and backward LSTMs: $\\overrightarrow{\\mathbf {h}}_{i} = LSTM^{f}(\\mathbf {w}_{i}, \\overrightarrow{\\mathbf {h}}_{i-1})$ and $\\overleftarrow{\\mathbf {h}}_{i} = LSTM^{b}(\\mathbf {w}_{i}, \\overleftarrow{\\mathbf {h}}_{i+1})$ . For every word pair $(\\mathbf {w}^a_{i}, \\mathbf {w}^b_{j})$ across sentences, the model directly calculates word pair interactions using cosine similarity, Euclidean distance, and dot product over the outputs of the encoding layer:","$$D(\\overrightarrow{\\mathbf {h}}_{i}, \\overrightarrow{\\mathbf {h}}_{j}) & = [cos(\\overrightarrow{\\mathbf {h}}_{i}, \\overrightarrow{\\mathbf {h}}_{j}), \\Vert \\overrightarrow{\\mathbf {h}}_{i} - \\overrightarrow{\\mathbf {h}}_{j}\\Vert , \\overrightarrow{\\mathbf {h}}_{i} \\cdot \\overrightarrow{\\mathbf {h}}_{j}]$$ (Eq. 21)","The above equation not only applies to forward hidden state $\\overrightarrow{\\mathbf {h}}_{i}$ and backward hidden state $\\overleftarrow{\\mathbf {h}}_{i}$ , but also to the concatenation $\\overleftrightarrow{\\mathbf {h}}_{i}= [\\overrightarrow{\\mathbf {h}}_{i}, \\overleftarrow{\\mathbf {h}}_{i}]$ and summation $\\mathbf {h}^{+}_{i}= \\overrightarrow{\\mathbf {h}}_{i} + \\overleftarrow{\\mathbf {h}}_{i}$ , resulting in a tensor $\\mathbf {D}^{13 \\times |sent1| \\times |sent2|}$ after padding one extra bias term. A “hard” attention is applied to the interaction tensor to build word alignment: selecting the most related word pairs and increasing the corresponding weights by 10 times. Then a 19-layer deep CNN is applied to aggregate the word interaction features for final classification."]},{"idx":7,"heading":"The Decomposable Attention Model (DecAtt)","paragraphs":["The Decomposable Attention model BIBREF10 is one of the earliest models to introduce attention-based alignment for sentence pair modeling, and it achieved state-of-the-art results on the SNLI dataset with about an order of magnitude fewer parameters than other models (see more in Table 5 ) without relying on word order information. It computes the word pair interaction between $\\mathbf {w}_{i}^{a}$ and $\\mathbf {w}_{j}^{b}$ (from input sentences $s_a$ and $s_b$ , each with $m$ and $n$ words, respectively) as ${e}_{ij} = {F(\\mathbf {w}_{i}^{a})}^{T} F(\\mathbf {w}_{j}^{b})$ , where $F$ is a feedforward network; then alignment is determined as follows:","$$&\\mathbf {\\beta }_{i} = \\sum _{j=1}^{n} \\frac{exp({e}_{ij})}{\\sum _{k=1}^{n} exp({e}_{ik})} \\mathbf {w}_{j}^{b} \\quad &\\mathbf {\\alpha }_{j} = \\sum _{i=1}^{m} \\frac{exp({e}_{ij})}{\\sum _{k=1}^{m} exp({e}_{kj})} \\mathbf {w}_{i}^{a} $$ (Eq. 23)","where $\\mathbf {\\beta }_{i}$ is the soft alignment between $\\mathbf {w}_{i}^{a}$ and subphrases $\\mathbf {w}_{j}^{b}$ in sentence $s_b$ , and vice versa for $\\mathbf {\\alpha }_{j}$ . The aligned phrases are fed into another feedforward network $G$ : $\\mathbf {v}_{i}^{a} = G([\\mathbf {w}_{i}^{a}; \\mathbf {\\beta }_{i}]) $ and $\\mathbf {v}_{j}^{b} = G([\\mathbf {w}_{j}^{b}; \\mathbf {\\alpha }_{j}]) $ to generate sets $\\lbrace \\mathbf {v}_{i}^{a}\\rbrace $ and $\\lbrace \\mathbf {v}_{j}^{b}\\rbrace $ , which are aggregated by summation and then concatenated together for classification."]},{"idx":8,"heading":"The Enhanced Sequential Inference Model (ESIM)","paragraphs":["The Enhanced Sequential Inference Model BIBREF0 is closely related to the DecAtt model, but it differs in a few aspects. First, Chen et al. Chen-Qian:2017:ACL demonstrated that using Bi-LSTM to encode sequential contexts is important for performance improvement. They used the concatenation $\\overline{\\mathbf {w}}_{i} = \\overleftrightarrow{\\mathbf {h}}_{i} = [\\overrightarrow{\\mathbf {h}}_{i}, \\overleftarrow{\\mathbf {h}}_{i}]$ of both directions as in the PWIM model. The word alignment $\\mathbf {\\beta }_{i}$ and $\\mathbf {\\alpha }_{j}$ between $\\overline{\\mathbf {w}}^{a}$ and $\\overline{\\mathbf {w}}^{b}$ are calculated the same way as in DecAtt. Second, they showed the competitive performance of recursive architecture with constituency parsing, which complements with sequential LSTM. The feedforward function $G$ in DecAtt is replaced with Tree-LSTM:","$$&\\mathbf {v}_{i}^{a} = TreeLSTM([\\overline{\\mathbf {w}}_{i}^{a}; \\mathbf {\\beta }_{i}; \\overline{\\mathbf {w}}_{i}^{a}-\\mathbf {\\beta }_{i}; \\overline{\\mathbf {w}}_{i}^{a} \\odot \\mathbf {\\beta }_{i}])\\\\ &\\mathbf {v}_{j}^{b} = TreeLSTM([\\overline{\\mathbf {w}}_{j}^{b}; \\mathbf {\\alpha }_{j}; \\overline{\\mathbf {w}}_{j}^{b}-\\mathbf {\\alpha }_{j}; \\overline{\\mathbf {w}}_{j}^{b} \\odot \\mathbf {\\alpha }_{j}])$$ (Eq. 25)","Third, instead of using summation in aggregation, ESIM adapts the average and max pooling and concatenation $\\mathbf {v}= [\\mathbf {v}_{ave}^{a}; \\mathbf {v}_{max}^{a}; \\mathbf {v}_{ave}^{b}; \\mathbf {v}_{max}^{b}]$ before passing through multi-layer perceptron (MLP) for classification:","$$& \\mathbf {v}_{ave}^{a} = \\sum _{i=1}^{m} \\frac{\\mathbf {v}_{i}^{a}}{m}, \\qquad \\mathbf {v}_{max}^{a} = \\max _{i=1}^{m}\\mathbf {v}_{i}^{a} , \\qquad \\mathbf {v}_{ave}^{b} = \\sum _{j=1}^{n} \\frac{\\mathbf {v}_{j}^{b}}{n}, \\qquad \\mathbf {v}_{max}^{b} = \\max _{j=1}^{n}\\mathbf {v}_{j}^{b}$$ (Eq. 26)"]},{"idx":9,"heading":"Datasets","paragraphs":["We conducted sentence pair modeling experiments on eight popular datasets: two NLI datasets, three PI datasets, one STS dataset and two QA datasets. Table 2 gives a comparison of these datasets:","SNLI BIBREF7 contains 570k hypotheses written by crowdsourcing workers given the premises. It focuses on three semantic relations: the premise entails the hypothesis (entailment), they contradict each other (contradiction), or they are unrelated (neutral).","Multi-NLI BIBREF33 extends the SNLI corpus to multiple genres of written and spoken texts with 433k sentence pairs.","Quora BIBREF34 contains 400k question pairs collected from the Quora website. This dataset has balanced positive and negative labels indicating whether the questions are duplicated or not.","Twitter-URL BIBREF35 includes 50k sentence pairs collected from tweets that share the same URL of news articles. This dataset contains both formal and informal language.","PIT-2015 BIBREF5 comes from SemEval-2015 and was collected from tweets under the same trending topic. It contains naturally occurred (i.e. written by independent Twitter users spontaneously) paraphrases and non-paraphrases with varied topics and language styles.","STS-2014 BIBREF36 is from SemEval-2014, constructed from image descriptions, news headlines, tweet news, discussion forums, and OntoNotes BIBREF37 .","WikiQA BIBREF8 is an open-domain question-answering dataset. Following He and Lin he-lin:2016:N16-1, questions without correct candidate answer sentences are excluded, and answer sentences are truncated to 40 tokens, resulting in 12k question-answer pairs for our experiments.","TrecQA BIBREF38 is an answer selection task of 56k question-answer pairs and created in Text Retrieval Conferences (TREC). For both WikiQA and TrecQA datasets, the best answer is selected according to the semantic relatedness with the question."]},{"idx":10,"heading":"Implementation Details","paragraphs":["We implement all the models with the same PyTorch framework. Below, we summarize the implementation details that are key for reproducing results for each model:","SSE: This model can converge very fast, for example, 2 or 3 epochs for the SNLI dataset. We control the convergence speed by updating the learning rate for each epoch: specifically, $lr=\\frac{1}{2^{\\frac{epoch\\_i}{2}}}*{init\\_lr}$ , where $init\\_lr$ is the initial learning rate and $epoch\\_i$ is the index of current epoch.","DecAtt: It is important to use gradient clipping for this model: for each gradient update, we check the L2 norm of all the gradient values, if it is greater than a threshold $b$ , we scale the gradient by a factor $\\alpha = b/L2\\_norm$ . Another useful procedure is to assemble batches of sentences with similar length.","ESIM: Similar but different from DecAtt, ESIM batches sentences with varied length and uses masks to filter out padding information. In order to batch the parse trees within Tree-LSTM recursion, we follow Bowman et al.'s bowman-EtAl:2016:P16-1 procedure that converts tree structures into the linear sequential structure of a shift reduce parser. Two additional masks are used for producing left and right children of a tree node.","PWIM: The cosine and Euclidean distances used in the word interaction layer have smaller values for similar vectors while dot products have larger values. The performance increases if we add a negative sign to make all the vector similarity measurements behave consistently."]},{"idx":11,"heading":"Analysis","paragraphs":["Table 3 and 4 show the results reported in the original papers and the replicated results with our implementation. We use accuracy, F1 score, Pearson's $r$ , Mean Average Precision (MAP), and Mean Reciprocal Rank (MRR) for evaluation on different datasets following the literature. Our reproduced results are slightly lower than the original results by 0.5 $\\sim $ 1.5 points on accuracy. We suspect the following potential reasons: (i) less extensive hyperparameter tuning for each individual dataset; (ii) only one run with random seeding to report results; and (iii) use of different neural network toolkits: for example, the original ESIM model was implemented with Theano, and PWIM model was in Torch.","Herein, we examine the main components that account for performance in sentence pair modeling.","How important is LSTM encoded context information for sentence pair modeling?","Regarding DecAtt, Parikh et al. parikh-EtAl:2016:EMNLP2016 mentioned that “intra-sentence attention is optional\"; they can achieve competitive results without considering context information. However, not surprisingly, our experiments consistently show that encoding sequential context information with LSTM is critical. Compared to DecAtt, ESIM shows better performance on every dataset (see Table 4 and Figure 3 ). The main difference between ESIM and DecAtt that contributes to performance improvement, we found, is the use of Bi-LSTM and Tree-LSTM for sentence encoding, rather than the different choices of aggregation functions.","Why does Tree-LSTM help with Twitter data?","Chen et al. Chen-Qian:2017:ACL offered a simple combination (ESIM $_{seq+tree}$ ) by averaging the prediction probabilities of two ESIM variants that use sequential Bi-LSTM and Tree-LSTM respectively, and suggested “parsing information complements very well with ESIM and further improves the performance\". However, we found that adding Tree-LSTM only helps slightly or not at all for most datasets, but it helps noticably with the two Twitter paraphrase datasets. We hypothesize the reason is that these two datasets come from real-world tweets which often contain extraneous text fragments, in contrast to SNLI and other datasets that have sentences written by crowdsourcing workers. For example, the segment “ever wondered ,” in the sentence pair ever wondered , why your recorded #voice sounds weird to you? and why do our recorded voices sound so weird to us? introduces a disruptive context into the Bi-LSTM encoder, while Tree-LSTM can put it in a less important position after constituency parsing.","How important is attentive interaction for sentence pair modeling? Why does SSE excel on Quora?","Both ESIM and DecAtt (Eq. 23 ) calculate an attention-based soft alignment between a sentence pair, which was also proposed in BIBREF27 and BIBREF29 for sentence pair modeling, whereas PWIM utilizes a hard attention mechanism. Both attention strategies are critical for model performance. In PWIM model BIBREF1 , we observed a 1 $\\sim $ 2 point performance drop after removing the hard attention, 0 $\\sim $ 3 point performance drop and $\\sim $ 25% training time reduction after removing the 19-layer CNN aggregation. Likely without even the authors of SSE knowing, the SSE model performs extraordinarily well on the Quora corpus, perhaps because Quora contains many sentence pairs with less complicated inter-sentence interactions (e.g., many identical words in the two sentences) and incorrect ground truth labels (e.g., What is your biggest regret in life? and What's the biggest regret you've had in life? are labeled as non-duplicate questions by mistake).","Figure 3 shows the learning curves. The DecAtt model converges quickly and performs well on large NLI datasets due to its design simplicity. PWIM is the slowest model (see time comparison in Table 5 ) but shows very strong performance on semantic similarity and paraphrase identification datasets. ESIM and SSE keep a good balance between training time and performance.","[3]This number was reported in BIBREF12 by co-authors of DecAtt BIBREF10 . [4]This number was reproduced by Williams et al. williams2017broad. [5]This number was generated by InferSent traind on SNLI and Multi-NLI datasets.","As shown in Figure 4 , we experimented with different training sizes of the largest SNLI dataset. All the models show improved performance as we increase the training size. ESIM and SSE have very similar trends and clearly outperform PWIM on the SNLI dataset. DecAtt shows a performance jump when the training size exceeds a threshold.","We conducted an in-depth analysis of model performance on the Multi-domain NLI dataset based on different categories: text genre, sentence pair overlap, and sentence length. As shown in Table 6 , all models have comparable performance between matched genre and unmatched genre. Sentence length and overlap turn out to be two important factors – the longer the sentences and the fewer tokens in common, the more challenging it is to determine their semantic relationship. These phenomena shared by the state-of-the-art systems reflect their similar design framework which is symmetric at processing both sentences in the pair, while question answering and natural language inference tasks are directional BIBREF30 . How to incorporate asymmetry into model design will be worth more exploration in future research.","In addition to the cross-domain study (Table 6 ), we conducted transfer learning experiments on three paraphrase identification datasets (Table 4 ). The most noteworthy phenomenon is that the SSE model performs better on Twitter-URL and PIT-2015 when trained on the large out-of-domain Quora data than the small in-domain training data. Two likely reasons are: (i) the SSE model with over 29 million parameters is data hungry and (ii) SSE model is a sentence encoding model, which generalizes better across domains/tasks than sentence pair interaction models. Sentence pair interaction models may encounter difficulties on Quora, which contains sentence pairs with the highest word overlap (51.5%) among all datasets and often causes the interaction patterns to focus on a few key words that differ. In contrast, the Twitter-URL dataset has the lowest overlap (23.0%) with a semantic relationship that is mainly based on the intention of the tweets."]},{"idx":12,"heading":"Conclusion","paragraphs":["We analyzed five different neural models (and their variations) for sentence pair modeling and conducted a series of experiments with eight representative datasets for different NLP tasks. We quantified the importance of the LSTM encoder and attentive alignment for inter-sentence interaction, as well as the transfer learning ability of sentence encoding based models. We showed that the SNLI corpus of over 550k sentence pairs cannot saturate the learning curve. We systematically compared the strengths and weaknesses of different network designs and provided insights for future work."]},{"idx":13,"heading":"Acknowledgements","paragraphs":["We thank Ohio Supercomputer Center BIBREF39 for computing resources. This work was supported in part by NSF CRII award (RI-1755898) and DARPA through the ARO (W911NF-17-C-0095). The content of the information in this document does not necessarily reflect the position or the policy of the U.S. Government, and no official endorsement should be inferred."]},{"idx":14,"heading":"Pretrained Word Embeddings","paragraphs":["We used the 200-dimensional GloVe word vectors BIBREF18 , trained on 27 billion words from Twitter (vocabulary size of 1.2 milion words) for Twitter URL BIBREF35 and PIT-2015 BIBREF5 datasets, and the 300-dimensional GloVe vectors, trained on 840 billion words (vocabulary size of 2.2 milion words) from Common Crawl for all other datasets. For out-of-vocabulary words, we initialized the word vectors using normal distribution with mean 0 and deviation 1."]},{"idx":15,"heading":"Hyper-parameter Settings","paragraphs":["We followed original papers or code implementations to set hyper-parameters for these models. In Infersent model BIBREF16 , the hidden dimension size for Bi-LSTM is 2048, and the fully connected layers have 512 hidden units. In SSE model BIBREF17 , the hidden size for three Bi-LSTMs is 512, 2014 and 2048, respectively. The fully connected layers have 1600 units. PWIM BIBREF1 and ESIM BIBREF0 both use Bi-LSTM for context encoding, having 200 hidden units and 300 hidden units respectively. The DecAtt model BIBREF10 uses three kinds of feed forward networks, all of which have 300 hidden units. Other parameters like learning rate, batch size, dropout rate, and all of them use the same settings as in original papers."]},{"idx":16,"heading":"Fine-tuning the Models","paragraphs":["It is not practical to fine tune every hyper-parameter in every model and every dataset, since we want to show how these models can generalize well on other datasets, we need try to avoid fine-tuning these parameters on some specific datasets, otherwise we can easily get over-fitted models. Therefore, we keep the hyper-parameters unchanged across different datasets, to demonstrate the generalization capability of each model. The default number of epochs for training these models is set to 20, if some models could converge earlier (no more performance gain on development set), we would stop running them before they approached epoch 20. The 20 epochs can guarantee every model get converged on every dataset."]},{"idx":17,"heading":"Figures and Tables","paragraphs":["Table 1: Summary of representative neural models for sentence pair modeling. The upper half contains sentence encoding models, and the lower half contains sentence pair interaction models.","Figure 1: Sentence encoding models focus on learning vector representations of individual sentences and then calculate the semantic relationship between sentences based on vector distance.","Figure 2: Sentence pair interaction models use different word alignment mechanisms before aggregation.","Table 2: Basic statistics and examples of different datasets for sentence pair modeling tasks.","Table 3: Reported results from original papers, which are mostly limited to a few datasets. For the Multi-NLI dataset, Acc m represents testing accuracy for the matched genre and Acc um for the unmatched genre.","Table 4: Replicated results with our reimplementation in PyTorch across multiple tasks and datasets. The best result in each dataset is denoted by a bold typeface, and the second best is denoted by an underline.","Figure 3: Training curves of ESIM, DecAtt, PWIM, SSE and InferSent models on eight datasets.","Table 5: Average training time per sentence pair in the Twitter-URL dataset (similar time for other datasets).","Figure 4: Performance vs. training size (log scale in x-axis) on SNLI dataset.","Table 6: Transfer learning experiments for paraphrase identification task.","Table 7: Categorical performance (accuracy) on Multi-NLI dataset. Overlap is the percentage of shared tokens between two sentences. Length is calculated based on the number of tokens of the longer sentence."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1808.09029","title":"Pyramidal Recurrent Unit for Language Modeling","sections":[{"idx":0,"heading":"Abstract","paragraphs":["LSTMs are powerful tools for modeling contextual information, as evidenced by their success at the task of language modeling. However, modeling contexts in very high dimensional space can lead to poor generalizability. We introduce the Pyramidal Recurrent Unit (PRU), which enables learning representations in high dimensional space with more generalization power and fewer parameters. PRUs replace the linear transformation in LSTMs with more sophisticated interactions including pyramidal and grouped linear transformations. This architecture gives strong results on word-level language modeling while reducing the number of parameters significantly. In particular, PRU improves the perplexity of a recent state-of-the-art language model Merity et al. (2018) by up to 1.3 points while learning 15-20% fewer parameters. For similar number of model parameters, PRU outperforms all previous RNN models that exploit different gating mechanisms and transformations. We provide a detailed examination of the PRU and its behavior on the language modeling tasks. Our code is open-source and available at https://sacmehta.github.io/PRU/"]},{"idx":1,"heading":"Introduction","paragraphs":["Long short term memory (LSTM) units BIBREF1 are popular for many sequence modeling tasks and are used extensively in language modeling. A key to their success is their articulated gating structure, which allows for more control over the information passed along the recurrence. However, despite the sophistication of the gating mechanisms employed in LSTMs and similar recurrent units, the input and context vectors are treated with simple linear transformations prior to gating. Non-linear transformations such as convolutions BIBREF2 have been used, but these have not achieved the performance of well regularized LSTMs for language modeling BIBREF3 .","A natural way to improve the expressiveness of linear transformations is to increase the number of dimensions of the input and context vectors, but this comes with a significant increase in the number of parameters which may limit generalizability. An example is shown in Figure FIGREF1 , where LSTMs performance decreases with the increase in dimensions of the input and context vectors. Moreover, the semantics of the input and context vectors are different, suggesting that each may benefit from specialized treatment.","Guided by these insights, we introduce a new recurrent unit, the Pyramidal Recurrent Unit (PRU), which is based on the LSTM gating structure. Figure FIGREF2 provides an overview of the PRU. At the heart of the PRU is the pyramidal transformation (PT), which uses subsampling to effect multiple views of the input vector. The subsampled representations are combined in a pyramidal fusion structure, resulting in richer interactions between the individual dimensions of the input vector than is possible with a linear transformation. Context vectors, which have already undergone this transformation in the previous cell, are modified with a grouped linear transformation (GLT) which allows the network to learn latent representations in high dimensional space with fewer parameters and better generalizability (see Figure FIGREF1 ).","We show that PRUs can better model contextual information and demonstrate performance gains on the task of language modeling. The PRU improves the perplexity of the current state-of-the-art language model BIBREF0 by up to 1.3 points, reaching perplexities of 56.56 and 64.53 on the Penn Treebank and WikiText2 datasets while learning 15-20% fewer parameters. Replacing an LSTM with a PRU results in improvements in perplexity across a variety of experimental settings. We provide detailed ablations which motivate the design of the PRU architecture, as well as detailed analysis of the effect of the PRU on other components of the language model."]},{"idx":2,"heading":"Related work","paragraphs":["Multiple methods, including a variety of gating structures and transformations, have been proposed to improve the performance of recurrent neural networks (RNNs). We first describe these approaches and then provide an overview of recent work in language modeling."]},{"idx":3,"heading":"Pyramidal Recurrent Units","paragraphs":["We introduce Pyramidal Recurrent Units (PRUs), a new RNN architecture which improves modeling of context by allowing for higher dimensional vector representations while learning fewer parameters. Figure FIGREF2 provides an overview of PRU. We first elaborate on the details of the pyramidal transformation and the grouped linear transformation. We then describe our recurrent unit, PRU."]},{"idx":4,"heading":"Pyramidal transformation for input","paragraphs":["The basic transformation in many recurrent units is a linear transformation INLINEFORM0 defined as: DISPLAYFORM0","where INLINEFORM0 are learned weights that linearly map INLINEFORM1 to INLINEFORM2 . To simplify notation, we omit the biases.","Motivated by successful applications of sub-sampling in computer vision (e.g., BIBREF22 , BIBREF23 , BIBREF9 , BIBREF24 ), we subsample input vector INLINEFORM0 into INLINEFORM1 pyramidal levels to achieve representation of the input vector at multiple scales. This sub-sampling operation produces INLINEFORM2 vectors, represented as INLINEFORM3 , where INLINEFORM4 is the sampling rate and INLINEFORM5 . We learn scale-specific transformations INLINEFORM6 for each INLINEFORM7 . The transformed subsamples are concatenated to produce the pyramidal analog to INLINEFORM8 , here denoted as INLINEFORM9 : DISPLAYFORM0","where INLINEFORM0 indicates concatenation. We note that pyramidal transformation with INLINEFORM1 is the same as the linear transformation.","To improve gradient flow inside the recurrent unit, we combine the input and output using an element-wise sum (when dimension matches) to produce residual analog of pyramidal transformation, as shown in Figure FIGREF2 BIBREF25 .","We sub-sample the input vector INLINEFORM0 into INLINEFORM1 pyramidal levels using the kernel-based approach BIBREF8 , BIBREF9 . Let us assume that we have a kernel INLINEFORM2 with INLINEFORM3 elements. Then, the input vector INLINEFORM4 can be sub-sampled as: DISPLAYFORM0","where INLINEFORM0 represents the stride and INLINEFORM1 .","The number of parameters learned by the linear transformation and the pyramidal transformation with INLINEFORM0 pyramidal levels to map INLINEFORM1 to INLINEFORM2 are INLINEFORM3 and INLINEFORM4 respectively. Thus, pyramidal transformation reduces the parameters of a linear transformation by a factor of INLINEFORM5 . For example, the pyramidal transformation (with INLINEFORM6 and INLINEFORM7 ) learns INLINEFORM8 fewer parameters than the linear transformation."]},{"idx":5,"heading":"Grouped linear transformation for context","paragraphs":["Many RNN architectures apply linear transformations to both the input and context vector. However, this may not be ideal due to the differing semantics of each vector. In many NLP applications including language modeling, the input vector is a dense word embedding which is shared across all contexts for a given word in a dataset. In contrast, the context vector is highly contextualized by the current sequence. The differences between the input and context vector motivate their separate treatment in the PRU architecture.","The weights learned using the linear transformation (Eq. EQREF9 ) are reused over multiple time steps, which makes them prone to over-fitting BIBREF26 . To combat over-fitting, various methods, such as variational dropout BIBREF26 and weight dropout BIBREF0 , have been proposed to regularize these recurrent connections. To further improve generalization abilities while simultaneously enabling the recurrent unit to learn representations at very high dimensional space, we propose to use grouped linear transformation (GLT) instead of standard linear transformation for recurrent connections BIBREF27 . While pyramidal and linear transformations can be applied to transform context vectors, our experimental results in Section SECREF39 suggests that GLTs are more effective.","The linear transformation INLINEFORM0 maps INLINEFORM1 linearly to INLINEFORM2 . Grouped linear transformations break the linear interactions by factoring the linear transformation into two steps. First, a GLT splits the input vector INLINEFORM3 into INLINEFORM4 smaller groups such that INLINEFORM5 . Second, a linear transformation INLINEFORM6 is applied to map INLINEFORM7 linearly to INLINEFORM8 , for each INLINEFORM9 . The INLINEFORM10 resultant output vectors INLINEFORM11 are concatenated to produce the final output vector INLINEFORM12 . DISPLAYFORM0","GLTs learn representations at low dimensionality. Therefore, a GLT requires INLINEFORM0 fewer parameters than the linear transformation. We note that GLTs are subset of linear transformations. In a linear transformation, each neuron receives an input from each element in the input vector while in a GLT, each neuron receives an input from a subset of the input vector. Therefore, GLT is the same as a linear transformation when INLINEFORM1 ."]},{"idx":6,"heading":"Pyramidal Recurrent Unit","paragraphs":["We extend the basic gating architecture of LSTM with the pyramidal and grouped linear transformations outlined above to produce the Pyramidal Recurrent Unit (PRU), whose improved sequence modeling capacity is evidenced in Section SECREF4 .","At time INLINEFORM0 , the PRU combines the input vector INLINEFORM1 and the previous context vector (or previous hidden state vector) INLINEFORM2 using the following transformation function as: DISPLAYFORM0","where INLINEFORM0 indexes the various gates in the LSTM model, and INLINEFORM1 and INLINEFORM2 represents the pyramidal and grouped linear transformations defined in Eqns. EQREF10 and EQREF15 , respectively.","We will now incorporate INLINEFORM0 into LSTM gating architecture to produce PRU. At time INLINEFORM1 , a PRU cell takes INLINEFORM2 , INLINEFORM3 , and INLINEFORM4 as inputs to produce forget INLINEFORM5 , input INLINEFORM6 , output INLINEFORM7 , and content INLINEFORM8 gate signals. The inputs are combined with these gate signals to produce context vector INLINEFORM9 and cell state INLINEFORM10 . Mathematically, the PRU with the LSTM gating architecture can be defined as: DISPLAYFORM0","where INLINEFORM0 represents the element-wise multiplication operation, and INLINEFORM1 and INLINEFORM2 are the sigmoid and hyperbolic tangent activation functions. We note that LSTM is a special case of PRU when INLINEFORM3 = INLINEFORM4 =1."]},{"idx":7,"heading":"Experiments","paragraphs":["To showcase the effectiveness of the PRU, we evaluate the performance on two standard datasets for word-level language modeling and compare with state-of-the-art methods. Additionally, we provide a detailed examination of the PRU and its behavior on the language modeling tasks."]},{"idx":8,"heading":"Set-up","paragraphs":["Following recent works, we compare on two widely used datasets, the Penn Treebank (PTB) BIBREF28 as prepared by BIBREF29 and WikiText2 (WT-2) BIBREF20 . For both datasets, we follow the same training, validation, and test splits as in BIBREF0 .","We extend the language model, AWD-LSTM BIBREF0 , by replacing LSTM layers with PRU. Our model uses 3-layers of PRU with an embedding size of 400. The number of parameters learned by state-of-the-art methods vary from 18M to 66M with majority of the methods learning about 22M to 24M parameters on the PTB dataset. For a fair comparison with state-of-the-art methods, we fix the model size to 19M and vary the value of INLINEFORM0 and hidden layer sizes so that total number of learned parameters is similar across different configurations. We use 1000, 1200, and 1400 as hidden layer sizes for values of INLINEFORM1 =1,2, and 4, respectively. We use the same settings for the WT-2 dataset. We set the number of pyramidal levels INLINEFORM2 to two in our experiments and use average pooling for sub-sampling. These values are selected based on our ablation experiments on the validation set (Section SECREF39 ). We measure the performance of our models in terms of word-level perplexity. We follow the same training strategy as in BIBREF0 .","To understand the effect of regularization methods on the performance of PRUs, we perform experiments under two different settings: (1) Standard dropout: We use a standard dropout BIBREF12 with probability of 0.5 after embedding layer, the output between LSTM layers, and the output of final LSTM layer. (2) Advanced dropout: We use the same dropout techniques with the same dropout values as in BIBREF0 . We call this model as AWD-PRU."]},{"idx":9,"heading":"Results","paragraphs":["Table TABREF23 compares the performance of the PRU with state-of-the-art methods. We can see that the PRU achieves the best performance with fewer parameters.","PRUs achieve either the same or better performance than LSTMs. In particular, the performance of PRUs improves with the increasing value of INLINEFORM0 . At INLINEFORM1 , PRUs outperform LSTMs by about 4 points on the PTB dataset and by about 3 points on the WT-2 dataset. This is explained in part by the regularization effect of the grouped linear transformation (Figure FIGREF1 ). With grouped linear and pyramidal transformations, PRUs learn rich representations at very high dimensional space while learning fewer parameters. On the other hand, LSTMs overfit to the training data at such high dimensions and learn INLINEFORM2 to INLINEFORM3 more parameters than PRUs.","With the advanced dropouts, the performance of PRUs improves by about 4 points on the PTB dataset and 7 points on the WT-2 dataset. This further improves with finetuning on the PTB (about 2 points) and WT-2 (about 1 point) datasets.","For similar number of parameters, the PRU with standard dropout outperforms most of the state-of-the-art methods by large margin on the PTB dataset (e.g. RAN BIBREF7 by 16 points with 4M less parameters, QRNN BIBREF33 by 16 points with 1M more parameters, and NAS BIBREF31 by 1.58 points with 6M less parameters). With advanced dropouts, the PRU delivers the best performance. On both datasets, the PRU improves the perplexity by about 1 point while learning 15-20% fewer parameters.","PRU is a drop-in replacement for LSTM, therefore, it can improve language models with modern inference techniques such as dynamic evaluation BIBREF21 . When we evaluate PRU-based language models (only with standard dropout) with dynamic evaluation on the PTB test set, the perplexity of PRU ( INLINEFORM0 ) improves from 62.42 to 55.23 while the perplexity of an LSTM ( INLINEFORM1 ) with similar settings improves from 66.29 to 58.79; suggesting that modern inference techniques are equally applicable to PRU-based language models."]},{"idx":10,"heading":"Analysis","paragraphs":["It is shown above that the PRU can learn representations at higher dimensionality with more generalization power, resulting in performance gains for language modeling. A closer analysis of the impact of the PRU in a language modeling system reveals several factors that help explain how the PRU achieves these gains.","As exemplified in Table TABREF34 , the PRU tends toward more confident decisions, placing more of the probability mass on the top next-word prediction than the LSTM. To quantify this effect, we calculate the entropy of the next-token distribution for both the PRU and the LSTM using 3687 contexts from the PTB validation set. Figure FIGREF32 shows a histogram of the entropies of the distribution, where bins of size 0.23 are used to effect categories. We see that the PRU more often produces lower entropy distributions corresponding to higher confidences for next-token choices. This is evidenced by the mass of the red PRU curve lying in the lower entropy ranges compared to the blue LSTM's curve. The PRU can produce confident decisions in part because more information is encoded in the higher dimensional context vectors.","The PRU has the ability to model individual words at different resolutions through the pyramidal transform; which provides multiple paths for the gradient to the embedding layer (similar to multi-task learning) and improves the flow of information. When considering the embeddings by part of speech, we find that the pyramid level 1 embeddings exhibit higher variance than the LSTM across all POS categories (Figure FIGREF33 ), and that pyramid level 2 embeddings show extremely low variance. We hypothesize that the LSTM must encode both coarse group similarities and individual word differences into the same vector space, reducing the space between individual words of the same category. The PRU can rely on the subsampled embeddings to account for coarse-grained group similarities, allowing for finer individual word distinctions in the embedding layer. This hypothesis is strengthened by the entropy results described above: a model which can make finer distinctions between individual words can more confidently assign probability mass. A model that cannot make these distinctions, such as the LSTM, must spread its probability mass across a larger class of similar words.","Saliency analysis using gradients help identify relevant words in a test sequence that contribute to the prediction BIBREF34 , BIBREF35 , BIBREF36 . These approaches compute the relevance as the squared norm of the gradients obtained through back-propagation. Table TABREF34 visualizes the heatmaps for different sequences. PRUs, in general, give more relevance to contextual words than LSTMs, such as southeast (sample 1), cost (sample 2), face (sample 4), and introduced (sample 5), which help in making more confident decisions. Furthermore, when gradients during back-propagation are visualized BIBREF37 (Table TABREF34 ), we find that PRUs have better gradient coverage than LSTMs, suggesting PRUs use more features than LSTMs that contributes to the decision. This also suggests that PRUs update more parameters at each iteration which results in faster training. Language model in BIBREF0 takes 500 and 750 epochs to converge with PRU and LSTM as a recurrent unit, respectively."]},{"idx":11,"heading":"Ablation studies","paragraphs":["In this section, we provide a systematic analysis of our design choices. Our training methodology is the same as described in Section SECREF19 with the standard dropouts. For a thorough understanding of our design choices, we use a language model with a single layer of PRU and fix the size of embedding and hidden layers to 600. The word-level perplexities are reported on the validation sets of the PTB and the WT-2 datasets.","The two hyper-parameters that control the trade-off between performance and number of parameters in PRUs are the number of pyramidal levels INLINEFORM0 and groups INLINEFORM1 . Figure FIGREF35 provides a trade-off between perplexity and recurrent unit (RU) parameters.","Variable INLINEFORM0 and fixed INLINEFORM1 : When we increase the number of pyramidal levels INLINEFORM2 at a fixed value of INLINEFORM3 , the performance of the PRU drops by about 1 to 4 points while reducing the total number of recurrent unit parameters by up to 15%. We note that the PRU with INLINEFORM4 at INLINEFORM5 delivers similar performance as the LSTM while learning about 15% fewer recurrent unit parameters.","Fixed INLINEFORM0 and variable INLINEFORM1 : When we vary the value of INLINEFORM2 at fixed number of pyramidal levels INLINEFORM3 , the total number of recurrent unit parameters decreases significantly with a minimal impact on the perplexity. For example, PRUs with INLINEFORM4 and INLINEFORM5 learns 77% fewer recurrent unit parameters while its perplexity (lower is better) increases by about 12% in comparison to LSTMs. Moreover, the decrease in number of parameters at higher value of INLINEFORM6 enables PRUs to learn the representations in high dimensional space with better generalizability (Table TABREF23 ).","Table TABREF43 shows the impact of different transformations of the input vector INLINEFORM0 and the context vector INLINEFORM1 . We make following observations: (1) Using the pyramidal transformation for the input vectors improves the perplexity by about 1 point on both the PTB and WT-2 datasets while reducing the number of recurrent unit parameters by about 14% (see R1 and R4). We note that the performance of the PRU drops by up to 1 point when residual connections are not used (R4 and R6). (2) Using the grouped linear transformation for context vectors reduces the total number of recurrent unit parameters by about 75% while the performance drops by about 11% (see R3 and R4). When we use the pyramidal transformation instead of the linear transformation, the performance drops by up to 2% while there is no significant drop in the number of parameters (R4 and R5).","We set sub-sampling kernel INLINEFORM0 (Eq. EQREF12 ) with stride INLINEFORM1 and size of 3 ( INLINEFORM2 ) in four different ways: (1) Skip: We skip every other element in the input vector. (2) Convolution: We initialize the elements of INLINEFORM3 randomly from normal distribution and learn them during training the model. We limit the output values between -1 and 1 using INLINEFORM4 activation function to make training stable. (3) Avg. pool: We initialize the elements of INLINEFORM5 to INLINEFORM6 . (4) Max pool: We select the maximum value in the kernel window INLINEFORM7 .","Table TABREF45 compares the performance of the PRU with different sampling methods. Average pooling performs the best while skipping give comparable performance. Both of these methods enable the network to learn richer word representations while representing the input vector in different forms, thus delivering higher performance. Surprisingly, a convolution-based sub-sampling method does not perform as well as the averaging method. The INLINEFORM0 function used after convolution limits the range of output values which are further limited by the LSTM gating structure, thereby impeding in the flow of information inside the cell. Max pooling forces the network to learn representations from high magnitude elements, thus distinguishing features between elements vanishes, resulting in poor performance."]},{"idx":12,"heading":"Conclusion","paragraphs":["We introduce the Pyramidal Recurrent Unit, which better model contextual information by admitting higher dimensional representations with good generalizability. When applied to the task of language modeling, PRUs improve perplexity across several settings, including recent state-of-the-art systems. Our analysis shows that the PRU improves the flow of gradient and expand the word embedding subspace, resulting in more confident decisions. Here we have shown improvements for language modeling. In future, we plan to study the performance of PRUs on different tasks, including machine translation and question answering. In addition, we will study the performance of the PRU on language modeling with more recent inference techniques, such as dynamic evaluation and mixture of softmax."]},{"idx":13,"heading":"Acknowledgments","paragraphs":["This research was supported by NSF (IIS 1616112, III 1703166), Allen Distinguished Investigator Award, and gifts from Allen Institute for AI, Google, Amazon, and Bloomberg. We are grateful to Aaron Jaech, Hannah Rashkin, Mandar Joshi, Aniruddha Kembhavi, and anonymous reviewers for their helpful comments."]},{"idx":14,"heading":"Figures and Tables","paragraphs":["Figure 1: Comparison of training (solid lines) and validation (dashed lines) perplexities on the Penn Treebank with standard dropout for pyramidal recurrent units (PRU) and LSTM. PRUs learn latent representations in very high-dimensional space with good generalizability and fewer parameters. See Section 3 for more details about PRUs. Best viewed in color.","Figure 2: Block diagram visualizing the transformations in pyramidal recurrent unit (left) and the LSTM (bottom right) along with the LSTM gating architecture (top right). Blue, red, green (or orange), and purple signify the current input xt, output of the previous cell ht−1, the output of transformations, and the fused output, respectively. The color intensity is used to represent sub-sampling and grouping operations.","Table 1: Comparison of single model word-level perplexity of our model with state-of-the-art on validation and test sets of Penn Treebank and Wikitext-2 dataset. For evaluation, we select the model with minimum validation loss. Lower perplexity value represents better performance.","Figure 3: Histogram of the entropies of next-token distributions predicted by the PRU (mean 3.80) and the LSTM (mean 3.93) on the PTB validation set. Lower entropy values indicate higher confidence decisions, which is desirable if decisions are often correct.","Figure 4: Variance of learned word embeddings for different categories of words on the PTB validation set. We compute the variance of a group of embeddings as the average squared euclidean distance to their mean. Higher variance may allow for better intra-category distinctions. The PRU with pyramid levels 1 and 2 is shown.","Table 2: Qualitative comparison between the LSTM and the PRU: (a) Gradient-based saliency analysis along with top-5 predicted words. (b) Gradients during back-propagation. For computing the gradients for a given test sequence, the top-1 predicted word was used as the true predicted word. Best viewed in color.","Figure 5: Impact of number of groups g and pyramidal levels K on the perplexity. Reduction in recurrent unit (RU) parameters is computed with respect to LSTM. Lower perplexity value represents better performance.","Table 4: Impact of different sub-sampling methods on the word-level perplexity (lower is better). We used g=1 and K=4 in our experiments.","Table 3: Impact of different transformations used for processing input and context vectors (LT - linear transformation, PT - pyramidal transformation, and GLT - grouped linear transformation). Here, † represents that PT was used without residual connection, PPL represents word-level perplexity (lower is better), and the number of parameters are in million. We used K=g=4 in our experiments."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1808.09716","title":"What can we learn from Semantic Tagging?","sections":[{"idx":0,"heading":"Abstract","paragraphs":["We investigate the effects of multi-task learning using the recently introduced task of semantic tagging. We employ semantic tagging as an auxiliary task for three different NLP tasks: part-of-speech tagging, Universal Dependency parsing, and Natural Language Inference. We compare full neural network sharing, partial neural network sharing, and what we term the learning what to share setting where negative transfer between tasks is less likely. Our findings show considerable improvements for all tasks, particularly in the learning what to share setting, which shows consistent gains across all tasks."]},{"idx":1,"heading":"Introduction","paragraphs":["Multi-task learning (MTL) is a recently resurgent approach to machine learning in which multiple tasks are simultaneously learned. By optimising the multiple loss functions of related tasks at once, multi-task learning models can achieve superior results compared to models trained on a single task. The key principle is summarized by BIBREF0 as “MTL improves generalization by leveraging the domain-specific information contained in the training signals of related tasks\". Neural MTL has become an increasingly successful approach by exploiting similarities between Natural Language Processing (NLP) tasks BIBREF1 , BIBREF2 , BIBREF3 . Our work builds upon BIBREF4 , who demonstrate that employing semantic tagging as an auxiliary task for Universal Dependency BIBREF5 part-of-speech tagging can lead to improved performance.","The objective of this paper is to investigate whether learning to predict lexical semantic categories can be beneficial to other NLP tasks. To achieve this we augment single-task models (ST) with an additional classifier to predict semantic tags and jointly optimize for both the original task and the auxiliary semantic tagging task. Our hypothesis is that learning to predict semantic tags as an auxiliary task can improve performance of single-task systems. We believe that this is, among other factors, due to the following:","We test our hypothesis on three disparate NLP tasks: (i) Universal Dependency part-of-speech tagging (UPOS), (ii) Universal Dependency parsing (UD DEP), a complex syntactic task; and (iii) Natural Language Inference (NLI), a complex task requiring deep natural language understanding."]},{"idx":2,"heading":"Semantic Tagging","paragraphs":["Semantic tagging BIBREF4 , BIBREF7 is the task of assigning language-neutral semantic categories to words. It is designed to overcome a lack of semantic information syntax-oriented part-of-speech tagsets, such as the Penn Treebank tagset BIBREF8 , usually have. Such tagsets exclude important semantic distinctions, such as negation and modals, types of quantification, named entity types, and the contribution of verbs to tense, aspect, or event.","The semantic tagset is language-neutral, abstracts over part-of-speech and named-entity classes, and includes fine-grained semantic information. The tagset consists of 80 semantic tags grouped in 13 coarse-grained classes. The tagset originated in the Parallel Meaning Bank (PMB) project BIBREF9 , where it contributes to compositional semantics and cross-lingual projection of semantic representations. Recent work has highlighted the utility of the tagset as a conduit for evaluating the semantics captured by vector representations BIBREF10 , or employed it in an auxiliary tagging task BIBREF4 , as we do in this work."]},{"idx":3,"heading":"Learning What to Share","paragraphs":["Recently, there has been an increasing interest in the development of models which are trained to learn what to (and what not to) share between a set of tasks, with the general aim of preventing negative transfer when the tasks are not closely related BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 . Our Learning What to Share setting is based on this idea and closely related to BIBREF15 's shared layer architecture.","Specifically, a layer $\\vec{h}_{X}$ which is shared between the main task and the auxiliary task is split into two subspaces: a shared subspace $\\vec{h}_{X_{S}}$ and a private subspace $\\vec{h}_{X_{P}}$ . The interaction between the shared subspaces is modulated via a sigmoidal gating unit applied to a set of learned weights, as seen in Equations ( 9 ) and () where $\\vec{h}_{X_{S(main)}}$ and $\\vec{h}_{X_{S(aux)}}$ are the main and auxiliary tasks' shared layers, $W_{a\\rightarrow m}$ and $W_{m\\rightarrow a}$ are learned weights, and $\\sigma $ is a sigmoidal function.","$$\\vec{h}_{X_{S(main)}} &= \\vec{h}_{X_{S(main)}} \\sigma (\\vec{h}_{X_{S(aux)}} W_{a\\rightarrow m})\\\\ \\vec{h}_{X_{S(aux)}} &= \\vec{h}_{X_{S(aux)}} \\sigma (\\vec{h}_{X_{S(main)}} W_{m\\rightarrow a})$$ (Eq. 9)","Unlike BIBREF15 's Shared-Layer Architecture, in our setup each task has its own shared subspace rather than one common shared layer. This enables the sharing of different parameters in each direction (i.e., from main to auxiliary task and from auxiliary to main task), allowing each task to choose what to learn from the other, rather than having “one shared layer to capture the shared information for all the tasks” as in BIBREF15 ."]},{"idx":4,"heading":"Multi-Task Learning Settings","paragraphs":["We implement three neural MTL settings, shown in Figure 1 . They differ in the way the network's parameters are shared between the tasks:"]},{"idx":5,"heading":"Data","paragraphs":["In the UPOS tagging experiments, we utilize the UD 2.0 English corpus BIBREF16 for the POS tagging and the semantically tagged PMB release 0.1.0 (sem-PMB) for the MTL settings. Note that there is no overlap between the two datasets. Conversely, for the UD DEP and NLI experiments there is a complete overlap between the datasets of main and auxiliary tasks, i.e., each instance is labeled with both the main task's labels and semantic tags. We use the Stanford POS Tagger BIBREF17 trained on sem-PMB to tag the UD corpus and NLI datasets with semantic tags, and then use those assigned tags for the MTL settings of our dependency parsing and NLI models. We find that this approach leads to better results when the main task is only loosely related to the auxiliary task. The UD DEP experiments use the English UD 2.0 corpus, and the NLI experiments use the SNLI BIBREF18 and SICK-E datasets BIBREF19 . The provided train, development, and test splits are used for all datasets. For sem-PMB, the silver and gold parts are used for training and testing respectively."]},{"idx":6,"heading":"Experiments","paragraphs":["We run four experiments for each of the four tasks (UPOS, UD DEP, SNLI, SICK-E), one using the ST model and one for each of the three MTL settings. Each experiment is run five times, and the average of the five runs is reported. We briefly describe the ST models and refer the reader to the original work for further details due to a lack of space. For reproducibility, detailed diagrams of the MTL models for each task and their hyperparameters can be found in Appendix \"MTL setting Diagrams, Preprocessing, and Hyperparameters\" ."]},{"idx":7,"heading":"Universal Dependency POS Tagging","paragraphs":["Our tagging model uses a basic contextual one-layer bi-LSTM BIBREF20 that takes in word embeddings and produces a sequence of recurrent states which can be viewed as contextualized representations. The recurrent $r_n$ state from the bi-LSTM corresponding to each time-step $t_n$ is passed through a dense layer with a softmax activation to predict the token's tag.","In each of the MTL settings a softmax classifier is added to predict a token's semantic tag and the model is then jointly trained on the concatenation of the sem-PMB and UPOS tagging data to minimize the sum of softmax cross-entropy losses of both the main (UPOS tagging) and auxiliary (semantic tagging) tasks."]},{"idx":8,"heading":"Universal Dependency Parsing","paragraphs":["We employ a parsing model that is based on BIBREF21 BIBREF21 . The model's embeddings layer is a concatenation of randomly initialized word embeddings and character-based word representations added to pre-trained word embeddings, which are passed through a 4-layer stacked bi-LSTM. Unlike BIBREF21 , our model jointly learns to perform UPOS tagging and parsing, instead of treating them as separate tasks. Therefore, instead of tag embeddings, we add a softmax classifier to predict UPOS tags after the first bi-LSTM layer. The outputs from that layer and the UPOS softmax prediction vectors are both concatenated to the original embedding layer and passed to the second bi-LSTM layer. The output of the last bi-LSTM is then used as input for four dense layers with a ReLU activation, producing four vector representations: a word as a dependent seeking its head; a word as a head seeking all its dependents; a word as a dependent deciding on its label; a word as head deciding on the labels of its dependents. These representations are then passed to biaffine and affine softmax classifiers to produce a fully-connected labeled probabilistic dependency graph BIBREF21 . Finally, a non-projective maximum spanning tree parsing algorithm BIBREF22 , BIBREF23 is used to obtain a well-formed dependency tree.","Similarly to UPOS tagging, an additional softmax classifier is used to predict a token's semantic tag in each of the MTL settings, as both tasks are jointly learned. In the FSN setting, the 4-layer stacked bi-LSTM is entirely shared. In the PSN setting the semantic tags are predicted from the second layer's hidden states, and the final two layers are devoted to the parsing task. In the LWS setting, the first two layers of the bi-LSTM are split into a private bi-LSTM $_{private}$ and a shared bi-LSTM $_{shared}$ for each of the tasks with the interaction between the shared subspaces being modulated via a gating unit. Then, two bi-LSTM layers that are devoted to parsing only are stacked on top."]},{"idx":9,"heading":"Natural Language Inference","paragraphs":["We base our NLI model on BIBREF25 's Enhanced Sequential Inference Model which uses a bi-LSTM to encode the premise and hypothesis, computes a soft-alignment between premise and hypothesis' representations using an attention mechanism, and employs an inference composition bi-LSTM to compose local inference information sequentially. The MTL settings are implemented by adding a softmax classifier to predict semantic tags at the level of the encoding bi-LSTM, with rest of the model unaltered. In the FSN setting, the hidden states of the encoding bi-LSTM are directly passed as input to the softmax classifier. In the PSN setting an earlier bi-LSTM layer is used to predict the semantic tags and the output from that is passed on to the encoding bi-LSTM which is stacked on top. This follows BIBREF26 's hierarchical approach. In the LWS setting, a bi-LSTM layer with private and shared subspaces is used for semantic tagging and for the ESIM model's encoding layer. In all MTL settings, the bi-LSTM used for semantic tagging is pre-trained on the sem-PMB data."]},{"idx":10,"heading":"Results and Discussion","paragraphs":["Results for all tasks are shown in Table 1 . In line with BIBREF4 's findings, the FSN setting leads to an improvement for UPOS tagging. POS tagging, a sequence labeling task, can be seen as the most closely related to semantic tagging, therefore negative transfer is minimal and the full sharing of parameters is beneficial. Surprisingly, the FSN setting also leads improvements for UD DEP. Indeed, for UD DEP, all of the MTL models outperform the ST model by increasing margins. For the NLI tasks, however, there is a clear degradation in performance.","The PSN setting shows mixed results and does not show a clear advantage over FSN for UPOS and UD DEP. This suggests that adding task-specific layers after fully-shared ones does not always enable sufficient task specialization. For the NLI tasks however, PSN is clearly preferable to FSN, especially for the small-sized SICK-E dataset where the FSN model fails to adequately learn.","As a sentence-level task, NLI is functionally dissimilar to semantic tagging. However, it is a task which requires deep understanding of natural language semantics and can therefore conceivably benefit from the signal provided by semantic tagging. Our results demonstrate that it is possible to leverage this signal given a selective sharing setup where negative transfer can be minimized. Indeed, for the NLI tasks, only the LWS setting leads to improvements over the ST models. The improvement is larger for the SICK-E task which has a much smaller training set and therefore stands to learn more from the semantic tagging signal. For all tasks, it can be observed that the LWS models outperform the rest of the models. This is in line with our expectations with the findings from previous work BIBREF12 , BIBREF15 that selective sharing outperforms full network and partial network sharing."]},{"idx":11,"heading":"Analysis","paragraphs":["In addition to evaluating performance directly, we attempt to qualify how semtags affect performance with respect to each of the SNLI MTL settings."]},{"idx":12,"heading":"Qualitative analyses","paragraphs":["The fact that NLI is a sentence-level task, while semantic tags are word-level annotations presents a difficulty in measuring the effect of semantic tags on the systems' performance, as there is no one-to-one correspondence between a correct label and a particular semantic tag. We therefore employ the following method in order to assess the contribution of semantic tags. Given the performance ranking of all our systems — $FSN < ST < PSN < LWS$ — we make a pairwise comparison between the output of a superior system $S_{sup}$ and an inferior system $S_{inf}$ . This involves taking the pairs of sentences that every $S_{sup}$ classifies correctly, but some $S_{inf}$ does not. Given that FSN is the worst performing system and, as such, has no `worse' system for comparison, we are left with six sets of sentences: ST-FSN, PSN-FSN, PSN-ST, LWS-PSN, LWS-ST, and LWS-FSN. To gain insight as to where a given system $S_{sup}$ performs better than a given $S_{inf}$ , we then sort each comparison sentence set by the frequency of semtags predicted therein, which are normalized by dividing by their frequency in the full SNLI test set.","We notice interesting patterns, visible in Figure 2 . Specifically, PSN appears markedly better at sentences with named entities (ART, PER, GEO, ORG) and temporal entities (DOM) than both ST and the FSN. Marginal improvements are also observed for sentences with negation and reflexive pronouns. The LWS setting continues this pattern, with additional improvements observable for sentences with the HAP tag for names of events, SST for subsective attributes, and the ROL tag for role nouns."]},{"idx":13,"heading":"Contribution of semantic tagging","paragraphs":["To assess the contribution of the semantic tagging auxiliary task independent of model architecture and complexity we run three additional SNLI experiments — one for each MTL setting — where the model architectures are unchanged but the auxiliary tasks are assigned no weight (i.e. do not affect the learning). The results confirm our previous findings that, for NLI, the semantic tagging auxiliary task only improves performance in a selective sharing setting, and hurts it otherwise: i) the FSN system which had performed below ST improves to equal it and ii) the PSN and LWS settings both see a drop to ST-level performance."]},{"idx":14,"heading":"Conclusions","paragraphs":["We present a comprehensive evaluation of MTL using a recently proposed task of semantic tagging as an auxiliary task. Our experiments span three types of NLP tasks and three MTL settings. The results of the experiments show that employing semantic tagging as an auxiliary task leads to improvements in performance for UPOS tagging and UD DEP in all MTL settings. For the SNLI tasks, requiring understanding of phrasal semantics, the selective sharing setup we term Learning What to Share holds a clear advantage. Our work offers a generalizable framework for the evaluation of the utility of an auxiliary task."]},{"idx":15,"heading":"UPOS Tagging","paragraphs":["fig:upos shows the three MTL models used for UPOS. All hyperparameters were tuned with respect to loss on the English UD 2.0 UPOS validation set. We trained for 20 epochs with a batch size of 128 and optimized using Adam BIBREF27 with a learning rate of $0.0001$ . We weight the auxiliary semantic tagging loss with $\\lambda $ = $0.1$ . The pre-trained word embeddings we used are GloVe embeddings BIBREF28 of dimension 100 trained on 6 billion tokens of Wikipedia 2014 and Gigaword 5. We applied dropout and recurrent dropout with a probability of $0.3$ to all bi-LSTMs."]},{"idx":16,"heading":"UD DEP","paragraphs":["fig:dep shows the three MTL models for UD DEP. We use the gold tokenization. All hyperparameters were tuned with respect to loss on the English UD 2.0 UD validation set. We trained for 15 epochs with a batch size of 50 and optimized using Adam with a learning rate of $2e-3$ . We weight the auxiliary semantic tagging loss with $\\lambda $ = $0.5$ . The pre-trained word embeddings we use are GloVe embeddings of dimension 100 trained on 6 billion tokens of Wikipedia 2014 and Gigaword 5. We applied dropout with a probability of $0.33$ to all bi-LSTM, embedding layers, and non-output dense layers."]},{"idx":17,"heading":"NLI","paragraphs":["fig:nli shows the three MTL models for NLI. All hyperparameters were tuned with respect to loss on the SNLI and SICK-E validation datasets (separately). For the SNLI experiments, we trained for 37 epochs with a batch size of 128. For the SICK-E experiments, we trained for 20 epochs with a batch size of 8. Note that the ESIM model was designed for the SNLI dataset, therefore performance is non-optimal for SICK-E. For both sets of experiments: we optimized using Adam with a learning rate of $0.00005$ ; we weight the auxiliary semantic tagging loss with $\\lambda $ = $0.1$ ; the pre-trained word embeddings we use are GloVe embeddings of dimension 300 trained on 840 billion tokens of Common Crawl; and we applied dropout and recurrent dropout with a probability of $0.3$ to all bi-LSTM, and non-output dense layers."]},{"idx":18,"heading":"SNLI model output analysis","paragraphs":["tab:examples shows demonstrative examples from the SNLI test set on which the Learning What to Share (LWS) model outperforms the single-task (ST) model. The examples cover all possible combinations of entailment classes. tab:semtags explains the relevant part of the semantic tagset. tab:fscore shows the per-label precision and recall scores."]},{"idx":19,"heading":"Figures and Tables","paragraphs":["Figure 1: Our three multi-task learning settings: (A) fully shared networks, (B) partially shared networks, and (C) Learning What to Share. Layers are mathematically denoted by vectors and the connections between them, represented by arrows, are mathematically denoted by matrices of weights. S indicates a shared layer, P a private layer, and X a layer with shared and private subspaces.","Table 1: Results for single-task models (ST), fullyshared networks (FSN), partially-shared networks (PSN), and learning what to share (LWS). All scores are reported as accuracy, except UD DEP for which we report LAS/UAS F1 score.","Figure 2: Normalized semantic tag frequencies for all six sets of sentences. X - Y denotes the set of sentences correctly classified by model X but misclassified by model Y.","Figure 3: The three MTL settings for each task. Layers dimensions are displayed in brackets.","Table 3: The list of semantic tags found in Table 2.","Table 4: Per-label precision (left) and recall (right) for all models.","Table 2: Examples of the entailment problems from SNLI which are incorrectly classified by the ST model but correctly classified by the LWS model. Automatically assigned semantic tags are in superscript."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1809.02279","title":"Cell-aware Stacked LSTMs for Modeling Sentences","sections":[{"idx":0,"heading":"Abstract","paragraphs":["We propose a method of stacking multiple long short-term memory (LSTM) layers for modeling sentences. In contrast to the conventional stacked LSTMs where only hidden states are fed as input to the next layer, our architecture accepts both hidden and memory cell states of the preceding layer and fuses information from the left and the lower context using the soft gating mechanism of LSTMs. Thus the proposed stacked LSTM architecture modulates the amount of information to be delivered not only in horizontal recurrence but also in vertical connections, from which useful features extracted from lower layers are effectively conveyed to upper layers. We dub this architecture Cell-aware Stacked LSTM (CAS-LSTM) and show from experiments that our models achieve state-of-the-art results on benchmark datasets for natural language inference, paraphrase detection, and sentiment classification."]},{"idx":1,"heading":"Introduction","paragraphs":["In the field of natural language processing (NLP), the most prevalent neural approach to obtaining sentence representations is to use recurrent neural networks (RNNs), where words in a sentence are processed in a sequential and recurrent manner. Along with their intuitive design, RNNs have shown outstanding performance across various NLP tasks e.g. language modeling BIBREF0 , BIBREF1 , machine translation BIBREF2 , BIBREF3 , BIBREF4 , text classification BIBREF5 , BIBREF6 , and parsing BIBREF7 , BIBREF8 .","Among several variants of the original RNN BIBREF9 , gated recurrent architectures such as long short-term memory (LSTM) BIBREF10 and gated recurrent unit (GRU) BIBREF2 have been accepted as de-facto standard choices for RNNs due to their capability of addressing the vanishing and exploding gradient problem and considering long-term dependencies. Gated RNNs achieve these properties by introducing additional gating units that learn to control the amount of information to be transferred or forgotten BIBREF11 , and are proven to work well without relying on complex optimization algorithms or careful initialization BIBREF12 .","Meanwhile, the common practice for further enhancing the expressiveness of RNNs is to stack multiple RNN layers, each of which has distinct parameter sets (stacked RNN) BIBREF13 , BIBREF14 . In stacked RNNs, the hidden states of a layer are fed as input to the subsequent layer, and they are shown to work well due to increased depth BIBREF15 or their ability to capture hierarchical time series BIBREF16 which are inherent to the nature of the problem being modeled.","However this setting of stacking RNNs might hinder the possibility of more sophisticated recurrence-based structures since the information from lower layers is simply treated as input to the next layer, rather than as another class of state that participates in core RNN computations. Especially for gated RNNs such as LSTMs and GRUs, this means that layer-to-layer connections cannot fully benefit from the carefully constructed gating mechanism used in temporal transitions. Some recent work on stacking RNNs suggests alternative methods that encourage direct and effective interaction between RNN layers by adding residual connections BIBREF17 , BIBREF18 , by shortcut connections BIBREF18 , BIBREF19 , or by using cell states of LSTMs BIBREF20 , BIBREF21 .","In this paper, we propose a method of constructing multi-layer LSTMs where cell states are used in controlling the vertical information flow. This system utilizes states from the left and the lower context equally in computation of the new state, thus the information from lower layers is elaborately filtered and reflected through a soft gating mechanism. Our method is easy-to-implement, effective, and can replace conventional stacked LSTMs without much modification of the overall architecture.","We call the proposed architecture Cell-aware Stacked LSTM, or CAS-LSTM, and evaluate our method on multiple benchmark datasets: SNLI BIBREF22 , MultiNLI BIBREF23 , Quora Question Pairs BIBREF24 , and SST BIBREF25 . From experiments we show that the CAS-LSTMs consistently outperform typical stacked LSTMs, opening the possibility of performance improvement of architectures that use stacked LSTMs.","Our contribution is summarized as follows.","This paper is organized as follows. We give a detailed description about the proposed method in § SECREF2 . Experimental results are given in § SECREF3 . We study prior work related to our objective in § SECREF4 and conclude in § SECREF5 ."]},{"idx":2,"heading":"Model Description","paragraphs":["In this section, we give a detailed formulation of the architectures used in experiments."]},{"idx":3,"heading":"Notation","paragraphs":["Throughout this paper, we denote matrices as boldface capital letters ( INLINEFORM0 ), vectors as boldface lowercase letters ( INLINEFORM1 ), and scalars as normal italic letters ( INLINEFORM2 ). For LSTM states, we denote a hidden state as INLINEFORM3 and a cell state as INLINEFORM4 . Also, a layer index of INLINEFORM5 or INLINEFORM6 is denoted by superscript and a time index is denoted by a subscript, i.e. INLINEFORM7 indicates the hidden state at time INLINEFORM8 and layer INLINEFORM9 . INLINEFORM10 means the element-wise multiplication between two vectors. We write INLINEFORM11 -th component of vector INLINEFORM12 as INLINEFORM13 . All vectors are assumed to be column vectors."]},{"idx":4,"heading":"Stacked LSTMs","paragraphs":["While there exist various versions of LSTM formulation, in this work we use the following, one of the most common versions: DISPLAYFORM0 DISPLAYFORM1","where INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 , INLINEFORM3 , INLINEFORM4 are trainable parameters. INLINEFORM5 and INLINEFORM6 are the sigmoid activation and the hyperbolic tangent activation function respectively. Also we assume that INLINEFORM7 where INLINEFORM8 is the INLINEFORM9 -th input to the network.","The input gate INLINEFORM0 and the forget gate INLINEFORM1 control the amount of information transmitted from INLINEFORM2 and INLINEFORM3 , the candidate cell state and the previous cell state, to the new cell state INLINEFORM4 . Similarly the output gate INLINEFORM5 soft-selects which portion of the cell state INLINEFORM6 is to be used in the final hidden state.","We can clearly see that cell states ( INLINEFORM0 , INLINEFORM1 , INLINEFORM2 ) play a crucial role in forming horizontal recurrence. However the current formulation does not consider INLINEFORM3 , the cell state from INLINEFORM4 -th layer, in computation and thus the lower context is reflected only through the rudimentary way, hindering the possibility of controlling vertical information flow."]},{"idx":5,"heading":"Cell-aware Stacked LSTMs","paragraphs":["Now we extend the stacked LSTM formulation defined above to address the problem noted in the previous subsection. To enhance the interaction between layers in a way similar to how LSTMs keep and forget the information from the previous time step, we introduce the additional forget gate INLINEFORM0 that determines whether to accept or ignore the signals coming from the previous layer. Therefore the proposed Cell-aware Stacked LSTM is formulated as follows: DISPLAYFORM0 DISPLAYFORM1","where INLINEFORM0 and INLINEFORM1 . INLINEFORM2 can either be a vector of constants or parameters. When INLINEFORM3 , the equations defined in the previous subsection are used. Therefore, it can be said that each non-bottom layer of CAS-LSTM accepts two sets of hidden and cell states—one from the left context and the other from the below context. The left and the below context participate in computation with the equivalent procedure so that the information from lower layers can be efficiently propagated. Fig. FIGREF1 compares CAS-LSTM to the conventional stacked LSTM architecture, and Fig. FIGREF8 depicts the computation flow of the CAS-LSTM.","We argue that considering INLINEFORM0 in computation is beneficial for the following reasons. First, INLINEFORM1 contains additional information compared to INLINEFORM2 since it is not filtered by INLINEFORM3 . Thus a model that directly uses INLINEFORM4 does not rely solely on INLINEFORM5 for extracting information, due to the fact that it has access to the raw information INLINEFORM6 , as in temporal connections. In other words, INLINEFORM7 no longer has to take all responsibility for selecting useful features for both horizontal and vertical transitions, and the burden of selecting information is shared with INLINEFORM8 .","Another advantage of using the INLINEFORM0 lies in the fact that it directly connects INLINEFORM1 and INLINEFORM2 . This direct connection helps and stabilizes training, since the terminal error signals can be easily backpropagated to model parameters. Fig. FIGREF23 illustrates paths between the two cell states.","We find experimentally that there is little difference between letting INLINEFORM0 be constant and letting it be trainable parameters, thus we set INLINEFORM1 in all experiments. We also experimented with the architecture without INLINEFORM2 i.e. two cell states are combined by unweighted summation similar to multidimensional RNNs BIBREF27 , and found that it leads to performance degradation and unstable convergence, likely due to mismatch in the range of cell state values between layers ( INLINEFORM3 for the first layer and INLINEFORM4 for the others). Experimental results on various INLINEFORM5 are presented in § SECREF3 .","The idea of having multiple states is also related to tree-structured RNNs BIBREF29 , BIBREF30 . Among them, tree-structured LSTMs (Tree-LSTMs) BIBREF31 , BIBREF32 , BIBREF33 are similar to ours in that they use both hidden and cell states from children nodes. In Tree-LSTMs, states for all children nodes are regarded as input, and they participate in the computation equally through weight-shared (in Child-Sum Tree-LSTMs) or weight-unshared (in INLINEFORM0 -ary Tree-LSTMs) projection. From this perspective, each CAS-LSTM layer (where INLINEFORM1 ) can be seen as a binary Tree-LSTM where the structures it operates on are fixed to right-branching trees. The use of cell state in computation could be one reason that Tree-LSTMs perform better than sequential LSTMs even when trivial trees (strictly left- or right-branching) are given BIBREF34 .","Multidimensional RNNs (MDRNN) are an extension of 1D sequential RNNs that can accept multidimensional input e.g. images, and have been successfully applied to image segmentation BIBREF26 and handwriting recognition BIBREF27 . Notably multidimensional LSTMs (MDLSTM) BIBREF27 have an analogous formulation to ours except the INLINEFORM0 term and the fact that we use distinct weights per column (or `layer' in our case). From this view, CAS-LSTM can be seen as a certain kind of MDLSTM that accepts a 2D input INLINEFORM1 . Grid LSTMs BIBREF21 also take INLINEFORM2 inputs but emit INLINEFORM3 outputs, which is different from our case where a single set of hidden and cell states is produced."]},{"idx":6,"heading":"Sentence Encoders","paragraphs":["The sentence encoder network we use in our experiments takes INLINEFORM0 words (assumed to be one-hot vectors) as input. The words are projected to corresponding word representations: INLINEFORM1 where INLINEFORM2 . Then INLINEFORM3 is fed to a INLINEFORM4 -layer CAS-LSTM model, resulting in the representations INLINEFORM5 . The sentence representation, INLINEFORM6 , is computed by max-pooling INLINEFORM7 over time as in the work of BIBREF35 . Similar to their results, from preliminary experiments we found that the max-pooling performs consistently better than mean- and last-pooling.","To make models more expressive, a bidirectional CAS-LSTM network may also be used. In the bidirectional case, the forward representations INLINEFORM0 and the backward representations INLINEFORM1 are concatenated and max-pooled to yield the sentence representation INLINEFORM2 . We call this bidirectional architecture Bi-CAS-LSTM in experiments."]},{"idx":7,"heading":"Top-layer Classifiers","paragraphs":["For the natural language inference experiments, we use the following heuristic function proposed by BIBREF36 in feature extraction: DISPLAYFORM0","where INLINEFORM0 means vector concatenation, and INLINEFORM1 and INLINEFORM2 are applied element-wise.","And we use the following function in paraphrase identification experiments: DISPLAYFORM0","as in the work of BIBREF37 .","For sentiment classification, we use the sentence representation itself. DISPLAYFORM0","We feed the feature extracted from INLINEFORM0 as input to the MLP classifier with ReLU activation followed by the fully-connected softmax layer to predict the label distribution: DISPLAYFORM0","where INLINEFORM0 , INLINEFORM1 is the number of label classes, and INLINEFORM2 the dimension of the MLP output,"]},{"idx":8,"heading":"Experiments","paragraphs":["We evaluate our method on natural language inference (NLI), paraphrase identification (PI), and sentiment classification. We also conduct analysis on gate values and experiments on model variants. For detailed experimental settings, we refer readers to the supplemental material.","For the NLI and PI tasks, there exists recent work specializing in sentence pair classification. However in this work we confine our model to the architecture that encodes each sentence using a shared encoder without any inter-sentence interaction, in order to focus on the effectiveness of the models in extracting semantics. But note that the applicability of CAS-LSTM is not limited to sentence encoding based approaches."]},{"idx":9,"heading":"Natural Language Inference","paragraphs":["For the evaluation of performance of the proposed method on the NLI task, SNLI BIBREF22 and MultiNLI BIBREF23 datasets are used. The objective of both datasets is to predict the relationship between a premise and a hypothesis sentence: entailment, contradiction, and neutral. SNLI and MultiNLI datasets are composed of about 570k and 430k premise-hypothesis pairs respectively.","GloVe pretrained word embeddings BIBREF49 are used and remain fixed during training. The dimension of encoder states ( INLINEFORM0 ) is set to 300 and a 1024D MLP with one or two hidden layers is used. We apply dropout BIBREF50 to the word embeddings and the MLP layers. The features used as input to the MLP classifier are extracted following Eq. EQREF28 .","Table TABREF32 and TABREF33 contain results of the models on SNLI and MultiNLI datasets. In SNLI, our best model achieves the new state-of-the-art accuracy of 87.0% with relatively fewer parameters. Similarly in MultiNLI, our models match the accuracy of state-of-the-art models in both in-domain (matched) and cross-domain (mismatched) test sets. Note that only the GloVe word vectors are used as word representations, as opposed to some models that introduce character-level features. It is also notable that our proposed architecture does not restrict the selection of pooling method; the performance could further be improved by replacing max-pooling with other advanced algorithms e.g. intra-sentence attention BIBREF39 and generalized pooling BIBREF19 ."]},{"idx":10,"heading":"Paraphrase Identification","paragraphs":["We use Quora Question Pairs dataset BIBREF24 in evaluating the performance of our method on the PI task. The dataset consists of over 400k question pairs, and each pair is annotated with whether the two sentences are paraphrase of each other or not.","Similar to the NLI experiments, GloVe pretrained vectors, 300D encoders, and 1024D MLP are used. The number of CAS-LSTM layers is fixed to 2 in PI experiments. Two sentence vectors are aggregated using Eq. EQREF29 and fed as input to the MLP. The results on the Quora Question Pairs dataset are summarized in Table TABREF34 . Again we can see that our models outperform other models by large margin, achieving the new state of the art."]},{"idx":11,"heading":"Sentiment Classification","paragraphs":["In evaluating sentiment classification performance, the Stanford Sentiment Treebank (SST) BIBREF25 is used. It consists of about 12,000 binary-parsed sentences where constituents (phrases) of each parse tree are annotated with a sentiment label (very positive, positive, neutral, negative, very negative). Following the convention of prior work, all phrases and their labels are used in training but only the sentence-level data are used in evaluation.","In evaluation we consider two settings, namely SST-2 and SST-5, the two differing only in their level of granularity with regard to labels. In SST-2, data samples annotated with `neutral' are ignored from training and evaluation. The two positive labels (very positive, positive) are considered as the same label, and similarly for the two negative labels. As a result 98,794/872/1,821 data samples are used in training/validation/test, and the task is considered as a binary classification problem. In SST-5, data are used as-is and thus the task is a 5-class classification problem. All 318,582/1,101/2,210 data samples for training/validation/test are used in the SST-5 setting.","We use 300D GloVe vectors, 2-layer 150D or 300D encoders, and a 300D MLP classifier for the models, however unlike previous experiments we tune the word embeddings during training. The results on SST are listed in Table TABREF35 . Our models achieve the new state-of-the-art accuracy on SST-2 and competitive accuracy on SST-5, without utilizing parse tree information."]},{"idx":12,"heading":"Forget Gate Analysis","paragraphs":["To inspect the effect of the additional forget gate, we investigate how the values of vertical forget gates are distributed. We sample 1,000 random sentences from the development set of the SNLI dataset, and use the 3-layer CAS-LSTM model trained on the SNLI dataset to compute gate values.","If all values from a vertical forget gate INLINEFORM0 were to be 0, this would mean that the introduction of the additional forget gate is meaningless and the model would reduce to a plain stacked LSTM. On the contrary if all values were 1, meaning that the vertical forget gates were always open, it would be impossible to say that the information is modulated effectively.","Fig. FIGREF40 and FIGREF40 represent histograms of the vertical forget gate values from the second and the third layer. From the figures we can validate that the trained model does not fall into the degenerate case where vertical forget gates are ignored. Also the figures show that the values are right-skewed, which we conjecture to be a result of focusing more on a strong interaction between adjacent layers.","To further verify that the gate values are diverse enough within each time step, we compute the distribution of the range of values per time step, INLINEFORM0 , where INLINEFORM1 . We plot the histograms in Fig. FIGREF40 and FIGREF40 . From the figure we see that a vertical forget gate controls the amount of information flow effectively, making the decision of retaining or discarding signals.","Finally, to investigate the argument presented in § SECREF2 that the additional forget gate helps the previous output gate with reducing the burden of extracting all needed information, we inspect the distribution of the values from INLINEFORM0 . This distribution indicates how differently the vertical forget gate and the previous output gate select information from INLINEFORM1 . From Fig. FIGREF40 and FIGREF40 we can see that the two gates make fairly different decisions, from which we demonstrate that the direct path between INLINEFORM2 and INLINEFORM3 enables a model to utilize signals overlooked by INLINEFORM4 ."]},{"idx":13,"heading":"Model Variations","paragraphs":["In this subsection, we see the influence of each component of a model on performance by removing or replacing its components. the SNLI dataset is used for experiments, and the best performing configuration is used as a baseline for modifications. We consider the following variants: (i) models that use plain stacked LSTMs, (ii) models with different INLINEFORM0 , (iii) models without INLINEFORM1 , and (iv) models that integrate lower contexts via peephole connections.","Variant (iv) integrates lower contexts via the following equations: DISPLAYFORM0 DISPLAYFORM1","where INLINEFORM0 represent peephole weights that take cell states into account. Among the above equations, those that use the lower cell state INLINEFORM1 are Eq. EQREF52 and EQREF55 . We can see that INLINEFORM2 affects the value of INLINEFORM3 only via peephole connections, which makes INLINEFORM4 independent of INLINEFORM5 .","Table TABREF36 summarizes the results of model variants. We can again see that the use of cell states clearly improves sentence modeling performance (baseline vs. (i) and (iv) vs. (i)). Also from the results of baseline and (ii), we validate that the selection of INLINEFORM0 does not significantly affect performance but introducing INLINEFORM1 is beneficial (baseline vs. (iii)) possibly due to its effect on normalizing information from multiple sources, as mentioned in § SECREF2 . Finally, from the comparison between baseline and (iv), we show that the proposed way of combining the left and the lower contexts leads to better modeling of sentence representations than that of BIBREF20 in encoding sentences."]},{"idx":14,"heading":"Conclusion","paragraphs":["In this paper, we proposed a method of stacking multiple LSTM layers for modeling sentences, dubbed CAS-LSTM. It uses not only hidden states but also cell states from the previous layer, for the purpose of controlling the vertical information flow in a more elaborate way. We evaluated the proposed method on various benchmark tasks: natural language inference, paraphrase identification, and sentiment classification. Our models achieve the new state-of-the-art accuracy on SNLI and Quora Question Pairs datasets and obtain comparable results on MultiNLI and SST datasets. The proposed architecture can replace any stacked LSTM under one weak restriction—the size of states should be identical across all layers.","For future work we plan to apply the CAS-LSTM architecture beyond sentence modeling tasks. Various problems e.g. sequence labeling, sequence generation, and language modeling might benefit from sophisticated modulation on context integration. Aggregating diverse contexts from sequential data, e.g. those from forward and backward reading of text, could also be an intriguing research direction."]},{"idx":15,"heading":"Acknowledgments","paragraphs":["We thank Dan Edmiston for the review of the manuscript."]},{"idx":16,"heading":"Figures and Tables","paragraphs":["Figure 1: Visualization of (a) plain stacked LSTM and (b) CAS-LSTM. The red nodes indicate the blocks whose cell states directly affect the cell state clt.","Figure 2: Schematic diagram of a CAS-LSTM block.","Figure 3: Visualization of paths between cl−1t and c l t. In CAS-LSTM, the direct connection between cl−1t and c l t exists (denoted as red dashed lines).","Figure 4: (a), (b): Histograms of vertical forget gate values. (c), (d): Histograms of the ranges of vertical forget gate per time step. (e), (f): Histograms of the absolute difference between the previous output gate and the current vertical forget gate values.","Table 1: Results of the models on the SNLI dataset.","Table 2: Results of the models on the MultiNLI dataset. ‘In Acc.’ and ‘Cross Acc.’ represent accuracy calculated from the matched and mismatched test set respectively. ∗: SNLI dataset is used as additional training data. ∗∗: computed from hyperparameters provided by the authors.","Table 3: Results of the models on the Quora Question Pairs dataset.","Table 4: Results of the models on the SST dataset. ∗: models pretrained on large external corpora are used.","Table 5: Results of model variants."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1809.02286","title":"Dynamic Compositionality in Recursive Neural Networks with Structure-aware Tag Representations","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Most existing recursive neural network (RvNN) architectures utilize only the structure of parse trees, ignoring syntactic tags which are provided as by-products of parsing. We present a novel RvNN architecture that can provide dynamic compositionality by considering comprehensive syntactic information derived from both the structure and linguistic tags. Specifically, we introduce a structure-aware tag representation constructed by a separate tag-level tree-LSTM. With this, we can control the composition function of the existing word-level tree-LSTM by augmenting the representation as a supplementary input to the gate functions of the tree-LSTM. In extensive experiments, we show that models built upon the proposed architecture obtain superior or competitive performance on several sentence-level tasks such as sentiment analysis and natural language inference when compared against previous tree-structured models and other sophisticated neural models."]},{"idx":1,"heading":"Introduction","paragraphs":["One of the most fundamental topics in natural language processing is how best to derive high-level representations from constituent parts, as natural language meanings are a function of their constituent parts. How best to construct a sentence representation from distributed word embeddings is an example domain of this larger issue. Even though sequential neural models such as recurrent neural networks (RNN) BIBREF0 and their variants including Long Short-Term Memory (LSTM) BIBREF1 and Gated Recurrent Unit (GRU) BIBREF2 have become the de-facto standard for condensing sentence-level information from a sequence of words into a fixed vector, there have been many lines of research towards better sentence representation using other neural architectures, e.g. convolutional neural networks (CNN) BIBREF3 or self-attention based models BIBREF4 .","From a linguistic point of view, the underlying tree structure—as expressed by its constituency and dependency trees—of a sentence is an integral part of its meaning. Inspired by this fact, some recursive neural network (RvNN) models are designed to reflect the syntactic tree structure, achieving impressive results on several sentence-level tasks such as sentiment analysis BIBREF5 , BIBREF6 , machine translation BIBREF7 , natural language inference BIBREF8 , and discourse relation classification BIBREF9 .","However, some recent works have BIBREF10 , BIBREF11 proposed latent tree models, which learn to construct task-specific tree structures without explicit supervision, bringing into question the value of linguistically-motivated recursive neural models. Witnessing the surprising performance of the latent tree models on some sentence-level tasks, there arises a natural question: Are linguistic tree structures the optimal way of composing sentence representations for NLP tasks?","In this paper, we demonstrate that linguistic priors are in fact useful for devising effective neural models for sentence representations, showing that our novel architecture based on constituency trees and their tag information obtains superior performance on several sentence-level tasks, including sentiment analysis and natural language inference.","A chief novelty of our approach is that we introduce a small separate tag-level tree-LSTM to control the composition function of the existing word-level tree-LSTM, which is in charge of extracting helpful syntactic signals for meaningful semantic composition of constituents by considering both the structures and linguistic tags of constituency trees simultaneously. In addition, we demonstrate that applying a typical LSTM to preprocess the leaf nodes of a tree-LSTM greatly improves the performance of the tree models. Moreover, we propose a clustered tag set to replace the existing tags on the assumption that the original syntactic tags are too fined-grained to be useful in neural models.","In short, our contributions in this work are as follows:"]},{"idx":2,"heading":"Related Work","paragraphs":["Recursive neural networks (RvNN) are a kind of neural architecture which model sentences by exploiting syntactic structure. While earlier RvNN models proposed utilizing diverse composition functions, including feed-forward neural networks BIBREF12 , matrix-vector multiplication BIBREF5 , and tensor computation BIBREF6 , tree-LSTMs BIBREF13 remain the standard for several sentence-level tasks.","Even though classic RvNNs have demonstrated superior performance on a variety of tasks, their inflexibility, i.e. their inability to handle dynamic compositionality for different syntactic configurations, is a considerable weakness. For instance, it would be desirable if our model could distinguish e.g. adjective-noun composition from that of verb-noun or preposition-noun composition, as models failing to make such a distinction ignore real-world syntactic considerations such as `-arity' of function words (i.e. types), and the adjunct/argument distinction.","To enable dynamic compositionality in recursive neural networks, many previous works BIBREF14 , BIBREF15 , BIBREF16 , BIBREF9 , BIBREF17 , BIBREF18 , BIBREF19 have proposed various methods.","One main direction of research leverages tag information, which is produced as a by-product of parsing. In detail, BIBREF16 ( BIBREF16 ) suggested TG-RNN, a model employing different composition functions according to POS tags, and TE-RNN/TE-RNTN, models which leverage tag embeddings as additional inputs for the existing tree-structured models. Despite the novelty of utilizing tag information, the explosion of the number of parameters (in case of the TG-RNN) and the limited performance of the original models (in case of the TE-RNN/TE-RNTN) have prevented these models from being widely adopted. Meanwhile, BIBREF9 ( BIBREF9 ) and BIBREF18 ( BIBREF18 ) proposed models based on a tree-LSTM which also uses the tag vectors to control the gate functions of the tree-LSTM. In spite of their impressive results, there is a limitation that the trained tag embeddings are too simple to reflect the rich information which tags provide in different syntactic structures. To alleviate this problem, we introduce structure-aware tag representations in the next section.","Another way of building dynamic compositionality into RvNNs is to take advantage of a meta-network (or hyper-network). Inspired by recent works on dynamic parameter prediction, DC-TreeLSTMs BIBREF17 dynamically create the parameters for compositional functions in a tree-LSTM. Specifically, the model has two separate tree-LSTM networks whose architectures are similar, but the smaller of the two is utilized to calculate the weights of the bigger one. A possible problem for this model is that it may be easy to be trained such that the role of each tree-LSTM is ambiguous, as they share the same input, i.e. word information. Therefore, we design two disentangled tree-LSTMs in our model so that one focuses on extracting useful features from only syntactic information while the other composes semantic units with the aid of the features. Furthermore, our model reduces the complexity of computation by utilizing typical tree-LSTM frameworks instead of computing the weights for each example.","Finally, some recent works BIBREF10 , BIBREF11 have proposed latent tree-structured models that learn how to formulate tree structures from only sequences of tokens, without the aid of syntactic trees or linguistic information. The latent tree models have the advantage of being able to find the optimized task-specific order of composition rather than a sequential or syntactic one. In experiments, we compare our model with not only syntactic tree-based models but also latent tree models, demonstrating that modeling with explicit linguistic knowledge can be an attractive option."]},{"idx":3,"heading":"Model","paragraphs":["In this section, we introduce a novel RvNN architecture, called SATA Tree-LSTM (Structure-Aware Tag Augmented Tree-LSTM). This model is similar to typical Tree-LSTMs, but provides dynamic compositionality by augmenting a separate tag-level tree-LSTM which produces structure-aware tag representations for each node in a tree. In other words, our model has two independent tree-structured modules based on the same constituency tree, one of which (word-level tree-LSTM) is responsible for constructing sentence representations given a sequence of words as usual, while the other (tag-level tree-LSTM) provides supplementary syntactic information to the former.","In section 3.1, we first review tree-LSTM architectures. Then in section 3.2, we introduce a tag-level tree-LSTM and structure-aware tag representations. In section 3.3, we discuss an additional technique to boost the performance of tree-structured models, and in section 3.4, we describe the entire architecture of our model in detail."]},{"idx":4,"heading":"Tree-LSTM","paragraphs":["The LSTM BIBREF1 architecture was first introduced as an extension of the RNN architecture to mitigate the vanishing and exploding gradient problems. In addition, several works have discovered that applying the LSTM cell into tree structures can be an effective means of modeling sentence representations.","To be formal, the composition function of the cell in a tree-LSTM can be formulated as follows:","$$ \\begin{bmatrix} \\mathbf {i} \\\\ \\mathbf {f}_l \\\\ \\mathbf {f}_r \\\\ \\mathbf {o} \\\\ \\mathbf {g} \\end{bmatrix} = \\begin{bmatrix} \\sigma \\\\ \\sigma \\\\ \\sigma \\\\ \\sigma \\\\ \\tanh \\end{bmatrix} \\Bigg ( \\mathbf {W} \\begin{bmatrix} \\mathbf {h}_l \\\\ \\mathbf {h}_r \\\\ \\end{bmatrix} + \\mathbf {b} \\Bigg )$$ (Eq. 8)","$$ \\mathbf {c} = \\mathbf {f}_l \\odot \\mathbf {c}_l + \\mathbf {f}_r \\odot \\mathbf {c}_r + \\mathbf {i} \\odot \\mathbf {g}\\\\$$ (Eq. 9)","where $\\mathbf {h}, \\mathbf {c} \\in \\mathbb {R}^{d}$ indicate the hidden state and cell state of the LSTM cell, and $\\mathbf {h}_l, \\mathbf {h}_r, \\mathbf {c}_l, \\mathbf {c}_r \\in \\mathbb {R}^{d}$ the hidden states and cell states of a left and right child. $\\mathbf {g} \\in \\mathbb {R}^{d}$ is the newly composed input for the cell and $\\mathbf {i}, \\mathbf {f}_{l}, \\mathbf {f}_{r}, \\mathbf {o} \\in \\mathbb {R}^{d}$ represent an input gate, two forget gates (left, right), and an output gate respectively. $\\mathbf {W} \\in \\mathbb {R}^{5d\\times 2d}$ and $\\mathbf {b} \\in \\mathbb {R}^{5d}$ are trainable parameters. $\\sigma $ corresponds to the sigmoid function, $\\tanh $ to the hyperbolic tangent, and $\\odot $ to element-wise multiplication.","Note the equations assume that there are only two children for each node, i.e. binary or binarized trees, following the standard in the literature. While RvNN models can be constructed on any tree structure, in this work we only consider constituency trees as inputs.","In spite of the obvious upside that recursive models have in being so flexible, they are known for being difficult to fully utilize with batch computations as compared to other neural architectures because of the diversity of structure found across sentences. To alleviate this problem, BIBREF8 ( BIBREF8 ) proposed the SPINN model, which brings a shift-reduce algorithm to the tree-LSTM. As SPINN simplifies the process of constructing a tree into only two operations, i.e. shift and reduce, it can support more effective parallel computations while enjoying the advantages of tree structures. For efficiency, our model also starts from our own SPINN re-implementation, whose function is exactly the same as that of the tree-LSTM."]},{"idx":5,"heading":"Structure-aware Tag Representation","paragraphs":["In most previous works using linguistic tag information BIBREF16 , BIBREF9 , BIBREF18 , tags are usually represented as simple low-dimensional dense vectors, similar to word embeddings. This approach seems reasonable in the case of POS tags that are attached to the corresponding words, but phrase-level constituent tags (e.g. NP, VP, ADJP) vary greatly in size and shape, making them less amenable to uniform treatment. For instance, even the same phrase tags within different syntactic contexts can vary greatly in size and internal structure, as the case of NP tags in Figure 1 shows. Here, the NP consisting of DT[the]-NN[stories] has a different internal structure than the NP consisting of NP[the film 's]-NNS[shortcomings].","One way of deriving structure-aware tag representations from the original tag embeddings is to introduce a separate tag-level tree-LSTM which accepts the typical tag embeddings at each node of a tree and outputs the computed structure-aware tag representations for the nodes. Note that the module concentrates on extracting useful syntactic features by considering only the tags and structures of the trees, excluding word information.","Formally, we denote a tag embedding for the tag attached to each node in a tree as $\\textbf {e} \\in \\mathbb {R}^{d_\\text{T}}$ . Then, the function of each cell in the tag tree-LSTM is defined in the following way. Leaf nodes are defined by the following:","$$ \\begin{bmatrix} \\hat{\\mathbf {c}} \\\\ \\hat{\\mathbf {h}} \\\\ \\end{bmatrix} = \\tanh {\\left(\\mathbf {U}_\\text{T} \\mathbf {e} + \\mathbf {a}_\\text{T}\\right)}$$ (Eq. 13)","while non-leaf nodes are defined by the following:","$$ \\begin{bmatrix} \\hat{\\mathbf {i}} \\\\ \\hat{\\mathbf {f}}_l \\\\ \\hat{\\mathbf {f}}_r \\\\ \\hat{\\mathbf {o}} \\\\ \\hat{\\mathbf {g}} \\end{bmatrix} = \\begin{bmatrix} \\sigma \\\\ \\sigma \\\\ \\sigma \\\\ \\sigma \\\\ \\tanh \\end{bmatrix} \\Bigg ( \\mathbf {W_\\text{T}} \\begin{bmatrix} \\hat{\\mathbf {h}}_l \\\\ \\hat{\\mathbf {h}}_r \\\\ \\mathbf {e} \\\\ \\end{bmatrix} + \\mathbf {b}_\\text{T} \\Bigg )$$ (Eq. 14)","$$ \\hat{\\mathbf {c}} = \\hat{\\mathbf {f}}_l \\odot \\hat{\\mathbf {c}}_l + \\hat{\\mathbf {f}}_r \\odot \\hat{\\mathbf {c}}_r + \\hat{\\mathbf {i}} \\odot \\hat{\\mathbf {g}}\\\\$$ (Eq. 15)","where $\\hat{\\mathbf {h}}, \\hat{\\mathbf {c}} \\in \\mathbb {R}^{d_\\text{T}}$ represent the hidden state and cell state of each node in the tag tree-LSTM. We regard the hidden state ( $\\hat{\\mathbf {h}}$ ) as a structure-aware tag representation for the node. $ \\mathbf {U}_\\text{T} \\in \\mathbb {R}^{2d_\\text{T} \\times d_\\text{T}}, \\textbf {a}_\\text{T} \\in \\mathbb {R}^{2d_\\text{T}}, \\mathbf {W}_\\text{T} \\in \\mathbb {R}^{5d_\\text{T} \\times 3d_\\text{T}}$ , and $\\mathbf {b}_\\text{T} \\in \\mathbb {R}^{5d_\\text{T}}$ are trainable parameters. The rest of the notation follows equations 8 , 9 , and 10 . In case of leaf nodes, the states are computed by a simple non-linear transformation. Meanwhile, the composition function in a non-leaf node absorbs the tag embedding ( $\\mathbf {e}$ ) as an additional input as well as the hidden states of the two children nodes. The benefit of revising tag representations according to the internal structure is that the derived embedding is a function of the corresponding makeup of the node, rather than a monolithic, categorical tag.","With regard to the tags themselves, we conjecture that the taxonomy of the tags currently in use in many NLP systems is too complex to be utilized effectively in deep neural models, considering the specificity of many tag sets and the limited amount of data with which to train. Thus, we cluster POS (word-level) tags into 12 groups following the universal POS tagset BIBREF20 and phrase-level tags into 11 groups according to criteria analogous to the case of words, resulting in 23 tag categories in total. In this work, we use the revised coarse-grained tags instead of the original ones. For more details, we refer readers to the supplemental materials."]},{"idx":6,"heading":"Leaf-LSTM","paragraphs":["An inherent shortcoming of RvNNs relative to sequential models is that each intermediate representation in a tree is unaware of its external context until all the information is gathered together at the root node. In other words, each composition process is prone to be locally optimized rather than globally optimized.","To mitigate this problem, we propose using a leaf-LSTM following the convention of some previous works BIBREF21 , BIBREF7 , BIBREF11 , which is a typical LSTM that accepts a sequence of words in order. Instead of leveraging word embeddings directly, we can use each hidden state and cell state of the leaf-LSTM as input tokens for leaf nodes in a tree-LSTM, anticipating the proper contextualization of the input sequence.","Formally, we denote a sequence of words in an input sentence as $w_{1:n}$ ( $n$ : the length of the sentence), and the corresponding word embeddings as $\\mathbf {x}_{1:n}$ . Then, the operation of the leaf-LSTM at time $t$ can be formulated as,","$$ \\begin{bmatrix} \\tilde{\\mathbf {i}} \\\\ \\tilde{\\mathbf {f}} \\\\ \\tilde{\\mathbf {o}} \\\\ \\tilde{\\mathbf {g}} \\end{bmatrix} = \\begin{bmatrix} \\sigma \\\\ \\sigma \\\\ \\sigma \\\\ \\tanh \\end{bmatrix} \\Bigg ( \\mathbf {W}_\\text{L} \\begin{bmatrix} \\tilde{\\mathbf {h}}_{t-1} \\\\ \\mathbf {x}_t \\\\ \\end{bmatrix} + \\mathbf {b}_\\text{L} \\Bigg )$$ (Eq. 18)","$$ \\tilde{\\mathbf {c}}_t = \\tilde{\\mathbf {f}} \\odot \\tilde{\\mathbf {c}}_{t-1} + \\tilde{\\mathbf {i}} \\odot \\tilde{\\mathbf {g}}\\\\$$ (Eq. 19)","where $\\mathbf {x}_t \\in \\mathbb {R}^{d_w}$ indicates an input word vector and $\\tilde{\\mathbf {h}}_t$ , $\\tilde{\\mathbf {c}}_t \\in \\mathbb {R}^{d_h}$ represent the hidden and cell state of the LSTM at time $t$ ( $\\tilde{\\mathbf {h}}_{t-1}$ corresponds to the hidden state at time $t$ -1). $\\mathbf {W}_\\text{L}$ and $\\mathbf {b}_\\text{L} $ are learnable parameters. The remaining notation follows that of the tree-LSTM above.","In experiments, we demonstrate that introducing a leaf-LSTM fares better at processing the input words of a tree-LSTM compared to using a feed-forward neural network. We also explore the possibility of its bidirectional setting in ablation study."]},{"idx":7,"heading":"SATA Tree-LSTM","paragraphs":["In this section, we define SATA Tree-LSTM (Structure-Aware Tag Augmented Tree-LSTM, see Figure 2 ) which joins a tag-level tree-LSTM (section 3.2), a leaf-LSTM (section 3.3), and the original word tree-LSTM together.","As above we denote a sequence of words in an input sentence as $w_{1:n}$ and the corresponding word embeddings as $\\mathbf {x}_{1:n}$ . In addition, a tag embedding for the tag attached to each node in a tree is denoted by $\\textbf {e} \\in \\mathbb {R}^{d_\\text{T}}$ . Then, we derive the final sentence representation for the input sentence with our model in two steps.","First, we compute structure-aware tag representations ( $\\hat{\\mathbf {h}}$ ) for each node of a tree using the tag tree-LSTM (the right side of Figure 2 ) as follows:","$$ \\begin{bmatrix} \\hat{\\mathbf {c}} \\\\ \\hat{\\mathbf {h}} \\\\ \\end{bmatrix} = {\\left\\lbrace \\begin{array}{ll} \\text{Tag-Tree-LSTM}(\\mathbf {e}) & \\text{if a leaf node} \\\\ \\text{Tag-Tree-LSTM}(\\hat{\\mathbf {h}}_l, \\hat{\\mathbf {h}}_r, \\mathbf {e}) & \\text{otherwise} \\end{array}\\right.}$$ (Eq. 23)","where Tag-Tree-LSTM indicates the module we described in section 3.2.","Second, we combine semantic units recursively on the word tree-LSTM in a bottom-up fashion. For leaf nodes, we leverage the Leaf-LSTM (the bottom-left of Figure 2 , explained in section 3.3) to compute $\\tilde{\\mathbf {c}}_{t}$ and $\\tilde{\\mathbf {h}}_{t}$ in sequential order, with the corresponding input $\\mathbf {x}_t$ .","$$ \\begin{bmatrix} \\tilde{\\mathbf {c}}_{t} \\\\ \\tilde{\\mathbf {h}}_{t} \\\\ \\end{bmatrix} = \\text{Leaf-LSTM}(\\tilde{\\textbf {h}}_{t-1}, \\textbf {x}_t)$$ (Eq. 24)","Then, the $\\tilde{\\mathbf {c}}_{t}$ and $\\tilde{\\mathbf {h}}_{t}$ can be utilized as input tokens to the word tree-LSTM, with the left (right) child of the target node corresponding to the $t$ th word in the input sentence.","$$ \\begin{bmatrix} \\check{\\textbf {c}}_{\\lbrace l, r\\rbrace } \\\\ \\check{\\textbf {h}}_{\\lbrace l, r\\rbrace } \\end{bmatrix} = \\begin{bmatrix} \\tilde{\\textbf {c}}_{t} \\\\ \\tilde{\\textbf {h}}_{t} \\end{bmatrix}$$ (Eq. 25)","In the non-leaf node case, we calculate phrase representations for each node in the word tree-LSTM (the upper-left of Figure 2 ) recursively as follows:","$$ \\check{\\mathbf {g}} = \\tanh {\\left( \\mathbf {U}_\\text{w} \\begin{bmatrix} \\check{\\mathbf {h}}_l \\\\ \\check{\\mathbf {h}}_r \\\\ \\end{bmatrix} + \\mathbf {a}_\\text{w} \\right)}$$ (Eq. 26)","$$ \\begin{bmatrix} \\check{\\mathbf {i}} \\\\ \\check{\\mathbf {f}}_l \\\\ \\check{\\mathbf {f}}_r \\\\ \\check{\\mathbf {o}} \\end{bmatrix} = \\begin{bmatrix} \\sigma \\\\ \\sigma \\\\ \\sigma \\\\ \\sigma \\end{bmatrix} \\Bigg ( \\mathbf {W_\\text{w}} \\begin{bmatrix} \\check{\\mathbf {h}}_l \\\\ \\check{\\mathbf {h}}_r \\\\ \\hat{\\mathbf {h}} \\\\ \\end{bmatrix} + \\mathbf {b}_\\text{w} \\Bigg )$$ (Eq. 27)","where $\\check{\\mathbf {h}}$ , $\\check{\\mathbf {c}} \\in \\mathbb {R}^{d_h}$ represent the hidden and cell state of each node in the word tree-LSTM. $\\mathbf {U}_\\text{w} \\in \\mathbb {R}^{d_h \\times 2d_h}$ , $\\mathbf {W}_\\text{w} \\in \\mathbb {R}^{4d_h \\times \\left(2d_h+d_\\text{T}\\right)}$ , $\\mathbf {a}_\\text{w} \\in \\mathbb {R}^{d_h}$ , $\\mathbf {b}_\\text{w} \\in \\mathbb {R}^{4d_h}$ are learned parameters. The remaining notation follows those of the previous sections. Note that the structure-aware tag representations ( $\\hat{\\mathbf {h}}$ ) are only utilized to control the gate functions of the word tree-LSTM in the form of additional inputs, and are not involved in the semantic composition ( $\\check{\\mathbf {g}}$ ) directly.","Finally, the hidden state of the root node ( $\\check{\\mathbf {h}}_\\text{root}$ ) in the word-level tree-LSTM becomes the final sentence representation of the input sentence."]},{"idx":8,"heading":"Quantitative Analysis","paragraphs":["One of the most basic approaches to evaluate a sentence encoder is to measure the classification performance with the sentence representations made by the encoder. Thus, we conduct experiments on the following five datasets. (Summary statistics for the datasets are reported in the supplemental materials.)","MR: A group of movie reviews with binary (positive / negative) classes. BIBREF22","SST-2: Stanford Sentiment Treebank BIBREF6 . Similar to MR, but each review is provided in the form of a binary parse tree whose nodes are annotated with numeric sentiment values. For SST-2, we only consider binary (positive / negative) classes.","SST-5: Identical to SST-2, but the reviews are grouped into fine-grained (very negative, negative, neutral, positive, very positive) classes.","SUBJ: Sentences grouped as being either subjective or objective (binary classes). BIBREF23","TREC: A dataset which groups questions into six different question types (classes). BIBREF24","As a preprocessing step, we construct parse trees for the sentences in the datasets using the Stanford PCFG parser BIBREF25 . Because syntactic tags are by-products of constituency parsing, we do not need further preprocessing.","To classify the sentence given our sentence representation ( $\\check{\\mathbf {h}}_\\text{root}$ ), we use one fully-connected layer with a ReLU activation, followed by a softmax classifier. The final predicted probability distribution of the class $y$ given the sentence $w_{1:n}$ is defined as follows,","$$\\mathbf {s} = \\text{ReLU}(\\mathbf {W}_\\text{s} \\check{\\mathbf {h}}_\\text{root}+ \\mathbf {b}_\\text{s})$$ (Eq. 37)","$$p(y|w_{1:n}) = \\text{softmax}(\\mathbf {W}_\\text{c}\\mathbf {s} + \\mathbf {b}_\\text{c})$$ (Eq. 38)","where $\\textbf {s} \\in \\mathbb {R}^{d_\\text{s}}$ is the computed task-specific sentence representation for the classifier, and $\\textbf {W}_\\text{s} \\in \\mathbb {R}^{d_\\text{s} \\times d_h}$ , $\\textbf {W}_\\text{c} \\in \\mathbb {R}^{d_\\text{c} \\times d_s}$ , $\\textbf {b}_\\text{s} \\in \\mathbb {R}^{d_s}$ , $\\textbf {b}_\\text{c} \\in \\mathbb {R}^{d_c}$ are trainable parameters. As an objective function, we use the cross entropy of the predicted and true class distributions.","The results of the experiments on the five datasets are shown in table 1 . In this table, we report the test accuracy of our model and various other models on each dataset in terms of percentage. To consider the effects of random initialization, we report the best numbers obtained from each several runs with hyper-parameters fixed.","Compared with the previous syntactic tree-based models as well as other neural models, our SATA Tree-LSTM shows superior or competitive performance on all tasks. Specifically, our model achieves new state-of-the-art results within the tree-structured model class on 4 out of 5 sentence classification tasks—SST-2, SST-5, MR, and TREC. The model shows its strength, in particular, when the datasets provide phrase-level supervision to facilitate tree structure learning (i.e. SST-2, SST-5). Moreover, the numbers we report for SST-5 and TREC are competitive to the existing state-of-the-art results including ones from structurally pre-trained models such as ELMo BIBREF26 , proving our model's superiority. Note that the SATA Tree-LSTM also outperforms the recent latent tree-based model, indicating that modeling a neural model with explicit linguistic knowledge can be an attractive option.","On the other hand, a remaining concern is that our SATA Tree-LSTM is not robust to random seeds when the size of a dataset is relatively small, as tag embeddings are randomly initialized rather than relying on pre-trained ones in contrast with the case of words. From this observation, we could find out there needs a direction of research towards pre-trained tag embeddings.","To estimate the performance of our model beyond the tasks requiring only one sentence at a time, we conduct an experiment on the Stanford Natural Language Inference BIBREF34 dataset, each example of which consists of two sentences, the premise and the hypothesis. Our objective given the data is to predict the correct relationship between the two sentences among three options— contradiction, neutral, or entailment.","We use the siamese architecture to encode both the premise ( $p_{1:m}$ ) and hypothesis ( $h_{1:n}$ ) following the standard of sentence-encoding models in the literature. (Specifically, $p_{1:m}$ is encoded as $\\check{\\mathbf {h}}_\\text{root}^p \\in \\mathbb {R}^{d_h}$ and $h_{1:n}$ is encoded as $\\check{\\mathbf {h}}_\\text{root}^h \\in \\mathbb {R}^{d_h}$ with the same encoder.) Then, we leverage some heuristics BIBREF35 , followed by one fully-connected layer with a ReLU activation and a softmax classifier. Specifically,","$$\\mathbf {z} = \\left[ \\check{\\mathbf {h}}_\\text{root}^p; \\check{\\mathbf {h}}_\\text{root}^h; | \\check{\\mathbf {h}}_\\text{root}^p - \\check{\\mathbf {h}}_\\text{root}^h |; \\check{\\mathbf {h}}_\\text{root}^p \\odot \\check{\\mathbf {h}}_\\text{root}^h \\right]$$ (Eq. 41)","$$\\mathbf {s} = \\text{ReLU}(\\mathbf {W}_\\text{s} \\mathbf {z} + \\mathbf {b}_\\text{s})$$ (Eq. 42)","where $\\textbf {z} \\in \\mathbb {R}^{4d_h}$ , $\\textbf {s} \\in \\mathbb {R}^{d_s}$ are intermediate features for the classifier and $\\textbf {W}_\\text{s} \\in \\mathbb {R}^{d_\\text{s} \\times 4d_h}$ , $\\textbf {W}_\\text{c} \\in \\mathbb {R}^{d_\\text{c} \\times d_s}$ , $\\textbf {b}_\\text{s} \\in \\mathbb {R}^{d_s}$ , $\\textbf {b}_\\text{c} \\in \\mathbb {R}^{d_c}$ are again trainable parameters.","Our experimental results on the SNLI dataset are shown in table 2 . In this table, we report the test accuracy and number of trainable parameters for each model. Our SATA-LSTM again demonstrates its decent performance compared against the neural models built on both syntactic trees and latent trees, as well as the non-tree models. (Latent Syntax Tree-LSTM: BIBREF10 ( BIBREF10 ), Tree-based CNN: BIBREF35 ( BIBREF35 ), Gumbel Tree-LSTM: BIBREF11 ( BIBREF11 ), NSE: BIBREF36 ( BIBREF36 ), Reinforced Self-Attention Network: BIBREF4 ( BIBREF4 ), Residual stacked encoders: BIBREF37 ( BIBREF37 ), BiLSTM with generalized pooling: BIBREF38 ( BIBREF38 ).) Note that the number of learned parameters in our model is also comparable to other sophisticated models, showing the efficiency of our model.","Even though our model has proven its mettle, the effect of tag information seems relatively weak in the case of SNLI, which contains a large amount of data compared to the others. One possible explanation is that neural models may learn some syntactic rules from large amounts of text when the text size is large enough, reducing the necessity of external linguistic knowledge. We leave the exploration of the effectiveness of tags relative to data size for future work.","Here we go over the settings common across our models during experimentation. For more task-specific details, refer to the supplemental materials.","For our input embeddings, we used 300 dimensional 840B GloVe BIBREF39 as pre-trained word embeddings, and tag representations were randomly sampled from the uniform distribution [-0.005, 0.005]. Tag vectors are revised during training while the fine-tuning of the word embedding depends on the task. Our models were trained using the Adam BIBREF40 or Adadelta BIBREF41 optimizer, depending on task. For regularization, weight decay is added to the loss function except for SNLI following BIBREF42 ( BIBREF42 ) and Dropout BIBREF43 is also applied for the word embeddings and task-specific classifiers. Moreover, batch normalization BIBREF44 is adopted for the classifiers. As a default, all the weights in the model are initialized following BIBREF45 ( BIBREF45 ) and the biases are set to 0. The total norm of the gradients of the parameters is clipped not to be over 5 during training.","Our best models for each dataset were chosen by validation accuracy in cases where a validation set was provided as a part of the dataset. Otherwise, we perform a grid search on probable hyper-parameter settings, or run 10-fold cross-validation in cases where even a test set does not exist."]},{"idx":9,"heading":"Ablation Study","paragraphs":["In this section, we design an ablation study on the core modules of our model to explore their effectiveness. The dataset used in this experiment is SST-2. To conduct the experiment, we only replace the target module with other candidates while maintaining the other settings. To be specific, we focus on two modules, the leaf-LSTM and structure-aware tag embeddings (tag-level tree-LSTM). In the first case, the leaf-LSTM is replaced with a fully-connected layer with a $\\tanh $ activation or Bi-LSTM. In the second case, we replace the structure-aware tag embeddings with naive tag embeddings or do not employ them at all.","The experimental results are depicted in Figure 3 . As the chart shows, our model outperforms all the other options we have considered. In detail, the left part of the chart shows that the leaf-LSTM is the most effective option compared to its competitors. Note that the sequential leaf-LSTM is somewhat superior or competitive than the bidirectional leaf-LSTM when both have a comparable number of parameters. We conjecture this may because a backward LSTM does not add additional useful knowledge when the structure of a sentence is already known. In conclusion, we use the uni-directional LSTM as a leaf module because of its simplicity and remarkable performance.","Meanwhile, the right part of the figure demonstrates that our newly introduced structure-aware embeddings have a real impact on improving the model performance. Interestingly, employing the naive tag embeddings made no difference in terms of the test accuracy, even though the absolute validation accuracy increased (not reported in the figure). This result supports our assumption that tag information should be considered in the structure."]},{"idx":10,"heading":"Qualitative Analysis","paragraphs":["In previous sections, we have numerically demonstrated that our model is effective in encouraging useful composition of semantic units. Here, we directly investigate the computed representations for each node of a tree, showing that the remarkable performance of our model is mainly due to the gradual and recursive composition of the intermediate representations on the syntactic structure.","To observe the phrase-level embeddings at a glance, we draw a scatter plot in which a point represents the corresponding intermediate representation. We utilize PCA (Principal Component Analysis) to project the representations into a two-dimensional vector space. As a target parse tree, we reuse the one seen in Figure 1 . The result is shown in Figure 4 .","From this figure, we confirm that the intermediate representations have a hierarchy in the semantic space, which is very similar to that of the parse tree. In other words, as many tree-structured models pursue, we can see the tendency of constructing the representations from the low-level (the bottom of the figure) to the high-level (the top-left and top-right of the figure), integrating the meaning of the constituents recursively. An interesting thing to note is that the final sentence representation is near that of the phrase `, the stories are quietly moving.' rather than that of `Despite the film's shortcomings', catching the main meaning of the sentence."]},{"idx":11,"heading":"Conclusion","paragraphs":["We have proposed a novel RvNN architecture to fully utilize linguistic priors. A newly introduced tag-level tree-LSTM demonstrates that it can effectively control the composition function of the corresponding word-level tree-LSTM. In addition, the proper contextualization of the input word vectors results in significant performance improvements on several sentence-level tasks. For future work, we plan to explore a new way of exploiting dependency trees effectively, similar to the case of constituency trees."]},{"idx":12,"heading":"Acknowledgments","paragraphs":["We thank anonymous reviewers for their constructive and fruitful comments. This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (NRF2016M3C4A7952587)."]},{"idx":13,"heading":"Figures and Tables","paragraphs":["Figure 1: A constituency tree example from Stanford Sentiment Treebank.","Figure 2: A diagram of SATA Tree-LSTM. The model has two separate tree-LSTM modules, the right of which (tag tree-LSTM) extracts a structure-aware tag representation to control the composition function of the remaining tree-LSTM (word tree-LSTM). Fully-connected: onelayered non-linear transformation.","Table 1: The comparison of various models on different sentence classification tasks. We report the test accuracy of each model in percentage. Our SATA Tree-LSTM shows superior or competitive performance on all tasks, compared to previous treestructured models as well as other sophisticated models. ?: Latent tree-structured models. †: Models which are pre-trained with large external corpora.","Table 2: The accuracy of diverse models on Stanford Natural Language Inference. For fair comparison, we only consider sentence-encoding based models. Our model achieves a comparable result with a moderate number of parameters. ?: Latent tree models.","Figure 4: A scatter plot whose points represent the intermediate representations for each node of the tree in Figure 1. From this figure, we can see the tendency of constructing the representations recursively from the low to the high level.","Figure 3: An ablation study on the core modules of our model. The test accuracy of each model on SST-2 is reported. The results demonstrate that the modules play an important role for achieving the superior performance of our model. FC: A fully connected-layer with a tanh function. w/o tags: Tag embeddings are not used. w/ tags: The naive tag embeddings are directly inserted into each node of a tree."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1809.05807","title":"Dual Memory Network Model for Biased Product Review Classification","sections":[{"idx":0,"heading":"Abstract","paragraphs":["In sentiment analysis (SA) of product reviews, both user and product information are proven to be useful. Current tasks handle user profile and product information in a unified model which may not be able to learn salient features of users and products effectively. In this work, we propose a dual user and product memory network (DUPMN) model to learn user profiles and product reviews using separate memory networks. Then, the two representations are used jointly for sentiment prediction. The use of separate models aims to capture user profiles and product information more effectively. Compared to state-of-the-art unified prediction models, the evaluations on three benchmark datasets, IMDB, Yelp13, and Yelp14, show that our dual learning model gives performance gain of 0.6%, 1.2%, and 0.9%, respectively. The improvements are also deemed very significant measured by p-values."]},{"idx":1,"heading":"Introduction","paragraphs":["Written text is often meant to express sentiments of individuals. Recognizing the underlying sentiment expressed in the text is essential to understand the full meaning of the text. The SA community is increasingly interested in using natural language processing (NLP) techniques as well as sentiment theories to identify sentiment expressions in the text.","Recently, deep learning based methods have taken over feature engineering approaches to gain further performance improvement in SA. Typical neural network models include Convolutional Neural Network (CNN) BIBREF0 , Recursive auto-encoders BIBREF1 , Long-Short Term Memory (LSTM) BIBREF2 , and many more.","Attention-based models are introduced to highlight important words and sentences in a piece of text. Different attention models are built using information embedded in the text including users, products and text in local context BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 . In order to incorporate other aspects of knowledge, Qian et al. qian2016linguistically developed a model to employ additional linguistic resources to benefit sentiment classification. Long et al.long2017cognition and Mishra et al.mishra2016leveraging proposed cognition-based attention models learned from cognition grounded eye-tracking data.","Most text-based SA is modeled as sentiment classification tasks. In this work, SA is for product reviews. We use the term users to refer to writers of text, and products to refer to the targets of reviews in the text. A user profile is defined by the collection of reviews a user writes. Product information defined for a product is the collection of reviews for this product. Note that user profiles and product information are not independent of each other. That is one reason why previous works use unified models. By commonsense we know that review text written by a person may be subjective or biased towards his/her own preferences. Lenient users tend to give higher ratings than finicky ones even if they review the same products. Popular products do receive higher ratings than those unpopular ones because the aggregation of user reviews still shows the difference in opinion for different products. While users and products both play crucial roles in sentiment analysis, they are fundamentally different.","Reviews written by a user can be affected by user preference which is more subjective whereas reviews for a product are useful only if they are from a collection of different reviewers, because we know individual reviews can be biased. The popularity of a product tends to reflect the general impression of a collection of users as an aggregated result. Therefore, sentiment prediction of a product should give dual consideration to individual users as well as all reviews as a collection.","In this paper, we address the aforementioned issue by proposing to learn user profiles and product review information separately before making a joint prediction on sentiment classification. In the proposed Dual User and Product Memory Network (DUPMN) model, we first build a hierarchical LSTM BIBREF7 model to generate document representations. Then a user memory network (UMN) and a product memory network (PMN) are separately built based on document representation of user comments and product reviews. Finally, sentiment prediction is learned from a dual model.","To validate the effectiveness of our proposed model, evaluations are conducted on three benchmarking review datasets from IMDB and Yelp data challenge (including Yelp13 and Yelp14) BIBREF2 . Experimental results show that our algorithm can outperform baseline methods by large margins. Compared to the state-of-the-art method, DUPMN made 0.6%, 1.2%, and 0.9% increase in accuracy with p-values 0.007, 0.004, and 0.001 in the three benchmark datasets respectively. Results show that leveraging user profile and product information separately can be more effective for sentiment predictions.","The rest of this paper is organized as follows. Section SECREF2 gives related work, especially memory network models. Section SECREF3 introduces our proposed DUPMN model. Section SECREF4 gives the evaluation compared to state-of-the-art methods on three datasets. Section SECREF5 concludes this paper and gives some future directions in sentiment analysis models to consider individual bias."]},{"idx":2,"heading":"Related Work","paragraphs":["Related work includes neural network models and the use of user/product information in sentiment analysis."]},{"idx":3,"heading":"Neural Network Models","paragraphs":["In recent years, deep learning has greatly improved the performance of sentiment analysis. Commonly used models include Convolutional Neural Networks (CNNs) BIBREF8 , Recursive Neural Network (ReNNs) BIBREF1 , and Recurrent Neural Networks (RNNs) BIBREF9 . RNN naturally benefits sentiment classification because of its ability to capture sequential information in text. However, standard RNNs suffer from the so-called gradient vanishing problem BIBREF10 where gradients may grow or decay exponentially over long sequences. LSTM models are adopted to solve the gradient vanishing problem. An LSTM model provides a gated mechanism to keep the long-term memory. Each LSTM layer is generally followed by mean pooling and the output is fed into the next layer. Experiments in datasets which contain sentences and long documents demonstrate that LSTM model outperforms the traditional RNNs BIBREF2 , BIBREF11 . Attention mechanism is also added to LSTM models to highlight important segments at both sentence level and document level. Attention models can be built from text in local context BIBREF4 , user/production information BIBREF5 , BIBREF12 and other information such as cognition grounded eye tracking data BIBREF13 . LSTM models with attention mechanism are currently the state-of-the-art models in document sentiment analysis tasks BIBREF5 , BIBREF13 .","Memory networks are designed to handle larger context for a collection of documents. Memory networks introduce inference components combined with a so called long-term memory component BIBREF14 . The long-term memory component is a large external memory to represent data as a collection. This collective information can contain local context BIBREF15 or external knowledge base BIBREF16 . It can also be used to represent the context of users and products globally BIBREF17 . Dou uses dou2017capturing a memory network model in document level sentiment analysis and makes comparable result to the state-of-the-art model BIBREF5 ."]},{"idx":4,"heading":"Incorporating User and Product Information","paragraphs":["Both user profile and product information have crucial effects on sentiment polarities. Tang et al. tang2015learning proposed a model by incorporating user and product information into a CNN network for document level sentiment classification. User ids and product names are included as features in a unified document vector using the vector space model such that document vectors capture important global clues include individual preferences and product information.","Nevertheless, this method suffers from high model complexity and only word-level preference is considered rather than information at the semantic level BIBREF5 . Gui et al. gui2016intersubjectivity introduce an inter-subjectivity network to link users to the terms they used as well as the polarities of the terms. The network aims to learn writer embeddings which are subsequently incorporated into a CNN network for sentiment analysis. Chen et al. chen2016neural propose a model to incorporate user and product information into an LSTM with attention mechanism. This model is reported to produce the state-of-the-art results in the three benchmark datasets (IMDB, Yelp13, and Yelp14). Dou dou2017capturing also proposes a deep memory network to integrate user profile and product information in a unified model. However, the model only achieves a comparable result to the state-of-the-art attention based LSTM BIBREF5 ."]},{"idx":5,"heading":"The DUPMN Model","paragraphs":["We propose a DUPMN model. Firstly, document representation is learned by a hierarchical LSTM network to obtain both sentence-level representation and document level representation BIBREF18 . A memory network model is then trained using dual memory networks, one for training user profiles and the other for training product reviews. Both of them are joined together to predict sentiment for documents."]},{"idx":6,"heading":"Task Definition","paragraphs":["Let INLINEFORM0 be the set of review documents for classification, INLINEFORM1 be the set of users, and INLINEFORM2 be the set of products. For each document INLINEFORM3 ( INLINEFORM4 ), user INLINEFORM5 ( INLINEFORM6 ) is the writer of INLINEFORM7 on product INLINEFORM8 ( INLINEFORM9 ). Let INLINEFORM10 be all documents posted by INLINEFORM11 and INLINEFORM12 be all documents on INLINEFORM13 . INLINEFORM14 and INLINEFORM15 define the user context and the product context of INLINEFORM16 , respectively. For simplicity, we use INLINEFORM17 and INLINEFORM18 directly. The goal of a sentiment analysis task is to predict the sentiment label for each INLINEFORM19 ."]},{"idx":7,"heading":"Document Embedding","paragraphs":["Since review documents for sentiment classification such as restaurant reviews and movie comments are normally very long, a proper method to embed the documents is needed to speed up the training process and achieve better accuracy. Inspired by the work of Chen BIBREF5 , a hierarchical LSTM network is used to obtain embedding representation of documents. The first LSTM layer is used to obtain sentence representation by the hidden state of an LSTM network. The same mechanism is also used for document level representation with sentence-level representation as input. User and product attentions are included in the network so that all salient features are included in document representation. For document INLINEFORM0 , its embedding is denoted as INLINEFORM1 . INLINEFORM2 is a vector representation with dimension size INLINEFORM3 . In principle, the embedding representation of user context of INLINEFORM4 , denoted by INLINEFORM5 , and product context INLINEFORM6 vary depending on INLINEFORM7 . For easy matrix calculation, we take INLINEFORM8 as our model parameter so that INLINEFORM9 and INLINEFORM10 are two fixed INLINEFORM11 matrices."]},{"idx":8,"heading":"Memory Network Structure","paragraphs":["Inspired by the successful use of memory networks in language modeling, question answering, and sentiment analysis BIBREF19 , BIBREF17 , BIBREF20 , we propose our DUPMN by extending a single memory network model to two memory networks to reflect different influences from users' perspective and products' perspective. The structure of the model is shown in Figure FIGREF6 with 3 hops as an example although in principle a memory network can have INLINEFORM0 computational hops.","The DUPMN model has two separate memory networks: the UMN and the PMN. Each hop in a memory network includes an attention layer INLINEFORM0 and a linear addition INLINEFORM1 . Since the external memory INLINEFORM2 and INLINEFORM3 have the same structure, we use a generic notation INLINEFORM4 to denote them in the following explanations. Each document vector INLINEFORM5 is fed into the first hop of the two networks ( INLINEFORM6 = INLINEFORM7 ). Each INLINEFORM8 ( k= 1 ...... K-1) passes through the attention layer using an attention mechanism defined by a softmax function to obtain the attention weights INLINEFORM9 for document INLINEFORM10 : DISPLAYFORM0","And to produce an attention weighted vector INLINEFORM0 by DISPLAYFORM0","INLINEFORM0 is then linearly added to INLINEFORM1 to produce the output of this hop as INLINEFORM2 .","After completing the INLINEFORM0 th hop, the output INLINEFORM1 in UMN and INLINEFORM2 in PMN are joined together using a weighted mechanism to produce the output of DUPMN, INLINEFORM3 , is given below: DISPLAYFORM0","Two different weight vectors INLINEFORM0 and INLINEFORM1 in Formula EQREF9 can be trained for UMN and PMN. INLINEFORM2 and INLINEFORM3 are two constant weights to reflect the relative importance of user profile INLINEFORM4 and product information INLINEFORM5 . The parameters in the model including INLINEFORM6 , INLINEFORM7 , INLINEFORM8 and INLINEFORM9 . By minimizing the loss, those parameters can be optimized.","Sentiment prediction is obtained through a INLINEFORM0 layer. The loss function is defined by the cross entropy between the prediction from INLINEFORM1 and the ground truth labels."]},{"idx":9,"heading":"Experiment and Result Analysis","paragraphs":["Performance evaluations are conducted on three datasets and DUPMN is compared with a set of commonly used baseline methods including the state-of-the-art LSTM based method BIBREF5 , BIBREF21 ."]},{"idx":10,"heading":"Datasets","paragraphs":["The three benchmarking datasets include movie reviews from IMDB, restaurant reviews from Yelp13 and Yelp14 developed by Tang tang2015document. All datasets are tokenized using the Stanford NLP tool BIBREF22 . Table TABREF11 lists statistics of the datasets including the number of classes, number of documents, average length of sentences, the average number of documents per user, and the average number of documents per product. Since postings in social networks by both users and products follow the long tail distribution BIBREF23 , we only show the distribution of total number of posts for different products. For example, #p(0-50) means the number of products which have reviews between the size of 0 to 50. We split train/development/test sets at the rate of 8:1:1 following the same setting in BIBREF3 , BIBREF5 . The best configuration by the development dataset is used for the test set to obtain the final result."]},{"idx":11,"heading":"Baseline Methods","paragraphs":["In order to make a systematic comparison, three groups of baselines are used in the evaluation. Group 1 includes all commonly used feature sets mentioned in Chen et al. chen2016neural including Majority, Trigram, Text features (TextFeatures), and AveWordvec. All feature sets in Group 1 except Majority use the SVM classifier.","Group 2 methods include the recently published sentiment analysis models which only use context information, including:","SSWE BIBREF24 — An SVM model using sentiment specific word embedding.","RNTN+RNN BIBREF1 — A Recursive Neural Tensor Network (RNTN) to represent sentences.","CLSTM BIBREF25 — A Cached LSTM model to capture overall semantic information in long text.","LSTM+LA BIBREF5 — A state-of-the-art LSTM using local context as attention mechanism at both sentence level and document level.","LSTM+CBA BIBREF13 — A state-of-the-art LSTM model using cognition based data to build attention mechanism.","Group 3 methods are recently published neural network models which incorporate user and product information, including:","UPNN BIBREF3 — User and product information for sentiment classification at document level based on a CNN network.","UPDMN BIBREF20 — A deep memory network for document level sentiment classification by including user and product information in a unified model. Hop 1 gives the best result, and thus K=1 is used.","InterSub BIBREF6 — A CNN model making use of user and product information.","LSTM+UPA BIBREF5 — The state-of-the-art LSTM including both local context based attentions and user/product in the attention mechanism.","For the DUPMN model, we also include two variations which use only one memory network. The first variation only includes user profiles in the memory network, denoted as DUPMN-U. The second variation only uses product information, denoted as DUPMN-P."]},{"idx":12,"heading":"Performance Evaluation","paragraphs":["Four sets of experiments are conducted. The first experiment compares DUPMN with other sentiment analysis methods. The second experiment evaluates the effectiveness of different hop size INLINEFORM0 of memory network. The third experiment evaluates the effectiveness of UMN and PMN in different datasets. The fourth set of experiment examines the effect of memory size INLINEFORM1 on the performance of DUPMN. Performance measures include Accuracy (ACC), Root-Mean-Square-Error (RMSE), and Mean Absolute Error (MAE) for our model. For other baseline methods in Group 2 and Group 3, their reported results are used. We also show the p-value by comparing the result of 10 random tests for both our model and the state-of-the-art model in the t-test .","Table TABREF12 shows the result of the first experiment. DUPMN uses one hop (the best performer) with INLINEFORM0 being set at 100, a commonly used memory size for memory networks.","Generally speaking, Group 2 performs better than Group 1. This is because Group 1 uses a traditional SVM with feature engineering BIBREF26 and Group 2 uses more advanced deep learning methods proven to be effective by recent studies BIBREF0 , BIBREF5 . However, some feature engineering methods are no worse than some deep learning methods. For example, the TextFeature model outperforms SSWE by a significant margin.","When comparing Group 2 and Group 3 methods, we can see that user profiles and product information can improve performance as most of the methods in Group 3 perform better than methods in Group 2. This is more obvious in the IMDB dataset which naturally contains more subjectivity. In the IMDB dataset, almost all models with user and product information outperform the text-only models in Group 2 except LSTM+CBA BIBREF13 . However, the two LSTM models in Group 2 which include local attention mechanism do show that attention base methods can outperform methods using user profile and product information. In fact, the LSTM+CBA model using attention mechanism based on cognition grounded eye-tracking data in Group 2 outperforms quite a number of methods in Group 3. LSTM+CBA in Group 2 is only inferior to LSTM+UPA in Group 3 because of the additional user profile and production information used in LSTM+UPA.","Most importantly, the DUPMN model with both user memory and product memory significantly outperforms all the baseline methods including the state-of-the-art LSTM+UPA model BIBREF5 . By using user profiles and product information in memory networks, DUPMN outperforms LSTM+UPA in all three datasets. In the IMDB dataset, our model makes 0.6 % improvement over LSTM+UPA in accuracy with INLINEFORM0 of 0.007. Our model also achieves lower RMSE value. In the Yelp review dataset, the improvement is even more significant. DUPMN achieves 1.2% improvement in accuracy in Yelp13 with INLINEFORM1 of 0.004 and 0.9% in Yelp14 with INLINEFORM2 of 0.001, and the lower RMSE obtained by DUPMN also indicates that the proposed model can predict review ratings more accurately.","Best results are marked in bold; second best are underlined in the table","The second set of experiments evaluates the effectiveness of DUPMN using different number of hops INLINEFORM0 . Table TABREF26 shows the evaluation results. The number in the brackets after each model name indicates the number of hops used. Two conclusions can be obtained from Table TABREF26 . We find that more hops do not bring benefit. In all the three models, the single hop model obtains the best performance. Unlike video and image information, written text is grammatically structured and contains abstract information such that multiple hops may introduce more information distortion. Another reason may be due to over-fitting by the additional hops.","Comparing the performance of DUPMN-U and DUPMN-P in Table TABREF26 , it also shows that user memory and product memory indeed provide different kinds of information and thus their usefulness are different in different datasets. For the movie review dataset, IMDB, which is more subjective, results show that user profile information using DUPMN-U outperforms DUPMN-P as there is a 1.3% gain compared to that of DUPMN-P. However, on restaurant reviews in Yelp datasets, DUPMN-P performs better than DUPMN-U indicating product information is more valuable.","To further examine the effects of UMN and PMN on sentiment classification, we observe the difference of optimized values of the constant weights INLINEFORM0 and INLINEFORM1 between the UMN and the PMN given in Formula EQREF9 . The difference in their values indicates the relative importance of the two networks. The optimized weights given in Table TABREF27 on the three datasets show that user profile has a higher weight than product information in IMDB because movie review is more related to personal preferences whereas product information has a higher weight in the two restaurant review datasets. This result is consistent with the evaluation in Table TABREF26 on DUPMN-U and DUPMN-P.","Most social network data follows the long tail distribution. If the memory size to represent the data is too small, some context information will be lost. On the other hand, too large memory size which requires more resources in computation and storage may not introduce much benefit. Thus, the fourth set of experiments evaluates the effect of dimension size INLINEFORM0 in the DUPMN memory networks. Figure FIGREF28 shows the result of the evaluation for 1 hop configuration with memory size starting at 1 with 10 points at each increment until size of 75, the increment set to 25 from 75 to 200 to cover most postings. Results show that when memory size increases from 10 to 100, the performance of DUPMN steadily increases. Once it goes beyond 100, DUPMN is no longer sensitive to memory size. This is related to the distribution of document frequency rated by user/product in Table TABREF11 as the average is around 50. With long tail distribution, after 75, not many new documents will be included in the context. To improve algorithm efficiency without much compromise on performance, INLINEFORM1 can be any value that doubles the average. So, values between 100-200 in our algorithm should be quite sufficient."]},{"idx":13,"heading":"Case Analysis","paragraphs":["The review text below is for a sci-fi movie which has the golden label 10 (most positive). However, if it is read as an isolated piece of text, identifying its sentiment is difficult. The LSTM+LA model gives it the rating of 1 (most negative), perhaps because on the surface, there are many negative words like unacceptable, criticize and sucks even though the reviewer is praising the movie. Since our user memory can learn that the reviewer is a fan of sci-fi movies, our DUPMN model indeed gives the correct rating of 10.","okay, there are two types of movie lovers: ... they expect to see a Titanic every time they go to the cinema ... this movie sucks? ... it is definitely better than other sci-fi ..... the audio and visual effects are simply terrific and Travolta's performance is brilliant-funny and interesting. what people expect from sci-fi is beyond me ... the rating for Battlefield Earth is below 2.5, which is unacceptable for a movie with such craftsmanship. Scary movie, possibly the worst of all time - ..., has a 6! maybe we should all be a little more subtle when we criticize movies... especially sci-fi.., since they have become an endangered genre ... give this movie the recognition it deserves."]},{"idx":14,"heading":"Conclusion and Future Work","paragraphs":["We propose a novel dual memory network model for sentiment predictions. We argue that user profile and product information are fundamentally different as user profiles reflect more on subjectivity whereas product information reflects more on salient features of products at aggregated level. Based on this hypothesis, two separate memory networks for user context and product context are built at the document level through a hierarchical learning model. The inclusion of an attention layer can further capture semantic information more effectively. Evaluation on three benchmark review datasets shows that the proposed DUPMN model outperforms the current state-of-the-art systems with significant improvements shown in p-value of 0.007, 0.004 and 0.001 respectively. We also show that single hop memory networks is the most effective model. Evaluation results show that user profile and product information are indeed different and have different effects on different datasets. In more subjective datasets such as IMDB, the inclusion of user profile information is more important. Whereas on more objective datasets such as Yelp data, collective information of restaurant plays a more important role in classification.","Future works include two directions. One direction is to explore the contribution of user profiles and product information in aspects level sentiment analysis tasks. Another direction is to explore how knowledge-based information can be incorporated to further improve sentiment classification tasks."]},{"idx":15,"heading":"Acknowledgments","paragraphs":["The work is partially supported by the research grants from Hong Kong Polytechnic University (PolyU RTVU) and GRF grant (CERG PolyU 15211/14E, PolyU 152006/16E)."]},{"idx":16,"heading":"Figures and Tables","paragraphs":["Figure 1: Structure for Proposed DUPMN Model","Table 1: Statistics of the three benchmark datasets","Table 2: Evaluation of different methods; best result/group in accuracy is marked in bold; second best is underlined.","Table 3: Evaluation of different memory network hops and user and product information utilization3","Table 4: Average weight of UMN and PMN in different datasets","Figure 2: Effect of different memory sizes"]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1810.02268","title":"A Large-Scale Test Set for the Evaluation of Context-Aware Pronoun Translation in Neural Machine Translation","sections":[{"idx":0,"heading":"Abstract","paragraphs":["The translation of pronouns presents a special challenge to machine translation to this day, since it often requires context outside the current sentence. Recent work on models that have access to information across sentence boundaries has seen only moderate improvements in terms of automatic evaluation metrics such as BLEU. However, metrics that quantify the overall translation quality are ill-equipped to measure gains from additional context. We argue that a different kind of evaluation is needed to assess how well models translate inter-sentential phenomena such as pronouns. This paper therefore presents a test suite of contrastive translations focused specifically on the translation of pronouns. Furthermore, we perform experiments with several context-aware models. We show that, while gains in BLEU are moderate for those systems, they outperform baselines by a large margin in terms of accuracy on our contrastive test set. Our experiments also show the effectiveness of parameter tying for multi-encoder architectures."]},{"idx":1,"heading":"Introduction","paragraphs":["Even though machine translation has improved considerably with the advent of neural machine translation (NMT) BIBREF0 , BIBREF1 , the translation of pronouns remains a major issue. They are notoriously hard to translate since they often require context outside the current sentence.","As an example, consider the sentences in Figure FIGREF1 . In both languages, there is a pronoun in the second sentence that refers to the European Central Bank. When the second sentence is translated from English to German, the translation of the pronoun it is ambiguous. This ambiguity can only be resolved with context awareness: if a translation system has access to the previous English sentence, the previous German translation, or both, it can determine the antecedent the pronoun refers to. In this German sentence, the antecedent Europäische Zentralbank dictates the feminine gender of the pronoun sie.","It is unfortunate, then, that current NMT systems generally operate on the sentence level BIBREF2 , BIBREF3 , BIBREF4 . Documents are translated sentence-by-sentence for practical reasons, such as line-based processing in a pipeline and reduced computational complexity. Furthermore, improvements of larger-context models over baselines in terms of document-level metrics such as BLEU or RIBES have been moderate, so that their computational overhead does not seem justified, and so that it is hard to develop more effective context-aware architectures and empirically validate them.","To address this issue, we present an alternative way of evaluating larger-context models on a test set that allows to specifically measure a model's capability to correctly translate pronouns. The test suite consists of pairs of source and target sentences, in combination with contrastive translation variants (for evaluation by model scoring) and additional linguistic and contextual information (for further analysis). The resource is freely available. Additionally, we evaluate several context-aware models that have recently been proposed in the literature on this test set, and extend existing models with parameter tying.","The main contributions of our paper are:","Section SECREF2 explains how our paper relates to existing work on context-aware models and the evaluation of pronoun translation. Section SECREF3 describes our test suite. The context-aware models we use in our experiments are detailed in Section SECREF4 . We discuss our experiments in Section SECREF5 and the results in Section SECREF6 ."]},{"idx":2,"heading":"Related Work","paragraphs":["Two lines of work are related to our paper: research on context-aware translation (described in Section SECREF8 ) and research on focused evaluation of pronoun translation (described in Section SECREF11 )."]},{"idx":3,"heading":"Context-Aware NMT Models","paragraphs":["If the translation of a pronoun requires context beyond the current sentence (see the example in Figure FIGREF1 ), a natural extension of sentence-level NMT models is to condition the model prediction on this necessary context. In the following, we describe a number of existing approaches to making models “aware” of additional context.","The simplest possible extension is to translate units larger than sentences. BIBREF5 concatenate each sentence with the sentence that precedes it, for the source side of the corpus or both sides. All of their models are standard sequence-to-sequence models built with recurrent neural networks (RNNs), since the method does not require any architectural change. BIBREF11 use the same concatenation technique with a Transformer architecture BIBREF2 , and experiment with wider context.","A number of works do propose changes to the NMT architecture. A common technique is to extend a standard encoder-decoder model by additional encoders for the context sentence(s), with a modified attention mechanism BIBREF6 , BIBREF9 , BIBREF8 . One aspect that differs between these works is the architecture of the encoder and attention. While BIBREF6 , BIBREF9 extend an RNN encoder-decoder with a second encoder that the decoder attends to, BIBREF8 extend the Transformer architecture with an encoder that is attended to by the main encoder. BIBREF8 also introduce parameter sharing between the main encoder and the context encoder, but do not empirically demonstrate its importance.","While the number of encoded sentences in the previous work is fixed, BIBREF7 , BIBREF10 explore the integration of variable-size context through a hierarchical architecture, where a first-level RNN reads in words to produce sentence vectors, which are then fed into a second-level RNN to produce a document summary.","Apart from differences in the architectures, related work varies in whether it considers source context, target context, or both (see Table TABREF9 for an overview of language arcs and context types). Some work considers only source context, but for pronoun translation, target-side context is intuitively important for disambiguation, especially if the antecedent itself is ambiguous. In our evaluation, we therefore emphasize models that take into account both source and target context.","Our experiments are based on models from BIBREF9 , who have released their source code. We extend their models with parameter sharing, which was shown to be beneficial by BIBREF8 . Additionally, we consider a concatenative baseline, similar to BIBREF5 , and Transformer-based models BIBREF8 .","This section describes several context-aware NMT models that we use in our experiments. They fall into two major categories: models based on RNNs and models based on the Transformer architecture BIBREF2 . We experiment with additional context on the source side and target side."]},{"idx":4,"heading":"Evaluation of Pronoun Translation","paragraphs":["Pronouns can serve a variety of functions with complex cross-lingual variation BIBREF12 , and hand-picked, manually annotated test suites have been presented for the evaluation of pronoun translation BIBREF13 , BIBREF14 , BIBREF9 . While suitable for analysis, the small size of the test suites makes it hard to make statistically confident comparisons between systems, and the hand-picked nature of the test suites introduces biases. To overcome these problems, we opted for a fully automatic approach to constructing a large-scale test suite.","Conceptually, our test set is most similar to the “cross-lingual pronoun prediction” task held at DiscoMT and WMT in recent years BIBREF15 , BIBREF16 , BIBREF17 : participants are asked to fill a gap in a target sentence, where gaps correspond to pronouns.","The first edition of the task focused on English INLINEFORM0 French, and it was found that local context (such as the verb group) was a strong signal for pronoun prediction. Hence, future editions only provided target-side lemmas instead of fully inflected forms, which makes the task less suitable to evaluate end-to-end neural machine translation systems, although such systems have been trained on the task BIBREF18 .","BIBREF17 do not report on the proportion of intra-sentential and inter-sentential anaphora in their test set, but the two top-performing systems only made use of intra-sentential information. Our test suite focuses on allowing the comparison of end-to-end context-aware NMT systems, and we thus extract a large number of inter-sentential anaphora, with meta-data allowing for a focus on inter-sentential anaphora with a long distance between the pronoun and its antecedent. Our focus on evaluating end-to-end NMT systems also relieves us from having to provide annotated training sets, and reduces pressure to achieve balance and full coverage of phenomena.","An alternative approach to automatically evaluate pronoun translation are reference-based methods that produce a score based on word alignment between source, translation output, and reference translation, and identification of pronouns in them, such as AutoPRF BIBREF19 and APT BIBREF20 . BIBREF21 perform a human meta-evaluation and show substantial disagreement between reference-based metrics and human judges, especially because there often exist valid alternative translations that use different pronouns than the reference. Our test set, and our protocol of generating contrastive examples, is focused on selected pronouns to minimize the risk of producing contrastive examples that are actually valid translations."]},{"idx":5,"heading":"Test set with contrastive examples","paragraphs":["Contrastive evaluation requires a large set of suitable examples that involve the translation of pronouns. As additional goals, our test set is designed to 1) focus on hard cases, so that it can be used as a benchmark to track progress in context-aware translation and 2) allow for fine-grained analysis.","Section SECREF14 describes how we extract our data set. Section SECREF26 explains how, given a set of contrastive examples, contrastive evaluation works."]},{"idx":6,"heading":"Automatic extraction of contrastive examples from corpora","paragraphs":["We automatically create a test set from the OpenSubtitles corpus BIBREF22 . The goal is to provide a large number of difficult test cases where an English pronoun has to be translated to a German pronoun.","The most challenging cases are translating it to either er, sie or es, depending on the grammatical gender of the antecedent. Not only is the translation of it ambiguous, there is also class imbalance in the training data (see Table TABREF18 ). There is roughly a 30% probability that it is aligned to es, which makes it difficult to learn to translate er and sie. We use parsing and automatic co-reference resolution to find translation pairs that satisfy our constraints.","To provide a basis for filtering with constraints, we tokenize the whole data set with the Moses tokenizer, generate symmetric word alignments with fast_align BIBREF23 , parse the English text with CoreNLP BIBREF24 , parse the German text with ParZu BIBREF25 and perform coreference resolution on both sides. The coreference chains are obtained with the neural model of CoreNLP for English, and with CorZu for German BIBREF26 , respectively.","Then we opt for high-precision, aggressive filtering, according to the following protocol: for each pair of sentences INLINEFORM0 in English and German, extract iff","INLINEFORM0 contains the English pronoun it, and INLINEFORM1 contains a German pronoun that is third person singular (er, sie or es), as indicated by their part-of-speech tags;","those pronouns are aligned to each other;","both pronouns are in a coreference chain;","their nominal antecedents in the coreference chain are aligned on word level.","This removes most candidate pairs, but is necessary to overcome the noise introduced by our preprocessing pipeline, most notably coreference resolution. From the filtered set, we create a balanced test set by randomly sampling 4000 instances of each of the three translations of it under consideration (er, sie, es). We do not balance antecedent distance. See Table TABREF25 for the distribution of pronoun pairs and antecedent distance in the test set.","For each sentence pair in the resulting test set, we introduce contrastive translations. A contrastive translation is a translation variant where the correct pronoun is swapped with an incorrect one. For an example, see Table TABREF19 , where the pronoun it in the original translation corresponds to sie because the antecedent bat is a feminine noun in German (Fledermaus). We produce wrong translations by replacing sie with one of the other pronouns (er, es).","Note that, by themselves, these contrastive translations are grammatically correct if the antecedent is outside the current sentence. The test set also contains pronouns with an antecedent in the same sentence (antecedent distance 0). Those examples do not require any additional context for disambiguation and we therefore expect the sentence-level baseline to perform well on them.","We take extra care to ensure that the resulting contrastive translations are grammatically correct, because ungrammatical sentences are easily dismissed by an NMT system. For instance, if there are any possessive pronouns (such as seine) in the sentence, we also change their gender to match the personal pronoun replacement.","The German coreference resolution system does not resolve es because most instances of es in German are either non-referential forms, or they refer to a clause instead of a nominal antecedent. We limit the test set to nominal antecedents, as these are the only ambiguous cases with respect to translation. For this reason, we have to rely entirely on the English coreference links for the extraction of sentence pairs with it INLINEFORM0 es, as opposed to pairs with it INLINEFORM1 er and it INLINEFORM2 sie where we have coreference chains in both languages.","Our extraction process respects document boundaries, to ensure we always search for the right context. We extract additional information from the annotated documents, such as the distance (in sentences) between pronouns and their antecedents, the document of origin, lemma, morphology and dependency information if available."]},{"idx":7,"heading":"Evaluation by scoring","paragraphs":["Contrastive evaluation is different from conventional evaluation of machine translation in that it does not require any translation. Rather than testing a model's ability to translate, it is a method to test a model's ability to discriminate between given good and bad translations.","We exploit the fact that NMT systems are in fact language models of the target language, conditioned on source text. Like language models, NMT systems can be used to compute a model score (the negative log probability) for an existing translation. Contrastive evaluation, then, means to compare the model score of two pairs of inputs: INLINEFORM0 and INLINEFORM1 . If the model score of the actual reference translation is higher, we assume that this model can detect wrong pronoun translations.","However, this does not mean that systems actually produce the reference translation when given the source sentence for translation. An entirely different target sequence might rank higher in the system's beam during decoding. The only conclusion permitted by contrastive evaluation is whether or not the reference translation is more probable than a contrastive variant.","If the model score of the reference is indeed higher, we refer to this outcome as a “correct decision” by the model. The model's decision is only correct if the reference translation has a higher score than any contrastive translation. In our evaluation, we aggregate model decisions on the whole test set and report the overall percentage of correct decisions as accuracy.","During scoring, the model is provided with reference translations as target context, while during translation, the model needs to predict the full sequence. It is an open question to what extent performance deteriorates when context is itself predicted, and thus noisy. We highlight that the same problem arises for sentence-level NMT, and has been addressed with alternative training strategies BIBREF27 ."]},{"idx":8,"heading":"Recurrent Models","paragraphs":["We consider the following recurrent baselines:","baseline Our baseline model is a standard bidirectional RNN model with attention, trained with Nematus. It operates on the sentence level and does not see any additional context. The input and output embeddings of the decoder are tied, encoder embeddings are not.","concat22 We concatenate each sentence with one preceding sentence, for both the source and target side of the corpus. Then we train on this new data set without any changes to the model architecture. This very simple method is inspired by BIBREF5 .","The following models are taken, or slightly adapted, from BIBREF9 . For this reason, we give only a very short description of them here and the reader is referred to their work for details.","s-hier A multi-encoder architecture with hierarchical attention. This model has access to one additional context: the previous source sentence. It is read by a separate encoder, and attended to by an additional attention network. The output of the resulting two attention vectors is combined with yet another attention network.","s-t-hier Identical to s-hier, except that it considers two additional contexts: the previous source sentence and previous target sentence. Both are read by separate encoders, and sequences from all encoders are combined with hierarchical attention.","s-hier-to-2 The model has an additional encoder for source context, whereas the target side of the corpus is concatenated, in the same way as for concat22. This model achieved the best results in BIBREF9 .","For each variant, we also introduce and test weight tying: we share the parameters of embedding matrices between encoders that read the same kind of text (source or target side)."]},{"idx":9,"heading":"Transformer Models","paragraphs":["All remaining models are based on the Transformer architecture BIBREF2 . A Transformer avoids recurrence completely: it follows an encoder-decoder architecture using stacked self-attention and fully connected layers for both the encoder and decoder.","baseline A standard context-agnostic Transformer. All model parameters are identical to a Transformer-base in BIBREF2 .","concat22 A simple concatentation model where only the training data is modified, in the same way as for the recurrent concat22 model.","concat21 Trained on data where the preceding sentence is concatenated to the current one only on the source side. This model is also taken from BIBREF5 .","BIBREF8 A more sophisticated context-aware Transformer that uses source context only. It has a separate encoder for source context, but all layers except the last one are shared between encoders. A source and context sentence are first encoded independently, and then a single attention layer and a gating function are used to produce a context-aware representation of the source sentence. Such restricted interaction with context is shown to be beneficial for analysis of contextual phenomena captured by the model. For details the reader is referred to their work."]},{"idx":10,"heading":"Experiments","paragraphs":["We train all models on the data from the WMT 2017 English INLINEFORM0 German news translation shared task ( INLINEFORM1 5.8 million sentence pairs). These corpora do not have document boundaries, therefore a small fraction of sentences will be paired with wrong context, but we expect the model to be robust against occasional random context (see also BIBREF8 ). Experimental setups for the RNN and Transformer models are different, and we describe them separately.","All RNN-based models are trained with Nematus BIBREF28 . We learn a joint BPE model with 89.5k merge operations BIBREF29 . We train shallow models with an embedding size of 512, a hidden layer size of 1024 and layer normalization. Models are trained with Adam BIBREF30 , with an initial learning rate of 0.0001. We apply early stopping based on validation perplexity. The batch size for training is 80, and the maximum length of training sequences is 100 (if input sentences are concatenated) or 50 (if input lines are single sentences).","For our Transformer-based experiments, we use a custom implementation and follow the hyperparameters from BIBREF2 , BIBREF8 . Systems are trained on lowercased text that was encoded using BPE (32k merge operations). Models consist of 6 encoder and decoder layers with 8 attention heads. The hidden state size is 512, the size of feedforward layers is 2048.","Model performance is evaluated in terms of BLEU, on newstest2017, newstest2018 and all sentence pairs from our pronoun test set. We compute scores with SacreBLEU BIBREF31 . Evaluation with BLEU is done mainly to control for overall translation quality.","To evaluate pronoun translation, we perform contrastive evaluation and report the accuracy of models on our contrastive test set."]},{"idx":11,"heading":"Evaluation","paragraphs":["The BLEU scores in Table TABREF30 show a moderate improvement for most context-aware systems. This suggests that the architectural changes for the context-aware models do not degrade overall translation quality. The contrastive evaluation on our test set on the other hand shows a clear increase in the accuracy of pronoun translation: The best model s-hier-to-2.tied achieves a total of +16 percentage points accuracy on the test set over the baseline, see Table TABREF31 .","Table TABREF32 shows that context-aware models perform better than the baseline when the antecedent is outside the current sentence. In our experiments, all context-aware models consider one preceding sentence as context. The evaluation according to the distance of the antecedent in Table TABREF35 confirms that the subset of sentences with antecedent distance 1 benefits most from the tested context-aware models (up to +20 percentage points accuracy). However, we note two surprising patterns:","The first observation can be explained by the distribution of German pronouns in the test set. The further away the antecedent, the higher the percentage of it INLINEFORM0 es cases, which are the majority class, and thus the class that will be predicted most often if evidence for other classes is lacking. We speculate that this is due to our more permissive extraction heuristics for it INLINEFORM1 es.","We attribute the second observation to the existence of coreference chains where the preceding sentence contains a pronoun that refers to the same nominal antecedent as the pronoun in the current sentence. Consider the example in Table TABREF36 : The nominal antecedent of it in the current sentence is door, Tür in German with feminine gender. The nominal antecedent occurs two sentences before the current sentence, but the German sentence in between contains the pronoun sie, which is a useful signal for the context-aware models, even though they cannot know the nominal antecedent.","Note that only models aware of target-side context can benefit from such circumstances: The s-hier models as well as the Transformer model by BIBREF8 only see source side context, which results in lower accuracy if the distance to the antecedent is INLINEFORM0 1, see Table TABREF35 .","While such coreference chains complicate the interpretation of the results, we note that improvements on inter-sentential anaphora with antecedent distance INLINEFORM0 are relatively small (compared to distance 1), and that performance is still relatively poor (especially for the minority classes er and sie). We encourage evaluation of wider-context models on this subset, which is still large thanks to the size of the full test set.","Regarding the comparison of different context-aware architectures, our results demonstrate the effectiveness of parameter sharing between the main encoder (or decoder) and the contextual encoder. We observe an improvement of 5 percentage points from s-hier-to-2 to s-hier-to-2.tied, and 4 percentage points from s-t-hier to s-t-hier.tied. Context encoders introduce a large number of extra parameters, while inter-sentential context is only relevant for a relatively small number of predictions. We hypothesize that the training signal is thus too weak to train a strong contextual encoder in an end-to-end fashion without parameter sharing. Our results also confirm the finding by BIBREF9 that multi-encoder architectures, specifically s-hier-to-2(.tied), can outperform a simple concatenation system in the translation of coreferential pronouns.","The Transformer-based models perform strongest on pronouns with intra-segmental antecedent, outperforming the recurrent baseline by 9–18 percentage points. This is likely an effect of increased model depth and the self-attentional architecture in this set of experiments. The model by BIBREF8 only uses source context, and outperforms the most comparable RNN system, s-hier.tied. However, the Transformer-based concat22 slightly underperforms the RNN-based concat22, and we consider it future research how to better exploit target context with Transformer-based models."]},{"idx":12,"heading":"Conclusions","paragraphs":["We present a large-scale test suite to specifically test the capacity of NMT models to translate pronouns correctly. The test set contains 12,000 difficult cases of pronoun translations from English it to its German counterparts er, sie and es, extracted automatically from OpenSubtitles BIBREF22 .","We evaluate recently proposed context-aware models on our test set. Even though the increase in BLEU score is moderate for all context-aware models, the improvement in the translation of pronouns is considerable: The best model (s-hier-to-2.tied) achieves a +16 percentage points gain in accuracy over the baseline.","Our experiments confirm the importance of careful architecture design, with multi-encoder architectures outperforming a model that simply concatenates context sentences. We also demonstrate the effectiveness of parameter sharing between encoders of a context-aware model.","We hope the test set will prove useful for empirically validating novel architectures for context-aware NMT. So far, we have only evaluated models that consider one sentence of context, but the nominal antecedent is more distant for a sizable proportion of the test set, and the evaluation of variable-size context models BIBREF7 , BIBREF10 is interesting future work."]},{"idx":13,"heading":"Acknowledgements","paragraphs":["We are grateful to the Swiss National Science Foundation (SNF) for supporting the project CoNTra (grant number 105212_169888)."]},{"idx":14,"heading":"Figures and Tables","paragraphs":["Table 1: Overview of context-aware translation models in related work.","Table 2: Frequency and probability of alignments of it in the training data of our systems (all data from the WMT 2017 news translation task). Alignments are produced by a fast_align model.","Table 3: Example sentence pair with contrastive translations. An antecedent distance of 1 means that the antecedent is in the immediately preceding sentence.","Table 4: Test set frequencies of pronoun pairs and antecedent distance (measured in sentences).","Table 5: English→German BLEU scores on newstest2017, newstest2018 and all sentence pairs from our pronoun test set. Case-sensitive and case-insensitive (uncased) scores are reported. Higher is better, and the best scores are marked in bold.","Table 6: Accuracy on contrastive test set (N=4000 per pronoun) with regard to reference pronoun.","Table 7: Accuracy on contrastive test set with regard to antecedent location (within segment vs. outside segment).","Table 8: Accuracy on contrastive test set with regard to antecedent distance of antecedent (in sentences).","Table 9: Example where 1) antecedent distance is >1 and 2) the context given contains another pronoun as an additional hint."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1810.12196","title":"ReviewQA: a relational aspect-based opinion reading dataset","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Deep reading models for question-answering have demonstrated promising performance over the last couple of years. However current systems tend to learn how to cleverly extract a span of the source document, based on its similarity with the question, instead of seeking for the appropriate answer. Indeed, a reading machine should be able to detect relevant passages in a document regarding a question, but more importantly, it should be able to reason over the important pieces of the document in order to produce an answer when it is required. To motivate this purpose, we present ReviewQA, a question-answering dataset based on hotel reviews. The questions of this dataset are linked to a set of relational understanding competencies that we expect a model to master. Indeed, each question comes with an associated type that characterizes the required competency. With this framework, it is possible to benchmark the main families of models and to get an overview of what are the strengths and the weaknesses of a given model on the set of tasks evaluated in this dataset. Our corpus contains more than 500.000 questions in natural language over 100.000 hotel reviews. Our setup is projective, the answer of a question does not need to be extracted from a document, like in most of the recent datasets, but selected among a set of candidates that contains all the possible answers to the questions of the dataset. Finally, we present several baselines over this dataset."]},{"idx":1,"heading":"Introduction","paragraphs":["A large majority of the human knowledge is recorded through text documents. That is why ability for a system to automatically infer information from text without any structured data has become a major challenge. Answering questions about a given document is a relevant proxy task that has been proposed as a way to evaluate the reading ability of a given model. In this configuration, a text document such as a news article, a document from Wikipedia or any type of text is presented to a machine with an associated set of questions. The system is then expected to answer these questions and evaluated by its accuracy on this task. The machine reading framework is very general and we can imagine a large panel of questions that can possibly handle most of the standard natural language processing tasks. For example, the task of named entities recognition can be formulated as a machine reading one where your document is the sentence and the question would be 'What are the named entities mentioned in this sentence?'. These natural language interactions are an important objective for reading systems.","Recently, many datasets have been proposed to build and evaluate reading models BIBREF0 , BIBREF1 . From cloze style questions BIBREF2 to open questions BIBREF3 , from synthetic data BIBREF4 to human written articles BIBREF5 , many styles of documents and questions have been proposed to challenge reading models. The correct answer to the questions proposed in most of these datasets is a span of text of the source document, which can be restricted to a single word in several cases. It means that the answer should explicitly be present in the source document and that the model should be able to locate it.","Different models have already shown superhuman performance on several of these datasets and particularly on the SQuAD dataset composed of Wikipedia articles BIBREF6 , BIBREF7 . However, some limits of such models have been highlighted when they encounter perturbations into the input documents BIBREF8 . Indeed almost all of the state of the art models on the SQuAD dataset suffer from a lack of robustness against adversarial examples. Once the model is trained, a meaningless sentence added at the end of the text document can completely disturb the reading system. Conversely, these adversarial examples do not seem to fool a human reader who will be capable of answering the questions as well as without this perturbation. One possible explanation of this phenomenon is that computers are good at extracting patterns in the document that match the representation of the question. If multiple spans of the documents look similar to the questions, the reader might not be able to decide which one is relevant. Moreover, Wikipedia articles tend to be written with the same standard writing style, factual, unambiguous. Such writing style tends to favor the pattern matching between the questions and the documents. This format of documents/questions has certainly influenced the design of the comprehension models that have been proposed so far. Most of them are composed of stacked attention layers that match question and document representations.","Following concepts proposed in the 20 bAbI tasks BIBREF4 or in the visual question-answering dataset CLEVR BIBREF9 , we think that the challenge, limited to the detection of relevant passages in a document, is only the first step in building systems that truly understand text. The second step is the ability of reasoning with the relevant information extracted from a document. To set up this challenge, we propose to leverage on a hotel reviews corpus that requires reasoning skills to answer natural language questions. The reviews we used have been extracted from TripAdvisor and originally proposed in BIBREF10 , BIBREF11 . In the original data, each review comes with a set of rated aspects among the seventh available: Business service, Check in / Front Desk, Cleanliness, Location, Room, Sleep Quality, Value and for all the reviews an Overall rating. In this articles we propose to exploit these data to create a dataset of question-answering that will challenge 8 competencies of the reader.","Our contributions can be summarized as follow:"]},{"idx":2,"heading":"Machine comprehension datasets","paragraphs":["ReviewQA is proposed as a novel dataset regarding the collection of the existing ones. Indeed a large panel of available datasets, that evaluate models on different types of documents, can only be valuable for designing efficient models and learning protocols. In this following part, we describe several of these datasets.","SQuAD: The Standford Question Answering Dataset (SQuAD) introduced in BIBREF0 is a large dataset of natural questions over the 500 most popular articles of Wikipedia. All the questions have been crowdsourced and answers are spans of text extracted from source documents. This dataset has been very popular these last two years and the performance of the architectures that have been proposed have rapidly increased until several models surpass the human score. Indeed, in the original paper human performance has been measured at 82.304 points for the exact match metric and at the time we are writing this paper four models have already a higher score. In another hand BIBREF8 has shown that these models suffer from a lack of robustness against adversarial examples that are meaningless from a human point of view. This suggests the need for a more challenging dataset that will allow developing strongest reasoning architectures.","NewsQA: NewsQA BIBREF1 is a dataset very similar to SQuAD. It contains 120.000 human generated questions over 12.000 articles form CNN originally introduced in BIBREF5 . It has been designed to be more challenging than SQuAD with questions that might require to extract multiple spans of text or not be answerable.","WikiHop and MedHop: These are two recent datasets introduced in BIBREF13 . Unlike SQuAD and NewsQA, important facts are spread out across multiple documents and, in order to answer a question, it is necessary to jump over a set of passages to collect the required information. The relevant passages are not explicitly mentioned in the data so this dataset measures the ability that a model has to navigate across multiple documents. The questions come with a set of candidates which are all present in the text.","MS Marco: This dataset has been released in BIBREF14 . The documents come from the internet and the questions are real user queries asked through the bing search engine. The dataset contains around 100.000 queries and each of them comes with a set of approximatively 10 relevant passages. Like in SQuAD, several models are already doing superhuman performances on this dataset.","Facebook bAbI tasks: This is a set of 20 toy tasks proposed in BIBREF4 and designed to measure text understanding. Each task requires a certain capability to be completed like induction, deduction and more. Documents are synthetic stories, composed of few sentences that describe a set of actions. This dataset was one of the first attempt to introduce a general set of prerequisite capabilities required for the reading task. Although it has been a very challenging framework, beneficial to the emergence of the attention mechanism inside the reading architectures, a Gated end-to-end memory network BIBREF15 now succeed in almost all of the 20 tasks. One of the possible reason is that the data are synthetic data, without noise or ambiguity. We propose a comparable framework with understanding and reasoning tasks based on user-generated comments that are much more realistic and that required language competencies to be understood.","CLEVR: Beyond textual question-answering, Visual Question-Answering (VQA) has been largely studied during the last couple of years. More recently, the problem of relational reasoning has been introduced through this dataset BIBREF9 . The main original idea was to introduce relational reasoning questions over object shapes and placements. This dataset has already motivated the development of original deep models. To the best of our knowledge, no natural language question-answering corpus has been designed to investigate such capabilities. As we will present in the following of this paper, we think sentiment analysis is particularly suited for this task and we will introduce a novel machine reading corpus with such capability requirements."]},{"idx":3,"heading":"Attention-based models for aspect-based sentiment analysis","paragraphs":["Sentiment analysis is one of the historical tasks of Natural Language Processing. It is an important challenge for companies, restaurants, hotels that aim to analyze customer satisfaction regarding products and quality of services. Given a text document, the objective is to predict its overall polarity. Generally, it can be positive, negative or neutral. This analysis gives a quick overview of a general sentiment over a set of documents, but this framework tends to be restrictive. Indeed, one document tends to express multiple opinions of different aspects. For instance, in the sentence: The fish was very good but the service was terrible, there is not a general dominant sentiment, and a finer analysis is needed. The task of aspect-based sentiment analysis aims to predict a polarity of a sentence regarding a given aspect. In the previous example a positive polarity should be associated to the aspect food, and on the contrary, a negative sentiment is expressed regarding the quality of the service.","The idea of using models originally designed for question-answering, for the sentiment analysis task has been introduced in BIBREF16 , BIBREF17 . In these papers, several adaptations of the end-to-end memory network (MemN2N) BIBREF18 are used to predict the polarity of a review regarding a given aspect. In that configuration, the review is encoded into the memory cells and the controller, usually initialized with a representation of the question, is initialized with a representation of the aspect. The analysis of the attention between the values of the controller and the document has shown interesting results, by highlighting relevant part of a document regarding an aspect."]},{"idx":4,"heading":"ReviewQA dataset","paragraphs":["We think that evaluating the task of sentiment analysis through the setup of question-answering is a relevant playground for machine reading research. Indeed natural language questions about the different aspects of the targeted venues are typical kind of questions we want to be able to ask to a system. In this context, we introduce a set of reasoning questions types over the relationships between aspects. We propose ReviewQA, a dataset of natural language questions over hotel reviews. These questions are divided into 8 groups, regarding the competency required to be answered. In this section, we describe each task and the process followed to generate this dataset."]},{"idx":5,"heading":"Original data","paragraphs":["We used a set of reviews extracted from the TripAdvisor website and originally proposed in BIBREF10 and BIBREF11 . This corpus is available at http://www.cs.virginia.edu/~hw5x/Data/LARA/TripAdvisor/TripAdvisorJson.tar.bz2. Each review comes with the name of the associated hotel, a title, an overall rating, a comment and a list of rated aspects. From 0 to 7 aspects, among value, room, location, cleanliness, check-in/front desk, service, business service, can possibly be rated in a review. Figure FIGREF8 displays a review extracted from this dataset."]},{"idx":6,"heading":"Relational reasoning competencies","paragraphs":["Objective: Starting with the original corpus, we aim at building a machine reading task where natural language questions will challenge the model on its understanding of the reviews. Indeed learning relational reasoning competencies over natural language documents is a major challenge of the current reading models. These original raw data allow us to generate relational questions that can possibly require a global understanding of the comment and reasoning skills to be treated. For example, asking a question like What is the best aspect rated in this comment ? is not an easy question that can be answered without a deep understanding of the review. It is necessary to capture all the aspects mentioned in the text, to predict their rating and finally to select the best one. The tasks and the dataset we propose are publicly available at http://www.europe.naverlabs.com/Blog/ReviewQA-A-novel-relational-aspect-based-opinion-dataset-for-machine-reading","We introduce a list of 8 different competencies that a reading system should master in order to process reviews and text documents in general. These 8 tasks require different competencies and a different level of understanding of the document to be well answered. For instance, detecting if an aspect is mentioned in a review will require less understanding of the review than predicting explicitly the rating of this aspect. Table TABREF10 presents the 8 tasks we have introduced in this dataset with an example of a question that corresponds to each task. We also provide the expected type of the answer (Yes/No question, rating question...). It can be an additional tool to analyze the errors of the readers."]},{"idx":7,"heading":"Construction of the dataset","paragraphs":["We sample 100.000 reviews from the original corpus. Figure FIGREF12 presents the distribution of the number of words of the reviews in the dataset. We explicitly favor reviews which contain an important number of words. In average, a review contains 200 words. Indeed these long reviews are most likely to contain challenging relations between different aspects. A short review which deals with only a few aspects is more likely to not be very relevant to the challenge we want to propose in this dataset. Figure FIGREF14 displays the distribution of the ratings per aspects in the 100.000 reviews we based our dataset. We can see that the average values of these ratings tend to be quite high. It could have introduced bias if it was not the case for all the aspects. For example, we do not want that the model learns that in general, the service is rated better than the location and them answer without looking at the document. Since this situation is the same for all the aspects, the relational tasks introduced in this dataset remains extremely relevant.","Then we randomly select 6 tasks for each review (the same task can be selected multiple times) and randomly select a natural language question that corresponds to this task. The questions are human-generated patterns that we have crowdsourced in order to produce a dataset as rich as possible. To this end, we have generated several patterns that correspond to the capabilities we wanted to express in a given question and we have crowdsourced rephrasing of these patterns.","The final dataset we propose is composed of more than 500.000 questions about 100.000 reviews. Table TABREF13 shows the repartition of the documents and queries into the train and test set. Each review contains a maximum of 6 questions. Sometimes less when it is not possible to generate all. For example, if only two or three aspects are mentioned in a review, we will be able to generate only a little set of relational questions. Figure FIGREF15 depicts the repartition of the answers in the generated dataset. A majority of the tasks we introduced, even if they possibly require a high level of understanding of the document and the question, are binary questions. It means that in the generated dataset the answers yes and no tend to be more present than the others. To balance in a better way the distribution of the answers, we chose to affect a higher probability of sampling to the task 5, 6, 7.1, 8. Indeed, these tasks are not binary questions and required an aspect name as the answer. Figure FIGREF17 represents the repartition of question types in our dataset. Finally, figure FIGREF15 shows the repartition of the answers in the dataset."]},{"idx":8,"heading":"Paraphrase augmentation using backtranslation","paragraphs":["In order to generate more paraphrases of the questions, we used a backtranslation method to enrich them. The idea is to use a translation model that will translate our human-generated questions into another language, and then translate them back to English. This double translation will introduce rewordings of the questions that we will be able to integrate into this dataset. This approach has been used in BIBREF7 to perform data augmentation on the training set. For this purpose, we have trained a fairseq BIBREF19 model to translate sentences from English to French and for French to English. In order to preserve the quality of the sentences we have so far, we only keep the most probable translation of each original sentence. Indeed a beam search is used during the translation to predict the most probable translations which mean that we each translation comes with an associated probability. By selecting only the first translations, we almost double the number of questions without degrading the quality of the questions proposed in the dataset."]},{"idx":9,"heading":"Models","paragraphs":["In this section, we present the performance of four different models on our dataset: a logistic regression and three neural models. The first one is a basic LSTM BIBREF20 , the second a MemN2N BIBREF18 and the third one is a model of our own design. This fourth model reuses the encoding layers of the R-net BIBREF12 and we modify the final layers with a projection layer that will be able to select the answer among the set of candidates instead of pointing the answerer directly into the source document.","Logistic regression: To produce the representation of the input, we concatenate the Bag-Of-Words representation of the document with the Bag-Of-Words representation of the question. It produces an array of size INLINEFORM0 where INLINEFORM1 is the vocabulary size. Then we use a logistic regression to select the most probable answer among the INLINEFORM2 possibilities.","LSTM: We start with a concatenation of the sequence of indexes of the document with the sequence of indexes of the question. Them we feed an LSTM network with this vector and use the final state as the representation of the input. Finally, we apply a logistic regression over this representation to produce the final decision.","End-to-end memory networks: This architecture is based on two different memory cells (input and output) that contain a representation of the document. A controller, initialized with the encoding of the question, is used to calculate an attention between this controller and the representation of the document in the input memory. This attention is them used to re-weight the representation of the document in the output memory. This response from the output memory is them utilized to update the controller. After that, either a matrix is used to project this representation into the answer space either the controller is used to go through an over hop of memory. This architecture allows the model to sequentially look into the initial document seeking for important information regarding the current state of its controller. This model achieves very good performances on the 20 bAbI tasks dataset.","Deep projective reader: This is a model of our own design, largely inspired by the efficient R-net reader BIBREF12 . The overall architecture is composed of 4 stacked layers: an encoding layer, a question/document attention, a self-attention layer and a projection layer. The following paragraphs briefly describe the overall utility of each of these layers.","Encoding: The sentence is tokenized by words. Each token is represented by the concatenation of its embedding vector and the final state of a bidirectional recurrent network over the characters of this word. Finally, another bidirectional RNN on the top of this representation produce the encoding of the document and the question.","Question/document attention: We apply a question/document attention layer that matches the representation of the question with each token of the document individually to output an attention that gives more weight to the important tokens of the document regarding the question.","Self-attention layer: The previous layer has built a question-aware representation of the document. One problem with such representation is that form the moment each token has only a good knowledge of its closest neighbors. To tackle this problem, BIBREF12 have proposed to use a self-attention layer that matches each individual token with all the other tokens of the document. Doing that, each token is now aware of a larger context.","Output layer: A bidirectional RNN is applied on the top of the last layer and we use its final state as the representation of the input. We use a projection matrix to project this representation into the answer space and select the most probable one"]},{"idx":10,"heading":"Training details","paragraphs":["We propose to train these models on the entire set of tasks and them to measure the overall performance and the accuracy of each individual task. In all the models, we use the Adam optimizer BIBREF21 with a learning rate of 0.01 and the batch size is set to 64. All the parameter are initialized from a Gaussian distribution with mean 0 and a standard deviation of 0.01. The dimension of the word embeddings in the projective deep reading model and the LSTM model is 300 and we use Glove pre-trained vectors ( BIBREF22 ). We use a MemN2N with 5 memory hops and a linear start of 5 epochs. The reviews are split by sentence and each memory block corresponds to one sentence. Each sentence is represented by its bag-of-word representation augmented with temporal encoding as it is suggested in BIBREF18 ."]},{"idx":11,"heading":"Model performance","paragraphs":["Table TABREF19 displays the performance of the 4 baselines on the ReviewQA's test set. These results are the performance achieved by our own implementation of these 4 models. According to our results, the simple LSTM network and the MemN2N perform very poorly on this dataset. Especially on the most advanced reasoning tasks. Indeed, the task 5 which corresponds to the prediction of the exact rating of an aspect seems to be very challenging for these model. Maybe the tokenization by sentence to create the memory blocks of the MemN2N, which is appropriated in the case of the bAbI tasks, is not a good representation of the documents when it has to handle human generated comments. However, the logistic regression achieves reasonable performance on these tasks, and do not suffer from catastrophic performance on any tasks. Its worst result comes on task 6 and one of the reason is probably that this architecture is not designed to predict a list of answers. On the contrary, the deep projective reader achieves encouraging on this dataset. It outperforms all the other baselines, with very good scores on the first fourth tasks. The question/document and document/document attention layers proposed in BIBREF12 seem once again to produce rich encodings of the inputs which are relevant for our projection layer."]},{"idx":12,"heading":"Conclusion","paragraphs":["In this paper, we formalize the sentiment analysis task through the framework of machine reading and release ReviewQA, a relational question-answering corpus. This dataset allows evaluating a set of relational reasoning skills through natural language questions. It is composed of a large panel of human-generated questions. Moreover, we propose to augment the dataset with backtranslated reformulations of these questions. Finally, we evaluate 4 models on this dataset, including a projective model of our own design that seems to be a strong baseline for this dataset. We expect that this large dataset will encourage the research community to develop reasoning models and evaluate their models on this set of tasks."]},{"idx":13,"heading":"Acknowledgment","paragraphs":["We thank Vassilina Nikoulina and Stéphane Clinchant for the help regarding the backtranslation rewording of the questions."]},{"idx":14,"heading":"Figures and Tables","paragraphs":["Figure 2: Number of words per review.","Figure 1: An example from the original dataset.","Table 1: Descriptions and examples of the 8 tasks evaluated in ReviewQA.","Figure 3: Distribution of the ratings per aspect.","Figure 4: Distribution of answers in the generated dataset.","Table 2: Repartition of the questions into the train and test set.","Figure 5: Repartition of the tasks in the generated dataset.","Table 3: Accuracy (%) of 4 models on the ReviewQA dataset."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1901.08079","title":"A Question-Entailment Approach to Question Answering","sections":[{"idx":0,"heading":"Abstract","paragraphs":["One of the challenges in large-scale information retrieval (IR) is to develop fine-grained and domain-specific methods to answer natural language questions. Despite the availability of numerous sources and datasets for answer retrieval, Question Answering (QA) remains a challenging problem due to the difficulty of the question understanding and answer extraction tasks. One of the promising tracks investigated in QA is to map new questions to formerly answered questions that are `similar'. In this paper, we propose a novel QA approach based on Recognizing Question Entailment (RQE) and we describe the QA system and resources that we built and evaluated on real medical questions. First, we compare machine learning and deep learning methods for RQE using different kinds of datasets, including textual inference, question similarity and entailment in both the open and clinical domains. Second, we combine IR models with the best RQE method to select entailed questions and rank the retrieved answers. To study the end-to-end QA approach, we built the MedQuAD collection of 47,457 question-answer pairs from trusted medical sources, that we introduce and share in the scope of this paper. Following the evaluation process used in TREC 2017 LiveQA, we find that our approach exceeds the best results of the medical task with a 29.8% increase over the best official score. The evaluation results also support the relevance of question entailment for QA and highlight the effectiveness of combining IR and RQE for future QA efforts. Our findings also show that relying on a restricted set of reliable answer sources can bring a substantial improvement in medical QA."]},{"idx":1,"heading":"Introduction","paragraphs":["With the availability of rich data on users' locations, profiles and search history, personalization has become the leading trend in large-scale information retrieval. However, efficiency through personalization is not yet the most suitable model when tackling domain-specific searches. This is due to several factors, such as the lexical and semantic challenges of domain-specific data that often include advanced argumentation and complex contextual information, the higher sparseness of relevant information sources, and the more pronounced lack of similarities between users' searches.","A recent study on expert search strategies among healthcare information professionals BIBREF0 showed that, for a given search task, they spend an average of 60 minutes per collection or database, 3 minutes to examine the relevance of each document, and 4 hours of total search time. When written in steps, their search strategy spans over 15 lines and can reach up to 105 lines.","With the abundance of information sources in the medical domain, consumers are more and more faced with a similar challenge, one that needs dedicated solutions that can adapt to the heterogeneity and specifics of health-related information.","Dedicated Question Answering (QA) systems are one of the viable solutions to this problem as they are designed to understand natural language questions without relying on external information on the users.","In the context of QA, the goal of Recognizing Question Entailment (RQE) is to retrieve answers to a premise question ( INLINEFORM0 ) by retrieving inferred or entailed questions, called hypothesis questions ( INLINEFORM1 ) that already have associated answers. Therefore, we define the entailment relation between two questions as: a question INLINEFORM2 entails a question INLINEFORM3 if every answer to INLINEFORM4 is also a correct answer to INLINEFORM5 BIBREF1 .","RQE is particularly relevant due to the increasing numbers of similar questions posted online BIBREF2 and its ability to solve differently the challenging issues of question understanding and answer extraction. In addition to being used to find relevant answers, these resources can also be used in training models able to recognize inference relations and similarity between questions.","Question similarity has recently attracted international challenges BIBREF3 , BIBREF4 and several research efforts proposing a wide range of approaches, including Logistic Regression, Recurrent Neural Networks (RNNs), Long Short Term Memory cells (LSTMs), and Convolutional Neural Networks (CNNs) BIBREF5 , BIBREF6 , BIBREF1 , BIBREF7 .","In this paper, we study question entailment in the medical domain and the effectiveness of the end-to-end RQE-based QA approach by evaluating the relevance of the retrieved answers. Although entailment was attempted in QA before BIBREF8 , BIBREF9 , BIBREF10 , as far as we know, we are the first to introduce and evaluate a full medical question answering approach based on question entailment for free-text questions. Our contributions are:","The next section is dedicated to related work on question answering, question similarity and entailment. In Section SECREF3 , we present two machine learning (ML) and deep learning (DL) methods for RQE and compare their performance using open-domain and clinical datasets. Section SECREF4 describes the new collection of medical question-answer pairs. In Section SECREF5 , we describe our RQE-based approach for QA. Section SECREF6 presents our evaluation of the retrieved answers and the results obtained on TREC 2017 LiveQA medical questions."]},{"idx":2,"heading":"Background","paragraphs":["In this section we define the RQE task and describe related work at the intersection of question answering, question similarity and textual inference."]},{"idx":3,"heading":"Task Definition","paragraphs":["The definition of Recognizing Question Entailment (RQE) can have a significant impact on QA results. In related work, the meaning associated with Natural Language Inference (NLI) varies among different tasks and events. For instance, Recognizing Textual Entailment (RTE) was addressed by the PASCAL challenge BIBREF12 , where the entailment relation has been assessed manually by human judges who selected relevant sentences \"entailing\" a set of hypotheses from a list of documents returned by different Information Retrieval (IR) methods. In another definition, the Stanford Natural Language Inference corpus SNLI BIBREF13 , used three classification labels for the relations between two sentences: entailment, neutral and contradiction. For the entailment label, the annotators who built the corpus were presented with an image and asked to write a caption “that is a definitely true description of the photo”. For the neutral label, they were asked to provide a caption “that might be a true description of the label”. They were asked for a caption that “is definitely a false description of the photo” for the contradiction label.","More recently, the multiNLI corpus BIBREF14 was shared in the scope of the RepEval 2017 shared task BIBREF15 . To build the corpus, annotators were presented with a premise text and asked to write three sentences. One novel sentence, which is “necessarily true or appropriate in the same situations as the premise,” for the entailment label, a sentence, which is “necessarily false or inappropriate whenever the premise is true,” for the contradiction label, and a last sentence, “where neither condition applies,” for the neutral label.","Whereas these NLI definitions might be suitable for the broad topic of text understanding, their relation to practical information retrieval or question answering systems is not straightforward.","In contrast, RQE has to be tailored to the question answering task. For instance, if the premise question is \"looking for cold medications for a 30 yo woman\", a RQE approach should be able to consider the more general (less restricted) question \"looking for cold medications\" as relevant, since its answers are relevant for the initial question, whereas \"looking for medications for a 30 yo woman\" is a useless contextualization. The entailment relation we are seeking in the QA context should include relevant and meaningful relaxations of contextual and semantic constraints (cf. Section SECREF13 )."]},{"idx":4,"heading":"Related Work on Question Answering","paragraphs":["Classical QA systems face two main challenges related to question analysis and answer extraction. Several QA approaches were proposed in the literature for the open domain BIBREF16 , BIBREF17 and the medical domain BIBREF18 , BIBREF19 , BIBREF20 . A variety of methods were developed for question analysis, focus (topic) recognition and question type identification BIBREF21 , BIBREF22 , BIBREF23 , BIBREF24 . Similarly, many different approaches tackled document or passage retrieval and answer selection and (re)ranking BIBREF25 , BIBREF26 , BIBREF27 .","An alternative approach consists in finding similar questions or FAQs that are already answered BIBREF28 , BIBREF29 . One of the earliest question answering systems based on finding similar questions and re-using the existing answers was FAQ FINDER BIBREF30 . Another system that complements the existing Q&A services of NetWellness is SimQ BIBREF2 , which allows retrieval of similar web-based consumer health questions. SimQ uses syntactic and semantic features to compute similarity between questions, and UMLS BIBREF31 as a standardized semantic knowledge source. The system achieves 72.2% precision, 78.0% recall and 75.0% F-score on NetWellness questions. However, the method was evaluated only on one question similarity dataset, and the retrieved answers were not evaluated.","The aim of the medical task at TREC 2017 LiveQA was to develop techniques for answering complex questions such as consumer health questions, as well as to identify relevant answer sources that can comply with the sensitivity of medical information retrieval.","The CMU-OAQA system BIBREF32 achieved the best performance of 0.637 average score on the medical task by using an attentional encoder-decoder model for paraphrase identification and answer ranking. The Quora question-similarity dataset was used for training. The PRNA system BIBREF33 achieved the second best performance in the medical task with 0.49 average score using Wikipedia as the first answer source and Yahoo and Google searches as secondary answer sources. Each medical question was decomposed into several subquestions. To extract the answer from the selected text passage, a bi-directional attention model trained on the SQUAD dataset was used.","Deep neural network models have been pushing the limits of performance achieved in QA related tasks using large training datasets. The results obtained by CMU-OAQA and PRNA showed that large open-domain datasets were beneficial for the medical domain. However, the best system (CMU-OAQA) relying on the same training data obtained a score of 1.139 on the LiveQA open-domain task.","While this gap in performance can be explained in part by the discrepancies between the medical test questions and the open-domain questions, it also highlights the need for larger medical datasets to support deep learning approaches in dealing with the linguistic complexity of consumer health questions and the challenge of finding correct and complete answers.","Another technique was used by ECNU-ICA team BIBREF34 based on learning question similarity via two long short-term memory (LSTM) networks applied to obtain the semantic representations of the questions. To construct a collection of similar question pairs, they searched community question answering sites such as Yahoo! and Answers.com. In contrast, the ECNU-ICA system achieved the best performance of 1.895 in the open-domain task but an average score of only 0.402 in the medical task. As the ECNU-ICA approach also relied on a neural network for question matching, this result shows that training attention-based decoder-encoder networks on the Quora dataset generalized better to the medical domain than training LSTMs on similar questions from Yahoo! and Answers.com.","The CMU-LiveMedQA team BIBREF20 designed a specific system for the medical task. Using only the provided training datasets and the assumption that each question contains only one focus, the CMU-LiveMedQA system obtained an average score of 0.353. They used a convolutional neural network (CNN) model to classify a question into a restricted set of 10 question types and crawled \"relevant\" online web pages to find the answers. However, the results were lower than those achieved by the systems relying on finding similar answered questions. These results support the relevance of similar question matching for the end-to-end QA task as a new way of approaching QA instead of the classical QA approaches based on Question Analysis and Answer Retrieval."]},{"idx":5,"heading":"Related Work on Question Similarity and Entailment","paragraphs":["Several efforts focused on recognizing similar questions. Jeon et al. BIBREF35 showed that a retrieval model based on translation probabilities learned from a question and answer archive can recognize semantically similar questions. Duan et al. BIBREF36 proposed a dedicated language modeling approach for question search, using question topic (user's interest) and question focus (certain aspect of the topic).","Lately, these efforts were supported by a task on Question-Question similarity introduced in the community QA challenge at SemEval (task 3B) BIBREF3 . Given a new question, the task focused on reranking all similar questions retrieved by a search engine, assuming that the answers to the similar questions will be correct answers for the new question. Different machine learning and deep learning approaches were tested in the scope of SemEval 2016 BIBREF3 and 2017 BIBREF4 task 3B. The best performing system in 2017 achieved a MAP of 47.22% using supervised Logistic Regression that combined different unsupervised similarity measures such as Cosine and Soft-Cosine BIBREF37 . The second best system achieved 46.93% MAP with a learning-to-rank method using Logistic Regression and a rich set of features including lexical and semantic features as well as embeddings generated by different neural networks (siamese, Bi-LSTM, GRU and CNNs) BIBREF38 . In the scope of this challenge, a dataset was collected from Qatar Living forum for training. We refer to this dataset as SemEval-cQA.","In another effort, an answer-based definition of RQE was proposed and tested BIBREF1 . The authors introduced a dataset of clinical questions and used a feature-based method that provided an Accuracy of 75% on consumer health questions. We will call this dataset Clinical-QE. Dos Santos et al. BIBREF5 proposed a new approach to retrieve semantically equivalent questions combining a bag-of-words representation with a distributed vector representation created by a CNN and user data collected from two Stack Exchange communities. Lei et al. BIBREF7 proposed a recurrent and convolutional model (gated convolution) to map questions to their semantic representations. The models were pre-trained within an encoder-decoder framework."]},{"idx":6,"heading":"RQE Approaches and Experiments","paragraphs":["The choice of two methods for our empirical study is motivated by the best performance achieved by Logistic Regression in question-question similarity at SemEval 2017 (best system BIBREF37 and second best system BIBREF38 ), and the high performance achieved by neural networks on larger datasets such as SNLI BIBREF13 , BIBREF39 , BIBREF40 , BIBREF41 . We first define the RQE task, then present the two approaches, and evaluate their performance on five different datasets."]},{"idx":7,"heading":"Definition","paragraphs":["In the context of QA, the goal of RQE is to retrieve answers to a new question by retrieving entailed questions with associated answers. We therefore define question entailment as:","a question INLINEFORM0 entails a question INLINEFORM1 if every answer to INLINEFORM2 is also a complete or partial answer to INLINEFORM3 .","We present below two examples of consumer health questions INLINEFORM0 and entailed questions INLINEFORM1 :","Example 1 (each answer to the entailed question B1 is a complete answer to A1):","A1: What is the latest news on tennitis, or ringing in the ear, I am 75 years old and have had ringing in the ear since my mid 5os. Thank you.","B1: What is the latest research on Tinnitus?","Example 2 (each answer to the entailed question B2 is a partial answer to A2):","A2: My mother has been diagnosed with Alzheimer's, my father is not of the greatest health either and is the main caregiver for my mother. My question is where do we start with attempting to help our parents w/ the care giving and what sort of financial options are there out there for people on fixed incomes.","B2: What resources are available for Alzheimer's caregivers?","The inclusion of partial answers in the definition of question entailment also allows efficient relaxation of the contextual constraints of the original question INLINEFORM0 to retrieve relevant answers from entailed, but less restricted, questions."]},{"idx":8,"heading":"Deep Learning Model","paragraphs":["To recognize entailment between two questions INLINEFORM0 (premise) and INLINEFORM1 (hypothesis), we adapted the neural network proposed by Bowman et al. BIBREF13 . Our DL model, presented in Figure FIGREF20 , consists of three 600d ReLU layers, with a bottom layer taking the concatenated sentence representations as input and a top layer feeding a softmax classifier. The sentence embedding model sums the Recurrent neural network (RNN) embeddings of its words. The word embeddings are first initialized with pretrained GloVe vectors. This adaptation provided the best performance in previous experiments with RQE data.","GloVe is an unsupervised learning algorithm to generate vector representations for words BIBREF42 . Training is performed on aggregated word co-occurrence statistics from a large corpus, and the resulting representations show interesting linear substructures of the word vector space. We use the pretrained common crawl version with 840B tokens and 300d vectors, which are not updated during training."]},{"idx":9,"heading":"Logistic Regression Classifier","paragraphs":["In this feature-based approach, we use Logistic Regression to classify question pairs into entailment or no-entailment. Logistic Regression achieved good results on this specific task and outperformed other statistical learning algorithms such as SVM and Naive Bayes. In a preprocessing step, we remove stop words and perform word stemming using the Porter algorithm BIBREF43 for all ( INLINEFORM0 , INLINEFORM1 ) pairs.","We use a list of nine features, selected after several experiments on RTE datasets BIBREF12 . We compute five similarity measures between the pre-processed questions and use their values as features. We use Word Overlap, the Dice coefficient based on the number of common bigrams, Cosine, Levenshtein, and the Jaccard similarities. Our feature list also includes the maximum and average values obtained with these measures and the question length ratio (length( INLINEFORM0 )/length( INLINEFORM1 )). We compute a morphosyntactic feature indicating the number of common nouns and verbs between INLINEFORM2 and INLINEFORM3 . TreeTagger BIBREF44 was used for POS tagging.","For RQE, we add an additional feature specific to the question type. We use a dictionary lookup to map triggers to the question type (e.g. Treatment, Prognosis, Inheritance). Triggers are identified for each question type based on a manual annotation of a set of medical questions (cf. Section SECREF36 ). This feature has three possible values: 2 (Perfect match between INLINEFORM0 type(s) and INLINEFORM1 type(s)), 1 (Overlap between INLINEFORM2 type(s) and INLINEFORM3 type(s)) and 0 (No common types)."]},{"idx":10,"heading":"Datasets Used for the RQE Study","paragraphs":["We evaluate the RQE methods (i.e. deep learning model and logistic regression classifier) using two datasets of sentence pairs (SNLI and multiNLI), and three datasets of question pairs (Quora, Clinical-QE, and SemEval-cQA).","The Stanford Natural Language Inference corpus (SNLI) BIBREF13 contains 569,037 sentence pairs written by humans based on image captioning. The training set of the MultiNLI corpus BIBREF14 consists of 393,000 pairs of sentences from five genres of written and spoken English (e.g. Travel, Government). Two other \"matched\" and \"mismatched\" sets are also available for development (20,000 pairs). Both SNLI and multiNLI consider three types of relationships between sentences: entailment, neutral and contradiction. We converted the contradiction and neutral labels to the same non-entailment class.","The QUORA dataset of similar questions was recently published with 404,279 question pairs. We randomly selected three distinct subsets (80%/10%/10%) for training (323,423 pairs), development (40,428 pairs) and test (40,428 pairs).","The clinical-QE dataset BIBREF1 contains 8,588 question pairs and was constructed using 4,655 clinical questions asked by family doctors BIBREF45 . We randomly selected three distinct subsets (80%/10%/10%) for training (6,870 pairs), development (859 pairs) and test (859 pairs).","The question similarity dataset of SemEval 2016 Task 3B (SemEval-cQA) BIBREF3 contains 3,869 question pairs and aims to re-rank a list of related questions according to their similarity to the original question. The same dataset was used for SemEval 2017 Task 3 BIBREF4 .","To construct our test dataset, we used a publicly shared set of Consumer Health Questions (CHQs) received by the U.S. National Library of Medicine (NLM), and annotated with named entities, question types, and focus BIBREF46 , BIBREF47 . The CHQ dataset consists of 1,721 consumer information requests manually annotated with subquestions, each identified by a question type and a focus.","First, we selected automatically harvested FAQs, from U.S. National Institutes of Health (NIH) websites, that share both the same focus and the same question type with the CHQs. As FAQs are most often very short, we first assume that the CHQ entails the FAQ. Two sets of pairs were constructed: (i) positive pairs of CHQs and FAQs sharing at least one common question type and the question focus, and (ii) negative pairs corresponding to a focus mismatch or type mismatch. For each category of negative examples, we randomly selected the same number of pairs for a balanced dataset. Then, we manually validated the constructed pairs and corrected the positive and negative labels when needed. The final RQE dataset contains 850 CHQ-FAQ pairs with 405 positive and 445 negative pairs. Table TABREF26 presents examples from the five training datasets (SNLI, MultiNLI, SemEval-cQA, Clinical-QE and Quora) and the new test dataset of medical CHQ-FAQ pairs."]},{"idx":11,"heading":"Results of RQE Approaches","paragraphs":["In the first experiment, we evaluated the DL and ML methods on SNLI, multi-NLI, Quora, and Clinical-QE. For the datasets that did not have a development and test sets, we randomly selected two sets, each amounting to 10% of the data, for test and development, and used the remaining 80% for training. For MultiNLI, we used the dev1-matched set for validation and the dev2-mismatched set for testing.","Table TABREF28 presents the results of the first experiment. The DL model with GloVe word embeddings achieved better results on three datasets, with 82.80% Accuracy on SNLI, 78.52% Accuracy on MultiNLI, and 83.62% Accuracy on Quora. Logistic Regression achieved the best Accuracy of 98.60% on Clinical-RQE. We also performed a 10-fold cross-validation on the full Clinical-QE data of 8,588 question pairs, which gave 98.61% Accuracy.","In the second experiment, we used these datasets for training only and compared their performance on our test set of 850 consumer health questions. Table TABREF29 presents the results of this experiment. Logistic Regression trained on the clinical-RQE data outperformed DL models trained on all datasets, with 73.18% Accuracy.","To validate further the performance of the LR method, we evaluated it on question similarity detection. A typical approach to this task is to use an IR method to find similar question candidates, then a more sophisticated method to select and re-rank the similar questions. We followed a similar approach for this evaluation by combining the LR method with the IR baseline provided in the context of SemEval-cQA. The hybrid method combines the score provided by the Logistic Regression model and the reciprocal rank from the IR baseline using a weight-based combination:","INLINEFORM0","The weight INLINEFORM0 was set empirically through several tests on the cQA-2016 development set ( INLINEFORM1 ). Table TABREF30 presents the results on the cQA-2016 and cQA-2017 test datasets. The hybrid method (LR+IR) provided the best results on both datasets. On the 2016 test data, the LR+IR method outperformed the best system in all measures, with 80.57% Accuracy and 77.47% MAP (official system ranking measure in SemEval-cQA). On the cQA-2017 test data, the LR+IR method obtained 44.66% MAP and outperformed the cQA-2017 best system in Accuracy with 67.27%."]},{"idx":12,"heading":"Discussion of RQE Results","paragraphs":["When trained and tested on the same corpus, the DL model with GloVe embeddings gave the best results on three datasets (SNLI, MultiNLI and Quora). Logistic Regression gave the best Accuracy on the Clinical-RQE dataset with 98.60%. When tested on our test set (850 medical CHQs-FAQs pairs), Logistic Regression trained on Clinical-QE gave the best performance with 73.18% Accuracy.","The SNLI and multi-NLI models did not perform well when tested on medical RQE data. We performed additional evaluations using the RTE-1, RTE-2 and RTE-3 open-domain datasets provided by the PASCAL challenge and the results were similar. We have also tested the SemEval-cQA-2016 model and had a similar drop in performance on RQE data. This could be explained by the different types of data leading to wrong internal conceptualizations of medical terms and questions in the deep neural layers. This performance drop could also be caused by the complexity of the test consumer health questions that are often composed of several subquestions, contain contextual information, and may contain misspellings and ungrammatical sentences, which makes them more difficult to process BIBREF48 . Another aspect is the semantics of the task as discussed in Section SECREF6 . The definition of textual entailment in open-domain may not quite apply to question entailment due to the strict semantics. Also the general textual entailment definitions refer only to the premise and hypothesis, while the definition of RQE for question answering relies on the relationship between the sets of answers of the compared questions."]},{"idx":13,"heading":"Building a Medical QA Collection from Trusted Resources","paragraphs":["A RQE-based QA system requires a collection of question-answer pairs to map new user questions to the existing questions with an RQE approach, rank the retrieved questions, and present their answers to the user."]},{"idx":14,"heading":"Method","paragraphs":["To construct trusted medical question-answer pairs, we crawled websites from the National Institutes of Health (cf. Section SECREF56 ). Each web page describes a specific topic (e.g. name of a disease or a drug), and often includes synonyms of the main topic that we extracted during the crawl.","We constructed hand-crafted patterns for each website to automatically generate the question-answer pairs based on the document structure and the section titles. We also annotated each question with the associated focus (topic of the web page) as well as the question type identified with the designed patterns (cf. Section SECREF36 ).","To provide additional information about the questions that could be used for diverse IR and NLP tasks, we automatically annotated the questions with the focus, its UMLS Concept Unique Identifier (CUI) and Semantic Type. We combined two methods to recognize named entities from the titles of the crawled articles and their associated UMLS CUIs: (i) exact string matching to the UMLS Metathesaurus, and (ii) MetaMap Lite BIBREF49 . We then used the UMLS Semantic Network to retrieve the associated semantic types and groups."]},{"idx":15,"heading":"Question Types","paragraphs":["The question types were derived after the manual evaluation of 1,721 consumer health questions. Our taxonomy includes 16 types about Diseases, 20 types about Drugs and one type (Information) for the other named entities such as Procedures, Medical exams and Treatments. We describe below the considered question types and examples of associated question patterns.","Question Types about Diseases (16): Information, Research (or Clinical Trial), Causes, Treatment, Prevention, Diagnosis (Exams and Tests), Prognosis, Complications, Symptoms, Inheritance, Susceptibility, Genetic changes, Frequency, Considerations, Contact a medical professional, Support Groups.","Examples:","What research (or clinical trial) is being done for DISEASE?","What is the outlook for DISEASE?","How many people are affected by DISEASE?","When to contact a medical professional about DISEASE?","Who is at risk for DISEASE?","Where to find support for people with DISEASE?","Question Types About Drugs (20): Information, Interaction with medications, Interaction with food, Interaction with herbs and supplements, Important warning, Special instructions, Brand names, How does it work, How effective is it, Indication, Contraindication, Learn more, Side effects, Emergency or overdose, Severe reaction, Forget a dose, Dietary, Why get vaccinated, Storage and disposal, Usage, Dose.","Examples:","Are there interactions between DRUG and herbs and supplements?","What important warning or information should I know about DRUG?","Are there safety concerns or special precautions about DRUG?","What is the action of DRUG and how does it work?","Who should get DRUG and why is it prescribed?","What to do in case of a severe reaction to DRUG?","Question Type for other medical entities (e.g. Procedure, Exam, Treatment): Information.","What is Coronary Artery Bypass Surgery?","What are Liver Function Tests?"]},{"idx":16,"heading":"Medical Resources","paragraphs":["We used 12 trusted websites to construct a collection of question-answer pairs. For each website, we extracted the free text of each article as well as the synonyms of the article focus (topic). These resources and their brief descriptions are provided below:","National Cancer Institute (NCI) : We extracted free text from 116 articles on various cancer types (729 QA pairs). We manually restructured the content of the articles to generate complete answers (e.g. a full answer about the treatment of all stages of a specific type of cancer). Figure FIGREF54 presents examples of QA pairs generated from a NCI article.","Genetic and Rare Diseases Information Center (GARD): This resource contains information about various aspects of genetic/rare diseases. We extracted all disease question/answer pairs from 4,278 topics (5,394 QA pairs).","Genetics Home Reference (GHR): This NLM resource contains consumer-oriented information about the effects of genetic variation on human health. We extracted 1,099 articles about diseases from this resource (5,430 QA pairs).","MedlinePlus Health Topics: This portion of MedlinePlus contains information on symptoms, causes, treatment and prevention for diseases, health conditions and wellness issues. We extracted the free texts in summary sections of 981 articles (981 QA pairs).","National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) : We extracted text from 174 health information pages on diseases studied by this institute (1,192 QA pairs).","National Institute of Neurological Disorders and Stroke (NINDS): We extracted free text from 277 information pages on neurological and stroke-related diseases from this resource (1,104 QA pairs).","NIHSeniorHealth : This website contains health and wellness information for older adults. We extracted 71 articles from this resource (769 QA pairs).","National Heart, Lung, and Blood Institute (NHLBI) : We extracted text from 135 articles on diseases, tests, procedures, and other relevant topics on disorders of heart, lung, blood, and sleep (559 QA pairs).","Centers for Disease Control and Prevention (CDC) : We extracted text from 152 articles on diseases and conditions (270 QA pairs).","MedlinePlus A.D.A.M. Medical Encyclopedia: This resource contains 4,366 articles about conditions, tests, and procedures. 17,348 QA pairs were extracted from this resource. Figure FIGREF55 presents examples of QA pairs generated from A.D.A.M encyclopedia.","MedlinePlus Drugs: We extracted free text from 1,316 articles about Drugs and generated 12,889 QA pairs.","MedlinePlus Herbs and Supplements: We extracted free text from 99 articles and generated 792 QA pairs.","The final collection contains 47,457 annotated question-answer pairs about Diseases, Drugs and other named entities (e.g. Tests) extracted from these 12 trusted resources."]},{"idx":17,"heading":"The Proposed Entailment-based QA System","paragraphs":["Our goal is to generate a ranked list of answers for a given Premise Question INLINEFORM0 by ranking the recognized Hypothesis Questions INLINEFORM1 . Based on the RQE experiments above (Section SECREF27 ), we selected Logistic Regression trained on the clinical-RQE dataset to recognize entailed questions and rank them with their classification scores."]},{"idx":18,"heading":"RQE-based QA Approach","paragraphs":["Classifying the full QA collection for each test question is not feasible for real-time applications. Therefore, we first filter the questions with an IR method to retrieve candidate questions, then classify them as entailed (or not) by the user/test question. Based on the positive results of the combination method tested on SemEval-cQA data (Section SECREF27 ), we adopted a combination method to merge the results obtained by the search engine and the RQE scores. The answers are then combined from both methods and ranked using an aggregate score. Figure FIGREF82 presents the overall architecture of the proposed QA system. We describe each module in more details next."]},{"idx":19,"heading":"Finding Similar Question Candidates","paragraphs":["For each premise question INLINEFORM0 , we use the Terrier search engine to retrieve INLINEFORM1 relevant question candidates INLINEFORM2 and then apply the RQE classifier to predict the labels for the pairs ( INLINEFORM3 , INLINEFORM4 ).","We indexed the questions of our QA collection without the associated answers. In order to improve the indexing and the performance of question retrieval, we also indexed the synonyms of the question focus and the triggers of the question type with each question. This choice allowed us to avoid the shortcomings of query expansion, including incorrect or irrelevant synonyms and the increased execution time. The synonyms of the question focus (topic) were extracted automatically from the QA collection. The triggers of each question type were defined manually in the question types taxonomy. Below are two examples of indexed questions from our QA collection, with the automatically added focus synonyms and question type triggers:","What are the treatments for Torticollis?","Focus: Torticollis. Question type: Treatment.","Added focus synonyms: \"Spasmodic torticollis, Wry neck, Loxia, Cervical dystonia\". Added question type triggers: \"relieve, manage, cure, remedy, therapy\".","What is the outlook for Legionnaire disease?","Focus: Legionnaire disease. Question Type: Prognosis.","Added focus synonyms: \"Legionella pneumonia, Pontiac fever, Legionellosis\". Added question type triggers: \"prognosis, life expectancy\".","The IR task consists of retrieving hypothesis questions INLINEFORM0 relevant to the submitted question INLINEFORM1 . As fusion of IR result has shown good performance in different tracks in TREC, we merge the results of the TF-IDF weighting function and the In-expB2 DFR model BIBREF50 .","Let INLINEFORM0 = INLINEFORM1 , INLINEFORM2 , ..., INLINEFORM3 be the set of INLINEFORM4 questions retrieved by the first IR model INLINEFORM5 and INLINEFORM6 = INLINEFORM7 , INLINEFORM8 , ..., INLINEFORM9 be the set of INLINEFORM10 questions retrieved by the second IR model INLINEFORM11 . We merge both sets by summing the scores of each retrieved question INLINEFORM12 in both INLINEFORM13 and INLINEFORM14 lists, then we re-rank the hypothesis questions INLINEFORM15 ."]},{"idx":20,"heading":"Combining IR and RQE Methods","paragraphs":["The IR models and the RQE Logistic Regression model bring different perspectives to the search for relevant candidate questions. In particular, question entailment allows understanding the relations between the important terms, whereas the traditional IR methods identify the important terms, but will not notice if the relations are opposite. Moreover, some of the question types that the RQE classifier learns will not be deemed important terms by traditional IR and the most relevant questions will not be ranked at the top of the list.","Therefore, in our approach, when a question is submitted to the system, candidate questions are fetched using the IR models, then the RQE classifier is applied to filter out the non-entailed questions and re-rank the remaining candidates.","Specifically, we denote INLINEFORM0 the list of question candidates INLINEFORM1 returned by the IR system. The premise question INLINEFORM2 is then used to construct N question pairs INLINEFORM3 . The RQE classifier is then applied to filter out the question pairs that are not entailed and re-rank the remaining pairs.","More precisely, let INLINEFORM0 = INLINEFORM1 in INLINEFORM2 be the list of selected candidate questions that have a positive entailment relation with a given premise question INLINEFORM3 . We rank INLINEFORM4 by computing a hybrid score INLINEFORM5 for each candidate question INLINEFORM6 taking into account the score of the IR system INLINEFORM7 and the score of the RQE system INLINEFORM8 .","For each system INLINEFORM0 INLINEFORM1 , we normalize the associated score by dividing it by the maximum score among the INLINEFORM2 candidate questions retrieved by INLINEFORM3 for INLINEFORM4 :","INLINEFORM0","INLINEFORM0 INLINEFORM1","In our experiments, we fixed the value of INLINEFORM0 to 100. This threshold value was selected as a safe value for this task for the following reasons:","Our collection of 47,457 question-answer pairs was collected from only 12 NIH institutes and is unlikely to contain more than 100 occurrences of the same focus-type pair.","Each question was indexed with additional annotations for the question focus, its synonyms and the question type synonyms."]},{"idx":21,"heading":"Evaluating RQE for Medical Question Answering","paragraphs":["The objective of this evaluation is to study the effectiveness of RQE for Medical Question Answering, by comparing the answers retrieved by the hybrid entailment-based approach, the IR method and the other QA systems participating to the medical task at TREC 2017 LiveQA challenge (LiveQA-Med)."]},{"idx":22,"heading":"Evaluation Method","paragraphs":["We developed an interface to perform the manual evaluation of the retrieved answers. Figure 5 presents the evaluation interface showing, for each test question, the top-10 answers of the evaluated QA method and the reference answer(s) used by LiveQA assessors to help judging the retrieved answers by the participating systems.","We used the test questions of the medical task at TREC-2017 LiveQA BIBREF11 . These questions are randomly selected from the consumer health questions that the NLM receives daily from all over the world. The test questions cover different medical entities and have a wide list of question types such as Comparison, Diagnosis, Ingredient, Side effects and Tapering.","For a relevant comparison, we used the same judgment scores as the LiveQA Track:","Correct and Complete Answer (4)","Correct but Incomplete (3)","Incorrect but Related (2)","Incorrect (1)","We evaluated the answers returned by the IR-based method and the hybrid QA method (IR+RQE) according to the same reference answers used in LiveQA-Med. The answers were anonymized (the method names were blinded) and presented to 3 assessors: a medical doctor (Assessor A), a medical librarian (B) and a researcher in medical informatics (C). None of the assessors participated in the development of the QA methods. Assessors B and C evaluated 1,000 answers retrieved by each of the methods (IR and IR+RQE). Assessor A evaluated 2,000 answers from both methods.","Table TABREF103 presents the inter-annotator agreement (IAA) through F1 score computed by considering one of the assessors as reference. In the first evaluation, we computed the True Positives (TP) and False Positives (FP) over all ratings and the Precision and F1 score. As there are no negative labels (only true or false positives for each category), Recall is 100%. We also computed a partial IAA by grouping the \"Correct and Complete Answer\" and \"Correct but Incomplete\" ratings (as Correct), and the \"Incorrect but Related\" and \"Incorrect\" ratings (as Incorrect). The average agreement on distinguishing the Correct and Incorrect answers is 94.33% F1 score. Therefore, we used the evaluations performed by assessor A for both methods. The official results of the TREC LiveQA track relied on one assessor per question as well."]},{"idx":23,"heading":"Evaluation of the first retrieved answer","paragraphs":["We computed the measures used by TREC LiveQA challenges BIBREF51 , BIBREF11 to evaluate the first retrieved answer for each test question:","avgScore(0-3): the average score over all questions, transferring 1-4 level grades to 0-3 scores. This is the main score used to rank LiveQA runs.","succ@i+: the number of questions with score i or above (i INLINEFORM0 {2..4}) divided by the total number of questions.","prec@i+: the number of questions with score i or above (i INLINEFORM0 {2..4}) divided by number of questions answered by the system.","Table TABREF108 presents the average scores, success and precision results. The hybrid IR+RQE QA system achieved better results than the IR-based system with 0.827 average score. It also achieved a higher score than the best results achieved in the medical challenge at LiveQA'17. Evaluating the RQE system alone is not relevant, as applying RQE on the full collection for each user question is not feasible for a real-time system because of the extended execution time."]},{"idx":24,"heading":"Evaluation of the top ten answers","paragraphs":["In this evaluation, we used Mean Average Precision (MAP) and Mean Reciprocal Rank (MRR) which are commonly used in QA to evaluate the top-10 answers for each question. We consider answers rated as “Correct and Complete Answer” or “Correct but Incomplete” as correct answers, as the test questions contain multiple subquestions while each answer in our QA collection can cover only one subquestion.","MAP is the mean of the Average Precision (AvgP) scores over all questions.","(1) INLINEFORM0","Q is the number of questions. INLINEFORM0 is the AvgP of the INLINEFORM1 question.","INLINEFORM0","K is the number of correct answers. INLINEFORM0 is the rank of INLINEFORM1 correct answer.","MRR is the average of the reciprocal ranks for each question. The reciprocal rank of a question is the multiplicative inverse of the rank of the first correct answer.","(2) INLINEFORM0","Q is the number of questions. INLINEFORM0 is the rank of the first correct answer for the INLINEFORM1 question.","Table TABREF113 presents the MAP@10 and MRR@10 of our QA methods. The IR+RQE system outperforms the IR-based QA system with 0.311 MAP@10 and 0.333 MRR@10."]},{"idx":25,"heading":"Discussion of entailment-based QA for the medical domain","paragraphs":["In our evaluation, we followed the same LiveQA guidelines with the highest possible rigor. In particular, we consulted with NIST assessors who provided us with the paraphrases of the test questions that they used to judge the answers. Our IAA on the answers rating was also high compared to related tasks, with an 88.5% F1 agreement with the exact four categories and a 94.3% agreement when reducing the categories to two: “Correct” and “Incorrect” answers. Our results show that RQE improves the overall performance and exceeds the best results in the medical LiveQA'17 challenge by a factor of 29.8%. This performance improvement is particularly interesting as:","Our answer source has only 47K question-answer pairs when LiveQA participating systems relied on much larger collections, including the World Wide Web.","Our system answered one subquestion at most when many LiveQA test questions had several subquestions.","The latter observation, (b), makes the hybrid IR+RQE approach even more promising as it gives it a large potential for the improvement of answer completeness.","The former observation, (a), provides another interesting insight: restricting the answer source to only reliable collections can actually improve the QA performance without losing coverage (i.e., our QA approach provided at least one answer to each test question and obtained the best relevance score).","In another observation, the assessors reported that many of the returned answers had a correct question type but a wrong focus, which indicates that including a focus recognition module to filter such wrong answers can improve further the QA performance in terms of precision. Another aspect that was reported is the repetition of the same (or similar) answer from different websites, which could be addressed by improving answer selection with inter-answer comparisons and removal of near-duplicates. Also, half of the LiveQA test questions are about Drugs, when only two of our resources are specialized in Drugs, among 12 sub-collections overall. Accordingly, the assessors noticed that the performance of the QA systems was better on questions about diseases than on questions about drugs, which suggests a need for extending our medical QA collection with more information about drugs and associated question types.","We also looked closely at the private websites used by the LiveQA-Med annotators to provide some of the reference answers for the test questions. For instance, the ConsumerLab website was useful to answer a question about the ingredients of a Drug (COENZYME Q10). Similarly, the eHealthMe website was used to answer a test question asking about interactions between two drugs (Phentermine and Dicyclomine) when no information was found in DailyMed. eHealthMe provides healthcare big data analysis and private research and studies including self-reported adverse drug effects by patients.","But the question remains on the extent to which such big data and other private websites could be used to automatically answer medical questions if information is otherwise unavailable. Unlike medical professionals, patients do not necessarily have the knowledge and tools to validate such information. An alternative approach could be to put limitations on medical QA systems in terms of the questions that can be answered (e.g. \"What is my diagnosis for such symptoms\") and build classifiers to detect such questions and warn the users about the dangers of looking for their answers online.","More generally, medical QA systems should follow some strict guidelines regarding the goal and background knowledge and resources of each system in order to protect the consumers from misleading or harmful information. Such guidelines could be based (i) on the source of the information such as health and medical information websites sponsored by the U.S. government, not-for-profit health or medical organizations, and medical university centers, or (ii) on conventions such as the code of conduct of the HON Foundation (HONcode) that addresses the reliability and usefulness of medical information on the Internet.","Our experiments show that limiting the number of answer sources with such guidelines is not only feasible, but it could also enhance the performance of the QA system from an information retrieval perspective."]},{"idx":26,"heading":"Conclusion","paragraphs":["In this paper, we carried out an empirical study of machine learning and deep learning methods for Recognizing Question Entailment in the medical domain using several datasets. We developed a RQE-based QA system to answer new medical questions using existing question-answer pairs. We built and shared a collection of 47K medical question-answer pairs. Our QA approach outperformed the best results on TREC-2017 LiveQA medical test questions. The proposed approach can be applied and adapted to open-domain as well as specific-domain QA. Deep learning models achieved interesting results on open-domain and clinical datasets, but obtained a lower performance on consumer health questions. We will continue investigating other network architectures including transfer learning, as well as creation of a large collection of consumer health questions for training to improve the performance of DL models. Future work also includes exploring integration of a Question Focus Recognition module to enhance candidate question retrieval, and expanding our question-answer collection."]},{"idx":27,"heading":"Acknowledgements","paragraphs":["We thank Halil Kilicoglu (NLM/NIH) for his help with the crawling and the manual evaluation and Sonya E. Shooshan (NLM/NIH) for her help with the judgment of the retrieved answers. We also thank Ellen Voorhees (NIST) for her valuable support with the TREC LiveQA evaluation.","We consider the case of the question number 36 in the TREC-2017 LiveQA medical test dataset:","36. congenital diaphragmatic hernia. what are the causes of congenital diaphragmatic hernia? Can cousin marriage cause this? What kind of lung disease the baby might experience life long?","This question was answered by 5 participating runs (vs. 8 runs for other questions), and all submitted answers were wrong (scores of 1 or 2). However, our IR-based QA system retrieved one excellent answer (score 4) and our hybrid IR+RQE system provided 3 excellent answers.","A) TREC 2017 LiveQA-Med Participants' Results:","B) Our IR-based QA System:","C) Our IR+RQE QA System:"]},{"idx":28,"heading":"Figures and Tables","paragraphs":["Figure 1: Neural Network Architecture","Table 1: Description of training and test datasets.","Table 2: Accuracy (%) of Deep Learning and Machine Learning methods on four datasets SNLI, MultiNLI, Quora, and Clinical-QE.","Table 3: Performance on 850 consumer health questions: Accuracy (%) of Machine Learning and Deep Learning Methods.","Table 4: Results (%) of the hybrid method (Logistic Regression + IR) on community QA datasets (cQA-Test 2016 and cQA-Test 2017).","Figure 2: Examples of QA pairs generated from an article about Langerhans Cell Histiocytosis (NCI).","Figure 3: Examples of QA pairs generated from an article about Acromegaly (A.D.A.M encyclopedia).","Figure 4: Overview of the RQE-based Question Answering System.","Table 5: Inter-Annotator Agreement (IAA) over all ratings in the manual evaluation of the retrieved answers. Partial IAA over two ratings \"Correct\" and \"Incorrect\".","Table 6: LiveQA Measures: Average Score (main score), Success@i+ and Precision@i+ on LiveQA’17 Test Data. Evaluation of the first retrieved answer for each question. N.B. Evaluating the RQE System alone is not relevant as explained previously (Section 5.1).","Table 7: Common Measures: MAP and MRR on LiveQA’17 Test Questions. Evaluation of top 10 answers for each question."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1901.09381","title":"Dual Co-Matching Network for Multi-choice Reading Comprehension","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Multi-choice reading comprehension is a challenging task that requires complex reasoning procedure. Given passage and question, a correct answer need to be selected from a set of candidate answers. In this paper, we propose \\textbf{D}ual \\textbf{C}o-\\textbf{M}atching \\textbf{N}etwork (\\textbf{DCMN}) which model the relationship among passage, question and answer bidirectionally. Different from existing approaches which only calculate question-aware or option-aware passage representation, we calculate passage-aware question representation and passage-aware answer representation at the same time. To demonstrate the effectiveness of our model, we evaluate our model on a large-scale multiple choice machine reading comprehension dataset (i.e. RACE). Experimental result show that our proposed model achieves new state-of-the-art results."]},{"idx":1,"heading":"Introduction","paragraphs":["Machine reading comprehension and question answering has becomes a crucial application problem in evaluating the progress of AI system in the realm of natural language processing and understanding BIBREF0 . The computational linguistics communities have devoted significant attention to the general problem of machine reading comprehension and question answering.","However, most of existing reading comprehension tasks only focus on shallow QA tasks that can be tackled very effectively by existing retrieval-based techniques BIBREF1 . For example, recently we have seen increased interest in constructing extractive machine reading comprehension datasets such as SQuAD BIBREF2 and NewsQA BIBREF3 . Given a document and a question, the expected answer is a short span in the document. Question context usually contains sufficient information for identifying evidence sentences that entail question-answer pairs. For example, 90.2% questions in SQuAD reported by Min BIBREF4 are answerable from the content of a single sentence. Even in some multi-turn conversation tasks, the existing models BIBREF5 mostly focus on retrieval-based response matching.","In this paper, we focus on multiple-choice reading comprehension datasets such as RACE BIBREF6 in which each question comes with a set of answer options. The correct answer for most questions may not appear in the original passage which makes the task more challenging and allow a rich type of questions such as passage summarization and attitude analysis. This requires a more in-depth understanding of a single document and leverage external world knowledge to answer these questions. Besides, comparing to traditional reading comprehension problem, we need to fully consider passage-question-answer triplets instead of passage-question pairwise matching.","In this paper, we propose a new model, Dual Co-Matching Network, to match a question-answer pair to a given passage bidirectionally. Our network leverages the latest breakthrough in NLP: BERT BIBREF7 contextual embedding. In the origin BERT paper, the final hidden vector corresponding to first input token ([CLS]) is used as the aggregation representation and then a standard classification loss is computed with a classification layer. We think this method is too rough to handle the passage-question-answer triplet because it only roughly concatenates the passage and question as the first sequence and uses question as the second sequence, without considering the relationship between the question and the passage. So we propose a new method to model the relationship among the passage, the question and the candidate answer.","Firstly we use BERT as our encode layer to get the contextual representation of the passage, question, answer options respectively. Then a matching layer is constructed to get the passage-question-answer triplet matching representation which encodes the locational information of the question and the candidate answer matched to a specific context of the passage. Finally we apply a hierarchical aggregation method over the matching representation from word-level to sequence-level and then from sequence level to document-level. Our model improves the state-of-the-art model by 2.6 percentage on the RACE dataset with BERT base model and further improves the result by 3 percentage with BERT large model."]},{"idx":2,"heading":"Model","paragraphs":["For the task of multi-choice reading comprehension, the machine is given a passage, a question and a set of candidate answers. The goal is to select the correct answer from the candidates. P, Q, and A are used to represent the passage, the question and a candidate answer respectively. For each candidate answer, our model constructs a question-aware passage representation, a question-aware passage representation and a question-aware passage representation. After a max-pooling layer, the three representations are concatenated as the final representation of the candidate answer. The representations of all candidate answers are then used for answer selection.","In section \"Encoding layer\" , we introduce the encoding mechanism. Then in section \"Conclusions\" , we introduce the calculation procedure of the matching representation between the passage, the question and the candidate answer. In section \"Aggregation layer\" , we introduce the aggregation method and the objective function."]},{"idx":3,"heading":"Encoding layer","paragraphs":["This layer encodes each token in passage and question into a fixed-length vector including both word embedding and contextualized embedding. We utilize the latest result from BERT BIBREF7 as our encoder and the final hidden state of BERT is used as our final embedding. In the origin BERT BIBREF7 , the procedure of processing multi-choice problem is that the final hidden vector corresponding to first input token ([CLS]) is used as the aggregation representation of the passage, the question and the candidate answer, which we think is too simple and too rough. So we encode the passage, the question and the candidate answer respectively as follows:","$$\\begin{split} \\textbf {H}^p=&BERT(\\textbf {P}),\\textbf {H}^q=BERT(\\textbf {Q}) \\\\ &\\textbf {H}^a=BERT(\\textbf {A}) \\end{split}$$ (Eq. 3)","where $\\textbf {H}^p \\in R^{P \\times l}$ , $\\textbf {H}^q \\in R^{Q \\times l}$ and $\\textbf {H}^a \\in R^{A \\times l}$ are sequences of hidden state generated by BERT. $P$ , $Q$ , $A$ are the sequence length of the passage, the question and the candidate answer respectively. $l$ is the dimension of the BERT hidden state."]},{"idx":4,"heading":"Matching layer","paragraphs":["To fully mine the information in a {P, Q, A} triplet , We make use of the attention mechanism to get the bi-directional aggregation representation between the passage and the answer and do the same process between the passage and the question. The attention vectors between the passage and the answer are calculated as follows:","$$\\begin{split} \\textbf {W}&=SoftMax(\\textbf {H}^p({H^{a}G + b})^T), \\\\ \\textbf {M}^{p}&=\\textbf {W}\\textbf {H}^{a}, \\textbf {M}^{a}=\\textbf {W}^T\\textbf {H}^{p}, \\end{split}$$ (Eq. 5)","where $G \\in R^{l \\times l}$ and $b \\in R^{A \\times l}$ are the parameters to learn. $\\textbf {W} \\in R^{P \\times A}$ is the attention weight matrix between the passage and the answer. $\\textbf {M}^{p} \\in R^{P \\times l}$ represent how each hidden state in passage can be aligned to the answe rand $\\textbf {M}^{a} \\in R^{A \\times l}$ represent how the candidate answer can be aligned to each hidden state in passage. In the same method, we can get $\\textbf {W}^{\\prime } \\in R^{P \\times Q}$ and $\\textbf {M}^{q} \\in R^{Q \\times l}$ for the representation between the passage and the question.","To integrate the original contextual representation, we follow the idea from BIBREF8 to fuse $\\textbf {M}^{a}$ with original $\\textbf {H}^p$ and so is $\\textbf {M}^{p}$ . The final representation of passage and the candidate answer is calculated as follows:","$$\\begin{split} \\textbf {S}^{p}&=F([\\textbf {M}^{a} - \\textbf {H}^{a}; \\textbf {M}^{a} \\cdot \\textbf {H}^{a}]W_1 + b_1),\\\\ \\textbf {S}^{a}&=F([\\textbf {M}^{p} - \\textbf {H}^{p}; \\textbf {M}^{p} \\cdot \\textbf {H}^{p}]W_2 + b_2),\\\\ \\end{split}$$ (Eq. 6)","where $W_1, W_2 \\in R^{2l \\times l}$ and $b_1 \\in R^{P \\times l}, b_2 \\in R^{(A) \\times l}$ are the parameters to learn. $[ ; ]$ is the column-wise concatenation and $-, \\cdot $ are the element-wise subtraction and multiplication between two matrices. Previous work in BIBREF9 , BIBREF10 shows this method can build better matching representation. $F$ is the activation function and we choose $ReLU$ activation function there. $\\textbf {S}^{p} \\in R^{P \\times l}$ and $\\textbf {S}^{a} \\in R^{A \\times l}$ are the final representations of the passage and candidate answer. In the question side, we can get $\\textbf {S}^{p^{\\prime }} \\in R^{P \\times l}$ and $\\textbf {S}^{q} \\in R^{Q \\times l}$ in the same calculation method."]},{"idx":5,"heading":"Aggregation layer","paragraphs":["To get the final representation for each candidate answer, a row-wise max pooling operation is used to $\\textbf {S}^{p}$ and $\\textbf {S}^{a}$ . Then we get $\\textbf {C}^{p} \\in R^l$ and $\\textbf {C}^{a} \\in R^l$ respectively. In the question side, $\\textbf {C}^{p^{\\prime }} \\in R^l$ and $\\textbf {C}^{q} \\in R^l$ are calculated. Finally, we concatenate all of them as the final output $\\textbf {C} \\in R^{4l}$ for each {P, Q, A} triplet.","$$\\begin{split} \\textbf {C}^{p} = &Pooling(\\textbf {S}^{p}), \\textbf {C}^{a} = Pooling(\\textbf {S}^{a}),\\\\ \\textbf {C}^{p^{\\prime }} = &Pooling(\\textbf {S}^{p^{\\prime }}), \\textbf {C}^{q} = Pooling(\\textbf {S}^{q}),\\\\ \\textbf {C} &= [\\textbf {C}^{p}; \\textbf {C}^{a};\\textbf {C}^{p^{\\prime }};\\textbf {C}^{q}] \\end{split}$$ (Eq. 9)","For each candidate answer choice $i$ , its matching representation with the passage and question can be represented as $\\textbf {C}_i$ . Then our loss function is computed as follows:","$$\\begin{split} L(\\textbf {A}_i|\\textbf {P,Q}) = -log{\\frac{exp(V^T\\textbf {C}_i)}{\\sum _{j=1}^N{exp(V^T\\textbf {C}_j)}}}, \\end{split}$$ (Eq. 10)","where $V \\in R^l$ is a parameter to learn."]},{"idx":6,"heading":"Experiment","paragraphs":["We evaluate our model on RACE dataset BIBREF6 , which consists of two subsets: RACE-M and RACE-H. RACE-M comes from middle school examinations while RACE-H comes from high school examinations. RACE is the combination of the two.","We compare our model with the following baselines: MRU(Multi-range Reasoning) BIBREF12 , DFN(Dynamic Fusion Networks) BIBREF11 , HCM(Hierarchical Co-Matching) BIBREF8 , OFT(OpenAI Finetuned Transformer LM) BIBREF13 , RSM(Reading Strategies Model) BIBREF14 . We also compare our model with the BERT baseline and implement the method described in the original paper BIBREF7 , which uses the final hidden vector corresponding to the first input token ([CLS]) as the aggregate representation followed by a classification layer and finally a standard classification loss is computed.","Results are shown in Table 2 . We can see that the performance of BERT $_{base}$ is very close to the previous state-of-the-art and BERT $_{large}$ even outperforms it for 3.7%. But experimental result shows that our model is more powerful and we further improve the result for 2.2% computed to BERT $_{base}$ and 2.2% computed to BERT $_{large}$ ."]},{"idx":7,"heading":"Conclusions","paragraphs":["In this paper, we propose a Dual Co-Matching Network, DCMN, to model the relationship among the passage, question and the candidate answer bidirectionally. By incorporating the latest breakthrough, BERT, in an innovative way, our model achieves the new state-of-the-art in RACE dataset, outperforming the previous state-of-the-art model by 2.2% in RACE full dataset."]},{"idx":8,"heading":"Figures and Tables","paragraphs":["Table 1: An example passage with related question and options from RACE dataset. The ground-truth answer and the evidence sentences in the passage are in bold.","Figure 1: The framework of our model. P-Passage, Q-Question, O-Option.","Table 2: Analysis of the sentences in passage required to answer questions on RACE and COIN. 50 examples from each dataset are sampled randomly. N sent indicates the number of sentences required to answer the question. The evidence sentences in the passage are in emphasis and the correct answer is with bold.","Table 3: Statistics of multi-choice machine reading comprehension datasets. #o is the average number of candidate options for each question. #p is the number of documents included in the dataset. #q indicates the total number of questions in the dataset.","Table 4: Experiment results on RACE test set. All the results are from single models. PSS : Passage Sentence Selection; AOI : Answer Option Interaction. ∗ indicates our implementation.","Table 5: Ablation study on RACE dev set. PSS : Passage Sentence Selection. AOI : Answer Option Interaction. DCMN+: DCMN + PSS + AOI","Table 6: Results on the test set of SemEval Task 11, ROCStories, MCTest and the development set of COIN Task 1. The test set of COIN is not public. DCMN+: DCMN + PSS + AOI . Previous SOTA: previous state-of-the-art model. All the results are from single models.","Table 7: Performance comparison with different combination methods on the RACE dev set4. We use BERTbase as our encoder here. [; ] indicates the concatenation operation. SP O is the unidirectional matching referred in Eq. 24. MP O is the bidirectional matching representation referred in Eq. 27. Here uses our annotations to show previous matching strategies.","Figure 2: Results of sentence selection on the development set of RACE and COIN when selecting different number of sentences (Top K). We use BERTbase as encoder and cosine score method here. RACE/COIN-w indicates the results on RACE/COIN without sentence selection module.","Table 8: Results on RACE and COIN dev set with different scoring methods (cosine and bilinear score in PSS). We use BERTbase as encoder here.","Figure 3: Performance on different question types, tested on the RACE development set. BERTlarge is used as encoder here. OI: Answer Option Interaction. SS: Passage Sentence Selection."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1903.11283","title":"Grammatical Error Correction and Style Transfer via Zero-shot Monolingual Translation","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Both grammatical error correction and text style transfer can be viewed as monolingual sequence-to-sequence transformation tasks, but the scarcity of directly annotated data for either task makes them unfeasible for most languages. We present an approach that does both tasks within the same trained model, and only uses regular language parallel data, without requiring error-corrected or style-adapted texts. We apply our model to three languages and present a thorough evaluation on both tasks, showing that the model is reliable for a number of error types and style transfer aspects."]},{"idx":1,"heading":"Introduction","paragraphs":["Sequence-to-sequence (seq2seq) transformations have recently proven to be a successful framework for several natural language processing tasks, like: machine translation (MT) BIBREF0 , BIBREF1 , speech recognition BIBREF2 , speech synthesis BIBREF3 , natural language inference BIBREF4 and others. However, the success of these models depends on the availability of large amounts of directly annotated data for the task at hand (like translation examples, text segments and their speech recordings, etc.). This is a severe limitation for tasks where data is not abundantly available as well as for low-resource languages.","Here we focus on two such tasks: grammatical error correction (GEC) and style transfer. Modern approaches to GEC learn from parallel corpora of erroneous segments and their manual corrections BIBREF5 , BIBREF6 ; text style transfer also relies on supervised approaches that require texts of the same meaning and different styles BIBREF7 , BIBREF8 or imprecise unsupervised methods BIBREF9 , BIBREF10 .","In this paper we introduce an approach to performing both GEC and style transfer with the same trained model, while not using any supervised training data for either task. It is based on zero-shot neural machine translation (NMT) BIBREF11 , and as such, the only kind of data it uses is regular parallel corpora (with texts and their translations). However, we apply the model to do monolingual transfer, asking to translate the input segment into the same language. We show, that this “monolingual translation” is what enables the model to correct the errors in the input as well as adapt the output into a desired style. Moreover, the same trained model performs both tasks on several languages.","Our main contributions are thus: (i) a single method for both style transfer and grammatical error correction, without using annotated data for either task, (ii) support for both tasks on multiple languages within the same model, (iii) a thorough quantitative and qualitative manual evaluation of the model on both tasks, and (iv) highlighting of the model's reliability aspects on both tasks. We used publicly available software and data; an online demo of our results is available, but concealed for anonymization purposes.","We describe the details of our approach in Section SECREF2 , then evaluate it in terms of performance in grammatical error correction in Section SECREF3 and in style transfer in Section SECREF4 . The paper ends with a review of related work in Section SECREF5 and conclusions in Section SECREF6 ."]},{"idx":2,"heading":"Method","paragraphs":["As mentioned in the introduction, our approach is based on the idea of zero-shot MT BIBREF11 . There the authors show that after training a single model to translate from Portuguese to English as well as from English to Spanish, it can also translate Portuguese into Spanish, without seeing any translation examples for this language pair. We use the zero-shot effect to achieve monolingual translation by training the model on bilingual examples in both directions, and then doing translation into the same language as the input: illustrated on Figure FIGREF1 .","With regular sentences monolingual translation does not seem useful, as its behaviour mainly consists of copying. However, when the input sentence has characteristics unseen or rarely seen by the model at training time (like grammatical errors or different stylistic choices) – the decoder still generates the more regular version of the sentence (thus fixing the errors or adapting the style). Furthermore, in case of multilingual multi-domain NMT BIBREF12 , it is possible to switch between different domains or styles at runtime, thus performing “monolingual domain adaptation” or style transfer.","To create a multilingual multi-domain NMT system we use the self-attention architecture BIBREF13 . Instead of specifying the output language with a token inside the input sequence, as BIBREF11 did, we follow BIBREF12 and use word features (or factors). On one hand, this provides a stronger signal for the model, and on the other – allows for additional parametrization, which in our case is the text domain/style of the corpus.","As a result, a pre-processed English-Latvian training set sentence pair “Hello!”–“Sveiki!” looks like:","Here 2lv and 2os specify Latvian and OpenSubtitles as the output language and domain; the output text has no factors to predict. At application time we simply use the same input and output languages, for example the grammatically incorrect input “we is” looks like the following, after pre-processing:","The intuition behind our approach is that a multilingual shared encoder produces semantically rich latent sentence representations BIBREF14 , which provide a solid ground for the effective style transfer on top.","Next we present the technical details, the experiment setup and the data we used for training the model used in the experiments."]},{"idx":3,"heading":"Languages and Data","paragraphs":["We use three languages in our experiments: English, Estonian and Latvian. All three have different characteristics, for example Latvian and (especially) Estonian are morphologically complex and have loose word order, while English has a strict word order and the morphology is much simpler. Most importantly, all three languages have error-corrected corpora for testing purposes, though work on their automatic grammatical error correction is extremely limited (see Section SECREF3 ).","The corpora we use for training the model are OpenSubtitles2018 BIBREF15 , Europarl BIBREF16 , JRC-Acquis and EMEA BIBREF17 . We assume that there should be sufficient stylistic difference between these corpora, especially between the more informal OpenSubtitles2018 (comprised of movie and TV subtitles) on one hand and Europarl and JRC-Acquis (proceedings and documents of the European Parliament) on the other."]},{"idx":4,"heading":"Technical Details","paragraphs":["For Europarl, JRC-Acquis and EMEA we use all data available for English-Estonian, English-Latvian and Estonian-Latvian language pairs. From OpenSubtitles2018 we take a random subset of 3M sentence pairs for English-Estonian, which is still more than English-Latvian and Estonian-Latvian (below 1M; there we use the whole corpus). This is done to balance the corpora representation and to limit the size of training data.","Details on the model hyper-parameters, data pre-processing and training can be found in Appendix SECREF7 ."]},{"idx":5,"heading":"Evaluation","paragraphs":["First, we evaluate our model in the context of MT, as the translation quality can be expected to have influence on the other tasks that the model performs. We use public benchmarks for Estonian-English and Latvian-English translations from the news translation shared tasks of WMT 2017 and 2018 BIBREF18 , BIBREF19 . The BLEU scores for each translation direction and all included styles/domains are shown in Table TABREF6 .","Some surface notes on these results: the BLEU scores for translation from and into Latvian are below English-Estonian scores, which is likely explained by smaller datasets that include Latvian. Also, translation into English has higher scores than into Estonian/Latvian, which is also expected.","An interesting side-effect we have observed is the model's resilience to code-switching in the input text. The reason is that the model is trained with only the target language (and domain), and not the source language, as a result of which it learns language normalization of sorts. For example, the sentence “Ma tahan two saldējumus.” (“Ma tahan” / “I want” in Estonian, “two” and “saldējumus” / “ice-creams” in genitive, plural in Latvian) is correctly translated into English as “I want two ice creams.”. See more examples in Appendix SECREF8 ."]},{"idx":6,"heading":"Grammatical Error Correction","paragraphs":["In this section we evaluate our model's performance in the GEC task: for example, for the English input “huge fan I are”, our model's output is “I am a huge fan”; this section's goal is to systematically check, how reliable the corrections are.","Although GEC does not require any distinction in text style, the core idea of this article is to also perform style transfer with the same multilingual multi-domain model. That only means that for GEC we have to select an output domain/style when producing error corrections.","Naturally, the model only copes with some kinds of errors and fails on others – for instance, word order is restored, as long as it does not affect the perception of the meaning. On the other hand, we do not expect orthographic variations like typos to be fixed reliably, since they affect the sub-word segmentation of the input and thus can hinder the translation.","Below we present qualitative and quantitative analysis of our model's GEC results, showing its overall performance, as well as which kinds of errors are handled reliably and which are not."]},{"idx":7,"heading":"Test Data and Metrics","paragraphs":["We use the following error-corrected corpora both for scoring and as basis for manual analysis:","for English: CoNLL-2014 BIBREF5 and JFLEG BIBREF20 corpora","for Estonian: the Learner Language Corpus BIBREF21","for Latvian: the Error-annotated Corpus of Latvian BIBREF22","All of these are based on language learner (L2) essays and their manual corrections.","To evaluate the model quantitatively we used two metrics: the Max-Match (M INLINEFORM0 ) metric from the CoNLL-2014 shared task scorer, and the GLEU score BIBREF23 for the other corpora. The main difference is that M INLINEFORM1 is based on the annotation of error categories, while the GLEU score compares the automatic correction to a reference without any error categorization."]},{"idx":8,"heading":"Results","paragraphs":["The M INLINEFORM0 scores are computed based on error-annotated corpora. Since error annotations were only available for English, we calculated the scores on English CoNLL corpus, see Table TABREF12 ).","Our model gets the M INLINEFORM0 score of 32.1. While it does not reach the score of the best CoNLL model BIBREF24 or the state-of-the-art BIBREF25 , these use annotated corpora to train. Our results count as restricted in CoNLL definitions and are more directly comparable to the classifier-based approach trained on unannotated corpora by BIBREF26 , while requiring even less effort.","The GLEU scores can be seen in Table TABREF13 . We calculated GLEU for both formal and informal style models for all three languages. For English our model's best score was 45.9 and for Estonian it was 38.1. Latvian corrected output in fact get worse scores than the original uncorrected corpus, which can be explained by smaller training corpora and worse MT quality for Latvian (see Table TABREF6 )."]},{"idx":9,"heading":"Qualitative Analysis","paragraphs":["We looked at the automatic corrections for 100 erroneous sentences of English and Estonian each as well as 80 sentences of Latvian. The overall aim was to find the ratio of sentences where (1) all errors have been corrected (2) only some are corrected (3) only some are corrected and part of the meaning is changed and (4) all meaning is lost.","The analysis was done separately for four error types: spelling and grammatical errors, word choice and word order. In case a sentence included more that one error type it was counted once for each error type. For English the first two types were annotated in the corpus, the rest were annotated by us, separating the original third error category into two new ones. The results can be seen in Table TABREF15 .","Not all English sentences included errors. 30 sentences remained unchanged, out of which 17 had no mistakes in them. For the changed sentences 87% were fully or partially corrected. In case of Estonian, where all sentences had mistakes, 61 out of the 100 sentences were fully or partially corrected without loss of information. 12 sentences became nonsense, all of which originally had some lexical mistakes. For English the results are similar: the most confusing type of errors that leads to complete loss of meaning is word choice. On the other hand, this was also the most common error type for both languages and errors of that type were fully corrected in 45% of cases for Estonian and 72% for English. Using words in the wrong order is a typical learner's error for Estonian that has rather free word order. It is also difficult to describe or set rules for this type of error. Our model manages this type rather well, correcting 79% of sentences acceptably, only losing some meaning in 2 sentences including this error type.","A similar experiment using 80 Latvian sentences yielded 17 fully corrected sentences, 15, 22 and 26 respectively for the other categories. As the Latvian model is weaker in general, this also leads to more chances of losing some of the meaning; we exclude it from the more detailed analysis and focus on English and Estonian.","Our model handles punctuation, word order mistakes and grammatical errors well. For example the subject-verb disagreement in English UID16 and verb-object disagreement in Estonian UID19 have been corrected.","“When price of gas goes up , the consumer do not want buy gas for fuels”","“When the price of gas goes up, the consumer doesn't want to buy gas for fuels”","“Sellepärast ütleb ta filmi lõpus, et tahab oma unistuse tagasi”","“Sellepärast ütleb ta filmi lõpus, et tahab oma unistust tagasi”","that's-why says he film at-end, that (he)-wants his-own dream INLINEFORM0","Sentences that include several error types are generally noticeably more difficult to correct. Depending on the error types that have been combined our model manages quite well and corrects all or several errors present. The sentence UID23 includes mistakes with word order and word choice: the argument \"vabaainetele\" (to elective courses) here should precede the verb and the verb \"registreeruma\" (register oneself) takes no such argument. Our model corrects both mistakes while also replacing the word \"seejärel\" (after that) with its synonym.","Seejärel pidi igaüks ennast registreeruma vabaainetele.","then had-to everyone oneself register-oneself to-free-courses","Siis pidi igaüks end vabaainetele registreerima.","then had-to everyone oneself to-free-courses register","The model fixes typos, but it mainly manages cases where two letters are needed but one is written and vice versa, for example \"detailled\" is corrected to \"detailed\" and ‘planing’ to \"planning\". More complicated mistakes are missed, especially if combined with other error types, and in some sentences a misspelled word is changed into an incorrect form that has a common ending, like \"misundrestood\" to \"misundrested\". The results get better if the input has been automatically spell-checked.","The system does more changes than are strictly necessary and often replaces correct words and phrases, for example \"frequently\" was changed to to \"often\" or in Estonian “öelda” (\"say\") to “avaldada” (\"publish\"). Sometimes this also confuses the meaning: \"siblings\" was changed to \"friends\".","To conclude this section, our model reliable corrects grammatical, spelling and word order errors on , with more mixed performance on lexical choice errors and some unnecessary paraphrasing of the input. The error types that the model manages well can be traced back to having a strong monolingual language model, a common trait of a good NMT model. As the model operates on the level of word parts and its vocabulary is limited, this leads to combining wrong word parts, sometimes across languages. This could be fixed by either using character-based NMT or doing automatic spelling correction prior to applying our current model.","We limit further comparisons to two styles, translating sentences of the OpenSubtitles test set into the style of Europarl and vice versa. Our assumption is that, generally, movie subtitles gravitate towards the more informal style, and parliament proceedings towards the more formal (see examples of translations into those styles in Table TABREF30 ). Preliminary tests showed that JRC-Acquis and EMEA texts resulted in practically the same style as Europarl. We also leave Latvian out of the evaluations, assuming that its performance is weaker, similarly to GEC results.","Human evaluation was performed on a subset of 100 sentences, 50 of them selected randomly from the OpenSubtitles test set and the other 50 from Europarl. Each sentence was translated into the opposite style. The resulting pairs were presented to participants, who were asked the following questions about each of them: (1) Do the sentences differ in any way? (2) How fluent is the translated sentence? (On a scale of 1 to 4, where 1 is unreadable, and 4 is perfectly fluent); (3) How similar are the sentences in meaning? (With options \"exactly the same\", \"the same with minor changes\", \"more or less the same\", \"completely different or nonsensical\"); (4) Does the translated sentence sound more formal than the original, more informal, or neither? (5) What differences are there between the sentences? (E.g. grammatical, lexical, missing words or phrases, word order, contractions, the use of formal \"you\").","Two such surveys were conducted, one in English and one in Estonian. 3 people participated in each of them, each of the three evaluators presented with the same set of examples.","In evaluation of fluency, all three human evaluators gave the translated sentences the same score in 41 out of 55 cases in English (not taking into account sentences which were simply copied from their originals), and in 51 out of 68 cases in Estonian. In evaluation of direction of style transfer, all three evaluators agreed in 16 cases and at least two agreed in 43 cases in English, and in Estonian in 19 cases all three agreed and in 59 at least two.","Of the 100 translated sentences, 45 were marked by all participants as being the same as their original sentences in the English set and 32 in Estonian. The remaining 55 and 68, respectively, were used to quantify style transfer quality.","Being a reasonably strong MT system, our model scores quite high on fluency (3.84 for English, 3.64 for Estonian) and meaning preservation (3.67 for English, 3.35 for Estonian). For meaning preservation, the judgments were converted into a scale of 1-4, where 1 stands for completely different meanings or nonsensical sentences, and 4 for the exact same meaning.","We evaluated the style transfer itself in the following way. For each pair of sentences, the average score given by three evaluators was calculated, in which the answer that the translated sentence is more formal counts as +1, more informal as -1, and neither as 0. We calculated the root mean square error between these scores and desired values (+1 if we aimed to make the sentence more formal, -1 if more informal). RMSE of 0 would stand for always transferring style in the right direction as judged by all evaluators, and 2 for always transferring style in the wrong direction.","On the English set, the RMSE is 0.78, and on Estonian 0.89. These numbers show that style transfer generally happens in the right direction, but is not very strong. Of the 55 sentences in English that were different from their source sentences, in 33 cases the sign of the average human score matched the desired one, in 7 it did not, and in 15 no change in style was observed by humans. In Estonian 36 sentences showed the right direction of style transfer, 10 wrong, and 22 no change.","In English sentences where the direction of style transfer was found to be correct (Figure FIGREF32 ), changes in use of contractions were reported in 19 cases (e.g. I have just been vs. I've just been), lexical changes in 15 cases (e.g. 'cause vs. because, or sure vs. certainly), grammatical in 13 (e.g. replacing no one's gonna with no one will, or method of production with production method), missing or added words or phrases in 8 cases.","In Estonian correctly transferred sentences (Figure FIGREF32 ), the most frequently reported were lexical substitutions (30 cases), followed by missing of added words or phrases (24 cases), changes in grammar (22 cases) and in word order (16 cases).","To conclude this section, unlike many style transfer models which produce text with strong style characteristics (e.g. with strong positive or negative sentiment), often at the cost of preserving meaning and fluency, our model gravitates towards keeping the meaning and fluency of the original sentence intact and mimicking some of the desired stylistic traits."]},{"idx":10,"heading":"Style Transfer","paragraphs":["Next we move on to evaluating the same model in terms of its performance in the context of style transfer.","At first, we examined how often the sentences change when translated monolingually. The assumption is that passing modified style factors should prevent the model from simply copying the source sequences when translating inside a single language, and incentivize it to match its output to certain style characteristics typical for different corpora. Figure FIGREF28 shows the proportions of sentence pairs in the 1000-sentence test sets where there was a significant difference between translations into different styles. We can observe that English texts change less often than Estonian or Latvian, while Europarl sentences are changed more often than those of other corpora.","To assess whether these changes actually correspond to the model's capability for transferring style, we turned to help of human evaluators."]},{"idx":11,"heading":"Cross-lingual Style Transfer","paragraphs":["Being able to translate between languages and also to modify the output to match the desired style allows the model to essentially perform domain adaptation. When translating from a language which has no formal \"you\" (English) into one that does (Estonian or Latvian), it will quite consistently use the informal variant when the target style is OpenSubtitles and the formal when the target style is Europarl (you rock INLINEFORM0 sa rokid/te rokite). The model is also quite consistent in use of contractions in English (es esmu šeit INLINEFORM1 I am here/I'm here). Some lexical substitutions occur: need on Matti lapsed. INLINEFORM2 those are Matt's kids./these are Matt's children. Word order may change: Where is Anna's bag? is Kus on Anna kott? in the more formal variant, and Kus Anna kott on? in the more informal. This feature is useful, but out of scope of this article, as we focus on monolingual applications."]},{"idx":12,"heading":"Related Work","paragraphs":["Grammatical error correction: there have been four shared tasks for GEC with prepared error-tagged datasets for L2 learners of English in the last decade: HOO BIBREF27 , BIBREF28 and CoNLL BIBREF29 , BIBREF5 . This has given an opportunity to train new models on the shared datasets and get an objective comparison of results. The general approach for grammatical error correction has been to use either rule-based approach, machine learning on error-tagged corpora, MT models on parallel data of erroneous and corrected sentences, or a combination of these BIBREF5 . The top model of the CONLL shared task in 2014 used a combined model of rule-based approach and MT BIBREF24 . All of these require annotated data or considerable effort to create, whereas our model is much more resource-independent. Another focus of the newer research is on creating GEC models without human-annotated resources. For example BIBREF26 combine statistical MT with unsupervised classification using unannotated parallel data for MT and unannotated native data for the classification model. In this case parallel data of erroneous and corrected sentences is still necessary for MT; the classifier uses native data, but still needs definitions of possible error types to classify – this work needs to be done by a human and is difficult for some less clear error types. In our approach there is no need for parallel data nor to specify error types, only for native data.","There has been little work on Estonian and Latvian GEC, all limited with rule-based approaches BIBREF30 , BIBREF31 . For both languages, as well as any low-resourced languages, our approach gives a feasible way to do grammatical error correction without needing neither parallel nor error tagged corpora. Style transfer: Several approaches use directly annotated data: for example, BIBREF7 and BIBREF8 train MT systems on the corpus of modern English Shakespeare to original Shakespeare. BIBREF32 collect a dataset of 110K informal/formal sentence pairs and train rule-based, phrase-based, and neural MT systems using this data.","One line of work aims at learning a style-independent latent representation of content while building decoders that can generate sentences in the style of choice BIBREF9 , BIBREF33 , BIBREF34 , BIBREF35 , BIBREF36 , BIBREF37 , BIBREF38 , BIBREF39 . Unsupervised MT has also been adapted for the task BIBREF10 , BIBREF40 . Our system also does not require parallel data between styles, but leverages the stability of the off-the-shelf supervised NMT to avoid the hassle of training unsupervised NMT systems and making GANs converge. Another problem with many current (both supervised and unsupervised) style transfer methods is that they are bounded to solve a binary task, where only two styles are included (whether because of data or restrictions of the approach). Our method, on the other hand, can be extended to as many styles as needed as long as there are parallel MT corpora in these styles available.","Notably, BIBREF41 use side constrains in order to translate in polite/impolite German, while we rely on multilingual encoder representations and use the system monolingually at inference time.","Finally, the most similar to our work conceptually is the approach of BIBREF42 , where they translate a sentence into another language, hoping that it will lose some style indicators, and then translate it back into the original language with a desired style tag attached to the encoder latent space. We also use the MT encoder to obtain rich sentence representations, but learn them directly as a part of a single multilingual translation system."]},{"idx":13,"heading":"Conclusions","paragraphs":["We presented a simple approach where a single multilingual NMT model is adapted to monolingual transfer and performs grammatical error correction and style transfer. We experimented with three languages and presented extensive evaluation of the model on both tasks. We used publicly available software and data and believe that our work can be easily reproduced.","We showed that for GEC our approach reliably corrects spelling, word order and grammatical errors, while being less reliable on lexical choice errors. Applied to style transfer our model is very good at meaning preservation and output fluency, while reliably transferring style for English contractions, lexical choice and grammatical constructions. The main benefit is that no annotated data is used to train the model, thus making it very easy to train it for other (especially under-resourced) languages.","Future work includes exploring adaptations of this approach to both tasks separately, while keeping the low cost of creating such models."]},{"idx":14,"heading":"Model Training: Technical Details","paragraphs":["After rudimentary cleaning (removing pairs where at least one sentence is longer that 100 tokens, at least one sentence is an empty string or does not contain any alphabetic characters, and pairs with length ratio over 9) and duplication to accommodate both translation directions in each language pair, the total size of the training corpus is 22.9M sentence pairs; training set sizes per language and corpus are given in Table TABREF36 . Validation set consists of 12K sentence pairs, 500 for each combination of translation direction and corpus. We also keep a test set of 24K sentence pairs, 1000 for each translation direction and corpus.","The data preprocessing pipeline consists of tokenization with Moses tokenizer BIBREF44 , true-casing, and segmentation with SentencePiece BIBREF45 with a joint vocabulary of size 32 000.","We trained a Transformer NMT model using the Sockeye framework BIBREF43 , mostly following the so-called Transformer base model: we used 6 layers, 512 positions, 8 attention heads and ReLU activations for both the encoder and decoder; Adam optimizer was used. Source and target token embeddings were both of size 512, and factors determining target language and style had embeddings of size 4. Batch size was set to 2048 words, initial learning rate to 0.0002, reducing by a factor of 0.7 every time the validation perplexity had not improved for 8 checkpoints, which happened every 4000 updates. The model converged during the 17th epoch, when validation perplexity has not improved for 32 consecutive checkpoints. The parameters of a single best checkpoint were used for all translations, with beam size set to 5."]},{"idx":15,"heading":"Output Examples","paragraphs":["We present more examples of translation of code-switched input segments, error correction and style transfer in English, Estonian and Latvian, informal (inf) and formal (fml) output style:"]},{"idx":16,"heading":"Figures and Tables","paragraphs":["Figure 1: Schematic illustration of zero-shot monolingual translation. The model is trained on bilingual data in all translation directions (English-to-Estonian, Estonian-to-English, English-to-Latvian, etc.) and then applied in monolingual directions only (English-toEnglish, etc.), without having seen any sentence pairs for them. The illustration is simplified, as it does not show the style (text domain) parametrization.","Table 1: BLEU scores of the multilingual MT model on WMT’17 (Latvian↔English) and WMT’18 (Estonian↔English) test sets","Table 2: M2 scores on the CoNLL corpus, including precision and recall.","Table 3: GLEU scores for all three languages. No scores have been previously reported elsewhere for Estonian and Latvian.","Table 4: GEC results by error types; “grammar” stands for grammatical mistakes, “lex” stands for lexical choice and “order” – for word order errors.","Figure 2: Proportions of sentences with token-wise edit distance ≥ 3 from the original when translated monolingually from and into different styles","Table 5: Examples of style transfer","Figure 3: Number of sentences in which evaluators reported different types of changes","Table 6: Training set sizes (number of sentence pairs)"]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1903.11437","title":"Using Monolingual Data in Neural Machine Translation: a Systematic Study","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Neural Machine Translation (MT) has radically changed the way systems are developed. A major difference with the previous generation (Phrase-Based MT) is the way monolingual target data, which often abounds, is used in these two paradigms. While Phrase-Based MT can seamlessly integrate very large language models trained on billions of sentences, the best option for Neural MT developers seems to be the generation of artificial parallel data through \\textsl{back-translation} - a technique that fails to fully take advantage of existing datasets. In this paper, we conduct a systematic study of back-translation, comparing alternative uses of monolingual data, as well as multiple data generation procedures. Our findings confirm that back-translation is very effective and give new explanations as to why this is the case. We also introduce new data simulation techniques that are almost as effective, yet much cheaper to implement."]},{"idx":1,"heading":"Introduction","paragraphs":["The new generation of Neural Machine Translation (NMT) systems is known to be extremely data hungry BIBREF0 . Yet, most existing NMT training pipelines fail to fully take advantage of the very large volume of monolingual source and/or parallel data that is often available. Making a better use of data is particularly critical in domain adaptation scenarios, where parallel adaptation data is usually assumed to be small in comparison to out-of-domain parallel data, or to in-domain monolingual texts. This situation sharply contrasts with the previous generation of statistical MT engines BIBREF1 , which could seamlessly integrate very large amounts of non-parallel documents, usually with a large positive effect on translation quality.","Such observations have been made repeatedly and have led to many innovative techniques to integrate monolingual data in NMT, that we review shortly. The most successful approach to date is the proposal of BIBREF2 , who use monolingual target texts to generate artificial parallel data via backward translation (BT). This technique has since proven effective in many subsequent studies. It is however very computationally costly, typically requiring to translate large sets of data. Determining the “right” amount (and quality) of BT data is another open issue, but we observe that experiments reported in the literature only use a subset of the available monolingual resources. This suggests that standard recipes for BT might be sub-optimal.","This paper aims to better understand the strengths and weaknesses of BT and to design more principled techniques to improve its effects. More specifically, we seek to answer the following questions: since there are many ways to generate pseudo parallel corpora, how important is the quality of this data for MT performance? Which properties of back-translated sentences actually matter for MT quality? Does BT act as some kind of regularizer BIBREF3 ? Can BT be efficiently simulated? Does BT data play the same role as a target-side language modeling, or are they complementary? BT is often used for domain adaptation: can the effect of having more in-domain data be sorted out from the mere increase of training material BIBREF2 ? For studies related to the impact of varying the size of BT data, we refer the readers to the recent work of BIBREF4 .","To answer these questions, we have reimplemented several strategies to use monolingual data in NMT and have run experiments on two language pairs in a very controlled setting (see § SECREF2 ). Our main results (see § SECREF4 and § SECREF5 ) suggest promising directions for efficient domain adaptation with cheaper techniques than conventional BT."]},{"idx":2,"heading":"In-domain and out-of-domain data","paragraphs":["We are mostly interested with the following training scenario: a large out-of-domain parallel corpus, and limited monolingual in-domain data. We focus here on the Europarl domain, for which we have ample data in several languages, and use as in-domain training data the Europarl corpus BIBREF5 for two translation directions: English INLINEFORM0 German and English INLINEFORM1 French. As we study the benefits of monolingual data, most of our experiments only use the target side of this corpus. The rationale for choosing this domain is to (i) to perform large scale comparisons of synthetic and natural parallel corpora; (ii) to study the effect of BT in a well-defined domain-adaptation scenario. For both language pairs, we use the Europarl tests from 2007 and 2008 for evaluation purposes, keeping test 2006 for development. When measuring out-of-domain performance, we will use the WMT newstest 2014."]},{"idx":3,"heading":"NMT setups and performance","paragraphs":["Our baseline NMT system implements the attentional encoder-decoder approach BIBREF6 , BIBREF7 as implemented in Nematus BIBREF8 on 4 million out-of-domain parallel sentences. For French we use samples from News-Commentary-11 and Wikipedia from WMT 2014 shared translation task, as well as the Multi-UN BIBREF9 and EU-Bookshop BIBREF10 corpora. For German, we use samples from News-Commentary-11, Rapid, Common-Crawl (WMT 2017) and Multi-UN (see table TABREF5 ). Bilingual BPE units BIBREF11 are learned with 50k merge operations, yielding vocabularies of about respectively 32k and 36k for English INLINEFORM0 French and 32k and 44k for English INLINEFORM1 German.","Both systems use 512-dimensional word embeddings and a single hidden layer with 1024 cells. They are optimized using Adam BIBREF12 and early stopped according to the validation performance. Training lasted for about three weeks on an Nvidia K80 GPU card.","Systems generating back-translated data are trained using the same out-of-domain corpus, where we simply exchange the source and target sides. They are further documented in § SECREF8 . For the sake of comparison, we also train a system that has access to a large batch of in-domain parallel data following the strategy often referred to as “fine-tuning”: upon convergence of the baseline model, we resume training with a 2M sentence in-domain corpus mixed with an equal amount of randomly selected out-of-domain natural sentences, with the same architecture and training parameters, running validation every 2000 updates with a patience of 10. Since BPE units are selected based only on the out-of-domain statistics, fine-tuning is performed on sentences that are slightly longer (ie. they contain more units) than for the initial training. This system defines an upper-bound of the translation performance and is denoted below as natural.","Our baseline and topline results are in Table TABREF6 , where we measure translation performance using BLEU BIBREF13 , BEER BIBREF14 (higher is better) and characTER BIBREF15 (smaller is better). As they are trained from much smaller amounts of data than current systems, these baselines are not quite competitive to today's best system, but still represent serious baselines for these datasets. Given our setups, fine-tuning with in-domain natural data improves BLEU by almost 4 points for both translation directions on in-domain tests; it also improves, albeit by a smaller margin, the BLEU score of the out-of-domain tests."]},{"idx":4,"heading":"Using artificial parallel data in NMT","paragraphs":["A simple way to use monolingual data in MT is to turn it into synthetic parallel data and let the training procedure run as usual BIBREF16 . In this section, we explore various ways to implement this strategy. We first reproduce results of BIBREF2 with BT of various qualities, that we then analyze thoroughly."]},{"idx":5,"heading":"The quality of Back-Translation","paragraphs":["BT requires the availability of an MT system in the reverse translation direction. We consider here three MT systems of increasing quality:","backtrans-bad: this is a very poor SMT system trained using only 50k parallel sentences from the out-of-domain data, and no additional monolingual data. For this system as for the next one, we use Moses BIBREF17 out-of-the-box, computing alignments with Fastalign BIBREF18 , with a minimal pre-processing (basic tokenization). This setting provides us with a pessimistic estimate of what we could get in low-resource conditions.","backtrans-good: these are much larger SMT systems, which use the same parallel data as the baseline NMTs (see § SECREF4 ) and all the English monolingual data available for the WMT 2017 shared tasks, totalling approximately 174M sentences. These systems are strong, yet relatively cheap to build.","backtrans-nmt: these are the best NMT systems we could train, using settings that replicate the forward translation NMTs.","Note that we do not use any in-domain (Europarl) data to train these systems. Their performance is reported in Table TABREF7 , where we observe a 12 BLEU points gap between the worst and best systems (for both languages).","As noted eg. in BIBREF19 , BIBREF20 , artificial parallel data obtained through forward-translation (FT) can also prove advantageous and we also consider a FT system (fwdtrans-nmt): in this case the target side of the corpus is artificial and is generated using the baseline NMT applied to a natural source.","Our results (see Table TABREF6 ) replicate the findings of BIBREF2 : large gains can be obtained from BT (nearly INLINEFORM0 BLEU in French and German); better artificial data yields better translation systems. Interestingly, our best Moses system is almost as good as the NMT and an order of magnitude faster to train. Improvements obtained with the bad system are much smaller; contrary to the better MTs, this system is even detrimental for the out-of-domain test.","Gains with forward translation are significant, as in BIBREF21 , albeit about half as good as with BT, and result in small improvements for the in-domain and for the out-of-domain tests. Experiments combining forward and backward translation (backfwdtrans-nmt), each using a half of the available artificial data, do not outperform the best BT results.","We finally note the large remaining difference between BT data and natural data, even though they only differ in their source side. This shows that at least in our domain-adaptation settings, BT does not really act as a regularizer, contrarily to the findings of BIBREF4 , BIBREF11 . Figure FIGREF13 displays the learning curves of these two systems. We observe that backtrans-nmt improves quickly in the earliest updates and then stays horizontal, whereas natural continues improving, even after 400k updates. Therefore BT does not help to avoid overfitting, it actually encourages it, which may be due “easier” training examples (cf. § SECREF15 )."]},{"idx":6,"heading":"Properties of back-translated data","paragraphs":["Comparing the natural and artificial sources of our parallel data wrt. several linguistic and distributional properties, we observe that (see Fig. FIGREF21 - FIGREF22 ):","artificial sources are on average shorter than natural ones: when using BT, cases where the source is shorter than the target are rarer; cases when they have the same length are more frequent.","automatic word alignments between artificial sources tend to be more monotonic than when using natural sources, as measured by the average Kendall INLINEFORM0 of source-target alignments BIBREF22 : for French-English the respective numbers are 0.048 (natural) and 0.018 (artificial); for German-English 0.068 and 0.053. Using more monotonic sentence pairs turns out to be a facilitating factor for NMT, as also noted by BIBREF20 .","syntactically, artificial sources are simpler than real data; We observe significant differences in the distributions of tree depths.","distributionally, plain word occurrences in artificial sources are more concentrated; this also translates into both a slower increase of the number of types wrt. the number of sentences and a smaller number of rare events.","The intuition is that properties (i) and (ii) should help translation as compared to natural source, while property (iv) should be detrimental. We checked (ii) by building systems with only 10M words from the natural parallel data selecting these data either randomly or based on the regularity of their word alignments. Results in Table TABREF23 show that the latter is much preferable for the overall performance. This might explain that the mostly monotonic BT from Moses are almost as good as the fluid BT from NMT and that both boost the baseline."]},{"idx":7,"heading":"Stupid Back-Translation","paragraphs":["We now analyze the effect of using much simpler data generation schemes, which do not require the availability of a backward translation engine."]},{"idx":8,"heading":"Setups","paragraphs":["We use the following cheap ways to generate pseudo-source texts:","copy: in this setting, the source side is a mere copy of the target-side data. Since the source vocabulary of the NMT is fixed, copying the target sentences can cause the occurrence of OOVs. To avoid this situation, BIBREF24 decompose the target words into source-side units to make the copy look like source sentences. Each OOV found in the copy is split into smaller units until all the resulting chunks are in the source vocabulary.","copy-marked: another way to integrate copies without having to deal with OOVs is to augment the source vocabulary with a copy of the target vocabulary. In this setup, BIBREF25 ensure that both vocabularies never overlap by marking the target word copies with a special language identifier. Therefore the English word resume cannot be confused with the homographic French word, which is marked @fr@resume.","copy-dummies: instead of using actual copies, we replace each word with “dummy” tokens. We use this unrealistic setup to observe the training over noisy and hardly informative source sentences.","We then use the procedures described in § SECREF4 , except that the pseudo-source embeddings in the copy-marked setup are pretrained for three epochs on the in-domain data, while all remaining parameters are frozen. This prevents random parameters from hurting the already trained model."]},{"idx":9,"heading":"Copy+marking+noise is not so stupid","paragraphs":["We observe that the copy setup has only a small impact on the English-French system, for which the baseline is already strong. This is less true for English-German where simple copies yield a significant improvement. Performance drops for both language pairs in the copy-dummies setup.","We achieve our best gains with the copy-marked setup, which is the best way to use a copy of the target (although the performance on the out-of-domain tests is at most the same as the baseline). Such gains may look surprising, since the NMT model does not need to learn to translate but only to copy the source. This is indeed what happens: to confirm this, we built a fake test set having identical source and target side (in French). The average cross-entropy for this test set is 0.33, very close to 0, to be compared with an average cost of 58.52 when we process an actual source (in English). This means that the model has learned to copy words from source to target with no difficulty, even for sentences not seen in training. A follow-up question is whether training a copying task instead of a translation task limits the improvement: would the NMT learn better if the task was harder? To measure this, we introduce noise in the target sentences copied onto the source, following the procedure of BIBREF26 : it deletes random words and performs a small random permutation of the remaining words. Results (+ Source noise) show no difference for the French in-domain test sets, but bring the out-of-domain score to the level of the baseline. Finally, we observe a significant improvement on German in-domain test sets, compared to the baseline (about +1.5 BLEU). This last setup is even almost as good as the backtrans-nmt condition (see § SECREF8 ) for German. This shows that learning to reorder and predict missing words can more effectively serve our purposes than simply learning to copy."]},{"idx":10,"heading":"Towards more natural pseudo-sources","paragraphs":["Integrating monolingual data into NMT can be as easy as copying the target into the source, which already gives some improvement; adding noise makes things even better. We now study ways to make pseudo-sources look more like natural data, using the framework of Generative Adversarial Networks (GANs) BIBREF27 , an idea borrowed from BIBREF26 ."]},{"idx":11,"heading":"GAN setups","paragraphs":["In our setups, we use a marked target copy, viewed as a fake source, which a generator encodes so as to fool a discriminator trained to distinguish a fake from a natural source. Our architecture contains two distinct encoders, one for the natural source and one for the pseudo-source. The latter acts as the generator ( INLINEFORM0 ) in the GAN framework, computing a representation of the pseudo-source that is then input to a discriminator ( INLINEFORM1 ), which has to sort natural from artificial encodings. INLINEFORM2 assigns a probability of a sentence being natural.","During training, the cost of the discriminator is computed over two batches, one with natural (out-of-domain) sentences INLINEFORM0 and one with (in-domain) pseudo-sentences INLINEFORM1 . The discriminator is a bidirectional-Recurrent Neural Network (RNN) of dimension 1024. Averaged states are passed to a single feed-forward layer, to which a sigmoid is applied. It inputs encodings of natural ( INLINEFORM2 ) and pseudo-sentences ( INLINEFORM3 ) and is trained to optimize:","INLINEFORM0","INLINEFORM0 's parameters are updated to maximally fool INLINEFORM1 , thus the loss INLINEFORM2 : INLINEFORM3","Finally, we keep the usual MT objective. ( INLINEFORM0 is a real or pseudo-sentence): INLINEFORM1","We thus need to train three sets of parameters: INLINEFORM0 and INLINEFORM1 (MT parameters), with INLINEFORM2 . The pseudo-source encoder and embeddings are updated wrt. both INLINEFORM3 and INLINEFORM4 . Following BIBREF28 , INLINEFORM5 is updated only when INLINEFORM6 's accuracy exceeds INLINEFORM7 . On the other hand, INLINEFORM8 is not updated when its accuracy exceeds INLINEFORM9 . At each update, two batches are generated for each type of data, which are encoded with the real or pseudo-encoder. The encoder outputs serve to compute INLINEFORM10 and INLINEFORM11 . Finally, the pseudo-source is encoded again (once INLINEFORM12 is updated), both encoders are plugged into the translation model and the MT cost is back-propagated down to the real and pseudo-word embeddings. Pseudo-encoder and discriminator parameters are pre-trained for 10k updates. At test time, the pseudo-encoder is ignored and inference is run as usual."]},{"idx":12,"heading":"GANs can help","paragraphs":["Results are in Table TABREF32 , assuming the same fine-tuning procedure as above. On top of the copy-marked setup, our GANs do not provide any improvement in both language pairs, with the exception of a small improvement for English-French on the out-of-domain test, which we understand as a sign that the model is more robust to domain variations, just like when adding pseudo-source noise. When combined with noise, the French model yields the best performance we could obtain with stupid BT on the in-domain tests, at least in terms of BLEU and BEER. On the News domain, we remain close to the baseline level, with slight improvements in German.","A first observation is that this method brings stupid BT models closer to conventional BT, at a greatly reduced computational cost. While French still remains 0.4 to 1.0 BLEU below very good backtranslation, both approaches are in the same ballpark for German - may be because BTs are better for the former system than for the latter.","Finally note that the GAN architecture has two differences with basic copy-marked: (a) a distinct encoder for real and pseudo-sentence; (b) a different training regime for these encoders. To sort out the effects of (a) and (b), we reproduce the GAN setup with BT sentences, instead of copies. Using a separate encoder for the pseudo-source in the backtrans-nmt setup can be detrimental to performance (see Table TABREF32 ): translation degrades in French for all metrics. Adding GANs on top of the pseudo-encoder was not able to make up for the degradation observed in French, but allowed the German system to slightly outperform backtrans-nmt. Even though this setup is unrealistic and overly costly, it shows that GANs are actually helping even good systems."]},{"idx":13,"heading":"Using Target Language Models","paragraphs":["In this section, we compare the previous methods with the use of a target side Language Model (LM). Several proposals exist in the literature to integrate LMs in NMT: for instance, BIBREF3 strengthen the decoder by integrating an extra, source independent, RNN layer in a conventional NMT architecture. Training is performed either with parallel, or monolingual data. In the latter case, word prediction only relies on the source independent part of the network."]},{"idx":14,"heading":"LM Setup","paragraphs":["We have followed BIBREF29 and reimplemented their deep-fusion technique. It requires to first independently learn a RNN-LM on the in-domain target data with a cross-entropy objective; then to train the optimal combination of the translation and the language models by adding the hidden state of the RNN-LM as an additional input to the softmax layer of the decoder. Our RNN-LMs are trained using dl4mt with the target side of the parallel data and the Europarl corpus (about 6M sentences for both French and German), using a one-layer GRU with the same dimension as the MT decoder (1024)."]},{"idx":15,"heading":"LM Results","paragraphs":["Results are in Table TABREF33 . They show that deep-fusion hardly improves the Europarl results, while we obtain about +0.6 BLEU over the baseline on newstest-2014 for both languages. deep-fusion differs from stupid BT in that the model is not directly optimized on the in-domain data, but uses the LM trained on Europarl to maximize the likelihood of the out-of-domain training data. Therefore, no specific improvement is to be expected in terms of domain adaptation, and the performance increases in the more general domain. Combining deep-fusion and copy-marked + noise + GANs brings slight improvements on the German in-domain test sets, and performance out of the domain remains near the baseline level."]},{"idx":16,"heading":"Re-analyzing the effects of BT","paragraphs":["As a follow up of previous discussions, we analyze the effect of BT on the internals of the network. Arguably, using a copy of the target sentence instead of a natural source should not be helpful for the encoder, but is it also the case with a strong BT? What are the effects on the attention model?"]},{"idx":17,"heading":"Parameter freezing protocol","paragraphs":["To investigate these questions, we run the same fine-tuning using the copy-marked, backtrans-nmt and backtrans-nmt setups. Note that except for the last one, all training scenarios have access to same target training data. We intend to see whether the overall performance of the NMT system degrades when we selectively freeze certain sets of parameters, meaning that they are not updated during fine-tuning."]},{"idx":18,"heading":"Results","paragraphs":["BLEU scores are in Table TABREF39 . The backtrans-nmt setup is hardly impacted by selective updates: updating the only decoder leads to a degradation of at most 0.2 BLEU. For copy-marked, we were not able to freeze the source embeddings, since these are initialized when fine-tuning begins and therefore need to be trained. We observe that freezing the encoder and/or the attention parameters has no impact on the English-German system, whereas it slightly degrades the English-French one. This suggests that using artificial sources, even of the poorest quality, has a positive impact on all the components of the network, which makes another big difference with the LM integration scenario.","The largest degradation is for natural, where the model is prevented from learning from informative source sentences, which leads to a decrease of 0.4 to over 1.0 BLEU. We assume from these experiments that BT impacts most of all the decoder, and learning to encode a pseudo-source, be it a copy or an actual back-translation, only marginally helps to significantly improve the quality. Finally, in the fwdtrans-nmt setup, freezing the decoder does not seem to harm learning with a natural source."]},{"idx":19,"heading":"Related work","paragraphs":["The literature devoted to the use of monolingual data is large, and quickly expanding. We already alluded to several possible ways to use such data: using back- or forward-translation or using a target language model. The former approach is mostly documented in BIBREF2 , and recently analyzed in BIBREF19 , which focus on fully artificial settings as well as pivot-based artificial data; and BIBREF4 , which studies the effects of increasing the size of BT data. The studies of BIBREF20 , BIBREF19 also consider forward translation and BIBREF21 expand these results to domain adaptation scenarios. Our results are complementary to these earlier studies.","As shown above, many alternatives to BT exist. The most obvious is to use target LMs BIBREF3 , BIBREF29 , as we have also done here; but attempts to improve the encoder using multi-task learning also exist BIBREF30 .","This investigation is also related to recent attempts to consider supplementary data with a valid target side, such as multi-lingual NMT BIBREF31 , where source texts in several languages are fed in the same encoder-decoder architecture, with partial sharing of the layers. This is another realistic scenario where additional resources can be used to selectively improve parts of the model.","Round trip training is another important source of inspiration, as it can be viewed as a way to use BT to perform semi-unsupervised BIBREF32 or unsupervised BIBREF33 training of NMT. The most convincing attempt to date along these lines has been proposed by BIBREF26 , who propose to use GANs to mitigate the difference between artificial and natural data."]},{"idx":20,"heading":"Conclusion","paragraphs":["In this paper, we have analyzed various ways to integrate monolingual data in an NMT framework, focusing on their impact on quality and domain adaptation. While confirming the effectiveness of BT, our study also proposed significantly cheaper ways to improve the baseline performance, using a slightly modified copy of the target, instead of its full BT. When no high quality BT is available, using GANs to make the pseudo-source sentences closer to natural source sentences is an efficient solution for domain adaptation.","To recap our answers to our initial questions: the quality of BT actually matters for NMT (cf. § SECREF8 ) and it seems that, even though artificial source are lexically less diverse and syntactically complex than real sentence, their monotonicity is a facilitating factor. We have studied cheaper alternatives and found out that copies of the target, if properly noised (§ SECREF4 ), and even better, if used with GANs, could be almost as good as low quality BTs (§ SECREF5 ): BT is only worth its cost when good BT can be generated. Finally, BT seems preferable to integrating external LM - at least in our data condition (§ SECREF6 ). Further experiments with larger LMs are needed to confirm this observation, and also to evaluate the complementarity of both strategies. More work is needed to better understand the impact of BT on subparts of the network (§ SECREF7 ).","In future work, we plan to investigate other cheap ways to generate artificial data. The experimental setup we proposed may also benefit from a refining of the data selection strategies to focus on the most useful monolingual sentences."]},{"idx":21,"heading":"Figures and Tables","paragraphs":["Table 1: Size of parallel corpora","Table 2: Performance wrt. different BT qualities","Table 3: BLEU scores for (backward) translation into English","Figure 1: Learning curves from backtrans-nmt and natural. Artificial parallel data is more prone to overfitting than natural data.","Figure 2: Properties of pseudo-English data obtained with backtrans-nmt from French. The synthetic source contains shorter sentences (a) and slightly simpler syntax (b). The vocabulary growth wrt. an increasing number of observed sentences (c) and the token-type correlation (d) suggest that the natural source is lexically richer.","Table 4: Selection strategies for BT data (English-French)","Figure 3: Properties of pseudo-English data obtained with backtrans-nmt (back-translated from German). Tendencies similar to English-French can be observed and difference in syntax complexity is even more visible.","Table 5: Performance wrt. various stupid BTs","Table 6: Performance wrt. different GAN setups","Table 7: Deep-fusion model","Table 8: BLEU scores with selective parameter freezing"]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1904.02954","title":"Alternative Weighting Schemes for ELMo Embeddings","sections":[{"idx":0,"heading":"Abstract","paragraphs":["ELMo embeddings (Peters et. al, 2018) had a huge impact on the NLP community and may recent publications use these embeddings to boost the performance for downstream NLP tasks. However, integration of ELMo embeddings in existent NLP architectures is not straightforward. In contrast to traditional word embeddings, like GloVe or word2vec embeddings, the bi-directional language model of ELMo produces three 1024 dimensional vectors per token in a sentence. Peters et al. proposed to learn a task-specific weighting of these three vectors for downstream tasks. However, this proposed weighting scheme is not feasible for certain tasks, and, as we will show, it does not necessarily yield optimal performance. We evaluate different methods that combine the three vectors from the language model in order to achieve the best possible performance in downstream NLP tasks. We notice that the third layer of the published language model often decreases the performance. By learning a weighted average of only the first two layers, we are able to improve the performance for many datasets. Due to the reduced complexity of the language model, we have a training speed-up of 19-44% for the downstream task."]},{"idx":1,"heading":"Introduction","paragraphs":["Peters2018 presented in their work Deep Contextualized Word Representations (often referred to as ELMo embeddings) a method that uses a bidirectional language model (biLM) to derive word representations which are based on the complete context of a sentence. They demonstrated that these ELMo embeddings can substantially increase the performance for various NLP tasks. This new type of word representations had a big impact on the NLP community and many new architectures, for example, many from EMNLP 2018, report a better performance when using ELMo embeddings.","Traditional word embeddings, like word2vec or GloVe, mapped each token in a sentence to a single dense vector. In contrast to that, the published ELMo implementation computes three layers of a language model: The first layer is a CNN that computes a non-contextualized word representation based on the characters of a word, followed by two bidirectional LSTM layers that take the context of the sentence into account.","The output of the three layers is integrated into task-specific neural architectures. However, the integration of ELMo into neural architectures is not straightforward. For example, Peters et al. describe two methods for the integration: Either the output of the last layer is used for downstream tasks, or a task-specific weighting of the three layer outputs is learned: $\\text{ELMo}_{\\text{weighted\\_average}} = \\gamma \\sum _{j=0}^{2}s_j h_j$","with $s \\in \\mathbb {R}^3$ softmax-normalized weights, $h_j$ the output of the $j$ -th layer of the biLM and $\\gamma $ a scalar that is used to scale the entire ELMo vector.","Learning this weighted average is not always easy, as it can require substantial changes in existent network architectures and some deep learning frameworks (for example Keras) lack the possibility to easily implement such a learned weighted average. Further, for unsupervised tasks, such a weighted average cannot be learned.","Hence, several authors used simplified ways to integrate ELMo embeddings in their architectures. Some use the output of the last ELMo layer, some concatenate all three vectors BIBREF1 , BIBREF2 , and others compute a (fixed) average of the three layers.","It is unclear what the impact of these different weighting schemes is for downstream tasks. Is the (rather complicated) learned weighted average proposed by Peters et al. needed to achieve optimal performance? Will a simpler method, like computing a fixed average, decrease the performance?","In this paper, we evaluate different schemes to combine the three ELMo vectors. We analyze the impact of these schemes for downstream NLP tasks. First, we study this for a BiLSTM-CRF architecture which only uses ELMo embeddings as input representation. Next, we study the different weighting schemes for the more complex models included in AllenNLP, which concatenate ELMo embeddings with other input representations like GloVe word embeddings.","In this paper we show that 1) the weighting scheme can have a significant impact on downstream NLP tasks, 2) that the learned weighted average proposed by Peters et al. does not yield the optimal performance for all datasets, and 3) that the second layer of the biLM yields in many cases a better performance than the third (last) layer.","Surprisingly, using the output of the second layer of the biLM model yields a better performance than using the third (last) layer in many downstream NLP tasks. Using this insight, we present a weighting scheme that learns a weighted average of the first two layers of the biLM. This scheme outperforms the originally proposed weighting scheme by Peters et al. for several datasets. Further, it is computationally faster than the original method. For downstream tasks, we saw a training speed-up of 19-44%."]},{"idx":2,"heading":"Related Work","paragraphs":["To our knowledge, only Peters2018 evaluated different weighting schemes. They evaluated to use either the output of the last layer or to learn a task-specific weighted average of all three layer outputs. They compare these two options in their paper and show a slight advantage for learning a weighted average. However, the evaluation is in our opinion insufficient. First, they evaluate both options on the development set, so it remains unclear if there are changes for unseen data (test set). Further, they evaluate it only with a single random seed. As shown in BIBREF3 , the performance of a neural network can change significantly with a different random seed. For example, we observe test score differences of up to 1.5 percentage points when the same model is trained with a different random seed with the AllenNLP model for the Stanford Sentiment Treebank (SST-5). The differences Peters et al. report between using the last layer and learning a task-specific weighting are rather small (0.4 - 0.7 percentage points). It is not clear if these differences are due to the effect of different random seeds or due to the weighting scheme."]},{"idx":3,"heading":"Alternative Weighting Schemes","paragraphs":["The published bidirectional language model (biLM) produces three 1024 dimensional vectors for each token in a sentence. In this paper we systematically study the following methods to combine the three vectors returned by the biLM:","Individual Layers: Only a single layer is used for the downstream task.","Concatenation: All three vectors are concatenated.","Fixed Average: We compute an average of all three vectors.","Learned Weighted Average: We learn a task-specific weighted average (ELMo $_\\text{weighted\\_average}$ ).","Learned Weighted Average of the 1st and 2nd Layer: We learn a task-specific weighted average of the first two layers."]},{"idx":4,"heading":"Evaluation of Weighting Schemes","paragraphs":["We test the different weighting schemes with two experiments. For the first experiment, we evaluate a neural network that solely uses ELMo embeddings as a representation of the input. This experiment shows how suitable the schemes are when no other features are used. In the second experiment, we evaluate the schemes with the more advanced, state-of-the-art architectures from AllenNLP. These models often concatenate the ELMo embeddings with other input representations. For example, the NER model from AllenNLP concatenates the ELMo embedding with GloVe embeddings and with a task-specific character-based word representation (similar to Ma2016). We expect that the results in the second experiment vary from the first experiment. If a particular weighting scheme lacks specific information, the network might still retrieve it from the other input representations.","For the first experiment, we use a BiLSTM-CRF architecture for sequence tagging BIBREF4 . We use ELMo embeddings instead of word embeddings. Two bidirectional LSTM layers (with 100 recurrent units each) are followed by a conditional random field (CRF) to produce the most likely tag sequence. The network was trained using Adam optimizer BIBREF5 and a variational dropout BIBREF6 of 0.5 was added to recurrent and output units.","We trained this architecture for the following datasets: Arguments: Argument component detection (major claim, claim, premise) in 402 persuasive essays BIBREF7 . Development and test set were 80 randomly selected essays each. ACE Entities/Events: ACE 2005 dataset BIBREF8 consists of 599 annotated documents from six different domains (newswire, broadcast news, broadcast conversations, blogs, forums, and speeches). We train the architecture to either detect events or to detect entities in these documents. We used 90 randomly selected documents each for the development and test set. POS: We use the part-of-speech tags from Universal Dependencies v. 1.3 for English with the provided data splits. We reduced the training set to the first 500 sentences to increase the difficulty for the network. The development and test set were kept unchanged. Chunking: CoNLL 2000 shared task dataset on chunking. NER: CoNLL 2003 shared task on named entity recognition. GENIA NER: The Bio-Entity Recognition Task at JNLPBA BIBREF9 annotated Medline abstracts with information on bio-entities (like protein or DNA-names). The dataset consists of 2000 abstracts for training (we used 400 of those as development set) and the test set contains 404 abstracts. WNUT16: WNUT16 was a shared task on Named Entity Recognition over Twitter BIBREF10 . Training data are 2,394 annotated tweets, development data are 1,000 tweets, and test data are 3,856 tweets.","For the second experiment, we use the existent AllenNLP models that reproduce the experiments of Peters et al. We use the CoNLL 2003 NER model, the Stanford Sentiment Treebank (SST-5) model, the constituency parsing model for the Penn TreeBank, and the Stanford Natural Language Inference Corpus (SNLI) model. The $F_1$ -score is computed for the NER tasks and parsing; accuracy is computed for the SST-task and the SNLI-task.","Not all experiments from the paper of Peters et al. are reproducible with AllenNLP. AllenNLP currently has no model for the SQuAD task. For the Coref-task, the AllenNLP configuration is missing some features and does not use ELMo embeddings. For the SRL-task, AllenNLP uses a different metric that is not comparable to the official metric.","For both experiments, we use the pre-trained ELMo 5.5B model, which was trained on a dataset of 5.5 billion tokens. We trained each setup with ten different random seed and report average test scores."]},{"idx":5,"heading":"Results","paragraphs":["The results of the BiLSTM-CRF model, that uses only the ELMo embeddings as input representations, are shown in the upper part of Table 1 .","We observe that the output of the first layer yields in most cases the worst performance. This was expected, as the first layer is a CNN that computes a character-based word representation. It does not take the context of a word into account.","In our experiment, we don't observe a difference between the computation of an unweighted average and of learning a task-specific weighted average. For four datasets, the unweighted average yielded better performance, while for the other four other datasets, the learned weighted average yielded better performance. However, the differences are insignificant.","To our surprise, using the second layer of the biLM yields in most cases a better performance than using the third (last) layer of the biLM. For 7 out of 8 datasets it outperforms even the learned weighted average method proposed by Peters et al. Only for the GENIA dataset achieved the learned weighted average a significantly better performance than using the output of the second layer. However, for this dataset, it appears that context information is less critical, as the system achieves a high performance by using only the characters of a word (1. Layer).","The results for the second experiment, that uses AllenNLP and ELMo embeddings in combination with other input representations, are presented in the lower part of Table 1 .","In contrast to our first experiment, we notice much smaller differences between different weighting schemes. In most cases, the differences are not statistically significant. When a network solely depends on ELMo embeddings as input, all relevant information to solve the task must be included. However, if ELMo is combined with other input representations, like GloVe embeddings, the dependency on the ELMo embeddings decreases. If ELMo does not capture critical word properties, the network can still fall back on one of the other input representations.","We notice that computing a weighted average of the first two layers yields the highest performance in 4 out of 5 datasets. It appears that computing the third layer of the biLM does not necessarily add value for downstream tasks. Removing it increases training as well as inference speed, as computing the bidirectional LSTM of the third layer is rather slow. By not computing the third layer of the biLM, we observe a training speed-up of 38% for the NER model, 44% for the SST model, 19% for the parsing model, and 27% for the SNLI model. The training of the SNLI model requires on a Tesla P100 GPU about one day."]},{"idx":6,"heading":"Conclusion","paragraphs":["As noted by Peter2018b, the lower layers of the biLM specialize in local syntactic relationships, allowing the higher layers to model longer range relationships. Knowing beforehand which of these properties are relevant for an NLP task is difficult. In our experiments, we observed significant performance differences in downstream tasks for the three biLM layers. The last, most abstract layer, often yielded mediocre results when it was used as the only input representation. For most datasets, the output of the second layer appears to be the most relevant representation. It offers the best trade-off between local syntactic features and more abstract, long-range relationships.","As it is not known in advance which layer produces the best input representation, learning a task-specific weighted average of the three layers appears advisable. However, for several datasets, it appears that the output of the third layer does not add value as it is a too abstract representation for the NLP tasks. The learned weighted average method presented by Peters et al. regularizes the three (softmax-normalized) weights $s_j$ . As a consequence, a zero or small $s_j$ value is not possible, and all three vectors are used even if one vector (e.g. the third layer output) decreases the performance. This explains why removing the (sometimes harmful) third layer can improve the model performance. Further, by removing the last layer, we observe a significant training speed-up. For the models included in AllenNLP, we observed a training speed-up of 19-44%, while improving the test performance in 3 out of 5 datasets. This speed-up can be crucial for cases that require fast training of inference speeds.","The weighting scheme appears especially important when these vectors are used as the only input representation for the task. In that case, we advise testing different weighting schemes. If ELMo is used in conjunction with other input representations, the weighting scheme was less critical."]},{"idx":7,"heading":"Acknowledgements","paragraphs":["This work was supported by the German Research Foundation through the German-Israeli Project Cooperation (DIP, grant DA 1600/1-1 and grant GU 798/17-1). We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan X(p) Pascal GPU used for this research."]},{"idx":8,"heading":"Figures and Tables","paragraphs":["Table 1: Upper half: Average scores of 10 training runs with different random seeds using the BiLSTM-CRF architecture of Reimers and Gurevych (2017). ELMo is used as the only input representation. Lower half: Average scores of 10 training runs using AllenNLP. ELMo is used in combination with other input representations. Bold entries mark the best performance per row. Red entries indicate statistically significantly worse entries with p < 0.01. F.-Avg.: unweighted average of all three layers, W.-Avg.: Learned weighted average of the three layers, as proposed by Peters et al., W.-Avg. 1 & 2: Learned weighted average of the first and second layer of the biLM."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1904.09535","title":"NeuronBlocks: Building Your NLP DNN Models Like Playing Lego","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Deep Neural Networks (DNN) have been widely employed in industry to address various Natural Language Processing (NLP) tasks. However, many engineers find it a big overhead when they have to choose from multiple frameworks, compare different types of models, and understand various optimization mechanisms. An NLP toolkit for DNN models with both generality and flexibility can greatly improve the productivity of engineers by saving their learning cost and guiding them to find optimal solutions to their tasks. In this paper, we introduce NeuronBlocks, a toolkit encapsulating a suite of neural network modules as building blocks to construct various DNN models with complex architecture. This toolkit empowers engineers to build, train, and test various NLP models through simple configuration of JSON files. The experiments on several NLP datasets such as GLUE, WikiQA and CoNLL-2003 demonstrate the effectiveness of NeuronBlocks."]},{"idx":1,"heading":"Introduction","paragraphs":["Deep Neural Networks (DNN) have been widely employed in industry for solving various Natural Language Processing (NLP) tasks, such as text classification, sequence labeling, question answering, etc. However, when engineers apply DNN models to address specific NLP tasks, they often face the following challenges.","The above challenges often hinder the productivity of engineers, and result in less optimal solutions to their given tasks. This motivates us to develop an NLP toolkit for DNN models, which facilitates engineers to develop DNN approaches. Before designing this NLP toolkit, we conducted a survey among engineers and identified a spectrum of three typical personas.","To satisfy the requirements of all the above three personas, the NLP toolkit has to be generic enough to cover as many tasks as possible. At the same time, it also needs to be flexible enough to allow alternative network architectures as well as customized modules. Therefore, we analyzed the NLP jobs submitted to a commercial centralized GPU cluster. Table TABREF11 showed that about 87.5% NLP related jobs belong to a few common tasks, including sentence classification, text matching, sequence labeling, MRC, etc. It further suggested that more than 90% of the networks were composed of several common components, such as embedding, CNN/RNN, Transformer and so on.","Based on the above observations, we developed NeuronBlocks, a DNN toolkit for NLP tasks. The basic idea is to provide two layers of support to the engineers. The upper layer targets common NLP tasks. For each task, the toolkit contains several end-to-end network templates, which can be immediately instantiated with simple configuration. The bottom layer consists of a suite of reusable and standard components, which can be adopted as building blocks to construct networks with complex architecture. By following the interface guidelines, users can also contribute to this gallery of components with their own modules.","The technical contributions of NeuronBlocks are summarized into the following three aspects."]},{"idx":2,"heading":"Related Work","paragraphs":["There are several general-purpose deep learning frameworks, such as TensorFlow, PyTorch and Keras, which have gained popularity in NLP community. These frameworks offer huge flexibility in DNN model design and support various NLP tasks. However, building models under these frameworks requires a large overhead of mastering these framework details. Therefore, higher level abstraction to hide the framework details is favored by many engineers.","There are also several popular deep learning toolkits in NLP, including OpenNMT BIBREF0 , AllenNLP BIBREF1 etc. OpenNMT is an open-source toolkit mainly targeting neural machine translation or other natural language generation tasks. AllenNLP provides several pre-built models for NLP tasks, such as semantic role labeling, machine comprehension, textual entailment, etc. Although these toolkits reduce the development cost, they are limited to certain tasks, and thus not flexible enough to support new network architectures or new components."]},{"idx":3,"heading":"Design","paragraphs":["The Neuronblocks is built on PyTorch. The overall framework is illustrated in Figure FIGREF16 . It consists of two layers: the Block Zoo and the Model Zoo. In Block Zoo, the most commonly used components of deep neural networks are categorized into several groups according to their functions. Within each category, several alternative components are encapsulated into standard and reusable blocks with a consistent interface. These blocks serve as basic and exchangeable units to construct complex network architectures for different NLP tasks. In Model Zoo, the most popular NLP tasks are identified. For each task, several end-to-end network templates are provided in the form of JSON configuration files. Users can simply browse these configurations and choose one or more to instantiate. The whole task can be completed without any coding efforts."]},{"idx":4,"heading":"Block Zoo","paragraphs":["We recognize the following major functional categories of neural network components. Each category covers as many commonly used modules as possible. The Block Zoo is an open framework, and more modules can be added in the future.","[itemsep= -0.4em,topsep = 0.3em, align=left, labelsep=-0.6em, leftmargin=1.2em]","Embedding Layer: Word/character embedding and extra handcrafted feature embedding such as pos-tagging are supported.","Neural Network Layers: Block zoo provides common layers like RNN, CNN, QRNN BIBREF2 , Transformer BIBREF3 , Highway network, Encoder Decoder architecture, etc. Furthermore, attention mechanisms are widely used in neural networks. Thus we also support multiple attention layers, such as Linear/Bi-linear Attention, Full Attention BIBREF4 , Bidirectional attention flow BIBREF5 , etc. Meanwhile, regularization layers such as Dropout, Layer Norm, Batch Norm, etc are also supported for improving generalization ability.","Loss Function: Besides of the loss functions built in PyTorch, we offer more options such as Focal Loss BIBREF6 .","Metrics: For classification task, AUC, Accuracy, Precision/Recall, F1 metrics are supported. For sequence labeling task, F1/Accuracy are supported. For knowledge distillation task, MSE/RMSE are supported. For MRC task, ExactMatch/F1 are supported."]},{"idx":5,"heading":"Model Zoo","paragraphs":["In NeuronBlocks, we identify four types of most popular NLP tasks. For each task, we provide various end-to-end network templates.","[itemsep= -0.4em,topsep = 0.3em, align=left, labelsep=-0.6em, leftmargin=1.2em]","Text Classification and Matching. Tasks such as domain/intent classification, question answer matching are supported.","Sequence Labeling. Predict each token in a sequence into predefined types. Common tasks include NER, POS tagging, Slot tagging, etc.","Knowledge Distillation BIBREF7 . Teacher-Student based knowledge distillation is one common approach for model compression. NeuronBlocks provides knowledge distillation template to improve the inference speed of heavy DNN models like BERT/GPT.","Extractive Machine Reading Comprehension. Given a pair of question and passage, predict the start and end positions of the answer spans in the passage."]},{"idx":6,"heading":"User Interface","paragraphs":["NeuronBlocks provides convenient user interface for users to build, train, and test DNN models. The details are described in the following.","[itemsep= -0.4em,topsep = 0.3em, align=left, labelsep=-0.6em, leftmargin=1.2em]","I/O interface. This part defines model input/output, such as training data, pre-trained models/embeddings, model saving path, etc.","Model Architecture interface. This is the key part of the configuration file, which defines the whole model architecture. Figure FIGREF19 shows an example of how to specify a model architecture using the blocks in NeuronBlocks. To be more specific, it consists of a list of layers/blocks to construct the architecture, where the blocks are supplied in the gallery of Block Zoo.","Training Parameters interface. In this part, the model optimizer as well as all other training hyper parameters are indicated."]},{"idx":7,"heading":"Workflow","paragraphs":["Figure FIGREF34 shows the workflow of building DNN models in NeuronBlocks. Users only need to write a JSON configuration file. They can either instantiate an existing template from Model Zoo, or construct a new architecture based on the blocks from Block Zoo. This configuration file is shared across training, test, and prediction. For model hyper-parameter tuning or architecture modification, users just need to change the JSON configuration file. Advanced users can also contribute novel customized blocks into Block Zoo, as long as they follow the same interface guidelines with the existing blocks. These new blocks can be further shared across all users for model architecture design. Moreover, NeuronBlocks has flexible platform support, such as GPU/CPU, GPU management platforms like PAI."]},{"idx":8,"heading":"Experiments","paragraphs":["To verify the performance of NeuronBlocks, we conducted extensive experiments for common NLP tasks on public data sets including CoNLL-2003 BIBREF14 , GLUE benchmark BIBREF13 , and WikiQA corpus BIBREF15 . The experimental results showed that the models built with NeuronBlocks can achieve reliable and competitive results on various tasks, with productivity greatly improved."]},{"idx":9,"heading":"Sequence Labeling","paragraphs":["For sequence labeling task, we evaluated NeuronBlocks on CoNLL-2003 BIBREF14 English NER dataset, following most works on the same task. This dataset includes four types of named entities, namely, PERSON, LOCATION, ORGANIZATION, and MISC. We adopted the BIOES tagging scheme instead of IOB, as many previous works indicated meaningful improvement with BIOES scheme BIBREF16 , BIBREF17 . Table TABREF28 shows the results on CoNLL-2003 Englist testb dataset, with 12 different combinations of network layers/blocks, such as word/character embedding, CNN/LSTM and CRF. The results suggest that the flexible combination of layers/blocks in NeuronBlocks can easily reproduce the performance of original models, with comparative or slightly better performance."]},{"idx":10,"heading":"GLUE Benchmark","paragraphs":["The General Language Understanding Evaluation (GLUE) benchmark BIBREF13 is a collection of natural language understanding tasks. We experimented on the GLUE benchmark tasks using BiLSTM and Attention based models. As shown in Table TABREF29 , the models built by NeuronBlocks can achieve competitive or even better results on GLUE tasks with minimal coding efforts."]},{"idx":11,"heading":"Knowledge Distillation","paragraphs":["We evaluated Knowledge Distillation task in NeuronBlocks on a dataset collected from one commercial search engine. We refer to this dataset as Domain Classification Dataset. Each sample in this dataset consists of two parts, i.e., a question and a binary label indicating whether the question belongs to a specific domain. Table TABREF36 shows the results, where Area Under Curve (AUC) metric is used as the performance evaluation criteria and Queries per Second (QPS) is used to measure inference speed. By knowledge distillation training approach, the student model by NeuronBlocks managed to get 23-27 times inference speedup with only small performance regression compared with BERTbase fine-tuned classifier."]},{"idx":12,"heading":"WikiQA","paragraphs":["The WikiQA corpus BIBREF15 is a publicly available dataset for open-domain question answering. This dataset contains 3,047 questions from Bing query logs, each associated with some candidate answer sentences from Wikipedia. We conducted experiments on WikiQA dataset using CNN, BiLSTM, and Attention based models. The results are shown in Table TABREF41 . The models built in NeuronBlocks achieved competitive or even better results with simple model configurations."]},{"idx":13,"heading":"Conclusion and Future Work","paragraphs":["In this paper, we introduce NeuronBlocks, a DNN toolkit for NLP tasks built on PyTorch. NeuronBlocks targets three types of engineers, and provides a two-layer solution to satisfy the requirements from all three types of users. To be more specific, the Model Zoo consists of various templates for the most common NLP tasks, while the Block Zoo supplies a gallery of alternative layers/modules for the networks. Such design achieves a balance between generality and flexibility. Extensive experiments have verified the effectiveness of this approach. NeuronBlocks has been widely used in a product team of a commercial search engine, and significantly improved the productivity for developing NLP DNN approaches.","As an open-source toolkit, we will further extend it in various directions. The following names a few examples."]},{"idx":14,"heading":"Figures and Tables","paragraphs":["Table 1: Task analysis of NLP DNN jobs submitted to a commercial centralized GPU cluster.","Figure 1: The overall framework of NeuronBlocks.","Figure 2: A Model architecture interface example of sequence labeling model in NeuronBlocks.","Table 2: NeuronBlocks results on CoNLL-2003 English NER testb dataset. The abbreviation (C-16)= (Chiu and Nichols, 2016), (L-16)= (Lample et al., 2016), (M-16)= (Ma and Hovy, 2016), (N)= (Yang et al., 2018), (P-17)= (Peters et al., 2017).","Table 3: NeuronBlocks?results on GLUE benchmark development sets. As described in (Wang et al., 2019), for CoLA, we report Matthews correlation. For QQP, we report accuracy and F1. For MNLI, we report accuracy averaged over the matched and mismatched development sets. For all other tasks we report accuracy. All values have been scaled by 100. Please note that results on the development sets are reported, since GLUE does not distribute labels for the test sets.","Figure 3: The workflow of NeuronBlocks.","Table 4: NeuronBlocks results on Knowledge Distillation task.","Table 5: NeuronBlocks results on WikiQA."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1905.11268","title":"Combating Adversarial Misspellings with Robust Word Recognition","sections":[{"idx":0,"heading":"Abstract","paragraphs":["To combat adversarial spelling mistakes, we propose placing a word recognition model in front of the downstream classifier. Our word recognition models build upon the RNN semicharacter architecture, introducing several new backoff strategies for handling rare and unseen words. Trained to recognize words corrupted by random adds, drops, swaps, and keyboard mistakes, our method achieves 32% relative (and 3.3% absolute) error reduction over the vanilla semi-character model. Notably, our pipeline confers robustness on the downstream classifier, outperforming both adversarial training and off-the-shelf spell checkers. Against a BERT model fine-tuned for sentiment analysis, a single adversarially-chosen character attack lowers accuracy from 90.3% to 45.8%. Our defense restores accuracy to 75% 1 . Surprisingly, better word recognition does not always entail greater robustness. Our analysis reveals that robustness also depends upon a quantity that we denote the sensitivity."]},{"idx":1,"heading":"Introduction","paragraphs":["Despite the rapid progress of deep learning techniques on diverse supervised learning tasks, these models remain brittle to subtle shifts in the data distribution. Even when the permissible changes are confined to barely-perceptible perturbations, training robust models remains an open challenge. Following the discovery that imperceptible attacks could cause image recognition models to misclassify examples BIBREF0 , a veritable sub-field has emerged in which authors iteratively propose attacks and countermeasures.","For all the interest in adversarial computer vision, these attacks are rarely encountered outside of academic research. However, adversarial misspellings constitute a longstanding real-world problem. Spammers continually bombard email servers, subtly misspelling words in efforts to evade spam detection while preserving the emails' intended meaning BIBREF1 , BIBREF2 . As another example, programmatic censorship on the Internet has spurred communities to adopt similar methods to communicate surreptitiously BIBREF3 .","In this paper, we focus on adversarially-chosen spelling mistakes in the context of text classification, addressing the following attack types: dropping, adding, and swapping internal characters within words. These perturbations are inspired by psycholinguistic studies BIBREF4 , BIBREF5 which demonstrated that humans can comprehend text altered by jumbling internal characters, provided that the first and last characters of each word remain unperturbed.","First, in experiments addressing both BiLSTM and fine-tuned BERT models, comprising four different input formats: word-only, char-only, word+char, and word-piece BIBREF6 , we demonstrate that an adversary can degrade a classifier's performance to that achieved by random guessing. This requires altering just two characters per sentence. Such modifications might flip words either to a different word in the vocabulary or, more often, to the out-of-vocabulary token UNK. Consequently, adversarial edits can degrade a word-level model by transforming the informative words to UNK. Intuitively, one might suspect that word-piece and character-level models would be less susceptible to spelling attacks as they can make use of the residual word context. However, our experiments demonstrate that character and word-piece models are in fact more vulnerable. We show that this is due to the adversary's effective capacity for finer grained manipulations on these models. While against a word-level model, the adversary is mostly limited to UNK-ing words, against a word-piece or character-level model, each character-level add, drop, or swap produces a distinct input, providing the adversary with a greater set of options.","Second, we evaluate first-line techniques including data augmentation and adversarial training, demonstrating that they offer only marginal benefits here, e.g., a BERT model achieving $90.3$ accuracy on a sentiment classification task, is degraded to $64.1$ by an adversarially-chosen 1-character swap in the sentence, which can only be restored to $69.2$ by adversarial training.","Third (our primary contribution), we propose a task-agnostic defense, attaching a word recognition model that predicts each word in a sentence given a full sequence of (possibly misspelled) inputs. The word recognition model's outputs form the input to a downstream classification model. Our word recognition models build upon the RNN-based semi-character word recognition model due to BIBREF7 . While our word recognizers are trained on domain-specific text from the task at hand, they often predict UNK at test time, owing to the small domain-specific vocabulary. To handle unobserved and rare words, we propose several backoff strategies including falling back on a generic word recognizer trained on a larger corpus. Incorporating our defenses, BERT models subject to 1-character attacks are restored to $88.3$ , $81.1$ , $78.0$ accuracy for swap, drop, add attacks respectively, as compared to $69.2$ , $63.6$ , and $50.0$ for adversarial training","Fourth, we offer a detailed qualitative analysis, demonstrating that a low word error rate alone is insufficient for a word recognizer to confer robustness on the downstream task. Additionally, we find that it is important that the recognition model supply few degrees of freedom to an attacker. We provide a metric to quantify this notion of sensitivity in word recognition models and study its relation to robustness empirically. Models with low sensitivity and word error rate are most robust."]},{"idx":2,"heading":"Related Work","paragraphs":["Several papers address adversarial attacks on NLP systems. Changes to text, whether word- or character-level, are all perceptible, raising some questions about what should rightly be considered an adversarial example BIBREF8 , BIBREF9 . BIBREF10 address the reading comprehension task, showing that by appending distractor sentences to the end of stories from the SQuAD dataset BIBREF11 , they could cause models to output incorrect answers. Inspired by this work, BIBREF12 demonstrate an attack that breaks entailment systems by replacing a single word with either a synonym or its hypernym. Recently, BIBREF13 investigated the problem of producing natural-seeming adversarial examples, noting that adversarial examples in NLP are often ungrammatical BIBREF14 .","In related work on character-level attacks, BIBREF8 , BIBREF15 explored gradient-based methods to generate string edits to fool classification and translation systems, respectively. While their focus is on efficient methods for generating adversaries, ours is on improving the worst case adversarial performance. Similarly, BIBREF9 studied how synthetic and natural noise affects character-level machine translation. They considered structure invariant representations and adversarial training as defenses against such noise. Here, we show that an auxiliary word recognition model, which can be trained on unlabeled data, provides a strong defense.","Spelling correction BIBREF16 is often viewed as a sub-task of grammatical error correction BIBREF17 , BIBREF18 . Classic methods rely on a source language model and a noisy channel model to find the most likely correction for a given word BIBREF19 , BIBREF20 . Recently, neural techniques have been applied to the task BIBREF7 , BIBREF21 , which model the context and orthography of the input together. Our work extends the ScRNN model of BIBREF7 ."]},{"idx":3,"heading":"Robust Word Recognition","paragraphs":["To tackle character-level adversarial attacks, we introduce a simple two-stage solution, placing a word recognition model ( $W$ ) before the downstream classifier ( $C$ ). Under this scheme, all inputs are classified by the composed model $C \\circ W$ . This modular approach, with $W$ and $C$ trained separately, offers several benefits: (i) we can deploy the same word recognition model for multiple downstream classification tasks/models; and (ii) we can train the word recognition model with larger unlabeled corpora.","Against adversarial mistakes, two important factors govern the robustness of this combined model: $W$ 's accuracy in recognizing misspelled words and $W$ 's sensitivity to adversarial perturbations on the same input. We discuss these aspects in detail below."]},{"idx":4,"heading":"ScRNN with Backoff","paragraphs":["We now describe semi-character RNNs for word recognition, explain their limitations, and suggest techniques to improve them.","Inspired by the psycholinguistic studies BIBREF5 , BIBREF4 , BIBREF7 proposed a semi-character based RNN (ScRNN) that processes a sentence of words with misspelled characters, predicting the correct words at each step. Let $s = \\lbrace w_1, w_2, \\dots , w_n\\rbrace $ denote the input sentence, a sequence of constituent words $w_i$ . Each input word ( $w_i$ ) is represented by concatenating (i) a one hot vector of the first character ( $\\mathbf {w_{i1}}$ ); (ii) a one hot representation of the last character ( $\\mathbf {w_{il}}$ , where $l$ is the length of word $w_i$ ); and (iii) a bag of characters representation of the internal characters ( $\\sum _{j=2}^{l-1}\\mathbf {w_{ij}})$ . ScRNN treats the first and the last characters individually, and is agnostic to the ordering of the internal characters. Each word, represented accordingly, is then fed into a BiLSTM cell. At each sequence step, the training target is the correct corresponding word (output dimension equal to vocabulary size), and the model is optimized with cross-entropy loss.","While BIBREF7 demonstrate strong word recognition performance, a drawback of their evaluation setup is that they only attack and evaluate on the subset of words that are a part of their training vocabulary. In such a setting, the word recognition performance is unreasonably dependent on the chosen vocabulary size. In principle, one can design models to predict (correctly) only a few chosen words, and ignore the remaining majority and still reach 100% accuracy. For the adversarial setting, rare and unseen words in the wild are particularly critical, as they provide opportunities for the attackers. A reliable word-recognizer should handle these cases gracefully. Below, we explore different ways to back off when the ScRNN predicts UNK (a frequent outcome for rare and unseen words):","Pass-through: word-recognizer passes on the (possibly misspelled) word as is.","Backoff to neutral word: Alternatively, noting that passing $\\colorbox {gray!20}{\\texttt {UNK}}$ -predicted words through unchanged exposes the downstream model to potentially corrupted text, we consider backing off to a neutral word like `a', which has a similar distribution across classes.","Backoff to background model: We also consider falling back upon a more generic word recognition model trained upon a larger, less-specialized corpus whenever the foreground word recognition model predicts UNK. Figure 1 depicts this scenario pictorially.","Empirically, we find that the background model (by itself) is less accurate, because of the large number of words it is trained to predict. Thus, it is best to train a precise foreground model on an in-domain corpus and focus on frequent words, and then to resort to a general-purpose background model for rare and unobserved words. Next, we delineate our second consideration for building robust word-recognizers."]},{"idx":5,"heading":"Model Sensitivity","paragraphs":["In computer vision, an important factor determining the success of an adversary is the norm constraint on the perturbations allowed to an image ( $|| \\bf x - \\bf x^{\\prime }||_{\\infty } < \\epsilon $ ). Higher values of $\\epsilon $ lead to a higher chance of mis-classification for at least one $\\bf x^{\\prime }$ . Defense methods such as quantization BIBREF22 and thermometer encoding BIBREF23 try to reduce the space of perturbations available to the adversary by making the model invariant to small changes in the input.","In NLP, we often get such invariance for free, e.g., for a word-level model, most of the perturbations produced by our character-level adversary lead to an UNK at its input. If the model is robust to the presence of these UNK tokens, there is little room for an adversary to manipulate it. Character-level models, on the other hand, despite their superior performance in many tasks, do not enjoy such invariance. This characteristic invariance could be exploited by an attacker. Thus, to limit the number of different inputs to the classifier, we wish to reduce the number of distinct word recognition outputs that an attacker can induce, not just the number of words on which the model is “fooled”. We denote this property of a model as its sensitivity.","We can quantify this notion for a word recognition system $W$ as the expected number of unique outputs it assigns to a set of adversarial perturbations. Given a sentence $s$ from the set of sentences $\\mathcal {S}$ , let $A(s) = {s_1}^{\\prime } , {s_2}^{\\prime }, \\dots , {s_n}^{\\prime }$ denote the set of $n$ perturbations to it under attack type $A$ , and let $V$ be the function that maps strings to an input representation for the downstream classifier. For a word level model, $V$ would transform sentences to a sequence of word ids, mapping OOV words to the same UNK ID. Whereas, for a char (or word+char, word-piece) model, $V$ would map inputs to a sequence of character IDs. Formally, sensitivity is defined as","$$S_{W,V}^A=\\mathbb {E}_{s}\\left[\\frac{\\#_{u}(V \\circ W({s_1}^{\\prime }), \\dots , V \\circ W({s_n}^{\\prime }))}{n}\\right] ,$$ (Eq. 12)","where $V \\circ W (s_i)$ returns the input representation (of the downstream classifier) for the output string produced by the word-recognizer $W$ using $s_i$ and $\\#_{u}(\\cdot )$ counts the number of unique arguments.","Intuitively, we expect a high value of $S_{W, V}^A$ to lead to a lower robustness of the downstream classifier, since the adversary has more degrees of freedom to attack the classifier. Thus, when using word recognition as a defense, it is prudent to design a low sensitivity system with a low error rate. However, as we will demonstrate, there is often a trade-off between sensitivity and error rate."]},{"idx":6,"heading":"Synthesizing Adversarial Attacks","paragraphs":["Suppose we are given a classifier $C: \\mathcal {S} \\rightarrow \\mathcal {Y}$ which maps natural language sentences $s \\in \\mathcal {S}$ to a label from a predefined set $y \\in \\mathcal {Y}$ . An adversary for this classifier is a function $A$ which maps a sentence $s$ to its perturbed versions $\\lbrace s^{\\prime }_1, s^{\\prime }_2, \\ldots , s^{\\prime }_{n}\\rbrace $ such that each $s^{\\prime }_i$ is close to $s$ under some notion of distance between sentences. We define the robustness of classifier $C$ to the adversary $A$ as:","$$R_{C,A} = \\mathbb {E}_s \\left[\\min _{s^{\\prime } \\in A(s)} \\mathbb {1}[C(s^{\\prime }) = y]\\right],$$ (Eq. 14)","where $y$ represents the ground truth label for $s$ . In practice, a real-world adversary may only be able to query the classifier a few times, hence $R_{C,A}$ represents the worst-case adversarial performance of $C$ . Methods for generating adversarial examples, such as HotFlip BIBREF8 , focus on efficient algorithms for searching the $\\min $ above. Improving $R_{C,A}$ would imply better robustness against all these methods.","We explore adversaries which perturb sentences with four types of character-level edits:","(1) Swap: swapping two adjacent internal characters of a word. (2) Drop: removing an internal character of a word. (3) Keyboard: substituting an internal character with adjacent characters of QWERTY keyboard (4) Add: inserting a new character internally in a word. In line with the psycholinguistic studies BIBREF5 , BIBREF4 , to ensure that the perturbations do not affect human ability to comprehend the sentence, we only allow the adversary to edit the internal characters of a word, and not edit stopwords or words shorter than 4 characters.","For 1-character attacks, we try all possible perturbations listed above until we find an adversary that flips the model prediction. For 2-character attacks, we greedily fix the edit which had the least confidence among 1-character attacks, and then try all the allowed perturbations on the remaining words. Higher order attacks can be performed in a similar manner. The greedy strategy reduces the computation required to obtain higher order attacks, but also means that the robustness score is an upper bound on the true robustness of the classifier."]},{"idx":7,"heading":"Experiments and Results","paragraphs":["In this section, we first discuss our experiments on the word recognition systems."]},{"idx":8,"heading":"Word Error Correction","paragraphs":["Data: We evaluate the spell correctors from § \"Robust Word Recognition\" on movie reviews from the Stanford Sentiment Treebank (SST) BIBREF24 . The SST dataset consists of 8544 movie reviews, with a vocabulary of over 16K words. As a background corpus, we use the IMDB movie reviews BIBREF25 , which contain 54K movie reviews, and a vocabulary of over 78K words. The two datasets do not share any reviews in common. The spell-correction models are evaluated on their ability to correct misspellings. The test setting consists of reviews where each word (with length $\\ge 4$ , barring stopwords) is attacked by one of the attack types (from swap, add, drop and keyboard attacks). In the all attack setting, we mix all attacks by randomly choosing one for each word. This most closely resembles a real world attack setting.","In addition to our word recognition models, we also compare to After The Deadline (ATD), an open-source spell corrector. We found ATD to be the best freely-available corrector. We refer the reader to BIBREF7 for comparisons of ScRNN to other anonymized commercial spell checkers.","For the ScRNN model, we use a single-layer Bi-LSTM with a hidden dimension size of 50. The input representation consists of 198 dimensions, which is thrice the number of unique characters (66) in the vocabulary. We cap the vocabulary size to 10K words, whereas we use the entire vocabulary of 78470 words when we backoff to the background model. For training these networks, we corrupt the movie reviews according to all attack types, i.e., applying one of the 4 attack types to each word, and trying to reconstruct the original words via cross entropy loss.","We calculate the word error rates (WER) of each of the models for different attacks and present our findings in Table 2 . Note that ATD incorrectly predicts $11.2$ words for every 100 words (in the `all' setting), whereas, all of the backoff variations of the ScRNN reconstruct better. The most accurate variant involves backing off to the background model, resulting in a low error rate of $6.9\\%$ , leading to the best performance on word recognition. This is a $32\\%$ relative error reduction compared to the vanilla ScRNN model with a pass-through backoff strategy. We can attribute the improved performance to the fact that there are $5.25\\%$ words in the test corpus that are unseen in the training corpus, and are thus only recoverable by backing off to a larger corpus. Notably, only training on the larger background corpus does worse, at $8.7\\%$ , since the distribution of word frequencies is different in the background corpus compared to the foreground corpus."]},{"idx":9,"heading":"Robustness to adversarial attacks","paragraphs":["We use sentiment analysis and paraphrase detection as downstream tasks, as for these two tasks, 1-2 character edits do not change the output labels.","For sentiment classification, we systematically study the effect of character-level adversarial attacks on two architectures and four different input formats. The first architecture encodes the input sentence into a sequence of embeddings, which are then sequentially processed by a BiLSTM. The first and last states of the BiLSTM are then used by the softmax layer to predict the sentiment of the input. We consider three input formats for this architecture: (1) Word-only: where the input words are encoded using a lookup table; (2) Char-only: where the input words are encoded using a separate single-layered BiLSTM over their characters; and (3) Word $+$ Char: where the input words are encoded using a concatenation of (1) and (2) .","The second architecture uses the fine-tuned BERT model BIBREF26 , with an input format of word-piece tokenization. This model has recently set a new state-of-the-art on several NLP benchmarks, including the sentiment analysis task we consider here. All models are trained and evaluated on the binary version of the sentence-level Stanford Sentiment Treebank BIBREF24 dataset with only positive and negative reviews.","We also consider the task of paraphrase detection. Here too, we make use of the fine-tuned BERT BIBREF26 , which is trained and evaluated on the Microsoft Research Paraphrase Corpus (MRPC) BIBREF27 .","Two common methods for dealing with adversarial examples include: (1) data augmentation (DA) BIBREF28 ; and (2) adversarial training (Adv) BIBREF29 . In DA, the trained model is fine-tuned after augmenting the training set with an equal number of examples randomly attacked with a 1-character edit. In Adv, the trained model is fine-tuned with additional adversarial examples (selected at random) that produce incorrect predictions from the current-state classifier. The process is repeated iteratively, generating and adding newer adversarial examples from the updated classifier model, until the adversarial accuracy on dev set stops improving.","In Table 3 , we examine the robustness of the sentiment models under each attack and defense method. In the absence of any attack or defense, BERT (a word-piece model) performs the best ( $90.3\\%$ ) followed by word+char models ( $80.5\\%$ ), word-only models ( $79.2\\%$ ) and then char-only models ( $70.3\\%$ ). However, even single-character attacks (chosen adversarially) can be catastrophic, resulting in a significantly degraded performance of $46\\%$ , $57\\%$ , $59\\%$ and $33\\%$ , respectively under the `all' setting.","Intuitively, one might suppose that word-piece and character-level models would be more robust to such attacks given they can make use of the remaining context. However, we find that they are the more susceptible. To see why, note that the word `beautiful' can only be altered in a few ways for word-only models, either leading to an UNK or an existing vocabulary word, whereas, word-piece and character-only models treat each unique character combination differently. This provides more variations that an attacker can exploit. Following similar reasoning, add and key attacks pose a greater threat than swap and drop attacks. The robustness of different models can be ordered as word-only $>$ word+char $>$ char-only $\\sim $ word-piece, and the efficacy of different attacks as add $>$ key $>$ drop $>$ swap.","Next, we scrutinize the effectiveness of defense methods when faced against adversarially chosen attacks. Clearly from table 3 , DA and Adv are not effective in this case. We observed that despite a low training error, these models were not able to generalize to attacks on newer words at test time. ATD spell corrector is the most effective on keyboard attacks, but performs poorly on other attack types, particularly the add attack strategy.","The ScRNN model with pass-through backoff offers better protection, bringing back the adversarial accuracy within $5\\%$ range for the swap attack. It is also effective under other attack classes, and can mitigate the adversarial effect in word-piece models by $21\\%$ , character-only models by $19\\%$ , and in word, and word+char models by over $4.5\\%$ . This suggests that the direct training signal of word error correction is more effective than the indirect signal of sentiment classification available to DA and Adv for model robustness.","We observe additional gains by using background models as a backoff alternative, because of its lower word error rate (WER), especially, under the swap and drop attacks. However, these gains do not consistently translate in all other settings, as lower WER is necessary but not sufficient. Besides lower error rate, we find that a solid defense should furnish the attacker the fewest options to attack, i.e. it should have a low sensitivity.","As we shall see in section § \"Understanding Model Sensitivity\" , the backoff neutral variation has the lowest sensitivity due to mapping UNK predictions to a fixed neutral word. Thus, it results in the highest robustness on most of the attack types for all four model classes.","Table 4 shows the accuracy of BERT on 200 examples from the dev set of the MRPC paraphrase detection task under various attack and defense settings. We re-trained the ScRNN model variants on the MRPC training set for these experiments. Again, we find that simple 1-2 character attacks can bring down the accuracy of BERT significantly ( $89\\%$ to $31\\%$ ). Word recognition models can provide an effective defense, with both our pass-through and neutral variants recovering most of the accuracy. While the neutral backoff model is effective on 2-char attacks, it hurts performance in the no attack setting, since it incorrectly modifies certain correctly spelled entity names. Since the two variants are already effective, we did not train a background model for this task."]},{"idx":10,"heading":"Understanding Model Sensitivity","paragraphs":["To study model sensitivity, for each sentence, we perturb one randomly-chosen word and replace it with all possible perturbations under a given attack type. The resulting set of perturbed sentences is then fed to the word recognizer (whose sensitivity is to be estimated). As described in equation 12 , we count the number of unique predictions from the output sentences. Two corrections are considered unique if they are mapped differently by the downstream classifier.","The neutral backoff variant has the lowest sensitivity (Table 5 ). This is expected, as it returns a fixed neutral word whenever the ScRNN predicts an UNK, therefore reducing the number of unique outputs it predicts. Open vocabulary (i.e. char-only, word+char, word-piece) downstream classifiers consider every unique combination of characters differently, whereas word-only classifiers internally treat all out of vocabulary (OOV) words alike. Hence, for char-only, word+char, and word-piece models, the pass-through version is more sensitive than the background variant, as it passes words as is (and each combination is considered uniquely). However, for word-only models, pass-through is less sensitive as all the OOV character combinations are rendered identical.","Ideally, a preferred defense is one with low sensitivity and word error rate. In practice, however, we see that a low error rate often comes at the cost of sensitivity. We see this trade-off in Figure 2 , where we plot WER and sensitivity on the two axes, and depict the robustness when using different backoff variants. Generally, sensitivity is the more dominant factor out of the two, as the error rates of the considered variants are reasonably low.","We verify if the sentiment (of the reviews) is preserved with char-level attacks. In a human study with 50 attacked (and subsequently misclassified), and 50 unchanged reviews, it was noted that 48 and 49, respectively, preserved the sentiment."]},{"idx":11,"heading":"Conclusion","paragraphs":["As character and word-piece inputs become commonplace in modern NLP pipelines, it is worth highlighting the vulnerability they add. We show that minimally-doctored attacks can bring down accuracy of classifiers to random guessing. We recommend word recognition as a safeguard against this and build upon RNN-based semi-character word recognizers. We discover that when used as a defense mechanism, the most accurate word recognition models are not always the most robust against adversarial attacks. Additionally, we highlight the need to control the sensitivity of these models to achieve high robustness."]},{"idx":12,"heading":"Acknowledgements","paragraphs":["The authors are grateful to Graham Neubig, Eduard Hovy, Paul Michel, Mansi Gupta, and Antonios Anastasopoulos for suggestions and feedback."]},{"idx":13,"heading":"Figures and Tables","paragraphs":["Table 1: Adversarial spelling mistakes inducing sentiment misclassification and word-recognition defenses.","Figure 1: A schematic sketch of our proposed word recognition system, consisting of a foreground and a background model. We train the foreground model on the smaller, domain-specific dataset, and the background model on a larger dataset (e.g., the IMDB movie corpus). We train both models to reconstruct the correct word from the orthography and context of the individual words, using synthetically corrupted inputs during training. Subse-","Table 2: Word Error Rates (WER) of ScRNN with each backoff strategy, plus ATD and an ScRNN trained only on the background corpus (78K vocabulary) The error rates include 5.25% OOV words.","Table 3: Accuracy of various classification models, with and without defenses, under adversarial attacks. Even 1-character attacks significantly degrade classifier performance. Our defenses confer robustness, recovering over 76% of the original accuracy, under the ‘all’ setting for all four model classes.","Table 4: Accuracy of BERT, with and without defenses, on MRPC when attacked under the ‘all’ attack setting.","Figure 2: Effect of sensitivity and word error rate on robustness (depicted by the bubble sizes) in word-only models (left) and char-only models (right).","Table 5: Sensitivity values for word recognizers. Neutral backoff shows lowest sensitivity."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1906.01076","title":"Episodic Memory in Lifelong Language Learning","sections":[{"idx":0,"heading":"Abstract","paragraphs":["We introduce a lifelong language learning setup where a model needs to learn from a stream of text examples without any dataset identifier. We propose an episodic memory model that performs sparse experience replay and local adaptation to mitigate catastrophic forgetting in this setup. Experiments on text classification and question answering demonstrate the complementary benefits of sparse experience replay and local adaptation to allow the model to continuously learn from new datasets. We also show that the space complexity of the episodic memory module can be reduced significantly (~50-90%) by randomly choosing which examples to store in memory with a minimal decrease in performance. We consider an episodic memory component as a crucial building block of general linguistic intelligence and see our model as a first step in that direction."]},{"idx":1,"heading":"Introduction","paragraphs":["The ability to continuously learn and accumulate knowledge throughout a lifetime and reuse it effectively to adapt to a new problem quickly is a hallmark of general intelligence. State-of-the-art machine learning models work well on a single dataset given enough training examples, but they often fail to isolate and reuse previously acquired knowledge when the data distribution shifts (e.g., when presented with a new dataset)—a phenomenon known as catastrophic forgetting BIBREF0 , BIBREF1 .","The three main approaches to address catastrophic forgetting are based on: (i) augmenting the loss function that is being minimized during training with extra terms (e.g., a regularization term, an optimization constraint) to prevent model parameters learned on a new dataset from significantly deviating from parameters learned on previously seen datasets BIBREF2 , BIBREF3 , BIBREF4 , (ii) adding extra learning phases such as a knowledge distillation phase, an experience replay BIBREF5 , BIBREF6 , and (iii) augmenting the model with an episodic memory module BIBREF7 . Recent methods have shown that these approaches can be combined—e.g., by defining optimization constraints using samples from the episodic memory BIBREF8 , BIBREF9 .","In language learning, progress in unsupervised pretraining BIBREF10 , BIBREF11 , BIBREF12 has driven advances in many language understanding tasks BIBREF13 , BIBREF14 . However, these models have been shown to require a lot of in-domain training examples, rapidly overfit to particular datasets, and are prone to catastrophic forgetting BIBREF15 , making them unsuitable as a model of general linguistic intelligence.","In this paper, we investigate the role of episodic memory for learning a model of language in a lifelong setup. We propose to use such a component for sparse experience replay and local adaptation to allow the model to continually learn from examples drawn from different data distributions. In experience replay, we randomly select examples from memory to retrain on. Our model only performs experience replay very sparsely to consolidate newly acquired knowledge with existing knowledge in the memory into the model. We show that a 1% experience replay to learning new examples ratio is sufficient. Such a process bears some similarity to memory consolidation in human learning BIBREF16 . In local adaptation, we follow Memory-based Parameter Adaptation BIBREF7 and use examples retrieved from memory to update model parameters used to make a prediction of a particular test example.","Our setup is different than a typical lifelong learning setup. We assume that the model only makes one pass over the training examples, similar to BIBREF9 . However, we also assume neither our training nor test examples have dataset identifying information (e.g., a dataset identity, a dataset descriptor). Our experiments focus on lifelong language learning on two tasks—text classification and question answering. BIBREF17 show that many language processing tasks (e.g., classification, summarization, natural language inference, etc.) can be formulated as a question answering problem. We argue that our lifelong language learning setup—where a model is presented with question-answer examples without an explicit identifier about which dataset (distribution) the examples come from—is a more realistic setup to learn a general linguistic intelligence model.","Our main contributions in this paper are:"]},{"idx":2,"heading":"Model","paragraphs":["We consider a continual (lifelong) learning setup where a model needs to learn from a stream of training examples INLINEFORM0 . We assume that all our training examples in the series come from multiple datasets of the same task (e.g., a text classification task, a question answering task), and each dataset comes one after the other. Since all examples come from the same task, the same model can be used to make predictions on all examples. A crucial difference between our continual learning setup and previous work is that we do not assume that each example comes with a dataset descriptor (e.g., a dataset identity). As a result, the model does not know which dataset an example comes from and when a dataset boundary has been crossed during training. The goal of learning is to find parameters INLINEFORM1 that minimize the negative log probability of training examples under our model: INLINEFORM2","Our model consists of three main components: (i) an example encoder, (ii) a task decoder, and (iii) an episodic memory module. Figure FIGREF6 shows an illustration of our complete model. We describe each component in detail in the following."]},{"idx":3,"heading":"Example Encoder","paragraphs":["Our encoder is based on the Transformer architecture BIBREF19 . We use the state-of-the-art text encoder BERT BIBREF12 to encode our input INLINEFORM0 . BERT is a large Transformer pretrained on a large unlabeled corpus on two unsupervised tasks—masked language modeling and next sentence prediction. Other architectures such as recurrent neural networks or convolutional neural networks can also be used as the example encoder.","In text classification, INLINEFORM0 is a document to be classified; BERT produces a vector representation of each token in INLINEFORM1 , which includes a special beginning-of-document symbol CLS as INLINEFORM2 . In question answering, INLINEFORM3 is a concatenation of a context paragraph INLINEFORM4 and a question INLINEFORM5 separated by a special separator symbol SEP."]},{"idx":4,"heading":"Task Decoder","paragraphs":["In text classification, following the original BERT model, we take the representation of the first token INLINEFORM0 from BERT (i.e., the special beginning-of-document symbol) and add a linear transformation and a softmax layer to predict the class of INLINEFORM1 . INLINEFORM2","Note that since there is no dataset descriptor provided to our model, this decoder is used to predict all classes in all datasets, which we assume to be known in advance.","For question answering, our decoder predicts an answer span—the start and end indices of the correct answer in the context. Denote the length of the context paragraph by INLINEFORM0 , and INLINEFORM1 . Denote the encoded representation of the INLINEFORM2 -th token in the context by INLINEFORM3 . Our decoder has two sets of parameters: INLINEFORM4 and INLINEFORM5 . The probability of each context token being the start of the answer is computed as: INLINEFORM6","We compute the probability of the end index of the answer analogously using INLINEFORM0 . The predicted answer is the span with the highest probability after multiplying the start and end probabilities. We take into account that the start index of an answer needs to precede its end index by setting the probabilities of invalid spans to zero."]},{"idx":5,"heading":"Episodic Memory","paragraphs":["Our model is augmented with an episodic memory module that stores previously seen examples throughout its lifetime. The episodic memory module is used for sparse experience replay and local adaptation to prevent catastrophic forgetting and encourage positive transfer. We first describe the architecture of our episodic memory module, before discussing how it is used at training and inference (prediction) time in § SECREF3 .","The module is a key-value memory block. We obtain the key representation of INLINEFORM0 (denoted by INLINEFORM1 ) using a key network—which is a pretrained BERT model separate from the example encoder. We freeze the key network to prevent key representations from drifting as data distribution changes (i.e. the problem that the key of a test example tends to be closer to keys of recently stored examples).","For text classification, our key is an encoded representation of the first token of the document to be classified, so INLINEFORM0 (i.e., the special beginning-of-document symbol). For question answering, we first take the question part of the input INLINEFORM1 . We encode it using the key network and take the first token as the key vector INLINEFORM2 . For both tasks, we store the input and the label INLINEFORM3 as its associated memory value.","If we assume that the model has unlimited capacity, we can write all training examples into the memory. However, this assumption is unrealistic in practice. We explore a simple writing strategy that relaxes this constraint based on random write. In random write, we randomly decide whether to write a newly seen example into the memory with some probability. We find that this is a strong baseline that outperforms other simple methods based on surprisal BIBREF20 and the concept of forgettable examples BIBREF21 in our preliminary experiments. We leave investigations of more sophisticated selection methods to future work.","Our memory has two retrieval mechanisms: (i) random sampling and (ii) INLINEFORM0 -nearest neighbors. We use random sampling to perform sparse experience replay and INLINEFORM1 -nearest neighbors for local adaptation, which are described in § SECREF3 below."]},{"idx":6,"heading":"Training and Inference","paragraphs":["Algorithm UID14 and Algorithm UID14 outline our overall training and inference procedures."]},{"idx":7,"heading":"Experiments","paragraphs":["In this section, we evaluate our proposed model against several baselines on text classification and question answering tasks."]},{"idx":8,"heading":"Datasets","paragraphs":["We use publicly available text classification datasets from BIBREF22 to evaluate our models (http://goo.gl/JyCnZq). This collection of datasets includes text classification datasets from diverse domains such as news classification (AGNews), sentiment analysis (Yelp, Amazon), Wikipedia article classification (DBPedia), and questions and answers categorization (Yahoo). Specifically, we use AGNews (4 classes), Yelp (5 classes), DBPedia (14 classes), Amazon (5 classes), and Yahoo (10 classes) datasets. Since classes for Yelp and Amazon datasets have similar semantics (product ratings), we merge the classes for these two datasets. In total, we have 33 classes in our experiments. These datasets have varying sizes. For example, AGNews is ten times smaller than Yahoo. We create a balanced version all datasets used in our experiments by randomly sampling 115,000 training examples and 7,600 test examples from all datasets (i.e., the size of the smallest training and test sets). We leave investigations of lifelong learning in unbalanced datasets to future work. In total, we have 575,000 training examples and 38,000 test examples.","We use three question answering datasets: SQuAD 1.1 BIBREF23 , TriviaQA BIBREF24 , and QuAC BIBREF25 . These datasets have different characteristics. SQuAD is a reading comprehension dataset constructed from Wikipedia articles. It includes almost 90,000 training examples and 10,000 validation examples. TriviaQA is a dataset with question-answer pairs written by trivia enthusiasts and evidence collected retrospectively from Wikipedia and the Web. There are two sections of TriviaQA, Web and Wikipedia, which we treat as separate datasets. The Web section contains 76,000 training examples and 10,000 (unverified) validation examples, whereas the Wikipedia section has about 60,000 training examples and 8,000 validation examples. QuAC is an information-seeking dialog-style dataset where a student asks questions about a Wikipedia article and a teacher answers with a short excerpt from the article. It has 80,000 training examples and approximately 7,000 validation examples."]},{"idx":9,"heading":"Models","paragraphs":["We compare the following models in our experiments:","Enc-Dec: a standard encoder-decoder model without any episodic memory module.","A-GEM BIBREF9 : Average Gradient Episodic Memory model that defines constraints on the gradients that are used to update model parameters based on retrieved examples from the memory. In its original formulation, A-GEM requires dataset identifiers and randomly samples examples from previous datasets. We generalize it to the setting without dataset identities by randomly sampling from the episodic memory module at fixed intervals, similar to our method.","Replay: a model that uses stored examples for sparse experience replay without local adaptation. We perform experience replay by sampling 100 examples from the memory and perform a gradient update after every 10,000 training steps, which gives us a 1% replay rate.","MbPA BIBREF7 : an episodic memory model that uses stored examples for local adaptation without sparse experience replay. The original MbPA formulation has a trainable key network. Our MbPA baseline uses a fixed key network since MbPA with a trainable key network performs significantly worse.","MbPA INLINEFORM0 : an episodic memory model with randomly retrieved examples for local adaptation (no key network).","MbPA++: our episodic memory model described in § SECREF2 .","MTL: a multitask model trained on all datasets jointly, used as a performance upper bound."]},{"idx":10,"heading":"Implementation Details","paragraphs":["We use a pretrained INLINEFORM0 model BIBREF12 as our example encoder and key network. INLINEFORM1 has 12 Transformer layers, 12 self-attention heads, and 768 hidden dimensions (110M parameters in total). We use the default BERT vocabulary in our experiments.","We use Adam BIBREF26 as our optimizer. We set dropout BIBREF27 to 0.1 and INLINEFORM0 in Eq. EQREF16 to 0.001. We set the base learning rate to INLINEFORM1 (based on preliminary experiments, in line with the suggested learning rate for using BERT). For text classification, we use a training batch of size 32. For question answering, the batch size is 8. The only hyperparameter that we tune is the local adaptation learning rate INLINEFORM2 . We set the number of neighbors INLINEFORM3 and the number of local adaptation steps INLINEFORM4 . We show results with other INLINEFORM5 and sensitivity to INLINEFORM6 in § SECREF38 .","For each experiment, we use 4 Intel Skylake x86-64 CPUs at 2 GHz, 1 Nvidia Tesla V100 GPU, and 20 GB of RAM."]},{"idx":11,"heading":"Results","paragraphs":["The models are trained in one pass on concatenated training sets, and evaluated on the union of the test sets. To ensure robustness of models to training dataset orderings, we evaluate on four different orderings (chosen randomly) for each task. As the multitask model has no inherent dataset ordering, we report results on four different shufflings of combined training examples. We show the exact orderings in Appendix SECREF6 . We tune the local adaptation learning rate using the first dataset ordering for each task and only run the best setting on the other orderings.","A main difference between these two tasks is that in text classification the model acquires knowledge about new classes as training progresses (i.e., only a subset of the classes that corresponds to a particular dataset are seen at each training interval), whereas in question answering the span predictor works similarly across datasets.","Table TABREF33 provides a summary of our main results. We report (macro-averaged) accuracy for classification and INLINEFORM0 score for question answering. We provide complete per-dataset (non-averaged) results in Appendix SECREF7 . Our results show that A-GEM outperforms the standard encoder-decoder model Enc-Dec, although it is worse than MbPA on both tasks. Local adaptation (MbPA) and sparse experience replay (Replay) help mitigate catastrophic forgetting compared to Enc-Dec, but a combination of them is needed to achieve the best performance (MbPA++).","Our experiments also show that retrieving relevant examples from memory is crucial to ensure that the local adaptation phase is useful. Comparing the results from MbPA++ and MbPA INLINEFORM0 , we can see that the model that chooses neighbors randomly is significantly worse than the model that finds and uses similar examples for local adaptation. We emphasize that having a fixed key network is crucial to prevent representation drift. The original MbPA formulation that updates the key network during training results in a model that only performs slightly better than MbPA INLINEFORM1 in our preliminary experiments. Our results suggest that our best model can be improved further by choosing relevant examples for sparse experience replay as well. We leave investigations of such methods to future work.","Comparing to the performance of the multitask model MTL—which is as an upper bound on achievable performance—we observe that there is still a gap between continual models and the multitask model. MbPA++ has the smallest performance gap. For text classification, MbPA++ outperforms single-dataset models in terms of averaged performance (70.6 vs. 60.7), demonstrating the success of positive transfer. For question answering, MbPA++ still lags behind single dataset models (62.0 vs. 66.0). Note that the collection of single-dataset models have many more parameters since there is a different set of model parameters per dataset. See Appendix SECREF8 for detailed results of multitask and single-dataset models.","Figure FIGREF34 shows INLINEFORM0 score and accuracy of various models on the test set corresponding to the first dataset seen during training as the models are trained on more datasets. The figure illustrates how well each model retains its previously acquired knowledge as it learns new knowledge. We can see that MbPA++ is consistently better compared to other methods."]},{"idx":12,"heading":"Analysis","paragraphs":["Our results in § SECREF30 assume that we can store all examples in memory (for all models, including the baselines). We investigate variants of MbPA++ that store only 50% and 10% of training examples. We randomly decide whether to write an example to memory or not (with probability 0.5 or 0.1). We show the results in Table TABREF42 . The results demonstrate that while the performance of the model degrades as the number of stored examples decreases, the model is still able to maintain a reasonably high performance even with only 10% memory capacity of the full model.","We investigate the effect of the number of retrieved examples for local adaptation to the performance of the model in Table TABREF42 . In both tasks, the model performs better as the number of neighbors increases. Recall that the goal of the local adaptation phase is to shape the output distribution of a test example to peak around relevant classes (or spans) based on retrieved examples from the memory. As a result, it is reasonable for the performance of the model to increase with more neighbors (up to a limit) given a key network that can reliably compute similarities between the test example and stored examples in memory and a good adaptation method.","Training MbPA++ takes as much time as training an encoder-decoder model without an episodic memory module since experience replay is performed sparsely (i.e., every 10,000 steps) with only 100 examples. This cost is negligible in practice and we observe no significant difference in terms of wall clock time to the vanilla encoder-decoder baseline. MbPA++ has a higher space complexity for storing seen examples, which could be controlled by limiting the memory capacity.","At inference time, MbPA++ requires a local adaptation phase and is thus slower than methods without local adaptation. This can be seen as a limitation of MbPA++ (and MbPA). One way to speed it up is to parallelize predictions across test examples, since each prediction is independent of others. We set the number of local adaptation steps INLINEFORM0 in our experiments. Figure FIGREF44 shows INLINEFORM1 is needed to converge to an optimal performance.","Comparing MBpA++ to other episodic memory models, MBpA has roughly the same time and space complexity as MBpA++. A-GEM, on the other hand, is faster at prediction time (no local adaptation), although at training time it is slower due to extra projection steps and uses more memory since it needs to store two sets of gradients (one from the current batch, and one from samples from the memory). We find that this cost is not negligible when using a large encoder such as BERT.","We show examples of retrieved neighbors from our episodic memory model in Appendix SECREF9 . We observe that the model manages to retrieve examples that are both syntactically and semantically related to a given query derived from a test example."]},{"idx":13,"heading":"Conclusion","paragraphs":["We introduced a lifelong language learning setup and presented an episodic memory model that performs sparse experience replay and local adaptation to continuously learn and reuse previously acquired knowledge. Our experiments demonstrate that our proposed method mitigates catastrophic forgetting and outperforms baseline methods on text classification and question answering."]},{"idx":14,"heading":"Dataset Order","paragraphs":["We use the following dataset orders (chosen randomly) for text classification:","For question answering, the orders are:"]},{"idx":15,"heading":"Full Results","paragraphs":["We show per-dataset breakdown of results in Table TABREF33 in Table TABREF54 and Table TABREF55 for text classification and question answering respectively."]},{"idx":16,"heading":"Single Dataset Models","paragraphs":["We show results of a single dataset model that is only trained on a particular dataset in Table TABREF56 ."]},{"idx":17,"heading":"Retrieved Examples","paragraphs":["We show examples of retrieved examples from memory given a test example in Table TABREF57 ."]},{"idx":18,"heading":"Figures and Tables","paragraphs":["Figure 1: An illustration of our model and how it interacts with the key-value memory module during training (left) and inference (right). During training, newly seen examples are used to update the base model and stored in the memory. At certain intervals, we sample examples from the memory and perform gradient updates on the base model (experience replay). During inference, we retrieve examples whose keys are similar to a test example under consideration to fine-tune the model (local adaptation). We use the fine-tuned model to make a prediction and then discard it—keeping the base model for other predictions.","Figure 2: Performance on test examples corresponding to the first dataset seen during training as training progresses.","Table 3: Results for different # of retrieved examples K.","Figure 3: F1 scores for MBPA++ and MBPA as the # of local adaptation steps increases.","Table 4: Per-dataset results on text classification for each ordering and model.","Table 5: Per-dataset results on question answering for each ordering and model.","Table 6: Performance of a standard encoder-decoder model on each dataset in our experiments. We report accuracy for text classification and F1 score for question answering. We also show results from a multitask model for comparisons."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1906.05506","title":"Character n-gram Embeddings to Improve RNN Language Models","sections":[{"idx":0,"heading":"Abstract","paragraphs":["This paper proposes a novel Recurrent Neural Network (RNN) language model that takes advantage of character information. We focus on character n-grams based on research in the field of word embedding construction (Wieting et al. 2016). Our proposed method constructs word embeddings from character n-gram embeddings and combines them with ordinary word embeddings. We demonstrate that the proposed method achieves the best perplexities on the language modeling datasets: Penn Treebank, WikiText-2, and WikiText-103. Moreover, we conduct experiments on application tasks: machine translation and headline generation. The experimental results indicate that our proposed method also positively affects these tasks."]},{"idx":1,"heading":"Introduction","paragraphs":["Neural language models have played a crucial role in recent advances of neural network based methods in natural language processing (NLP). For example, neural encoder-decoder models, which are becoming the de facto standard for various natural language generation tasks including machine translation BIBREF1 , summarization BIBREF2 , dialogue BIBREF3 , and caption generation BIBREF4 can be interpreted as conditional neural language models. Moreover, neural language models can be used for rescoring outputs from traditional methods, and they significantly improve the performance of automatic speech recognition BIBREF5 . This implies that better neural language models improve the performance of application tasks.","In general, neural language models require word embeddings as an input BIBREF6 . However, as described by BIBREF7 , this approach cannot make use of the internal structure of words although the internal structure is often an effective clue for considering the meaning of a word. For example, we can comprehend that the word `causal' is related to `cause' immediately because both words include the same character sequence `caus'. Thus, if we incorporate a method that handles the internal structure such as character information, we can improve the quality of neural language models and probably make them robust to infrequent words.","To incorporate the internal structure, BIBREF7 concatenated character embeddings with an input word embedding. They demonstrated that incorporating character embeddings improved the performance of RNN language models. Moreover, BIBREF8 and BIBREF9 applied Convolutional Neural Networks (CNN) to construct word embeddings from character embeddings.","On the other hand, in the field of word embedding construction, some previous researchers found that character INLINEFORM0 -grams are more useful than single characters BIBREF0 , BIBREF10 . In particular, BIBREF0 demonstrated that constructing word embeddings from character INLINEFORM1 -gram embeddings outperformed the methods that construct word embeddings from character embeddings by using CNN or a Long Short-Term Memory (LSTM).","Based on their reports, in this paper, we propose a neural language model that utilizes character INLINEFORM0 -gram embeddings. Our proposed method encodes character INLINEFORM1 -gram embeddings to a word embedding with simplified Multi-dimensional Self-attention (MS) BIBREF11 . We refer to this constructed embedding as char INLINEFORM2 -MS-vec. The proposed method regards char INLINEFORM3 -MS-vec as an input in addition to a word embedding.","We conduct experiments on the well-known benchmark datasets: Penn Treebank, WikiText-2, and WikiText-103. Our experiments indicate that the proposed method outperforms neural language models trained with well-tuned hyperparameters and achieves state-of-the-art scores on each dataset. In addition, we incorporate our proposed method into a standard neural encoder-decoder model and investigate its effect on machine translation and headline generation. We indicate that the proposed method also has a positive effect on such tasks."]},{"idx":2,"heading":"RNN Language Model","paragraphs":["In this study, we focus on RNN language models, which are widely used in the literature. This section briefly overviews the basic RNN language model.","In language modeling, we compute joint probability by using the product of conditional probabilities. Let INLINEFORM0 be a word sequence with length INLINEFORM1 , namely, INLINEFORM2 . We formally obtain the joint probability of word sequence INLINEFORM3 as follows: DISPLAYFORM0","INLINEFORM0 is generally assumed to be 1 in this literature, i.e., INLINEFORM1 , and thus we can ignore its calculation.","To estimate the conditional probability INLINEFORM0 , RNN language models encode sequence INLINEFORM1 into a fixed-length vector and compute the probability distribution of each word from this fixed-length vector. Let INLINEFORM2 be the vocabulary size and let INLINEFORM3 be the probability distribution of the vocabulary at timestep INLINEFORM4 . Moreover, let INLINEFORM5 be the dimension of the hidden state of an RNN and let INLINEFORM6 be the dimensions of embedding vectors. Then, RNN language models predict the probability distribution INLINEFORM7 by the following equation: DISPLAYFORM0","where INLINEFORM0 is a weight matrix, INLINEFORM1 is a bias term, and INLINEFORM2 is a word embedding matrix. INLINEFORM3 and INLINEFORM4 are a one-hot vector of an input word INLINEFORM5 and the hidden state of the RNN at timestep INLINEFORM6 , respectively. We define INLINEFORM7 at timestep INLINEFORM8 as a zero vector, that is, INLINEFORM9 . Let INLINEFORM10 represent an abstract function of an RNN, which might be the LSTM, the Quasi-Recurrent Neural Network (QRNN) BIBREF12 , or any other RNN variants."]},{"idx":3,"heading":"Incorporating Character nn-gram Embeddings","paragraphs":["We incorporate char INLINEFORM0 -MS-vec, which is an embedding constructed from character INLINEFORM1 -gram embeddings, into RNN language models since, as discussed earlier, previous studies revealed that we can construct better word embeddings by using character INLINEFORM2 -gram embeddings BIBREF0 , BIBREF10 . In particular, we expect char INLINEFORM3 -MS-vec to help represent infrequent words by taking advantage of the internal structure.","Figure FIGREF4 is the overview of the proposed method using character 3-gram embeddings (char3-MS-vec). As illustrated in this figure, our proposed method regards the sum of char3-MS-vec and the standard word embedding as an input of an RNN. In other words, let INLINEFORM0 be char INLINEFORM1 -MS-vec and we replace Equation with the following: DISPLAYFORM0"]},{"idx":4,"heading":"Multi-dimensional Self-attention","paragraphs":["To compute INLINEFORM0 , we apply an encoder to character INLINEFORM1 -gram embeddings. Previous studies demonstrated that additive composition, which computes the (weighted) sum of embeddings, is a suitable method for embedding construction BIBREF13 , BIBREF0 . Thus, we adopt (simplified) multi-dimensional self-attention BIBREF11 , which computes weights for each dimension of given embeddings and sums up the weighted embeddings (i.e., element-wise weighted sum) as an encoder. Let INLINEFORM2 be the character INLINEFORM3 -gram embeddings of an input word, let INLINEFORM4 be the number of character INLINEFORM5 -grams extracted from the word, and let INLINEFORM6 be the matrix whose INLINEFORM7 -th column corresponds to INLINEFORM8 , that is, INLINEFORM9 . The multi-dimensional self-attention constructs the word embedding INLINEFORM10 by the following equations: DISPLAYFORM0","where INLINEFORM0 means element-wise product of vectors, INLINEFORM1 is a weight matrix, INLINEFORM2 is the INLINEFORM3 -th column of a given matrix, and INLINEFORM4 is the INLINEFORM5 -th element of a given vector. In short, Equation applies the softmax function to each row of INLINEFORM6 and extracts the INLINEFORM7 -th column as INLINEFORM8 .","Let us consider the case where an input word is `the' and we use character 3-gram in Figure FIGREF4 . We prepare special characters `' and `$' to represent the beginning and end of the word, respectively. Then, `the' is composed of three character 3-grams: `th', `the', and `he$'. We multiply the embeddings of these 3-grams by transformation matrix INLINEFORM0 and apply the softmax function to each row as in Equation . As a result of the softmax, we obtain a matrix that contains weights for each embedding. The size of the computed matrix is identical to the input embedding matrix: INLINEFORM1 . We then compute Equation EQREF7 , i.e., the weighted sum of the embeddings, and add the resulting vector to the word embedding of `the'. Finally, we input the vector into an RNN to predict the next word."]},{"idx":5,"heading":"Word Tying","paragraphs":["BIBREF14 and BIBREF15 proposed a word tying method (WT) that shares the word embedding matrix ( INLINEFORM0 in Equation ) with the weight matrix to compute probability distributions ( INLINEFORM1 in Equation EQREF3 ). They demonstrated that WT significantly improves the performance of RNN language models.","In this study, we adopt char INLINEFORM0 -MS-vec as the weight matrix in language modeling. Concretely, we use INLINEFORM1 instead of INLINEFORM2 in Equation EQREF3 , where INLINEFORM3 contains char INLINEFORM4 -MS-vec for all words in the vocabulary."]},{"idx":6,"heading":"Experiments on Language Modeling","paragraphs":["We investigate the effect of char INLINEFORM0 -MS-vec on the word-level language modeling task. In detail, we examine the following four research questions;"]},{"idx":7,"heading":"Datasets","paragraphs":["We used the standard benchmark datasets for the word-level language modeling: Penn Treebank (PTB) BIBREF16 , WikiText-2 (WT2), and WikiText-103 (WT103) BIBREF17 . BIBREF18 and BIBREF17 published pre-processed PTB, WT2, and WT103. Following the previous studies, we used these pre-processed datasets for our experiments.","Table TABREF14 describes the statistics of the datasets. Table TABREF14 demonstrates that the vocabulary size of WT103 is too large, and thus it is impractical to compute char INLINEFORM0 -MS-vec for all words at every step. Therefore, we did not use INLINEFORM1 for word tying. In other words, we used only word embeddings INLINEFORM2 as the weight matrix INLINEFORM3 in WT103.","For machine translation, we used two kinds of language pairs: English-French and English-German sentences in the IWSLT 2016 dataset. The dataset contains about 208K English-French pairs and 189K English-German pairs. We conducted four translation tasks: from English to each language (En-Fr and En-De), and their reverses (Fr-En and De-En).","For headline generation, we used sentence-headline pairs extracted from the annotated English Gigaword corpus BIBREF35 in the same manner as BIBREF2 . The training set contains about 3.8M sentence-headline pairs. For evaluation, we exclude the test set constructed by BIBREF2 because it contains some invalid instances, as reported in BIBREF33 . We instead used the test sets constructed by BIBREF33 and BIBREF34 ."]},{"idx":8,"heading":"Baseline RNN Language Model","paragraphs":["For base RNN language models, we adopted the state-of-the-art LSTM language model BIBREF19 for PTB and WT2, and QRNN for WT103 BIBREF12 . BIBREF20 demonstrated that the standard LSTM trained with appropriate hyperparameters outperformed various architectures such as Recurrent Highway Networks (RHN) BIBREF21 . In addition to several regularizations, BIBREF19 introduced Averaged Stochastic Gradient Descent (ASGD) BIBREF22 to train the 3-layered LSTM language model. As a result, their ASGD Weight-Dropped LSTM (AWD-LSTM) achieved state-of-the-art results on PTB and WT2. For WT103, BIBREF23 achieved the top score with the 4-layered QRNN. Thus, we used AWD-LSTM for PTB and WT2, and QRNN for WT103 as the base language models, respectively. We used their implementations for our experiments."]},{"idx":9,"heading":"Results","paragraphs":["Table TABREF15 shows perplexities of the baselines and the proposed method. We varied INLINEFORM0 for char INLINEFORM1 -MS-vec from 2 to 4. For the baseline, we also applied two word embeddings to investigate the performance in the case where we use more kinds of word embeddings. In detail, we prepared INLINEFORM2 and used INLINEFORM3 instead of INLINEFORM4 in Equation . Table TABREF15 also shows the number of character INLINEFORM5 -grams in each dataset. This table indicates that char INLINEFORM6 -MS-vec improved the performance of state-of-the-art models except for char4-MS-vec on WT103. These results indicate that char INLINEFORM7 -MS-vec can raise the quality of word-level language models. In particular, Table TABREF15 shows that char3-MS-vec achieved the best scores consistently. In contrast, an additional word embedding did not improve the performance. This fact implies that the improvement of char INLINEFORM8 -MS-vec is caused by using character INLINEFORM9 -grams. Thus, we answer yes to the first research question.","Table TABREF16 shows the training time spent on each epoch. We calculated it on the NVIDIA Tesla P100. Table TABREF16 indicates that the proposed method requires more computational time than the baseline unfortunately. We leave exploring a faster structure for our future work.","Table TABREF17 shows perplexities on the PTB dataset where the frequency of an input word is lower than 2,000 in the training data. This table indicates that the proposed method can improve the performance even if an input word is infrequent. In other words, char INLINEFORM0 -MS-vec helps represent the meanings of infrequent words. Therefore, we answer yes to the second research question in the case of our experimental settings.","We explored the effectiveness of multi-dimensional self-attention for word embedding construction. Table TABREF24 shows perplexities of using several encoders on the PTB dataset. As in BIBREF8 , we applied CNN to construct word embeddings (charCNN in Table TABREF24 ). Moreover, we applied the summation and standard self-attention, which computes the scalar value as a weight for a character INLINEFORM0 -gram embedding, to construct word embeddings (char INLINEFORM1 -Sum-vec and char INLINEFORM2 -SS-vec, respectively). For CNN, we used hyperparameters identical to BIBREF8 (“Original Settings” in Table TABREF24 ) but the setting has two differences from other architectures: 1. The dimension of the computed vectors is much larger than the dimension of the baseline word embeddings and 2. The dimension of the input character embeddings is much smaller than the dimension of the baseline word embeddings. Therefore, we added two configurations: assigning the dimension of the computed vectors and input character embeddings a value identical to the baseline word embeddings (in Table TABREF24 , “Small CNN result dims” and “Large embedding dims”, respectively).","Table TABREF24 shows that the proposed char INLINEFORM0 -MS-vec outperformed charCNN even though the original settings of charCNN had much larger parameters. Moreover, we trained charCNN with two additional settings but CNN did not improve the baseline performance. This result implies that char INLINEFORM1 -MS-vec is better embeddings than ones constructed by applying CNN to character embeddings. Table TABREF24 also indicates that char INLINEFORM2 -Sum-vec was harmful to the performance. Moreover, char INLINEFORM3 -SS-vec did not have a positive effect on the baseline. These results answer yes to the third research question; our use of multi-dimensional self-attention is more appropriate for constructing word embeddings from character INLINEFORM4 -gram embeddings.","Table TABREF24 also shows that excluding INLINEFORM0 from word tying (“Exclude INLINEFORM1 from word tying”) achieved almost the same score as the baseline. Moreover, this table indicates that performance fails as the the number of parameters is increased. Thus, we need to assign INLINEFORM2 to word tying to prevent over-fitting for the PTB dataset. In addition, this result implies that the performance of WT103 in Table TABREF15 might be raised if we can apply word tying to WT103.","Moreover, to investigate the effect of only char INLINEFORM0 -MS-vec, we ignore INLINEFORM1 in Equation EQREF5 . We refer to this setting as “Remove word embeddings INLINEFORM2 ” in Table TABREF24 . Table TABREF24 shows cahr3-MS-vec and char4-MS-vec are superior to char2-MS-vec. In the view of perplexity, char3-MS-vec and char4-MS-vec achieved comparable scores to each other. On the other hand, char3-MS-vec is composed of much smaller parameters. Furthermore, we decreased the embedding size INLINEFORM3 to adjust the number of parameters to the same size as the baseline (“Same #Params as baseline” in Table TABREF24 ). In this setting, char3-MS-vec achieved the best perplexity. Therefore, we consider that char3-MS-vec is more useful than char4-MS-vec, which is the answer to the fourth research question. We use the combination of the char3-MS-vec INLINEFORM4 and word embedding INLINEFORM5 in the following experiments.","Finally, we compare the proposed method with the published scores reported in previous studies. Tables TABREF25 , TABREF26 , and TABREF27 , respectively, show perplexities of the proposed method and previous studies on PTB, WT2, and WT103. Since AWD-LSTM-MoS BIBREF28 and AWD-LSTM-DOC BIBREF29 achieved the state-of-the-art scores on PTB and WT2, we combined char3-MS-vec with them. These tables show that the proposed method improved the performance of the base model and outperformed the state-of-the-art scores on all datasets. In particular, char3-MS-vec improved perplexity by at least 1 point from current best scores on the WT103 dataset.","Tables TABREF31 and TABREF32 show the results of machine translation and headline generation, respectively. These tables show that EncDec+char3-MS-vec outperformed EncDec in all test data. In other words, these results indicate that our proposed method also has a positive effect on the neural encoder-decoder model. Moreover, it is noteworthy that char3-MS-vec improved the performance of EncDec even though the vocabulary set constructed by BPE contains subwords. This implies that character INLINEFORM0 -gram embeddings improve the quality of not only word embeddings but also subword embeddings.","In addition to the results of our implementations, the lower portion of Table TABREF32 contains results reported in previous studies. Table TABREF32 shows that EncDec+char3-MS-vec also outperformed the methods proposed in previous studies. Therefore, EncDec+char3-MS-vec achieved the top scores in the test sets constructed by BIBREF33 and BIBREF34 even though it does not have a task-specific architecture such as the selective gate proposed by BIBREF33 .","In these experiments, we only applied char3-MS-vec to EncDec but BIBREF38 indicated that combining multiple kinds of subword units can improve the performance. We will investigate the effect of combining several character INLINEFORM0 -gram embeddings in future work."]},{"idx":10,"heading":"Experiments on Applications","paragraphs":["As described in Section SECREF1 , neural encoder-decoder models can be interpreted as conditional neural language models. Therefore, to investigate if the proposed method contributes to encoder-decoder models, we conduct experiments on machine translation and headline generation tasks."]},{"idx":11,"heading":"Experimental Settings","paragraphs":["We employed the neural encoder-decoder with attention mechanism described in BIBREF34 as the base model. Its encoder consists of a 2-layer bidirectional LSTM and its decoder consists of a 2-layer LSTM with attention mechanism proposed by BIBREF36 . We refer to this neural encoder-decoder as EncDec. To investigate the effect of the proposed method, we introduced char3-MS-vec into EncDec. Here, we applied char3-MS-vec to both the encoder and decoder. Moreover, we did not apply word tying technique to EncDec because it is default setting in the widely-used encoder-decoder implementation.","We set the embedding size and dimension of the LSTM hidden state to 500 for machine translation and 400 for headline generation. The mini-batch size is 64 for machine translation and 256 for headline generation. For other hyperparameters, we followed the configurations described in BIBREF34 . We constructed the vocabulary set by using Byte-Pair-Encoding (BPE) BIBREF37 because BPE is a currently widely-used technique for vocabulary construction. We set the number of BPE merge operations to 16K for machine translation and 5K for headline generation."]},{"idx":12,"heading":"Conclusion","paragraphs":["In this paper, we incorporated character information with RNN language models. Based on the research in the field of word embedding construction BIBREF0 , we focused on character INLINEFORM0 -gram embeddings to construct word embeddings. We used multi-dimensional self-attention BIBREF11 to encode character INLINEFORM1 -gram embeddings. Our proposed char INLINEFORM2 -MS-vec improved the performance of state-of-the-art RNN language models and achieved the best perplexities on Penn Treebank, WikiText-2, and WikiText-103. Moreover, we investigated the effect of char INLINEFORM3 -MS-vec on application tasks, specifically, machine translation and headline generation. Our experiments show that char INLINEFORM4 -MS-vec also improved the performance of a neural encoder-decoder on both tasks."]},{"idx":13,"heading":"Acknowledgments","paragraphs":["This work was supported by JSPS KAKENHI Grant Number JP18K18119. We would like to thank the anonymous reviewers for their helpful suggestions and comments."]},{"idx":14,"heading":"Figures and Tables","paragraphs":["Figure 1: Overview of the proposed method. The proposed method computes charn-MS-vec from character n-gram (3- gram in this figure) embeddings and inputs the sum of it and the standard word embedding into an RNN.","Table 1: Statistics of PTB, WT2, and WT103.","Table 2: Perplexities on each dataset. We varied the n for charn-MS-vec from 2 to 4.","Table 3: Computational speed of the baseline and proposed method on NVIDIA Tesla P100.","Table 4: Perplexities on the PTB dataset where an input word is infrequent in the training data, which means its frequency is lower than 2,000.","Table 5: Perplexities of each structure on PTB dataset.","Table 6: Perplexities of the proposed method and as reported in previous studies on the PTB dataset.","Table 7: Perplexities of the proposed method and as reported in previous studies on the WT2 dataset.","Table 8: Perplexities of the proposed method and as reported in previous studies on the WT103 dataset.","Table 9: BLEU scores on the IWSLT16 dataset. We report the average score of 3 runs.","Table 10: ROUGE F1 scores on the headline generation test sets provided by (Zhou et al. 2017) and (Kiyono et al. 2017). The upper part is the results of our implementation and the lower part shows the scores reported in previous studies. In the upper part, we report the average score of 3 runs."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1907.04433","title":"GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing","sections":[{"idx":0,"heading":"Abstract","paragraphs":["We present GluonCV and GluonNLP, the deep learning toolkits for computer vision and natural language processing based on Apache MXNet (incubating). These toolkits provide state-of-the-art pre-trained models, training scripts, and training logs, to facilitate rapid prototyping and promote reproducible research. We also provide modular APIs with flexible building blocks to enable efficient customization. Leveraging the MXNet ecosystem, the deep learning models in GluonCV and GluonNLP can be deployed onto a variety of platforms with different programming languages. Benefiting from open source under the Apache 2.0 license, GluonCV and GluonNLP have attracted 100 contributors worldwide on GitHub. Models of GluonCV and GluonNLP have been downloaded for more than 1.6 million times in fewer than 10 months."]},{"idx":1,"heading":"Introduction","paragraphs":["Deep learning, a sub-field of machine learning research, has driven the rapid progress in artificial intelligence research, leading to astonishing breakthroughs on long-standing problems in a plethora of fields such as computer vision and natural language processing. Tools powered by deep learning are changing the way movies are made, diseases are diagnosed, and play a growing role in understanding and communicating with humans.","Such development is made possible by deep learning frameworks, such as Caffe BIBREF0 , Chainer BIBREF1 , CNTK BIBREF2 , Apache (incubating) MXNet BIBREF3 , PyTorch BIBREF4 , TensorFlow BIBREF5 , and Theano BIBREF6 . These frameworks have been crucial in disseminating ideas in the field. Specifically, imperative tools, arguably spearheaded by Chainer, are easy to learn, read, and debug. Such benefits make imperative programming interface quickly adopted by the Gluon API of MXNet (while can be seamlessly switched to symbolic programming for high performance), PyTorch, and TensorFlow Eager.","Leveraging the imperative Gluon API of MXNet, we design and develop the GluonCV and GluonNLP (referred to as GluonCV/NLP hereinafter) toolkits for deep learning in computer vision and natural language processing. To the best of our knowledge, GluonCV/NLP are the first open source toolkits for deep learning in both computer vision and natural language processing that simultaneously i) provide modular APIs to allow customization by re-using efficient building blocks; ii) provide pre-trained state-of-the-art models, training scripts, and training logs to enable fast prototyping and promote reproducible research; iii) leverage the MXNet ecosystem so that models can be deployed in a wide variety of programming languages including C++, Clojure, Java, Julia, Perl, Python, R, and Scala."]},{"idx":2,"heading":"Design and Features","paragraphs":["In the following, we describe the design and features of GluonCV/NLP."]},{"idx":3,"heading":"Modular APIs","paragraphs":["GluonCV/NLP provide access to modular APIs to allow users to customize their model design, training, and inference by re-using efficient components across different models. Such common components include (but are not limited to) data processing utilities, models with individual components, initialization methods, and loss functions.","To elucidate how the modular API facilitates efficient implementation, let us take the data API of GluonCV/NLP as an example, which is used to build efficient data pipelines with popular benchmark data sets or those supplied by users. In computer vision and natural language processing tasks, inputs or labels often come in with different shapes, such as images with a varying number of objects and sentences of different lengths. Thus, the data API provides a collection of utilities to sample inputs or labels then transform them into mini-batches to be efficiently computed. Besides, users can access a wide range of popular data sets via the data API, including (but are not limited to) ImageNet of image classification, VOC of object detection, COCO of instance segmentation, SQuAD of question answering, and SST of sentiment analysis."]},{"idx":4,"heading":"Model Zoo","paragraphs":["Building upon those modular APIs, GluonCV/NLP provide pre-trained state-of-the-art models, training scripts, and training logs via the model zoo to enable fast prototyping and promote reproducible research. As of the time of writing, GluonCV/NLP have provided over 100 models for common computer vision and natural language processing tasks, such as image classification, object detection, semantic segmentation, instance segmentation, pose estimation, word embedding, language model, machine translation, sentiment analysis, natural language inference, dependency parsing, and question answering."]},{"idx":5,"heading":"Leveraging the MXNet Ecosystem","paragraphs":["GluonCV/NLP have benefitted from the MXNet ecosystem through use of MXNet. At the lowest level, MXNet provides high-performance C++ implementations of operators that are leveraged by GluonCV/NLP; thus, improvements in low-level components of MXNet often result in performance gains in GluonCV/NLP. Same as any other model implemented with MXNet, GluonCV/NLP can be used to train models on CPU, GPU (single or multiple), and multiple machines. In sharp contrast to building upon other deep learning frameworks, through the unique hybridizing mechanism by MXNet BIBREF7 , usually GluonCV/NLP models can be deployed with no or minimal configuration in a wide spectrum of programming languages including C++, Clojure, Java, Julia, Perl, Python, R, and Scala. There are also ongoing efforts to bring more quantization (int8 and float16 inference) benefits from MXNet to GluonCV/NLP to further accelerate model inference.","The documentation https://gluon-cv.mxnet.io/ and http://gluon-nlp.mxnet.io/ of GluonCV/NLP include installation instructions, contribution instructions, open source repositories, extensive API reference, and comprehensive tutorials. As another benefit of leveraging the MXNet ecosystem, the GluonCV/NLP documentation is supplemented by the interactive open source book Dive into Deep Learning (based on the Gluon API of MXNet) BIBREF7 , which provides sufficient background knowledge about GluonCV/NLP tasks, models, and building blocks. Notably, some users of Dive into Deep Learning have later become contributors of GluonCV/NLP."]},{"idx":6,"heading":"Requirement, Availability, and Community","paragraphs":["GluonCV/NLP are implemented in Python and are available for systems running Linux, macOS, and Windows since Python is platform agnostic. The minimum and open source package (e.g., MXNet) requirements are specified in the documentation. As of the time of writing, GluonCV/NLP have reached version 0.6 and 0.4 respectively, and have been open sourced under the Apache 2.0 license. Since the initial release of the source code in April 2018, GluonCV/NLP have attracted 100 contributors worldwide. Models of GluonCV/NLP have been downloaded for more than 1.6 million times in fewer than 10 months."]},{"idx":7,"heading":"Performance","paragraphs":["We demonstrate the performance of GluonCV/NLP models in various computer vision and natural language processing tasks. Specifically, we evaluate popular or state-of-the-art models on standard benchmark data sets. In the experiments, we compare model performance between GluonCV/NLP and other open source implementations with Caffe, Caffe2, Theano, and TensorFlow, including ResNet BIBREF8 and MobileNet BIBREF9 for image classification (ImageNet), Faster R-CNN BIBREF10 for object detection (COCO), Mask R-CNN BIBREF11 for instance segmentation, Simple Pose BIBREF12 for pose estimation (COCO), textCNN BIBREF13 for sentiment analysis (TREC), and BERT BIBREF14 for question answering (SQuAD 1.1), sentiment analysis (SST-2), natural langauge inference (MNLI-m), and paraphrasing (MRPC). Table TABREF5 shows that the GluonCV/GluonNLP implementation matches or outperforms the compared open source implementation for the same model evaluated on the same data set."]},{"idx":8,"heading":"Conclusion","paragraphs":["GluonCV/NLP provide modular APIs and the model zoo to allow users to rapidly try out new ideas or develop downstream applications in computer vision and natural language processing. GluonCV/NLP are in active development and our future works include further enriching the API and the model zoo, and supporting deployment in more scenarios.","We would like to thank all the contributors of GluonCV and GluonNLP (the git log command can be used to list all the contributors). Specifically, we thank Xiaoting He, Heewon Jeon, Kangjian Wu, and Luyu Xia for providing part of results in Table TABREF5 . We would also like to thank the entire MXNet community for their foundational contributions."]},{"idx":9,"heading":"Figures and Tables","paragraphs":["Table 1: Comparison of model performance (in percentage) on the validation data sets between GluonCV/NLP and other open source implementations (OOSI) across popular computer vision and natural language processing tasks and data sets."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1907.05338","title":"To Tune or Not To Tune? How About the Best of Both Worlds?","sections":[{"idx":0,"heading":"Abstract","paragraphs":["The introduction of pre-trained language models has revolutionized natural language research communities. However, researchers still know relatively little regarding their theoretical and empirical properties. In this regard, Peters et al. perform several experiments which demonstrate that it is better to adapt BERT with a light-weight task-specific head, rather than building a complex one on top of the pre-trained language model, and freeze the parameters in the said language model. However, there is another option to adopt. In this paper, we propose a new adaptation method which we first train the task model with the BERT parameters frozen and then fine-tune the entire model together. Our experimental results show that our model adaptation method can achieve 4.7% accuracy improvement in semantic similarity task, 0.99% accuracy improvement in sequence labeling task and 0.72% accuracy improvement in the text classification task."]},{"idx":1,"heading":"Introduction","paragraphs":["The introduction of pre-trained language models, such as BERT BIBREF1 and Open-GPT BIBREF2 , among many others, has brought tremendous progress to the NLP research and industrial communities. The contribution of these models can be categorized into two aspects. First, pre-trained language models allow modelers to achieve reasonable accuracy without the need an excessive amount of manually labeled data. This strategy is in contrast with the classical deep learning methods, which requires a multitude more data to reach comparable results. Second, for many NLP tasks, including but not limited to, SQuAD BIBREF3 , CoQA BIBREF4 , named entity recognition BIBREF5 , Glue BIBREF6 , machine translation BIBREF7 , pre-trained model allows the creation of new state-of-art, given a reasonable amount of labelled data.","In the post pre-trained language model era, to pursue new state-of-art, two directions can be followed. The first method, is to improve the pre-training process, such as in the work of ERNIE BIBREF8 , GPT2.0 BIBREF2 and MT-DNN BIBREF9 . The second method is to stand on the shoulder of the pre-trained language models. Among the many possibilities, one of them is to build new neural network structures on top of pre-trained language models.","In principles, there are three ways to train the networks with stacked neural networks on top of pre-trained language models, as shown in Table TABREF1 . In Peters et al . BIBREF0 , the authors compare the possibility of option stack-only and finetune-only, and conclude that option finetune-only is better than option stack-only. More specifically, Peter et al. BIBREF0 argue that it is better to add a task-specific head on top of BERT than to freeze the weights of BERT and add more complex network structures. However, Peters et al. BIBREF0 did not compare option stack-and-finetune and finetune-only. On the other hand, before pre-trained deep language models became popular, researchers often use a strategy analog to option stack-and-finetune. That is, modelers first train the model until convergence, and then fine-tune the word embeddings with a few epochs. If pre-trained language models can be understood as at least partially resemblance of word embeddings, then it will be imprudent not to consider the possibility of option stack-and-finetune.","In this study, we aim to compare the strategy stack-and-finetune and strategy finetune-only. More specifically, we perform three NLP tasks, sequence labeling, text classification, and question similarity. In the first tasks, we demonstrate that even without modifying the network structures, building networks on top of pre-trained language models might improve accuracy. In the second tasks, we show that by ensembling different neural networks, one can even improve the accuracy of fine-tuning only methods even further. Finally, in the last task, we demonstrate that if one can tailor-made a neural network that specifically fit the characteristics of the pre-trained language models, one can improve the accuracy even further. All the results indicate the strategy stack-and-finetune is superior to strategy finetune-only. This leads us to conclude that, at least, by overlooking the possibility strategy stack-and-finetune is imprudent.","The contribution of this paper is two-fold. First, we propose a new strategy to improve the fine-tune-only strategy proposed by Peter et al. BIBREF0 , this allows us to achieve better results, at least on the selected tasks. More importantly, the results of this study demonstrate the importance of neural networks design, even in the presence of all-powerful pre-trained language models. Second, during the experiment, we have found that although simply using the proposed training strategy can result in higher accuracies compared to that of Peter et al. BIBREF0 , it is still a challenging task to find the appropriate methods to design and to utilize pre-trained networks. In this regard, we find that pre-trained models differ significantly from word embeddings in terms of their training strategies. Especially, since word embeddings can be viewed as shallow transfer learning, while pre-trained model should be viewed as deep transfer learning, one must try to combat over-fitting problems with more care due to the enormous number of parameters presented in the pre-trained models. Besides, we also find that in order to achieve the maximal performance in the post-pre-trained language model era, one must design, either manually or via Auto ML, networks that best fit the structure, especially the depth of the pre-trained language models.","The rest of the paper is organized as follows. First, we review the relevant literature on pre-trained deep neural networks, the argument in Peter et al. BIBREF0 as well as fine-tuning strategies with word embeddings. Second, we present three experiments and showed the superiority of strategy stack-and-finetune compared to strategy finetune-only. Finally, we conclude with some remarks and future research possibilities."]},{"idx":2,"heading":"Related Studies","paragraphs":["Before the introduction of deep neural networks, researchers in the field of NLP have been using pre-trained models. Among all of them, one of the most famous is the word embeddings, which maps each word into a continuous vector, instead of one-hot encodings BIBREF10 . By doing so, not only are we able to reduce the dimensionality of the input features, which helps to avoid over-fitting, but also capture, at least partially, the internal meaning of each word.","However, since each word is only endowed with a fixed numerical vector in the methodology of word embeddings, word embeddings are unable to capture the contextual meaning in the text. For example, consider the word ”bank” sentences “I am walking on the bank of the river.” with “I am going to rob the bank”. It is obvious that the word “bank” represents completely different meaning, which the word embeddings techniques fail to capture.","The aforementioned deficiencies prompt researchers to propose deep neural networks that are able to be trained in an unsupervised fashion while being able to capture the contextual meaning of the words presented in the texts. Some early attempts include pre-trained models includes, CoVe BIBREF11 , CVT BIBREF12 , BIBREF13 , ELMo BIBREF14 and ULMFiT BIBREF15 . However, the most successful ones are BERT BIBREF1 and Open-GPT BIBREF2 . Unlike standard NLP deep learning model, BERT and Open-GPT are built on top of transformer BIBREF16 structures, instead of LSTM BIBREF17 or GRU BIBREF18 . The difference between BERT and Open-GPT is that BERT uses bi-directional self-attentions while Open-GPT uses only unidirectional ones, as shown in Figure FIGREF2 . The transformer structures differ from the LSTM's in the two important aspects. First, it allows for stacking of multiple layers with residual connections and batch normalizations, which allows for free gradient flow. Second, the core computational unit is matrix multiplications, which allows researchers to utilize the full computational potential of TPU BIBREF19 . After training on a large corpus, both BERT and Open-GPT are able to renew the SOTA of many important natural language tasks, such as such as SQuAD BIBREF3 , CoQA BIBREF4 , named entity recognition BIBREF5 , Glue BIBREF6 , machine translation BIBREF7 .","In the presence of the success of pre-trained language models, especially BERT BIBREF1 , it is natural to ask how to best utilize the pre-trained language models to achieve new state-of-the-art results. In this line of work, Liu et al. BIBREF20 investigated the linguistic knowledge and transferability of contextual representations by comparing BERT BIBREF1 with ELMo BIBREF14 , and concluded that while the higher levels of LSTM's are more task-specific, this trend does not exhibit in transformer based models. Stickland and Murray BIBREF21 invented projected attention layer for multi-task learning using BERT, which results in an improvement in various state-of-the-art results compared to the original work of Devlin et al. BIBREF1 . Xu et al. BIBREF22 propose a “post-training” algorithms, which does not directly fine-tune BERT, but rather first “post-train” BERT on the task related corpus using the masked language prediction task next sentence prediction task, which helps to reduce the bias in the training corpus. Finally, Sun et al. BIBREF23 added additional fine-tuning tasks based on multi-task training, which further improves the prediction power of BERT in the tasks of text classification.","In this aspect, however, there is a simple yet crucial question that needs to be addressed. That is, whether it is possible to top BERT with the commonly used or task specific layers, and if this is possible, how to best utilize the pre-trained language models in this situation. In this regards, Peters et al. BIBREF0 investigated how to best adapt the pre-trained model to a specific task, and focused on two different adaptation method,feature extraction and directly fine-tuning the pre-trained model, which corresponding to the strategy finetune-only and the strategy stack-only in Table TABREF1 . On this regard, Peters et al. BIBREF0 performs five experiments, including: (1) named entity recognition BIBREF5 ; (2) sentiment analysis BIBREF24 ; (3) natural language inference BIBREF25 ; (4) paraphrase detection BIBREF26 ; (5) semantic textual similarity BIBREF27 . By the results of these tasks, Peters et al. BIBREF0 concludes that adding a light task-specific head and performing fine-tuning on BERT is better than building a complex network on top without BERT fine-tuning."]},{"idx":3,"heading":"Methodology","paragraphs":["Under our strategy stack-and-finetune, the model training process is divided into two phases, which are described in detail below. In the first phase, the parameters of the pre-training model are fixed, and only the upper-level models added for a specific task is learned. In the second phase, we fine-tune the upper-level models together with the pre-trained language models. We choose this strategy for the following reasons. Pre-training models have been used to obtain more effective word representations through the study of a large number of corpora. In the paradigm proposed in the original work by Devlin et al. BIBREF1 , the author directly trained BERT along with with a light-weighted task-specific head. In our case though, we top BERT with a more complex network structure, using Kaiming initialization BIBREF28 . If one would fine-tune directly the top models along with the weights in BERT, one is faced with the following dilemma: on the one hand, if the learning rate is too large, it is likely to disturb the structure innate to the pre-trained language models; on the other hand, if the learning rate is too small, since we top BERT with relatively complex models, the convergence of the top models might be impeded. Therefore, in the first phase we fix the weights in the pre-training language models, and only train the model on top of it.","Another aspect that is worth commenting in the first phase is that it is most beneficial that one does not train the top model until it reaches the highest accuracy on the training or validation data sets, but rather only up to a point where the prediction accuracy of the training and validation data sets do not differ much. This is intuitively reasonable for the following reasons. Unlike word embeddings, the pre-trained language models possess a large number of parameters compared to the task-specific models we build on top them. Therefore, if one were to train the top models until they reach the highest prediction accuracy in the training or validation data sets, it would likely cause the models to over-fit. Therefore, in our experiment, we found that this leads to the highest performance increase in the fine-tuning stage."]},{"idx":4,"heading":"Overview","paragraphs":["We perform three different experiments to test our hypotheses. First, we perform a named entity recognition tasks, by adding a bi-LSTM on top of the BERT model. In this experiment, we hope to test whether, without any modification to the commonly used network structure, our proposed training strategy will improve the overall accuracy. Second, we perform a text classification experiments, in this experiments, we trained three models, and perform a model ensemble. We hope to show that even the added network has not contributed to significantly in improving the accuracy, it does provide opportunities for model ensembles. Finally, we perform the textual similarity tests, in which we show that if one can tailor make a network that specifically fit the characteristics of the pre-trained languages, more significant improvement can be expected.","Under the strategy finetune-only, we use only single BERT.In order to adapt to different tasks, we will add a fully connected layer upon BERT. In the sequence labeling task, the BERT word embedding of each word passes through two fully connected layers, and the prediction probability of named entity can be obtained. In the next two verification tasks, we use “[CLS]” for prediction and add two fully connected layers subsequently. Under our strategy stack-and-finetune, we set different learning rates for the two phases. We tried to set the learning rate of the first stage to INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 and INLINEFORM4 , and set it to a smaller number in the latter stage, such as INLINEFORM5 , INLINEFORM6 , INLINEFORM7 and INLINEFORM8 . After our experiments, we found that it gets better results while the learning rate is set to 0.001 in the stage of training only the upper model and set to INLINEFORM9 in the later stage. Since BERT-Adam BIBREF1 has excellent performance, in our experiments, we use it as an optimizer with INLINEFORM10 , INLINEFORM11 -weight decay of INLINEFORM12 .We apply a dropout trick on all layers and set the dropout probability as 0.1."]},{"idx":5,"heading":"Experiment A: Sequence Labeling","paragraphs":["In the sequence labeling task,we explore sub-task named entity recognition using CoNLL03 dataset BIBREF5 , which is a public available used in many studies to test the accuracy of their proposed methods BIBREF29 , BIBREF30 , BIBREF31 , BIBREF32 , BIBREF1 . For strategy finetune-only and strategy stack-and-finetune, we implemented two models: one with BERT and the other with BERT adding a Bi-LSTM on top. Eval measure is accuracy and F1 score.","As is shown in Table 2, even without modifying the networks to specifically adapt to the pre-trained model, our training strategy still brought improvement towards overall accuracy of 0.99% for the accuracy and 0.068 on the F1 score, proving the success of our proposed methods."]},{"idx":6,"heading":"Experiment B: Text Classification","paragraphs":["In the task of text categorization, we used Yahoo Answer Classification Dataset. The Dataset is consists of 10 classes, but due to the huge amount of the dataset, we just select two class of them. As for the upper model,we choose DenseNet BIBREF33 and HighwayLSTM BIBREF34 .","The DenseNet structure contains four independent blocks and each block has four CNNs connected by residual. We initialize word embedding in the word representation layer with BERT. We initialize each character as a 768-dimension vector. In the experiment of training DenseNet,we concat the output vector of DenseNet with [CLS] for prediction.","We find the ensembled model enjoys a 0.72% improvements compared to the fine-tune only model and 0.005 improvement for the F1 score."]},{"idx":7,"heading":"Experiment C: Semantic Similarity Tasks","paragraphs":["We use “Quora-Question-Pair” dataset 1. This is a commonly used dataset containing 400k question pairs, annotated manually to be semantically equivalent or not. Due to its high quality, it is a standard dataset to test the success of various semantic similarity tasks. Various models which are tested on this data set are proposed, including but not limited to BIBREF35 , BIBREF36 , BIBREF37 , BIBREF38 .","Apart from the BERT fine-tuning only model and BERT+ BIMPM model, we also devise two new network structures by modifying the BIMPM model. In the first model is to remove the first bi-LSTM of BIMPM, which is the input layer for the matching layer in BIMPM. In the second model, we combine the matching layer of BIMPM and with a transformer BIBREF16 , a model we call Sim-Transformer by replacing the output layer of the matching layer, originally a bi-LSTM model, with a transformer model. From the experimental results shown in Table 4, we can see that due to the strong expressive ability of the BERT, there is almost no difference in the experimental results of removing the first bi-LSTM and BIMPM. In addition, we also find that Sim-Transformer's performance without fine-tuning is nearly four percentage points lower than BIMPM, but it out-performs BIMPM after fine-tuning. In general, the results show that BERT + Sim-Transformer out-performs BERT-only model by 4.7%, thus confirming our hypotheses again."]},{"idx":8,"heading":"Discussions and Conclusions","paragraphs":["In summary, we find that in all the three tasks, our proposed method out-performs the methods of simply tuning pre-trained language models, as is proposed in BIBREF0 . However, we would like to caution the readers in two aspects when reading the conclusion of this study. First, this study does not argue that our proposed methods are always superior to fine-tuning only methods. For example, all the experiments in our study are based on data sets of relatively large size. In the other spectrum, if one is only given a limited data set, then building complex networks upon pre-trained language models might lead to disastrous over-fitting. If this is the case, then it is possible that deep domain adaptation BIBREF39 might be a better choice if one desires to stack neural networks on top of pre-trained language models. However, most domain adaptation applications belong to the field of computer vision, therefore, a call for domain adaptations research in the NLP fields.","During the experimentation, we also discover some tricks to obtain higher quality networks. The first is that due to the enormous number of parameters presented in the pre-trained language models, to achieve generalizable results on the test data sets, it is vital to combat over-fitting. In classical embedding + training networks, the general training method is to fix the word-embeddings, then train the top model until it converges, and finally fine-tuning the word-embeddings for a few epochs. This training strategy does not work when we replace pre-trained language models with word-embeddings. In our experiment, we first fix the pre-trained language models, and then we train the top neural networks only for a few epochs, until it reaches a reasonable accuracy, while closely monitoring the discrepancy between training accuracy and testing accuracy. After that, we fine-tune the pre-trained language model as well as our models on top together. This allows us to achieve better results on the experimentation. However, it is not yet clear to us when to stop the training of top neural networks. This poses an even more essential question for Auto ML researchers in the following sense. In the classical computer vision based Auto ML approaches, since one seldom build networks on already trained models, there is no particular need to auxiliary measure for over-fittings. While if Auto ML is to be performed on NLP tasks successfully, it might be essential that the gap between training accuracy and test accuracy to be incorporated when one evaluates the model.","Finally, it is not yet clear what is the most proper way to build networks that tops the pre-trained language models. However, there are several principles that we can follow when designing such networks. First, such networks must be able to ensure the gradient flow from the top of the model to the bottom. This is essential due to the depth of the pre-trained language model. Second, this also means, one does not need explicitly to build extremely complex networks on top of pre-trained language models unless it complements the mechanisms of self-attention. Finally, a challenge remains as to how to use the depth of pre-trained language models. The process of our experiment shows that utilizing deeper layers might be a fruitful way to achieve better accuracy."]},{"idx":9,"heading":"Figures and Tables","paragraphs":["Table 1: Methods to Stack Neural Networks on Top of Pre-trained Language Models","Figure 1: The Difference Between BERT and Open-GPT, extracted from Devlin et al. [2], Figure 1","Table 2: Results for named entity recognition","Table 3: Results for text classification","Table 4: Results for semantic similarity task"]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1908.05731","title":"Simple and Effective Noisy Channel Modeling for Neural Machine Translation","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Previous work on neural noisy channel modeling relied on latent variable models that incrementally process the source and target sentence. This makes decoding decisions based on partial source prefixes even though the full source is available. We pursue an alternative approach based on standard sequence to sequence models which utilize the entire source. These models perform remarkably well as channel models, even though they have neither been trained on, nor designed to factor over incomplete target sentences. Experiments with neural language models trained on billions of words show that noisy channel models can outperform a direct model by up to 3.2 BLEU on WMT'17 German-English translation. We evaluate on four language-pairs and our channel models consistently outperform strong alternatives such right-to-left reranking models and ensembles of direct models."]},{"idx":1,"heading":"Introduction","paragraphs":["Sequence to sequence models directly estimate the posterior probability of a target sequence $y$ given a source sequence $x$ BIBREF0, BIBREF1, BIBREF2, BIBREF3 and can be trained with pairs of source and target sequences. Unpaired sequences can be leveraged by data augmentation schemes such as back-translation, but direct models cannot naturally take advantage of unpaired data BIBREF4, BIBREF5.","The noisy channel approach is an alternative which is used in statistical machine translation BIBREF6, BIBREF7. It entails a channel model probability $p(x|y)$ that operates in the reverse direction as well as a language model probability $p(y)$. The language model can be estimated on unpaired data and can take a separate form to the channel model. Noisy channel modeling mitigates explaining away effects that result in the source being ignored for highly likely output prefixes BIBREF8.","Previous work on neural noisy channel modeling relied on a complex latent variable model that incrementally processes source and target prefixes BIBREF9. This trades efficiency for accuracy because their model performs significantly less well than a vanilla sequence to sequence model. For languages with similar word order, it can be sufficient to predict the first target token based on a short source prefix, but for languages where word order differs significantly, we may need to take the entire source sentence into account to make a decision.","In this paper, we show that a standard sequence to sequence model is an effective parameterization of the channel probability. We train the model on full sentences and apply it to score the source given an incomplete target sentence. This bases decoding decisions on scoring the entire source sequence and it is very simple and effective (§SECREF2). We analyze this approach for various target prefix sizes and find that it is most accurate for large target context sizes. Our simple noisy channel approach consistently outperforms strong baselines such as online ensembles and left-to-right re-ranking setups (§SECREF3)."]},{"idx":2,"heading":"Approach","paragraphs":["The noisy channel approach applies Bayes' rule to model $p(y|x) = p(x|y) p(y)/ p(x)$, that is, the channel model $p(x|y)$ operating from the target to the source and a language model $p(y)$. We do not model $p(x)$ since it is constant for all $y$. We compute the channel model probabilities as follows:","We refer to $p(y|x)$ as the direct model. A critical choice in our approach is to model $p(x|y)$ with a standard Transformer architecture BIBREF3 as opposed to a model which factors over target prefixes BIBREF9. This setup presents a clear train/test mismatch: we train $p(x|y)$ on complete sentence-pairs and perform inference with incomplete target prefixes of varying size $k$, i.e., $p(x|y_1,\\dots ,y_k)$. However, we find standard sequence to sequence models to be very robust to this mismatch (§SECREF3)."]},{"idx":3,"heading":"Approach ::: Decoding.","paragraphs":["To generate $y$ given $x$ with the channel model, we wish to compute $\\operatornamewithlimits{arg\\,max}_y \\log p(x|y) + \\log p(y)$. However, naïve decoding in this way is computationally expensive because the channel model $p(x|y)$ is conditional on each candidate target prefix. For the direct model, it is sufficient to perform a single forward pass over the network parameterizing $p(y|x)$ to obtain output word probabilities for the entire vocabulary. However, the channel model requires separate forward passes for each vocabulary word."]},{"idx":4,"heading":"Approach ::: Approximation.","paragraphs":["To mitigate this issue, we perform a two-step beam search where the direct model pre-prunes the vocabulary BIBREF9. For beam size $k_1$, and for each beam, we collect $k_2$ possible next word extensions according to the direct model. Next, we score the resulting $k_1 \\times k_2$ partial candidates with the channel model and then prune this set to size $k_1$. Other approaches to pre-pruning may be equally beneficial but we adopt this approach for simplicity. A downside of online decoding with the channel model approach is the high computational overhead: we need to invoke the channel model $k_1 \\times k_2$ times compared to just $k_1$ times for the direct model."]},{"idx":5,"heading":"Approach ::: Complexity.","paragraphs":["The model of BIBREF9 factorizes over source and target prefixes. During decoding, their model alternates between incrementally reading the target prefix and scoring a source prefix, resulting in a runtime of $O(n+m)$, where $n$ and $m$ are the source and target lengths, respectively. In comparison, our approach repeatedly scores the entire source for each target prefix, resulting in $O(mn)$ runtime. Although our approach has greater time complexity, the practical difference of scoring the tokens of a single source sentence instead of just one token is likely to be negligible on modern GPUs since all source tokens can be scored in parallel. Inference is mostly slowed down by the autoregressive nature of decoding. Scoring the entire source enables capturing more dependencies between the source and target, since the beginning of the target must explain the entire source, not just the beginning. This is especially critical when the word order between the source and target language varies considerably, and likely accounts for the lower performance of the direct model of BIBREF9 in comparison to a standard seq2seq model."]},{"idx":6,"heading":"Approach ::: Model combinaton.","paragraphs":["Since the direct model needs to be evaluated for pre-pruning, we also include these probabilities in making decoding decisions. We use the following linear combination of the channel model, the language model and the direct model for decoding:","where $t$ is the length of the target prefix $y$, $s$ is the source sentence length and $\\lambda $ is a tunable weight. Initially, we used separate weights for $p(x|y)$ and $p(y)$ but we found that a single weight resulted in the same accuracy and was easier to tune. Scaling by $t$ and $s$ makes the scores of the direct and channel model comparable to each other throughout decoding. In n-best re-ranking, we have complete target sentences which are of roughly equal length and therefore do not use per word scores."]},{"idx":7,"heading":"Experiments ::: Datasets.","paragraphs":["For English-German (En-De) we train on WMT'17 data, validate on news2016 and test on news2017. For reranking, we train models with a 40K joint byte pair encoding vocabulary (BPE; BIBREF11). To be able to use the language model during online decoding, we use the vocabulary of the langauge model on the target side. For the source vocabulary, we learn a 40K byte pair encoding on the source portion of the bitext; we find using LM and bitext vocabularies give similar accuracy. For Chinese-English (Zh-En), we pre-process WMT'17 data following BIBREF12, we develop on dev2017 and test on news2017. For IWSLT'14 De-En we follow the setup of BIBREF13 and measure case-sensitive tokenized BLEU. For WMT De-En, En-De and Zh-En we measure detokenized BLEU BIBREF14."]},{"idx":8,"heading":"Experiments ::: Language Models.","paragraphs":["We train two big Transformer language models with 12 blocks BIBREF15: one on the German newscrawl data distributed by WMT'18 comprising 260M sentences and another one on the English newscrawl data comprising 193M sentences. Both use a BPE vocabulary of 32K types. We train on 32 Nvidia V100 GPUs with 16-bit floating point operations BIBREF16 and training took four days."]},{"idx":9,"heading":"Experiments ::: Sequence to Sequence Model training.","paragraphs":["For En-De, De-En, Zh-En we use big Transformers and for IWSLT De-En a base Transformer BIBREF3 as implemented in fairseq BIBREF17. For online decoding experiments, we do not share encoder and decoder embeddings since the source and target vocabularies were learned separately. We report average accuracy of three random initializations of a each configuration. We generally use $k_1=5$ and $k_2=10$. We tune $\\lambda _1$, and a length penalty on the validation set."]},{"idx":10,"heading":"Experiments ::: Simple Channel Model","paragraphs":["We first motivate a standard sequence to sequence model to parameterize $p(x|y)$ as opposed to a model that is trained to operate over prefixes. We train Transformer models to translate from the target to the source (En-De) and compare two variants: i) a standard sequence to sequence model trained to predict full source sentences based on full targets (seq2seq). ii) a model trained to predict the full source based on a prefix of the target; we train on all possible prefixes of a target sentence, each paired with the full source (prefix-model).","Figure FIGREF12 shows that the prefix-model performs slightly better for short target prefixes but this advantage disappears after 15 tokens. On full target sentences seq2seq outperforms the prefix model by 5.7 BLEU. This is likely because the prefix-model needs to learn how to process both long and short prefixes which results in lower accuracy. The lower performance on long prefixes is even more problematic considering our subsequent finding that channel models perform over-proportionally well on long target prefixes (§SECREF18). The seq2seq model has not been trained to process incomplete targets but empirically it provides a simple and effective parameterization of $p(x|y)$."]},{"idx":11,"heading":"Experiments ::: Effect of Scoring the Entire Source Given Partial Target Prefixes","paragraphs":["The model of BIBREF9 uses a latent variable to incrementally score the source for prefixes of the target. Although this results in a faster run time, the model makes decoding decisions based on source prefixes even though the full source is available. In order to quantify the benefit of scoring the entire source instead of a learned prefix length, we simulate different fractions of the source and target in an n-best list reranking setup.","The n-best list is generated by the direct model and we re-rank the list in setups where we only have a fraction of the candidate hypothesis and the source sentence. We report BLEU of the selected full candidate hypothesis.","Figure FIGREF15 shows that for any given fraction of the target, scoring the entire source (src 1) has better or comparable performance than all other source prefix lengths. It is therefore beneficial to have a channel model that scores the entire source sentence."]},{"idx":12,"heading":"Experiments ::: Online Decoding","paragraphs":["Next, we evaluate online decoding with a noisy channel setup compared to just a direct model () as well as an ensemble of two direct models (). Table TABREF16 shows that adding a language model to () gives a good improvement BIBREF18 over a single direct model but ensembling two direct models is slightly more effective (). The noisy channel approach () improves by 1.9 BLEU over on news2017 and by 0.9 BLEU over the ensemble. Without per word scores, accuracy drops because the direct model and the channel model are not balanced and their weight shifts throughout decoding. Our simple approach outperforms strong online ensembles which illustrates the advantage over incremental architectures BIBREF9 that do not match vanilla seq2seq models by themselves."]},{"idx":13,"heading":"Experiments ::: Analysis","paragraphs":["Using the channel model in online decoding enables searching a much larger space compared to n-best list re-ranking. However, online decoding is also challenging because the channel model needs to score the entire source sequence given a partial target which can be hard. To measure this, we simulate different target prefix lengths in an n-best list re-ranking setup. The n-best list is generated by the direct model and we re-rank it for different target prefixes of the candidate hypothesis. As in SECREF14, we measure BLEU of the selected full candidate hypothesis. Figure FIGREF19 shows that the channel model enjoys much larger benefits from more target context than re-ranking with just the direct model and an LM () or re-ranking with a direct ensemble (). This experiment shows the importance of large context sizes for the channel approach to work well. It indicates that the channel approach may not be able to effectively exploit the large search space in online decoding due to the limited conditioning context provided by partial target prefixes."]},{"idx":14,"heading":"Experiments ::: Re-ranking","paragraphs":["Next, we switch to n-best re-ranking where we have the full target sentence available compared to online decoding. Noisy channel model re-ranking has been used by the top ranked entries of the WMT 2019 news translation shared task for English-German, German-English, Englsh-Russian and Russian-English BIBREF19. We compare to various baselines including right-to-left sequence to sequence models which are a popular choice for re-ranking and regularly feature in successful WMT submissions BIBREF20, BIBREF21, BIBREF22.","Table TABREF20 shows that the noisy channel model outperforms the baseline () by up to 4.0 BLEU for very large beams, the ensemble by up to 2.9 BLEU () and the best right-to-left configuration by 1.4 BLEU (). The channel approach improves more than other methods with larger n-best lists by adding 2.4 BLEU from $k_1=5$ to $k_1=100$. Other methods improve a lot less with larger beams, e.g., has the next largest improvement of 1.4 BLEU when increasing the beam size but this is still significantly lower than for the noisy channel approach. Adding a language model benefits all settings (, , ) but the channel approach benefits most ( vs ). The direct model with a language model () performs better than for online decoding, likely because the constrained re-ranking setup mitigates explaining away effects (cf. Table TABREF16).","Interestingly, both or give only modest improvements compared to . Although previous work demonstrated that reranking with can improve over , we show that the channel model is important to properly leverage the language model without suffering from explaining away effects BIBREF23, BIBREF24. Test results on all language directions confirm that performs best (Table TABREF21)."]},{"idx":15,"heading":"Conclusion","paragraphs":["Previous work relied on incremental channel models which do not make use of the entire source even though it is available and, as we demonstrate, beneficial. Standard sequence to sequence models are a simple parameterization for the channel probability that naturally exploits the entire source. This parameterization outperforms strong baselines such as ensembles of direct models and right-to-left models. Channel models are particularly effective with large context sizes and an interesting future direction is to iteratively refine the output while conditioning on previous contexts."]},{"idx":16,"heading":"Figures and Tables","paragraphs":["Figure 1: Comparison of two channel models: a standard seq2seq model trained on full sentence-pairs and a model trained on all possible target prefixes with the full source (prefix-model). We measure accuracy of predicting the full sourcewith increasing target prefixes for both models. Results are on news2016.","Figure 3: Impact of target prefix length for the channel model (CH+DIR+LM), direct model + LM (DIR+LM) and a direct ensemble (DIR ENS). We show detokenized BLEU on WMT De-En news2016 with beam 10.","Figure 2: For any given target prefix fraction, scoring the entire source has the best or comparable performance compared to other source prefixes. We show detokenized BLEU on the dev set of WMT17 Zh-En with beam 50.","Table 1: Online decoding accuracy for a direct model (DIR), ensembling two direct models (DIR ENS) and the channel approach (CH+DIR+LM). We ablate the impact of using per word scores. Results are on WMT De-En. Table 4 in the appendix shows standard deviations.","Table 2: Re-ranking BLEU with different n-best list sizes on news2016 of WMT De-En. We compare to decoding with a direct model only (DIR) and decoding with an ensemble of direct models (DIR ENS). Table 5 in the appendix shows standard deviations.","Table 3: Re-ranking accuracy with k1 = 50 on four language directions on the respective test sets. See Table 6 in the appendix for standard deviations.","Table 4: Online decoding accuracy for a direct model (DIR), ensembling two direct models (DIR ENS) and the channel approach (CH+DIR+LM). We ablate the impact of length normalization. Results are on news2017 of WMT De-En.","Table 5: Re-ranking BLEU with different n-best list sizes on news2016 of WMT De-En.","Table 6: Re-ranking accuracy with k1 = 50 on four language directions on the respective test sets."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1908.06259","title":"Hard but Robust, Easy but Sensitive: How Encoder and Decoder Perform in Neural Machine Translation","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Neural machine translation (NMT) typically adopts the encoder-decoder framework. A good understanding of the characteristics and functionalities of the encoder and decoder can help to explain the pros and cons of the framework, and design better models for NMT. In this work, we conduct an empirical study on the encoder and the decoder in NMT, taking Transformer as an example. We find that 1) the decoder handles an easier task than the encoder in NMT, 2) the decoder is more sensitive to the input noise than the encoder, and 3) the preceding words/tokens in the decoder provide strong conditional information, which accounts for the two observations above. We hope those observations can shed light on the characteristics of the encoder and decoder and inspire future research on NMT."]},{"idx":1,"heading":"Introduction","paragraphs":["The encoder-decoder based framework BIBREF0, BIBREF1, BIBREF2 is the dominant approach for neural machine translation (NMT) BIBREF3, BIBREF4. Although the encoder and decoder usually adopt the same model structure (RNN BIBREF5, CNN BIBREF6 or self-attention BIBREF3, BIBREF7) and the same number of layers, they perform different functionalities: the encoder extracts the hidden representations of the source sentence, and the decoder generates target tokens conditioned on the source hidden representations as well as the previous generated tokens.","While most existing works focus on the design and improvement of encoder-decoder framework for NMT BIBREF8, BIBREF6, BIBREF3, BIBREF9 as well as its detailed analyses BIBREF10, BIBREF11, BIBREF12, BIBREF13, few works concentrate on the characteristics and functionalities of the encoder and the decoder, which are valuable to understand this popular framework and improve its performance in NMT. Therefore, in this paper, we conduct a study and aim to understand the characteristics of the encoder and the decoder in NMT. We observe some interesting phenomena:","The decoder handles an easier task than the encoder. 1) We find that adding more layers to the encoder achieves larger improvements than adding more layers to the decoder. 2) We also compare the training time of the encoder and decoder by fixing the parameters of a well-trained decoder (encoder), and just update the parameters of the encoder (decoder). We found that the decoder converges faster than the encoder. These two results suggest that the decoder handles an easier task than the encoder in NMT.","The decoder is more sensitive to the input noise than the encoder. We randomly add different level of noise to the input of the encoder and decoder respectively during inference, and find that adding noise to the input of the decoder leads to better accuracy drop than that of the encoder.","We further analyze why the decoder is more sensitive by masking the previous tokens, and comparing autoregressive NMT with the non-autoregressive counterpart. We find that the preceding tokens in the decoder provide strong conditional information, which partially explain the previous two observations on the decoder.","We believe our studies on the different characteristics of the encoder and decoder will inspire the following research on the encoder-decoder framework as well as improve the performance on NMT and other encoder-decoder based tasks."]},{"idx":2,"heading":"Related Work","paragraphs":["In this section, we mainly review the work on the analysis of encoder-decoder framework in the field of NMT. BIBREF10 analyzed how NMT works based on the encoder-decoder framework and explained the translation errors with the proposed layer-wise relevance propagation. BIBREF14 focused on the representation of different layers of the encoder in NMT. BIBREF15 concluded that NMT systems are brittle. In the view of attention mechanism, BIBREF11 studied the encoder-decoder attention and provided detailed analysis of what is being learned by the attention mechanism in NMT model. BIBREF16 studied the effectiveness of hybrid attention in neural machine translation. BIBREF17 introduced dense connection in encoder-decoder attention. BIBREF13 analyzed the encoder-decoder attention mechanisms in the case of word sense disambiguation. For model architecture, BIBREF18 provided early study on RNN-based architectural hyperparameters. BIBREF19 proposed to pre-train the encoder-attention-decoder framework jointly. BIBREF12 conducted fine-grained analyses on various modules and made comparisons between RNN, CNN and self-attention. While the works above provide a better understanding of NMT model from the perspective of attention mechanism, network architecture and language modeling, few of them focus on the comparison of the characteristics for the encoder and the decoder in NMT. This is exactly what we focus in this paper."]},{"idx":3,"heading":"Hard for Encoder, Easy for Decoder","paragraphs":["In general, the encoder and decoder perform different functionalities in an NMT model. In this section, we compare the characteristics between the encoder and decoder by analyzing the difficulty of the corresponding task they handle in NMT. We investigate the task difficulty by comparing the training effort of the encoder and decoder from two perspectives: the number of layers and the convergence speed for the encoder and decoder respectively.","We train the Transformer BIBREF3 on IWSLT14 German$\\leftrightarrow $English (De$\\leftrightarrow $En) and Romanian$\\leftrightarrow $English (Ro$\\leftrightarrow $En) translation. The translation quality is measured by BLEU score. More details on datasets, model configurations and evaluation metrics can be found in the supplementary materials (Section 1.1)."]},{"idx":4,"heading":"Hard for Encoder, Easy for Decoder ::: Deeper Encoder Brings More Gains Than Deeper Decoder","paragraphs":["We vary the number of layers for the encoder and decoder respectively, to investigate which can achieve more gains with deeper layers. Deeper encoder (decoder) with more gains can indicate the task that encoder (decoder) handles needs more representation capability, which is harder to learn. We consider two different strategies when varying the number of layers: 1) increasing the number of layers for the encoder and decoder respectively based on a baseline model with a 2-layer encoder and a 2-layer decoder; 2) adjusting the number of layers for the encoder and decoder under the constraint that the total number of layers for encoder and decoder is fixed (totally 12 layers in our experiments).","The results are demonstrated in Figure FIGREF6 and FIGREF6. We have two observations: 1) adding more layers to the encoder brings more gain than the decoder (Figure FIGREF6); 2) the model with a shallow encoder typically performs worse than the model with a shallow decoder (Figure FIGREF6)."]},{"idx":5,"heading":"Hard for Encoder, Easy for Decoder ::: Encoder Converges Slower than Decoder","paragraphs":["We further investigate the difficulty of the task for the encoder and decoder by comparing their convergence speed. Encoder (decoder) that needs more training time indicates the task that encoder (decoder) handles is more difficult.","We evaluate the convergence speed for the encoder by initializing the model with the parameters of a well-trained decoder and fixing them during training, and initializing other components with random variables. We follow the opposite practice when evaluating the convergence speed for the decoder. We show the BLEU scores along the training process on all the four datasets in Figure FIGREF6, which demonstrates that the decoder converges faster than the encoder."]},{"idx":6,"heading":"Hard for Encoder, Easy for Decoder ::: Encoder Converges Slower than Decoder ::: Discussion","paragraphs":["From the analyses in the last two subsections, it can be seen that increasing the layer of encoder brings more gains than increasing the layer of the decoder, and encoder needs more training time than the decoder, which demonstrate that the encoder of the NMT model handles a more difficult task than the decoder."]},{"idx":7,"heading":"Robust Encoder, Sensitive Decoder","paragraphs":["In this section, we compare the characteristics between the encoder and decoder by analyzing their robustness according to the input noise in the inference phase. We simulate the input noise with three typical operations BIBREF20, BIBREF21: 1) random dropping: we randomly drop the input tokens of encoder and decoder respectively with different drop rates; 2) random noising: we randomly select tokens and replace its embedding with random noise; 3) random swapping: we randomly reverse the order for the adjacent tokens. The decoder in NMT model typically generates the current token one-by-one conditioned on the previous generated tokens, which suffers from error propagation BIBREF22: if a token is incorrectly predicted by the decoder, it will affect the prediction of the following tokens. Adding input noise to the decoder will further enhance the effect of error propagation, and thus influence our analysis. To eliminate the influence of error propagation, we apply teacher forcing BIBREF23 in the inference phase by feeding the previous ground-truth target tokens instead of the previously generated target tokens, following BIBREF24. We evaluate our model on IWSLT14 De$\\leftrightarrow $En, IWSLT14 Ro$\\leftrightarrow $En and WMT17 Chinese$\\leftrightarrow $English (Zh$\\leftrightarrow $En) translation tasks. More details on experimental configurations are described in supplementary materials (Section 1.2). The results are demonstrated in Figure FIGREF10. It can be seen that as the perturbation rate increases, adding different types of noise to the decoder input consistently achieves lower translation quality than adding noise to the encoder input."]},{"idx":8,"heading":"Robust Encoder, Sensitive Decoder ::: Discussion","paragraphs":["The consistent observations above suggest that the decoder is much more sensitive to the input noise than the encoder. Intuitively, the encoder aims at extracting abstract representations of the source sentence instead of depending on certain input tokens for prediction as the decoder does, demonstrating that the encoder is more robust than the decoder."]},{"idx":9,"heading":"Further Analysis","paragraphs":["Given the observations in Section SECREF3 and SECREF4, we find that the decoder is more sensitive to the input tokens compared with the encoder, and the task that decoder handles is easier. In this section, we give an explanation on this phenomenon. More details on experimental configurations are described in supplementary materials (Section 1.3). Besides, we also investigate which kind of input tokens the encoder and decoder are more sensitive to, from the perspective of Part-Of-Speech (POS), and show the results in the supplementary materials (Section 2) due to space limitation."]},{"idx":10,"heading":"Further Analysis ::: Why Decoder is Easier and More Sensitive","paragraphs":["The decoder in NMT model typically acts as a conditional language model, which generates tokens highly depending on the previous tokens, like the standard language model BIBREF25. We guess the conditional information (especially the tokens right before the predicted token) is too strong for the decoder. Therefore, we study the impact of the previous tokens as follows. For each predicted token $w_{t}$, where $t$ is the position in the target sentence, we drop its previous token $w_{t-n}$ from the decoder input and watch the performance changes, where $n\\in [1, t]$ is the distance between the dropping token and the current predicted token. Note that the experiments are conducted in the inference phase and evaluated with teacher forcing. As shown in Figure FIGREF14, when dropping the token close to the predicted token, the accuracy declines more heavily than dropping the token far away, which indicates the decoder depends more on the nearby tokens."]},{"idx":11,"heading":"Further Analysis ::: Comparison with Non-Autoregressive NMT","paragraphs":["Since an autoregressive decoding process highly depends on previous tokens, we further investigate the characteristics of non-autoregressive decoding in NMT BIBREF26, BIBREF27, where each target token is generated in parallel, without dependence on previous tokens. We compare the accuracy with different dropping rates for the encoder input, encoder output (the input of the encoder-decoder attention) and decoder input respectively, both for autoregressive NMT and non-autoregressive NMT, and illustrate the results in Figure FIGREF14 and FIGREF14. It can be observed that for autoregressive NMT, the model is more sensitive to the decoder input, and then the encoder input, and less sensitive to the encoder output. While for non-autoregressive NMT, the sensitivity is opposite to the autoregressive counterpart: the model is more sensitive to the encoder input, then encoder output, but less sensitive to the decoder input, due to the nature of non-autoregressive NMT that the decoder has to relying more on the source sentence."]},{"idx":12,"heading":"Conclusions and Future Work","paragraphs":["In this paper, we conducted a series of experiments to compare the characteristics of the encoder and decoder in NMT. We found that the decoder handles an easier task than the encoder, and the decoder is more sensitive to the input noise than the encoder. We further investigated why the decoder is more sensitive and the task it handles is easier, by analyzing the dependence of the decoder, and comparing the sensitivity to the input with non-autoregressive NMT. We hope our analyses inspire future research on NMT.","For future work, we will study the encoder-decoder framework in other machine learning tasks, such as text summarization, question answering and image captioning."]},{"idx":13,"heading":"Supplemental Material ::: Detailed Experimental Settings","paragraphs":["In this section, we give detailed descriptions on all the experimental settings in this work."]},{"idx":14,"heading":"Supplemental Material ::: Detailed Experimental Settings ::: Experimental Settings for Section 3","paragraphs":["We train the Transformer model BIBREF3 with different number of layers on IWSLT14 German$\\leftrightarrow $English (De$\\leftrightarrow $En) translation task and Romanian$\\leftrightarrow $English (Ro$\\leftrightarrow $En) translation task."]},{"idx":15,"heading":"Supplemental Material ::: Detailed Experimental Settings ::: Experimental Settings for Section 3 ::: Datasets","paragraphs":["The training sets of IWSLT14 De$\\leftrightarrow $En and IWSLT14 Ro$\\leftrightarrow $En contain $153K$ and $182K$ sentence pairs respectively. We use IWSLT14.TED.tst2013 as the validation set and IWSLT14.TED.tst2014 as the test set. For De$\\rightarrow $En translation, to be consistent with the previous work BIBREF29, BIBREF32, BIBREF28, we use the same validation and test sets as those used in BIBREF29, which consists of $7K$ and $7K$ sentences respectively, and follow the common practice to lowercase all words. All the datasets are preprocessed into workpieces following BIBREF5."]},{"idx":16,"heading":"Supplemental Material ::: Detailed Experimental Settings ::: Experimental Settings for Section 3 ::: Model Configurations","paragraphs":["Our experiments are based on transformer_small setting except for the number of layers. The model hidden size, feed-forward hidden size and the number of head are set to 256, 1024 and 4 respectively. We train each model on one NVIDIA GTX1080Ti GPU until convergence and the mini-batch size is fixed as 4096 tokens. We choose the Adam optimizer with $\\beta _{1} = 0.9$, $\\beta _{2} = 0.98$ and $\\varepsilon = 10^{-9}$. The learning rate schedule is consistent with BIBREF3."]},{"idx":17,"heading":"Supplemental Material ::: Detailed Experimental Settings ::: Experimental Settings for Section 3 ::: Evaluation","paragraphs":["We measure our translation quality by tokenized case-senstive BLEU BIBREF31 with multi-bleu.pl for De$\\leftrightarrow $En and sacreBLEU for Ro$\\leftrightarrow $En, which is consistent with previous methods. During inference, we generate target tokens autoregressively and use beam search with $beam=6$ and length penalty $\\alpha =1.1$. Larger BLEU score indicate better translation quality."]},{"idx":18,"heading":"Supplemental Material ::: Detailed Experimental Settings ::: Experimental Settings for Section 3 ::: Model Adaption","paragraphs":["We investigate the task difficulty by comparing the training effort of the encoder and decoder from two perspectives in the paper: 1) We vary the number of layers for the encoder and decoder respectively. All models are trained with the same configuration as described before except for the number of layers. 2) For training the encoder side, we initialize the decoder side with the parameters of a well-trained decoder, and initialize other components with random variable. For training decoder side, we follow the opposite operation."]},{"idx":19,"heading":"Supplemental Material ::: Detailed Experimental Settings ::: Experimental Settings for Section 4","paragraphs":["In Section 4, we do not train models from scratch and only evaluate well-trained models with different kind of perturbation to explore the robustness of the encoder and decoder."]},{"idx":20,"heading":"Supplemental Material ::: Detailed Experimental Settings ::: Experimental Settings for Section 4 ::: Datasets","paragraphs":["For IWSLT14 De$\\leftrightarrow $En and IWSLT14 Ro$\\leftrightarrow $En, we use the same validation and test sets as before. For WMT17 Chinese$\\leftrightarrow $English (Zh$\\leftrightarrow $En), there are $24M$ sentence pairs for training, $2K$ for both validation and test set. Sentences are encoded using sub-word types based on Byte-Pair-Encoding (BPE) BIBREF33, which has $40K$ BPE tokens for the source vocabulary and $37K$ BPE tokens for the target vocabulary. We use newsdev2017 and newstest2017 as validation and test sets respectively."]},{"idx":21,"heading":"Supplemental Material ::: Detailed Experimental Settings ::: Experimental Settings for Section 4 ::: Model Configuration","paragraphs":["For IWSLT14 De$\\leftrightarrow $En and IWSLT14 Ro$\\leftrightarrow $En, we use the same model configurations as before. The number of layers for the encoder and decoder are both fixed to 6. For WMT17 Zh$\\leftrightarrow $En translation, we use transformer_big setting. The model hidden size, feed-forward hidden size and the number of head are set to 1024, 4096 and 16 respectively. We train the model for Zh$\\leftrightarrow $En on 8 NVIDIA TESLA P40 GPUs until convergence. The optimizer and learning rate schedule is consistent with IWSLT14 transaltion tasks."]},{"idx":22,"heading":"Supplemental Material ::: Detailed Experimental Settings ::: Experimental Settings for Section 4 ::: Evaluation","paragraphs":["For IWSLT14 De$\\leftrightarrow $En and IWSLT14 Ro$\\leftrightarrow $En, we use the same evaluation metrics as before. For WMT17 Zh$\\leftrightarrow $En translation, we calculate the detokenized BLEU score by sacreBLEU. As described in the paper, to eliminate the influence of error propagation, we apply teacher forcing BIBREF23 in the inference phase by feeding the previous ground-truth target tokens instead of the previously generated tokens. Therefore, we just add the noise to the ground-truth target tokens when adding noise to the decoder input. We use greedy inference for all settings and length penalty $\\alpha =1.1$."]},{"idx":23,"heading":"Supplemental Material ::: Detailed Experimental Settings ::: Experimental Settings for Section 4 ::: Model Adaption","paragraphs":["For consistency, we use the same well-trained model for each translation task and simulate the input noise with three typical operations to explore the characteristics for the encoder and decoder. In Table TABREF41, we demonstrate the BLEU scores of all the models we used in Section 4 of the paper."]},{"idx":24,"heading":"Supplemental Material ::: Detailed Experimental Settings ::: Experimental Settings for Section 5","paragraphs":["We give an explanation on why decoder is easier and sensitive in Section 5 of the paper. We additionally train a non-autoregressive NMT model for comparison."]},{"idx":25,"heading":"Supplemental Material ::: Detailed Experimental Settings ::: Experimental Settings for Section 5 ::: Datasets","paragraphs":["We use the same validation and test sets as before for IWSLT14 translation tasks."]},{"idx":26,"heading":"Supplemental Material ::: Detailed Experimental Settings ::: Experimental Settings for Section 5 ::: Model Configuration","paragraphs":["We use the same model configurations as before for autoregressive model. As to non-autoregressive counterpart, we use the same network architecture as in NART BIBREF30. Both autoregressive and non-autoregressive model are based on transformer_small setting and the number of layers for the encoder and decoder are fixed to 6. The model hidden size, feed-forward hidden size and the number of head are set to 256, 1024 and 4 respectively. We train non-autoregressive NMT model on one NVIDIA GTX1080Ti GPU until convergence. The optimizer and learning rate schedule is the same as before."]},{"idx":27,"heading":"Supplemental Material ::: Detailed Experimental Settings ::: Experimental Settings for Section 5 ::: Evaluation","paragraphs":["We use the same evaluation metrics for autoregressive model and non-autoregressive model. We evaluate both models with greedy inference and $\\alpha =1.1$. The BLEU scores are demonstrated in Table TABREF41."]},{"idx":28,"heading":"Supplemental Material ::: Detailed Experimental Settings ::: Experimental Settings for Section 5 ::: Model Adaption","paragraphs":["In section 5 of the paper, we study the importance of previous tokens by dropping the input tokens $w_{t-n}$ for the decoder, where $w_t$ is the current token in target sentence and $n \\in [1,t]$ is the distance to the token dropped. We drop the same number of tokens for each $n$. We also compare the autoregressive model with non-autoregressive counterpart by randomly dropping the tokens from the encoder input, encoder output and decoder input. The dropping operation is consistent with Section 4 of the paper."]},{"idx":29,"heading":"Supplemental Material ::: How POS Influences the Encoder and Decoder","paragraphs":["Different kinds of Part-Of-Speech (POS) play different roles in a sentence. For example, nouns typically represents objects and verbs typically conveys actions. To study the impact of POS on the encoder and decoder in NMT, we drop the tokens belongs to each kind of POS and observe the accuracy changes on different kinds of POS.","Due to the definition of POS varies among different languages, we focus on our analysis on English for simplicity and evaluate our trained model on both De$\\rightarrow $En and En$\\rightarrow $De for fair comparison, i.e., we drop the input tokens of the encoder for En$\\rightarrow $De translation, and drop the input tokens of the decoder for De$\\rightarrow $En translation.","We show the results of the popular POS types (the POS that covers more than 3000 tokens in the test set) in Figure FIGREF48. Since different POS cover different number of tokens in the test set, the direct comparison between different POS is unfair. We introduce a null baseline by randomly dropping different number of tokens and observe the accuracy changes. Based on this data, we fit the curve between the translation accuracy and the number of dropping tokens. We compare each POS with the null baseline by computing the delta BLEU between the accuracy when dropping this kind of POS and the accuracy when dropping tokens randomly. If the delta BLEU is less than 0 (dropping a certain POS achieves lower BLEU score than the null baseline), the model is supposed to be more sensitive to this kind of POS.","The results are shown in Figure FIGREF48. It can be seen that both the encoder and decoder are more sensitive to noun (NN) and adjective (JJ). However, compared with the encoder, the decoder is less sensitive to preposition (IN), determiner (DT), pronoun (PRP), adverb (RB), verb (VB) and coordinating conjunction (CC) but much more sensitive to EOS, which is used to determine the end of a sentence."]},{"idx":30,"heading":"Figures and Tables","paragraphs":["Figure 1: Experimental results for analyzing the training effort for the encoder and decoder. X-axes in Figure 1a and 1b indicate the number of layers for encoder and decoder respectively (Encoder-Decoder). Y-axes in Figure 1a and 1b indicate the delta BLEU to the 2-2 layer model (Figure 1a) and the 6-6 layer model (Figure 1b).","Figure 2: The accuracy drop (compared to the model with no input noise) when adding noise to the input tokens of the encoder and decoder. X-axis indicates the perturbation rate (%), i.e., the ratio of tokens being perturbed by noise.","Figure 3: Further analyses on characteristics of the encoder and decoder. X-axis in Figure 3a indicates the distance between the dropped token and the current predicted token. X-axes in Figure 3b and 3c indicate the perturbation rate (%). See text for the detailed description.","Table 1: The BLEU scores of the models we used on various translation tasks. All models are evaluated with greedy inference and α = 1.1. TF indicates the model equipped with teacher forcing. (N) indicates the results from non-autoregressive NMT model.","Figure 4: The sensitivity of different kinds of POS for the encoder and the decoder. X-axis represents the number of dropped tokens. Y-axis represents the delta BLEU between dropping different kinds of POS and the null baseline. Negative ∆ BLEU score represents that dropping a certain type of POS results in worse accuracy than the null baseline, which indicates the encoder or the decoder is more sensitive to this kind of POS, and vice versa."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1908.06267","title":"Message Passing Attention Networks for Document Understanding","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Graph neural networks have recently emerged as a very effective framework for processing graph-structured data. These models have achieved state-of-the-art performance in many tasks. Most graph neural networks can be described in terms of message passing, vertex update, and readout functions. In this paper, we represent documents as word co-occurrence networks and propose an application of the message passing framework to NLP, the Message Passing Attention network for Document understanding (MPAD). We also propose several hierarchical variants of MPAD. Experiments conducted on 10 standard text classification datasets show that our architectures are competitive with the state-of-the-art. Ablation studies reveal further insights about the impact of the different components on performance. Code is publicly available at: https://github.com/giannisnik/mpad ."]},{"idx":1,"heading":"Introduction","paragraphs":["The concept of message passing over graphs has been around for many years BIBREF0, BIBREF1, as well as that of graph neural networks (GNNs) BIBREF2, BIBREF3. However, GNNs have only recently started to be closely investigated, following the advent of deep learning. Some notable examples include BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11, BIBREF12. These approaches are known as spectral. Their similarity with message passing (MP) was observed by BIBREF9 and formalized by BIBREF13 and BIBREF14.","The MP framework is based on the core idea of recursive neighborhood aggregation. That is, at every iteration, the representation of each vertex is updated based on messages received from its neighbors. All spectral GNNs can be described in terms of the MP framework.","GNNs have been applied with great success to bioinformatics and social network data, for node classification, link prediction, and graph classification. However, a few studies only have focused on the application of the MP framework to representation learning on text. This paper proposes one such application. More precisely, we represent documents as word co-occurrence networks, and develop an expressive MP GNN tailored to document understanding, the Message Passing Attention network for Document understanding (MPAD). We also propose several hierarchical variants of MPAD. Evaluation on 10 document classification datasets show that our architectures learn representations that are competitive with the state-of-the-art. Furthermore, ablation experiments shed light on the impact of various architectural choices.","In what follows, we first provide some background about the MP framework (in sec. SECREF2), thoroughly describe and explain MPAD (sec. SECREF3), present our experimental framework (sec. SECREF4), report and interpret our results (sec. SECREF5), and provide a review of the relevant literature (sec. SECREF6)."]},{"idx":2,"heading":"Message Passing Neural Networks","paragraphs":["BIBREF13 proposed a MP framework under which many of the recently introduced GNNs can be reformulated. MP consists in an aggregation phase followed by a combination phase BIBREF14. More precisely, let $G(V,E)$ be a graph, and let us consider $v \\in V$. At time $t+1$, a message vector $\\mathbf {m}_v^{t+1}$ is computed from the representations of the neighbors $\\mathcal {N}(v)$ of $v$:","The new representation $\\mathbf {h}^{t+1}_v$ of $v$ is then computed by combining its current feature vector $\\mathbf {h}^{t}_v$ with the message vector $\\mathbf {m}_v^{t+1}$:","Messages are passed for $T$ time steps. Each step is implemented by a different layer of the MP network. Hence, iterations correspond to network depth. The final feature vector $\\mathbf {h}_v^T$ of $v$ is based on messages propagated from all the nodes in the subtree of height $T$ rooted at $v$. It captures both the topology of the neighborhood of $v$ and the distribution of the vertex representations in it.","If a graph-level feature vector is needed, e.g., for classification or regression, a READOUT pooling function, that must be invariant to permutations, is applied:","Next, we present the MP network we developed for document understanding."]},{"idx":3,"heading":"Message Passing Attention network for Document understanding (MPAD) ::: Word co-occurrence networks","paragraphs":["We represent a document as a statistical word co-occurrence network BIBREF18, BIBREF19 with a sliding window of size 2 overspanning sentences. Let us denote that graph $G(V,E)$. Each unique word in the preprocessed document is represented by a node in $G$, and an edge is added between two nodes if they are found together in at least one instantiation of the window. $G$ is directed and weighted: edge directions and weights respectively capture text flow and co-occurrence counts.","$G$ is a compact representation of its document. In $G$, immediate neighbors are consecutive words in the same sentence. That is, paths of length 2 correspond to bigrams. Paths of length more than 2 can correspond either to traditional $n$-grams or to relaxed $n$-grams, that is, words that never appear in the same sentence but co-occur with the same word(s). Such nodes are linked through common neighbors.","Master node. Inspired by BIBREF3, our $G$ also includes a special document node, linked to all other nodes via unit weight bi-directional edges. In what follows, let us denote by $n$ the number of nodes in $G$, including the master node."]},{"idx":4,"heading":"Message Passing Attention network for Document understanding (MPAD) ::: Message passing","paragraphs":["We formulate our AGGREGATE function as:","where $\\mathbf {H}^t \\in \\mathbb {R}^{n \\times d}$ contains node features ($d$ is a hyperparameter), and $\\mathbf {A} \\in \\mathbb {R}^{n \\times n}$ is the adjacency matrix of $G$. Since $G$ is directed, $\\mathbf {A}$ is asymmetric. Also, $\\mathbf {A}$ has zero diagonal as we choose not to consider the feature of the node itself, only that of its incoming neighbors, when updating its representation. Since $G$ is weighted, the $i^{th}$ row of $A$ contains the weights of the edges incoming on node $v_i$. $\\mathbf {D} \\in \\mathbb {R}^{n \\times n}$ is the diagonal in-degree matrix of $G$. MLP denotes a multi-layer perceptron, and $\\mathbf {M}^{t+1} \\in \\mathbb {R}^{n \\times d}$ is the message matrix.","The use of a MLP was motivated by the observation that for graph classification, MP neural nets with 1-layer perceptrons are inferior to their MLP counterparts BIBREF14. Indeed, 1-layer perceptrons are not universal approximators of multiset functions. Note that like in BIBREF14, we use a different MLP at each layer.","Renormalization. The rows of $\\mathbf {D}^{-1}\\mathbf {A}$ sum to 1. This is equivalent to the renormalization trick of BIBREF9, but using only the in-degrees. That is, instead of computing a weighted sum of the incoming neighbors' feature vectors, we compute a weighted average of them. The coefficients are proportional to the strength of co-occurrence between words. One should note that by averaging, we lose the ability to distinguish between different neighborhood structures in some special cases, that is, we lose injectivity. Such cases include neighborhoods in which all nodes have the same representations, and neighborhoods of different sizes containing various representations in equal proportions BIBREF14. As suggested by the results of an ablation experiment, averaging is better than summing in our application (see subsection SECREF30). Note that instead of simply summing/averaging, we also tried using GAT-like attention BIBREF11 in early experiments, without obtaining better results.","As far as our COMBINE function, we use the Gated Recurrent Unit BIBREF20, BIBREF21:","Omitting biases for readability, we have:","where the $\\mathbf {W}$ and $\\mathbf {U}$ matrices are trainable weight matrices not shared across time steps, $\\sigma (\\mathbf {x}) = 1/(1+\\exp (-\\mathbf {x}))$ is the sigmoid function, and $\\mathbf {R}$ and $\\mathbf {Z}$ are the parameters of the reset and update gates. The reset gate controls the amount of information from the previous time step (in $\\mathbf {H}^t$) that should propagate to the candidate representations, $\\tilde{\\mathbf {H}}^{t+1}$. The new representations $\\mathbf {H}^{t+1}$ are finally obtained by linearly interpolating between the previous and the candidate ones, using the coefficients returned by the update gate.","Interpretation. Updating node representations through a GRU should in principle allow nodes to encode a combination of local and global signals (low and high values of $t$, resp.), by allowing them to remember about past iterations. In addition, we also explicitly consider node representations at all iterations when reading out (see Eq. DISPLAY_FORM18)."]},{"idx":5,"heading":"Message Passing Attention network for Document understanding (MPAD) ::: Readout","paragraphs":["After passing messages and performing updates for $T$ iterations, we obtain a matrix $\\mathbf {H}^T \\in \\mathbb {R}^{n \\times d}$ containing the final vertex representations. Let $\\hat{G}$ be graph $G$ without the special document node, and matrix $\\mathbf {\\hat{H}}^T \\in \\mathbb {R}^{(n-1) \\times d}$ be the corresponding representation matrix (i.e., $\\mathbf {H}^T$ without the row of the document node).","We use as our READOUT function the concatenation of self-attention applied to $\\mathbf {\\hat{H}}^T$ with the final document node representation. More precisely, we apply a global self-attention mechanism BIBREF22 to the rows of $\\mathbf {\\hat{H}}^T$. As shown in Eq. DISPLAY_FORM17, $\\mathbf {\\hat{H}}^T$ is first passed to a dense layer parameterized by matrix $\\mathbf {W}_A^T \\in \\mathbb {R}^{d \\times d}$. An alignment vector $\\mathbf {a}$ is then derived by comparing, via dot products, the rows of the output of the dense layer $\\mathbf {Y}^T \\in \\mathbb {R}^{(n-1) \\times d}$ with a trainable vector $\\mathbf {v}^T \\in \\mathbb {R}^d$ (initialized randomly) and normalizing with a softmax. The normalized alignment coefficients are finally used to compute the attentional vector $\\mathbf {u}^T \\in \\mathbb {R}^d$ as a weighted sum of the final representations $\\mathbf {\\hat{H}}^T$.","Note that we tried with multiple context vectors, i.e., with a matrix $\\mathbf {V}^T$ instead of a vector $\\mathbf {v}^T$, like in BIBREF22, but results were not convincing, even when adding a regularization term to the loss to favor diversity among the rows of $\\mathbf {V}^T$.","Master node skip connection. $\\mathbf {h}_G^T \\in \\mathbb {R}^{2d}$ is obtained by concatenating $\\mathbf {u}^T$ and the final master node representation. That is, the master node vector bypasses the attention mechanism. This is equivalent to a skip or shortcut connection BIBREF23. The reason behind this choice is that we expect the special document node to learn a high-level summary about the document, such as its size, vocabulary, etc. (more details are given in subsection SECREF30). Therefore, by making the master node bypass the attention layer, we directly inject global information about the document into its final representation.","Multi-readout. BIBREF14, inspired by Jumping Knowledge Networks BIBREF12, recommend to not only use the final representations when performing readout, but also that of the earlier steps. Indeed, as one iterates, node features capture more and more global information. However, retaining more local, intermediary information might be useful too. Thus, instead of applying the readout function only to $t=T$, we apply it to all time steps and concatenate the results, finally obtaining $\\mathbf {h}_G \\in \\mathbb {R}^{T \\times 2d}$ :","In effect, with this modification, we take into account features based on information aggregated from subtrees of different heights (from 1 to $T$), corresponding to local and global features."]},{"idx":6,"heading":"Message Passing Attention network for Document understanding (MPAD) ::: Hierarchical variants of MPAD","paragraphs":["Through the successive MP iterations, it could be argued that MPAD implicitly captures some soft notion of the hierarchical structure of documents (words $\\rightarrow $ bigrams $\\rightarrow $ compositions of bigrams, etc.). However, it might be beneficial to explicitly capture document hierarchy. Hierarchical architectures have brought significant improvements to many NLP tasks, such as language modeling and generation BIBREF24, BIBREF25, sentiment and topic classification BIBREF26, BIBREF27, and spoken language understanding BIBREF28, BIBREF29. Inspired by this line of research, we propose several hierarchical variants of MPAD, detailed in what follows. In all of them, we represent each sentence in the document as a word co-occurrence network, and obtain an embedding for it by applying MPAD as previously described.","MPAD-sentence-att. Here, the sentence embeddings are simply combined through self-attention.","MPAD-clique. In this variant, we build a complete graph where each node represents a sentence. We then feed that graph to MPAD, where the feature vectors of the nodes are initialized with the sentence embeddings previously obtained.","MPAD-path. This variant is similar to the clique one, except that instead of a complete graph, we build a path according to the natural flow of the text. That is, two nodes are linked by a directed edge if the two sentences they represent follow each other in the document."]},{"idx":7,"heading":"Experiments ::: Datasets","paragraphs":["We evaluate the quality of the document embeddings learned by MPAD on 10 document classification datasets, covering the topic identification, coarse and fine sentiment analysis and opinion mining, and subjectivity detection tasks. We briefly introduce the datasets next. Their statistics are reported in Table TABREF21.","(1) Reuters. This dataset contains stories collected from the Reuters news agency in 1987. Following common practice, we used the ModApte split and considered only the 10 classes with the highest number of positive training examples. We also removed documents belonging to more than one class and then classes left with no document (2 classes).","(2) BBCSport BIBREF30 contains documents from the BBC Sport website corresponding to 2004-2005 sports news articles.","(3) Polarity BIBREF31 features positive and negative labeled snippets from Rotten Tomatoes.","(4) Subjectivity BIBREF32 contains movie review snippets from Rotten Tomatoes (subjective sentences), and Internet Movie Database plot summaries (objective sentences).","(5) MPQA BIBREF33 is made of positive and negative phrases, annotated as part of the summer 2002 NRRC Workshop on Multi-Perspective Question Answering.","(6) IMDB BIBREF34 is a collection of highly polarized movie reviews from IMDB (positive and negative). There are at most 30 reviews for each movie.","(7) TREC BIBREF35 consists of questions that are classified into 6 different categories.","(8) SST-1 BIBREF36 contains the same snippets as Polarity. The authors used the Stanford Parser to parse the snippets and split them into multiple sentences. They then used Amazon Mechanical Turk to annotate the resulting phrases according to their polarity (very negative, negative, neutral, positive, very positive).","(9) SST-2 BIBREF36 is the same as SST-1 but with neutral reviews removed and snippets classified as positive or negative.","(10) Yelp2013 BIBREF26 features reviews obtained from the 2013 Yelp Dataset Challenge."]},{"idx":8,"heading":"Experiments ::: Baselines","paragraphs":["We evaluate MPAD against multiple state-of-the-art baseline models, including hierarchical ones, to enable fair comparison with the hierarchical MPAD variants.","doc2vec BIBREF37. Doc2vec (or paragraph vector) is an extension of word2vec that learns vectors for documents in a fully unsupervised manner. Document embeddings are then fed to a logistic regression classifier.","CNN BIBREF38. The convolutional neural network architecture, well-known in computer vision, is applied to text. There is one spatial dimension and the word embeddings are used as channels (depth dimensions).","DAN BIBREF39. The Deep Averaging Network passes the unweighted average of the embeddings of the input words through multiple dense layers and a final softmax.","Tree-LSTM BIBREF40 is a generalization of the standard LSTM architecture to constituency and dependency parse trees.","DRNN BIBREF41. Recursive neural networks are stacked and applied to parse trees.","LSTMN BIBREF42 is an extension of the LSTM model where the memory cell is replaced by a memory network which stores word representations.","C-LSTM BIBREF43 combines convolutional and recurrent neural networks. The region embeddings provided by a CNN are fed to a LSTM.","SPGK BIBREF44 also models documents as word co-occurrence networks. It computes a graph kernel that compares shortest paths extracted from the word co-occurrence networks and then uses a SVM to categorize documents.","WMD BIBREF45 is an application of the well-known Earth Mover's Distance to text. A k-nearest neighbor classifier is used.","S-WMD BIBREF46 is a supervised extension of the Word Mover's Distance.","Semantic-CNN BIBREF47. Here, a CNN is applied to semantic units obtained by clustering words in the embedding space.","LSTM-GRNN BIBREF26 is a hierarchical model where sentence embeddings are obtained with a CNN and a GRU-RNN is fed the sentence representations to obtain a document vector.","HN-ATT BIBREF27 is another hierarchical model, where the same encoder architecture (a bidirectional GRU-RNN) is used for both sentences and documents, with different parameters. A self-attention mechanism is applied to the RNN annotations at each level."]},{"idx":9,"heading":"Experiments ::: Model configuration and training","paragraphs":["We preprocess all datasets using the code of BIBREF38. On Yelp2013, we also replace all tokens appearing strictly less than 6 times with a special UNK token, like in BIBREF27. We then build a directed word co-occurrence network from each document, with a window of size 2.","We use two MP iterations ($T$=2) for the basic MPAD, and two MP iterations at each level, for the hierarchical variants. We set $d$ to 64, except on IMDB and Yelp on which $d=128$, and use a two-layer MLP. The final graph representations are passed through a softmax for classification. We train MPAD in an end-to-end fashion by minimizing the cross-entropy loss function with the Adam optimizer BIBREF48 and an initial learning rate of 0.001.","To regulate potential differences in magnitude, we apply batch normalization after concatenating the feature vector of the master node with the self-attentional vector, that is, after the skip connection (see subsection SECREF16). To prevent overfitting, we use dropout BIBREF49 with a rate of 0.5. We select the best epoch, capped at 200, based on the validation accuracy. When cross-validation is used (see 3rd column of Table TABREF21), we construct a validation set by randomly sampling 10% of the training set of each fold.","On all datasets except Yelp2013, we use the publicly available 300-dimensional pre-trained Google News vectors ($D$=300) BIBREF50 to initialize the node representations $\\mathbf {H}^0$. On Yelp2013, we follow BIBREF27 and learn our own word vectors from the training and validation sets with the gensim implementation of word2vec BIBREF51.","MPAD was implemented in Python 3.6 using the PyTorch library BIBREF52. All experiments were run on a single machine consisting of a 3.4 GHz Intel Core i7 CPU with 16 GB of RAM and an NVidia GeForce Titan Xp GPU."]},{"idx":10,"heading":"Results and ablations ::: Results","paragraphs":["Experimental results are shown in Table TABREF28. For the baselines, the best scores reported in each original paper are shown. MPAD reaches best performance on 7 out of 10 datasets, and is close second elsewhere. Moreover, the 7 datasets on which MPAD ranks first widely differ in training set size, number of categories, and prediction task (topic, sentiment, subjectivity), which indicates that MPAD can perform well in different settings.","MPAD vs. hierarchical variants. On 9 datasets out of 10, one or more of the hierarchical variants outperform the vanilla MPAD architecture, highlighting the benefit of explicitly modeling the hierarchical nature of documents.","However, on Subjectivity, standard MPAD outperforms all hierarchical variants. On TREC, it reaches the same accuracy. We hypothesize that in some cases, using a different graph to separately encode each sentence might be worse than using one single graph to directly encode the document. Indeed, in the single document graph, some words that never appear in the same sentence can be connected through common neighbors, as was explained in subsection SECREF7. So, this way, some notion of cross-sentence context is captured while learning representations of words, bigrams, etc. at each MP iteration. This creates better informed representations, resulting in a better document embedding. With the hierarchical variants, on the other hand, each sentence vector is produced in isolation, without any contextual information about the other sentences in the document. Therefore, the final sentence embeddings might be of lower quality, and as a group might also contain redundant/repeated information. When the sentence vectors are finally combined into a document representation, it is too late to take context into account."]},{"idx":11,"heading":"Results and ablations ::: Ablation studies","paragraphs":["To understand the impact of some hyperparameters on performance, we conducted additional experiments on the Reuters, Polarity, and IMDB datasets, with the non-hierarchical version of MPAD. Results are shown in Table TABREF29.","Number of MP iterations. First, we varied the number of message passing iterations from 1 to 4. We can clearly see in Table TABREF29 that having more iterations improves performance. We attribute this to the fact that we are reading out at each iteration from 1 to $T$ (see Eq. DISPLAY_FORM18), which enables the final graph representation to encode a mixture of low-level and high-level features. Indeed, in initial experiments involving readout at $t$=$T$ only, setting $T\\ge 2$ was always decreasing performance, despite the GRU-based updates (Eq. DISPLAY_FORM14). These results were consistent with that of BIBREF53 and BIBREF9, who both are reading out only at $t$=$T$ too. We hypothesize that node features at $T\\ge 2$ are too diffuse to be entirely relied upon during readout. More precisely, initially at $t$=0, node representations capture information about words, at $t$=1, about their 1-hop neighborhood (bigrams), at $t$=2, about compositions of bigrams, etc. Thus, pretty quickly, node features become general and diffuse. In such cases, considering also the lower-level, more precise features of the earlier iterations when reading out may be necessary.","Undirected edges. On Reuters, using an undirected graph leads to better performance, while on Polarity and IMDB, it is the opposite. This can be explained by the fact that Reuters is a topic classification task, for which the presence or absence of some patterns is important, but not necessarily the order in which they appear, while Polarity and IMDB are sentiment analysis tasks. To capture sentiment, modeling word order is crucial, e.g., in detecting negation.","No master node. Removing the master node deteriorates performance across all datasets, clearly showing the value of having such a node. We hypothesize that since the special document node is connected to all other nodes, it is able to encode during message passing a summary of the document.","No renormalization. Here, we do not use the renormalization trick of BIBREF9 during MP (see subsection SECREF10). That is, Eq. DISPLAY_FORM11 becomes $\\mathbf {M}^{t+1} = \\textsc {MLP}^{t+1}\\big (\\mathbf {A}\\mathbf {H}^{t}\\big )$. In other words, instead of computing a weighted average of the incoming neighbors' feature vectors, we compute a weighted sum of them. Unlike the mean, which captures distributions, the sum captures structural information BIBREF14. As shown in Table TABREF29, using sum instead of mean decreases performance everywhere, suggesting that in our application, capturing the distribution of neighbor representations is more important that capturing their structure. We hypothesize that this is the case because statistical word co-occurrence networks tend to have similar structural properties, regardless of the topic, polarity, sentiment, etc. of the corresponding documents.","Neighbors-only. In this experiment, we replaced the GRU combine function (see Eq. DISPLAY_FORM14) with the identity function. That is, we simply have $\\mathbf {H}^{t+1}$=$\\mathbf {M}^{t+1}$. Since $\\mathbf {A}$ has zero diagonal, by doing so, we completely ignore the previous feature of the node itself when updating its representation. That is, the update is based entirely on its neighbors. Except on Reuters (almost no change), performance always suffers, stressing the need to take into account the root node during updates, not only its neighborhood."]},{"idx":12,"heading":"Related work","paragraphs":["In what follows, we offer a brief review of relevant studies, ranked by increasing order of similarity with our work.","BIBREF9, BIBREF54, BIBREF11, BIBREF10 conduct some node classification experiments on citation networks, where nodes are scientific papers, i.e., textual data. However, text is only used to derive node feature vectors. The external graph structure, which plays a central role in determining node labels, is completely unrelated to text.","On the other hand, BIBREF55, BIBREF7 experiment on traditional document classification tasks. They both build $k$-nearest neighbor similarity graphs based on the Gaussian diffusion kernel. More precisely, BIBREF55 build one single graph where nodes are documents and distance is computed in the BoW space. Node features are then used for classification. Closer to our work, BIBREF7 represent each document as a graph. All document graphs are derived from the same underlying structure. Only node features, corresponding to the entries of the documents' BoW vectors, vary. The underlying, shared structure is that of a $k$-NN graph where nodes are vocabulary terms and similarity is the cosine of the word embedding vectors. BIBREF7 then perform graph classification. However they found performance to be lower than that of a naive Bayes classifier.","BIBREF56 use a GNN for hierarchical classification into a large taxonomy of topics. This task differs from traditional document classification. The authors represent documents as unweighted, undirected word co-occurrence networks with word embeddings as node features. They then use the spatial GNN of BIBREF15 to perform graph classification.","The work closest to ours is probably that of BIBREF53. The authors adopt the semi-supervised node classification approach of BIBREF9. They build one single undirected graph from the entire dataset, with both word and document nodes. Document-word edges are weighted by TF-IDF and word-word edges are weighted by pointwise mutual information derived from co-occurrence within a sliding window. There are no document-document edges. The GNN is trained based on the cross-entropy loss computed only for the labeled nodes, that is, the documents in the training set. When the final node representations are obtained, one can use that of the test documents to classify them and evaluate prediction performance.","There are significant differences between BIBREF53 and our work. First, our approach is inductive, not transductive. Indeed, while the node classification approach of BIBREF53 requires all test documents at training time, our graph classification model is able to perform inference on new, never-seen documents. The downside of representing documents as separate graphs, however, is that we lose the ability to capture corpus-level dependencies. Also, our directed graphs capture word ordering, which is ignored by BIBREF53. Finally, the approach of BIBREF53 requires computing the PMI for every word pair in the vocabulary, which may be prohibitive on datasets with very large vocabularies. On the other hand, the complexity of MPAD does not depend on vocabulary size."]},{"idx":13,"heading":"Conclusion","paragraphs":["We have proposed an application of the message passing framework to NLP, the Message Passing Attention network for Document understanding (MPAD). Experiments conducted on 10 standard text classification datasets show that our architecture is competitive with the state-of-the-art. By processing weighted, directed word co-occurrence networks, MPAD is sensitive to word order and word-word relationship strength. To explicitly capture the hierarchical structure of documents, we also propose three hierarchical variants of MPAD, that we show bring improvements over the vanilla architecture."]},{"idx":14,"heading":"Acknowledgments","paragraphs":["We thank the NVidia corporation for the donation of a GPU as part of their GPU grant program."]},{"idx":15,"heading":"Figures and Tables","paragraphs":["Table 1: Statistics of the datasets used in our experiments. CV indicates that cross-validation was used. # pretrained words refers to the number of words in the vocabulary having an entry in the Google News word vectors (except for Yelp2013).","Table 2: Classification accuracy on the 10 datasets. Best performance per column in bold, *best MPAD variant.","Table 3: Ablation results. The n in nMP refers to the number of message passing iterations. *vanilla model (MPAD in Table 2)."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1908.07195","title":"ARAML: A Stable Adversarial Training Framework for Text Generation","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Most of the existing generative adversarial networks (GAN) for text generation suffer from the instability of reinforcement learning training algorithms such as policy gradient, leading to unstable performance. To tackle this problem, we propose a novel framework called Adversarial Reward Augmented Maximum Likelihood (ARAML). During adversarial training, the discriminator assigns rewards to samples which are acquired from a stationary distribution near the data rather than the generator's distribution. The generator is optimized with maximum likelihood estimation augmented by the discriminator's rewards instead of policy gradient. Experiments show that our model can outperform state-of-the-art text GANs with a more stable training process."]},{"idx":1,"heading":"Introduction","paragraphs":["Natural text generation, as a key task in NLP, has been advanced substantially thanks to the flourish of neural models BIBREF0 , BIBREF1 . Typical frameworks such as sequence-to-sequence (seq2seq) have been applied to various generation tasks, including machine translation BIBREF2 and dialogue generation BIBREF3 . The standard paradigm to train such neural models is maximum likelihood estimation (MLE), which maximizes the log-likelihood of observing each word in the text given the ground-truth proceeding context BIBREF4 .","Although widely used, MLE suffers from the exposure bias problem BIBREF5 , BIBREF6 : during test, the model sequentially predicts the next word conditioned on its previous generated words while during training conditioned on ground-truth words. To tackle this problem, generative adversarial networks (GAN) with reinforcement learning (RL) training approaches have been introduced to text generation tasks BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , where the discriminator is trained to distinguish real and generated text samples to provide reward signals for the generator, and the generator is optimized via policy gradient BIBREF7 .","However, recent studies have shown that potential issues of training GANs on discrete data are more severe than exposure bias BIBREF14 , BIBREF15 . One of the fundamental issues when generating discrete text samples with GANs is training instability. Updating the generator with policy gradient always leads to an unstable training process because it's difficult for the generator to derive positive and stable reward signals from the discriminator even with careful pre-training BIBREF8 . As a result, the generator gets lost due to the high variance of reward signals and the training process may finally collapse BIBREF16 .","In this paper, we propose a novel adversarial training framework called Adversarial Reward Augmented Maximum Likelihood (ARAML) to deal with the instability issue of training GANs for text generation. At each iteration of adversarial training, we first train the discriminator to assign higher rewards to real data than to generated samples. Then, inspired by reward augmented maximum likelihood (RAML) BIBREF17 , the generator is updated on the samples acquired from a stationary distribution with maximum likelihood estimation (MLE), weighted by the discriminator's rewards. This stationary distribution is designed to guarantee that training samples are surrounding the real data, thus the exploration space of our generator is indeed restricted by the MLE training objective, resulting in more stable training. Compared to other text GANs with RL training techniques, our framework acquires samples from the stationary distribution rather than the generator's distribution, and uses RAML training paradigm to optimize the generator instead of policy gradient. Our contributions are mainly as follows:"]},{"idx":2,"heading":"Related Work","paragraphs":["Recently, text generation has been widely studied with neural models trained with maximum likelihood estimation BIBREF4 . However, MLE tends to generate universal text BIBREF18 . Various methods have been proposed to enhance the generation quality by refining the objective function BIBREF18 , BIBREF19 or modifying the generation distribution with external information like topic BIBREF20 , sentence type BIBREF21 , emotion BIBREF22 and knowledge BIBREF23 .","As mentioned above, MLE suffers from the exposure bias problem BIBREF5 , BIBREF6 . Thus, reinforcement learning has been introduced to text generation tasks such as policy gradient BIBREF6 and actor-critic BIBREF24 . BIBREF17 proposed an efficient and stable approach called Reward Augmented Maximum Likelihood (RAML), which connects the log-likelihood and expected rewards to incorporate MLE training objective into RL framework.","Since some text generation tasks have no explicit metrics to be directly optimized, adversarial training has been applied to generating discrete text samples with a discriminator to learn a proper reward. For instance, SeqGAN BIBREF7 devised a discriminator to distinguish the real data and generated samples, and a generator to maximize the reward from the discriminator via policy gradient. Other variants of GANs have been proposed to improve the generator or the discriminator. To improve the generator, MaliGAN BIBREF8 developed a normalized maximum likelihood optimization target for the generator to stably model the discrete sequences. LeakGAN BIBREF11 guided the generator with reward signals leaked from the discriminator at all generation steps to deal with long text generation task. MaskGAN BIBREF10 employed an actor-critic architecture to make the generator fill in missing text conditioned on the surrounding context, which is expected to mitigate the problem of mode collapse. As for the discriminator, RankGAN BIBREF9 replaced traditional discriminator with a ranker to learn the relative ranking information between the real texts and generated ones. Inverse reinforcement learning BIBREF12 used a trainable reward approximator as the discriminator to provide dense reward signals at each generation step. DPGAN BIBREF13 introduced a language model based discriminator and regarded cross-entropy as rewards to promote the diversity of generation results.","The most similar works to our model are RAML BIBREF17 and MaliGAN BIBREF8 : 1) Compared with RAML, our model adds a discriminator to learn the reward signals instead of choosing existing metrics as rewards. We believe that our model can adapt to various text generation tasks, particularly those without explicit evaluation metrics. 2) Unlike MaliGAN, we acquire samples from a fixed distribution near the real data rather than the generator's distribution, which is expected to make the training process more stable."]},{"idx":3,"heading":"Task Definition and Model Overview","paragraphs":["Text generation can be formulated as follows: given the real data distribution INLINEFORM0 , the task is to train a generative model INLINEFORM1 where INLINEFORM2 can fit INLINEFORM3 well. In this formulation, INLINEFORM4 and INLINEFORM5 denotes a word in the vocabulary INLINEFORM6 .","Figure FIGREF3 shows the overview of our model ARAML. This adversarial training framework consists of two phases: 1) The discriminator is trained to assign higher rewards to real data than to generated data. 2) The generator is trained on the samples acquired from a stationary distribution with reward augmented MLE training objective. This training paradigm of the generator indeed constrains the search space with the MLE training objective, which alleviates the issue of unstable training."]},{"idx":4,"heading":"Discriminator","paragraphs":["The discriminator INLINEFORM0 aims to distinguish real data and generated data like other GANs. Inspired by Least-Square GAN BIBREF25 , we devise the loss function as follows: DISPLAYFORM0","This loss function forces the discriminator to assign higher rewards to real data than to generated data, so the discriminator can learn to provide more proper rewards as the training proceeds."]},{"idx":5,"heading":"Generator","paragraphs":["The training objective of our generator INLINEFORM0 is derived from the objective of other discrete GANs with RL training method: DISPLAYFORM0","where INLINEFORM0 denotes the rewards from the discriminator INLINEFORM1 and the entropy regularized term INLINEFORM2 encourages INLINEFORM3 to generate diverse text samples. INLINEFORM4 is a temperature hyper-parameter to balance these two terms.","As mentioned above, discrete GANs suffer from the instability issue due to policy gradient, thus they are consequently difficult to train. Inspired by RAML BIBREF17 , we introduce an exponential payoff distribution INLINEFORM0 to connect RL loss with RAML loss: DISPLAYFORM0","where INLINEFORM0 . Thus, we can rewrite INLINEFORM1 with INLINEFORM2 and INLINEFORM3 as follows: DISPLAYFORM0","Following RAML, we remove the constant term and optimize the KL divergence in the opposite direction: DISPLAYFORM0","where INLINEFORM0 is a constant in the training phase of the generator. It has been proved that INLINEFORM1 and INLINEFORM2 are equivalent up to their first order Taylor approximations, and they have the same global optimum BIBREF17 . INLINEFORM3 can be trained in a MLE-like fashion but sampling from the distribution INLINEFORM4 is intractable in the adversarial setting, because INLINEFORM5 varies with the discriminator INLINEFORM6 . Thus, we introduce importance sampling to separate sampling process from INLINEFORM7 and obtain the final loss function: DISPLAYFORM0","where INLINEFORM0 denotes a stationary distribution and INLINEFORM1 . To optimize this loss function, we first construct the fixed distribution INLINEFORM2 to get samples, and devise the proper reward function INLINEFORM3 to train the generator in a stable and effective way.","We construct the distribution INLINEFORM0 based on INLINEFORM1 : DISPLAYFORM0","In this way, INLINEFORM0 can be designed to guarantee that INLINEFORM1 is near INLINEFORM2 , leading to a more stable training process. To obtain a new sample INLINEFORM3 from a real data sample INLINEFORM4 , we can design three steps which contain sampling an edit distance INLINEFORM5 , the positions INLINEFORM6 for substitution and the new words INLINEFORM7 filled into the corresponding positions. Thus, INLINEFORM8 can be decomposed into three terms: DISPLAYFORM0","The first step is to sample an edit distance based on a real data sample INLINEFORM0 , where INLINEFORM1 is a sequence of length INLINEFORM2 . The number of sentences which have the edit distance INLINEFORM3 to some input sentence can be computed approximately as below: DISPLAYFORM0","where INLINEFORM0 denotes the number of sentences which have an edit distance INLINEFORM1 to a sentence of length INLINEFORM2 , and INLINEFORM3 indicates the size of vocabulary. We then follow BIBREF17 to re-scale the counts by INLINEFORM4 and do normalization, so that we can sample an edit distance INLINEFORM5 from: DISPLAYFORM0","where INLINEFORM0 , as a temperature hyper-parameter, restricts the search space surrounding the original sentence. Larger INLINEFORM1 brings more samples with long edit distances.","The next step is to select positions for substitution based on the sampled edit distance INLINEFORM0 . Intuitively, we can randomly choose INLINEFORM1 distinct positions in INLINEFORM2 to be replaced by new words. The probability of choosing the position INLINEFORM3 is calculated as follows: DISPLAYFORM0","Following this sampling strategy, we can obtain the position set INLINEFORM0 . This strategy approximately guarantees that the edit distance between a new sentence and the original sentence is INLINEFORM1 .","At the final step, our model determines new words for substitution at each sampled position INLINEFORM0 . We can formulate this sampling process from the original sequence INLINEFORM1 to a new sample INLINEFORM2 as a sequential transition INLINEFORM3 . At each step from INLINEFORM4 to INLINEFORM5 INLINEFORM6 , we first sample a new word INLINEFORM7 from the distribution INLINEFORM8 , then replace the old word at position INLINEFORM9 of INLINEFORM10 to obtain INLINEFORM11 . The whole sampling process can be decomposed as follows: DISPLAYFORM0","There are two common sampling strategies to model INLINEFORM0 , i.e. random sampling and constrained sampling. Random sampling strategy samples a new word INLINEFORM1 according to the uniform distribution over the vocabulary INLINEFORM2 BIBREF17 , while constrained sampling strategy samples INLINEFORM3 to maximize the language model score of the target sentence INLINEFORM4 BIBREF26 , BIBREF27 . Here, we adopt constrained sampling in our model and compare the performances of two strategies in the experiment.","We devise the reward function INLINEFORM0 according to the discriminator's output INLINEFORM1 and the stationary distribution INLINEFORM2 : DISPLAYFORM0","Intuitively, this reward function encourages the generator to generate sentences with large sampling probability and high rewards from the discriminator. Thus, the weight of samples INLINEFORM0 can be calculated as follows: DISPLAYFORM0","So far, we can successfully optimize the generator's loss INLINEFORM0 via Equation EQREF12 . This training paradigm makes our generator avoid possible variances caused by policy gradient and get more stable reward signals from the discriminator, because our generator is restricted to explore the training samples near the real data.","[htb] Adversarial Reward Augmented Maximum Likelihood [1]","Total adversarial training iterations: INLINEFORM0","Steps of training generator: INLINEFORM0","Steps of training discriminator: INLINEFORM0","Pre-train the generator INLINEFORM0 with MLE loss Generate samples from INLINEFORM1 Pre-train the discriminator INLINEFORM2 via Eq.( EQREF6 ) Construct INLINEFORM3 via Eq.( EQREF14 ) - Eq.( EQREF19 ) each INLINEFORM4 each INLINEFORM5 Update INLINEFORM6 via Eq.( EQREF12 ) each INLINEFORM7 Update INLINEFORM8 via Eq.( EQREF6 )"]},{"idx":6,"heading":"Extension to Conditional Text Generation","paragraphs":["We have shown our adversarial training framework for text generation tasks without an input. Actually, it can also be extended to conditional text generation tasks like dialogue generation. Given the data distribution INLINEFORM0 where INLINEFORM1 denote contexts and responses respectively, the objective function of ARAML's generator can be modified as below: DISPLAYFORM0","where INLINEFORM0 and INLINEFORM1 is trained to distinguish whether INLINEFORM2 is the true response to INLINEFORM3 ."]},{"idx":7,"heading":"Comparison with RAML and MaliGAN","paragraphs":["The most similar works to our framework are RAML BIBREF17 and MaliGAN BIBREF8 . The main difference among them is the training objective of their generators. We have shown different objective functions in Table TABREF26 . For comparison, we use the form with no input for all the three models.","Our model is greatly inspired by RAML, which gets samples from a non-parametric distribution INLINEFORM0 constructed based on a specific reward. Compared to RAML, our reward comes from a learnable discriminator which varies as the adversarial training proceeds rather than a specific reward function. This difference equips our framework with the ability to adapt to the text generation tasks with no explicit evaluation metrics as rewards.","Our model is also similar to MaliGAN, which gets samples from the generator's distribution. In MaliGAN's training objective, INLINEFORM0 also indicates the generator's distribution but it's used in the sampling phase and fixed at each optimization step. The weight of samples INLINEFORM1 . Different from our model, MaliGAN acquires samples from the generator's distribution INLINEFORM2 , which usually brings samples with low rewards even with careful pre-training for the generator, leading to training instability. Instead, our framework gets samples from a stationary distribution INLINEFORM3 around real data, thus our training process is more stable."]},{"idx":8,"heading":"Datasets","paragraphs":["We evaluated ARAML on three datasets: COCO image caption dataset BIBREF28 , EMNLP2017 WMT dataset and WeiboDial single-turn dialogue dataset BIBREF29 . COCO and EMNLP2017 WMT are the common benchmarks with no input to evaluate the performance of discrete GANs, and we followed the existing works to preprocess these datasets BIBREF12 , BIBREF11 . WeiboDial, as a dialogue dataset, was applied to test the performance of our model with input trigger. We simply removed post-response pairs containing low-frequency words and randomly selected a subset for our training/test set. The statistics of three datasets are presented in Table TABREF28 ."]},{"idx":9,"heading":"Baselines","paragraphs":["We compared our model with MLE, RL and GAN baselines. Since COCO and EMNLP2017 WMT don't have input while WeiboDial regards posts as input, we chose the following baselines respectively:","MLE: a RNN model trained with MLE objective BIBREF4 . Its extension, Seq2Seq, can work on the dialogue dataset BIBREF2 .","SeqGAN: The first text GAN model that updates the generator with policy gradient based on the rewards from the discriminator BIBREF7 .","LeakGAN: A variant of SeqGAN that provides rewards based on the leaked information of the discriminator for the generator BIBREF11 .","MaliGAN: A variant of SeqGAN that optimizes the generator with a normalized maximum likelihood objective BIBREF8 .","IRL: This inverse reinforcement learning method replaces the discriminator with a reward approximator to provide dense rewards BIBREF12 .","RAML: A RL approach to incorporate MLE objective into RL training framework, which regards BLEU as rewards BIBREF17 .","DialogGAN: An extension of SeqGAN tuned to dialogue generation task with MLE objective added to the adversarial objective BIBREF16 .","DPGAN: A variant of DialogGAN which uses a language model based discriminator and regards cross-entropy as rewards BIBREF13 .","Note that MLE, SeqGAN, LeakGAN, MaliGAN and IRL are the baselines on COCO and EMNLP2017 WMT, while MLE, RAML, DialogGAN, and DPGAN on WeiboDial. The original codes are used to test the baselines."]},{"idx":10,"heading":"Implementation Details","paragraphs":["The implementation details of our model are shown in Table TABREF31 . For COCO / EMNLP2017, the generator is a LSTM unit BIBREF30 with 128 cells, and the discriminator is implemented based on BIBREF7 . For WeiboDial, the generator is an encoder-decoder structure with attention mechanism, where both the encoder and the decoder consist of a two-layer GRU BIBREF31 with 128 cells. The discriminator is implemented based on BIBREF32 . The language model used in the constrained sampling of ARAML is implemented in the same setting as the generators, and is pre-trained on the training set of each dataset. The codes and the datasets are available at https://github.com/kepei1106/ARAML.","As for the details of the baselines, the generators of all the baselines except LeakGAN are the same as ours. Note that the generator of LeakGAN consists of a hierarchical LSTM unit, thus we followed the implementation in the original paper. In terms of the differences, the discriminators of GAN baselines are implemented based on the original papers. Other hyper-parameters of baselines including batch size, learning rate, and pre-training epochs, were set based on the original codes, because the convergence of baselines is sensitive to these hyper-parameters."]},{"idx":11,"heading":"Language Generation on COCO and EMNLP2017 WMT","paragraphs":["We adopted forward/reverse perplexity BIBREF33 and Self-BLEU BIBREF34 to evaluate the quality of generated texts. Forward perplexity (PPL-F) indicates the perplexity on the generated data provided by a language model trained on real data to measure the fluency of generated samples. Reverse perplexity (PPL-R) switches the roles of generated data and real data to reflect the discrepancy between the generated distribution and the data distribution. Self-BLEU (S-BLEU) regards each sentence in the generated collection as hypothesis and the others as reference to obtain BLEU scores, which evaluates the diversity of generated results.","Results are shown in Table TABREF33 . LeakGAN performs best on forward perplexity because it can generate more fluent samples. As for reverse perplexity, our model ARAML beats other baselines, showing that our model can fit the data distribution better. Other GANs, particularly LeakGAN, obtain high reverse perplexity due to mode collapse BIBREF12 , thus they only capture limited fluent expressions, resulting in large discrepancy between the generated distribution and data distribution. ARAML also outperforms the baselines in terms of Self-BLEU, indicating that our model doesn't fall into mode collapse with the help of the MLE training objective and has the ability to generate more diverse sentences.","We also provide standard deviation of each metric in Table TABREF33 , reflecting the stability of each model's performance. Our model ARAML nearly achieves the smallest standard deviation in all the metrics, indicating that our framework outperforms policy gradient in the stability of adversarial training."]},{"idx":12,"heading":"Dialogue Generation on WeiboDial","paragraphs":["Dialogue evaluation is an open problem and existing works have found that automatic metrics have low correlation to human evaluation BIBREF35 , BIBREF36 , BIBREF37 . Thus, we resorted to manual evaluation to assess the generation quality on WeiboDial. We randomly sampled 200 posts from the test set and collected the generated results from all the models. For each pair of responses (one from ARAML and the other from a baseline, given the same input post), five annotators were hired to label which response is better (i.e. win, lose or tie) in terms of grammaticality (whether a response itself is grammatical and logical) and relevance (whether a response is appropriate and relevant to the post). The two metrics were evaluated independently.","The evaluation results are shown in Table TABREF35 . To measure the inter-annotator agreement, we calculated Fleiss' kappa BIBREF38 for each pair-wise comparison where results show moderate agreement ( INLINEFORM0 ). We also conducted sign test to check the significance of the differences.","As shown in Table TABREF35 , ARAML performs significantly better than other baselines in all the cases. This result indicates that the samples surrounding true responses provide stable rewards for the generator, and stable RAML training paradigm significantly enhances the performance in both metrics."]},{"idx":13,"heading":"Further Analysis on Stability","paragraphs":["To verify the training stability, we conducted experiments on COCO many times and chose the best 5 trials for SeqGAN, LeakGAN, IRL, MaliGAN and ARAML, respectively. Then, we presented the forward/reverse perplexity in the training process in Figure FIGREF38 . We can see that our model with smaller standard deviation is more stable than other GAN baselines in both metrics. Although LeakGAN reaches the best forward perplexity, its standard deviation is extremely large and it performs badly in reverse perplexity, indicating that it generates limited expressions that are grammatical yet divergent from the data distribution."]},{"idx":14,"heading":"Ablation Study","paragraphs":["The temperature INLINEFORM0 controls the search space surrounding the real data as we analyze in Section UID13 . To investigate its impact on the performance of our model, we fixed all the other hyper-parameters and test ARAML with different temperatures on COCO.","The experimental results are shown in Figure FIGREF41 . We can see that as the temperature becomes larger, forward perplexity increases gradually while Self-BLEU decreases. As mentioned in Section UID13 , large temperatures encourage our generator to explore the samples that are distant from real data distribution, thus the diversity of generated results will be improved. However, these samples distant from the data distribution are more likely to be poor in fluency, leading to worse forward perplexity. Reverse perplexity is influenced by both generation quality and diversity, so the correlation between temperature and reverse perplexity is not intuitive. We can observe that the model with INLINEFORM0 reaches the best reverse perplexity.","We have mentioned two common sampling strategies in Section UID13 , i.e. random sampling and constrained sampling. To analyze their impact, we keep all the model structures and hyper-parameters fixed and test ARAML with these two strategies on COCO.","Table TABREF45 shows the results. It's obvious that random sampling hurts the model performance except Self-BLEU-1, because it indeed allows low-quality samples available to the generator. Exploring these samples degrades the quality and diversity of generated results. Despite the worse performance on automatic metrics, random sampling doesn't affect the training stability of our framework. The standard deviation of ARAML-R is still smaller than other GAN baselines."]},{"idx":15,"heading":"Case Study","paragraphs":["Table TABREF47 presents the examples generated by the models on COCO. We can find that other baselines suffer from grammatical errors (e.g. “in front of flying her kite\" from MLE), repetitive expressions (e.g. “A group of people\" from IRL) and incoherent statements (e.g. “A group of people sitting on a cell phone” from IRL). By contrast, our model performs well in these sentences and has the ability to generate grammatical and coherent results.","Table TABREF48 shows the generated examples on WeiboDial. It's obvious that other baselines don't capture the topic word “late\" in the post, thus generate irrelevant responses. ARAML can provide a response that is grammatical and closely relevant to the post."]},{"idx":16,"heading":"Conclusion","paragraphs":["We propose a novel adversarial training framework to deal with the instability problem of current GANs for text generation. To address the instability issue caused by policy gradient, we incorporate RAML into the advesarial training paradigm to make our generator acquire stable rewards. Experiments show that our model performs better than several state-of-the-art GAN baselines with lower training variance, yet producing better performance on three text generation tasks."]},{"idx":17,"heading":"Acknowledgments","paragraphs":["This work was supported by the National Science Foundation of China (Grant No. 61936010/61876096) and the National Key R&D Program of China (Grant No. 2018YFC0830200). We would like to thank THUNUS NExT Joint-Lab for the support."]},{"idx":18,"heading":"Figures and Tables","paragraphs":["Figure 1: Overview of ARAML. The training samples are acquired from a stationary distribution Ps based on the real data. The generator is then trained on the samples augmented by the discriminator’s rewards. The discriminator is trained to distinguish real data and generated data.","Table 2: Statistics of COCO, EMNLP2017 WMT and WeiboDial. The average lengths 7.3/10.8 of WeiboDial indicate the lengths of posts and responses, respectively.","Table 1: Training objectives of generators for RAML, MaliGAN and ARAML.","Table 3: Implementation details of ARAML. G/D/LM indicates the generator / discriminator / language model used in constrained sampling, respectively.","Table 4: Automatic evaluation on COCO and EMNLP2017 WMT. Each metric is presented with mean and standard deviation.","Figure 2: PPL-F/PPL-R curves of ARAML, SeqGAN, LeakGAN, MaliGAN and IRL in the training process. The shade area indicates the standard deviation at each data point. The dotted vertical lines separate pre-training and adversarial training phases (50 for ARAML, IRL and MaliGAN, 80 for SeqGAN and LeakGAN).","Table 5: Human evaluation on WeiboDial. The scores represent the percentages of Win, Lose or Tie when our model is compared with a baseline. κ denotes Fleiss’ kappa (all are moderate agreement). The scores marked with * mean p-value< 0.05 and ** indicates p-value< 0.01 in sign test.","Table 6: PPL-F, PPL-R and S-BLEU of ARAML with random sampling (ARAML-R) and constrained sampling (ARAML-C) on COCO.","Table 7: Examples of generated sentences on COCO. Grammatical errors are in red, while blue text represents repetitive expressions and green part indicates incoherent statements.","Table 8: Examples of generated responses on WeiboDial."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1908.08345","title":"Text Summarization with Pretrained Encoders","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Bidirectional Encoder Representations from Transformers (BERT) represents the latest incarnation of pretrained language models which have recently advanced a wide range of natural language processing tasks. In this paper, we showcase how BERT can be usefully applied in text summarization and propose a general framework for both extractive and abstractive models. We introduce a novel document-level encoder based on BERT which is able to express the semantics of a document and obtain representations for its sentences. Our extractive model is built on top of this encoder by stacking several inter-sentence Transformer layers. For abstractive summarization, we propose a new fine-tuning schedule which adopts different optimizers for the encoder and the decoder as a means of alleviating the mismatch between the two (the former is pretrained while the latter is not). We also demonstrate that a two-staged fine-tuning approach can further boost the quality of the generated summaries. Experiments on three datasets show that our model achieves state-of-the-art results across the board in both extractive and abstractive settings. Our code is available at this https URL"]},{"idx":1,"heading":"Introduction","paragraphs":["Language model pretraining has advanced the state of the art in many NLP tasks ranging from sentiment analysis, to question answering, natural language inference, named entity recognition, and textual similarity. State-of-the-art pretrained models include ELMo BIBREF1, GPT BIBREF2, and more recently Bidirectional Encoder Representations from Transformers (Bert; BIBREF0). Bert combines both word and sentence representations in a single very large Transformer BIBREF3; it is pretrained on vast amounts of text, with an unsupervised objective of masked language modeling and next-sentence prediction and can be fine-tuned with various task-specific objectives.","In most cases, pretrained language models have been employed as encoders for sentence- and paragraph-level natural language understanding problems BIBREF0 involving various classification tasks (e.g., predicting whether any two sentences are in an entailment relationship; or determining the completion of a sentence among four alternative sentences). In this paper, we examine the influence of language model pretraining on text summarization. Different from previous tasks, summarization requires wide-coverage natural language understanding going beyond the meaning of individual words and sentences. The aim is to condense a document into a shorter version while preserving most of its meaning. Furthermore, under abstractive modeling formulations, the task requires language generation capabilities in order to create summaries containing novel words and phrases not featured in the source text, while extractive summarization is often defined as a binary classification task with labels indicating whether a text span (typically a sentence) should be included in the summary.","We explore the potential of Bert for text summarization under a general framework encompassing both extractive and abstractive modeling paradigms. We propose a novel document-level encoder based on Bert which is able to encode a document and obtain representations for its sentences. Our extractive model is built on top of this encoder by stacking several inter-sentence Transformer layers to capture document-level features for extracting sentences. Our abstractive model adopts an encoder-decoder architecture, combining the same pretrained Bert encoder with a randomly-initialized Transformer decoder BIBREF3. We design a new training schedule which separates the optimizers of the encoder and the decoder in order to accommodate the fact that the former is pretrained while the latter must be trained from scratch. Finally, motivated by previous work showing that the combination of extractive and abstractive objectives can help generate better summaries BIBREF4, we present a two-stage approach where the encoder is fine-tuned twice, first with an extractive objective and subsequently on the abstractive summarization task.","We evaluate the proposed approach on three single-document news summarization datasets representative of different writing conventions (e.g., important information is concentrated at the beginning of the document or distributed more evenly throughout) and summary styles (e.g., verbose vs. more telegraphic; extractive vs. abstractive). Across datasets, we experimentally show that the proposed models achieve state-of-the-art results under both extractive and abstractive settings. Our contributions in this work are three-fold: a) we highlight the importance of document encoding for the summarization task; a variety of recently proposed techniques aim to enhance summarization performance via copying mechanisms BIBREF5, BIBREF6, BIBREF7, reinforcement learning BIBREF8, BIBREF9, BIBREF10, and multiple communicating encoders BIBREF11. We achieve better results with a minimum-requirement model without using any of these mechanisms; b) we showcase ways to effectively employ pretrained language models in summarization under both extractive and abstractive settings; we would expect any improvements in model pretraining to translate in better summarization in the future; and c) the proposed models can be used as a stepping stone to further improve summarization performance as well as baselines against which new proposals are tested."]},{"idx":2,"heading":"Background ::: Pretrained Language Models","paragraphs":["Pretrained language models BIBREF1, BIBREF2, BIBREF0, BIBREF12, BIBREF13 have recently emerged as a key technology for achieving impressive gains in a wide variety of natural language tasks. These models extend the idea of word embeddings by learning contextual representations from large-scale corpora using a language modeling objective. Bidirectional Encoder Representations from Transformers (Bert; BIBREF0) is a new language representation model which is trained with a masked language modeling and a “next sentence prediction” task on a corpus of 3,300M words.","The general architecture of Bert is shown in the left part of Figure FIGREF2. Input text is first preprocessed by inserting two special tokens. [cls] is appended to the beginning of the text; the output representation of this token is used to aggregate information from the whole sequence (e.g., for classification tasks). And token [sep] is inserted after each sentence as an indicator of sentence boundaries. The modified text is then represented as a sequence of tokens $X=[w_1,w_2,\\cdots ,w_n]$. Each token $w_i$ is assigned three kinds of embeddings: token embeddings indicate the meaning of each token, segmentation embeddings are used to discriminate between two sentences (e.g., during a sentence-pair classification task) and position embeddings indicate the position of each token within the text sequence. These three embeddings are summed to a single input vector $x_i$ and fed to a bidirectional Transformer with multiple layers:","where $h^0=x$ are the input vectors; $\\mathrm {LN}$ is the layer normalization operation BIBREF14; $\\mathrm {MHAtt}$ is the multi-head attention operation BIBREF3; superscript $l$ indicates the depth of the stacked layer. On the top layer, Bert will generate an output vector $t_i$ for each token with rich contextual information.","Pretrained language models are usually used to enhance performance in language understanding tasks. Very recently, there have been attempts to apply pretrained models to various generation problems BIBREF15, BIBREF16. When fine-tuning for a specific task, unlike ELMo whose parameters are usually fixed, parameters in Bert are jointly fine-tuned with additional task-specific parameters."]},{"idx":3,"heading":"Background ::: Extractive Summarization","paragraphs":["Extractive summarization systems create a summary by identifying (and subsequently concatenating) the most important sentences in a document. Neural models consider extractive summarization as a sentence classification problem: a neural encoder creates sentence representations and a classifier predicts which sentences should be selected as summaries. SummaRuNNer BIBREF7 is one of the earliest neural approaches adopting an encoder based on Recurrent Neural Networks. Refresh BIBREF8 is a reinforcement learning-based system trained by globally optimizing the ROUGE metric. More recent work achieves higher performance with more sophisticated model structures. Latent BIBREF17 frames extractive summarization as a latent variable inference problem; instead of maximizing the likelihood of “gold” standard labels, their latent model directly maximizes the likelihood of human summaries given selected sentences. Sumo BIBREF18 capitalizes on the notion of structured attention to induce a multi-root dependency tree representation of the document while predicting the output summary. NeuSum BIBREF19 scores and selects sentences jointly and represents the state of the art in extractive summarization."]},{"idx":4,"heading":"Background ::: Abstractive Summarization","paragraphs":["Neural approaches to abstractive summarization conceptualize the task as a sequence-to-sequence problem, where an encoder maps a sequence of tokens in the source document $\\mathbf {x} = [x_1, ..., x_n]$ to a sequence of continuous representations $\\mathbf {z} = [z_1, ..., z_n]$, and a decoder then generates the target summary $\\mathbf {y} = [y_1, ..., y_m]$ token-by-token, in an auto-regressive manner, hence modeling the conditional probability: $p(y_1, ..., y_m|x_1, ..., x_n)$.","BIBREF20 and BIBREF21 were among the first to apply the neural encoder-decoder architecture to text summarization. BIBREF6 enhance this model with a pointer-generator network (PTgen) which allows it to copy words from the source text, and a coverage mechanism (Cov) which keeps track of words that have been summarized. BIBREF11 propose an abstractive system where multiple agents (encoders) represent the document together with a hierarchical attention mechanism (over the agents) for decoding. Their Deep Communicating Agents (DCA) model is trained end-to-end with reinforcement learning. BIBREF9 also present a deep reinforced model (DRM) for abstractive summarization which handles the coverage problem with an intra-attention mechanism where the decoder attends over previously generated words. BIBREF4 follow a bottom-up approach (BottomUp); a content selector first determines which phrases in the source document should be part of the summary, and a copy mechanism is applied only to preselected phrases during decoding. BIBREF22 propose an abstractive model which is particularly suited to extreme summarization (i.e., single sentence summaries), based on convolutional neural networks and additionally conditioned on topic distributions (TConvS2S)."]},{"idx":5,"heading":"Fine-tuning Bert for Summarization ::: Summarization Encoder","paragraphs":["Although Bert has been used to fine-tune various NLP tasks, its application to summarization is not as straightforward. Since Bert is trained as a masked-language model, the output vectors are grounded to tokens instead of sentences, while in extractive summarization, most models manipulate sentence-level representations. Although segmentation embeddings represent different sentences in Bert, they only apply to sentence-pair inputs, while in summarization we must encode and manipulate multi-sentential inputs. Figure FIGREF2 illustrates our proposed Bert architecture for Summarization (which we call BertSum).","In order to represent individual sentences, we insert external [cls] tokens at the start of each sentence, and each [cls] symbol collects features for the sentence preceding it. We also use interval segment embeddings to distinguish multiple sentences within a document. For $sent_i$ we assign segment embedding $E_A$ or $E_B$ depending on whether $i$ is odd or even. For example, for document $[sent_1, sent_2, sent_3, sent_4, sent_5]$, we would assign embeddings $[E_A, E_B, E_A,E_B, E_A]$. This way, document representations are learned hierarchically where lower Transformer layers represent adjacent sentences, while higher layers, in combination with self-attention, represent multi-sentence discourse.","Position embeddings in the original Bert model have a maximum length of 512; we overcome this limitation by adding more position embeddings that are initialized randomly and fine-tuned with other parameters in the encoder."]},{"idx":6,"heading":"Fine-tuning Bert for Summarization ::: Extractive Summarization","paragraphs":["Let $d$ denote a document containing sentences $[sent_1, sent_2, \\cdots , sent_m]$, where $sent_i$ is the $i$-th sentence in the document. Extractive summarization can be defined as the task of assigning a label $y_i \\in \\lbrace 0, 1\\rbrace $ to each $sent_i$, indicating whether the sentence should be included in the summary. It is assumed that summary sentences represent the most important content of the document.","With BertSum, vector $t_i$ which is the vector of the $i$-th [cls] symbol from the top layer can be used as the representation for $sent_i$. Several inter-sentence Transformer layers are then stacked on top of Bert outputs, to capture document-level features for extracting summaries:","where $h^0=\\mathrm {PosEmb}(T)$; $T$ denotes the sentence vectors output by BertSum, and function $\\mathrm {PosEmb}$ adds sinusoid positional embeddings BIBREF3 to $T$, indicating the position of each sentence.","The final output layer is a sigmoid classifier:","where $h^L_i$ is the vector for $sent_i$ from the top layer (the $L$-th layer ) of the Transformer. In experiments, we implemented Transformers with $L=1, 2, 3$ and found that a Transformer with $L=2$ performed best. We name this model BertSumExt.","The loss of the model is the binary classification entropy of prediction $\\hat{y}_i$ against gold label $y_i$. Inter-sentence Transformer layers are jointly fine-tuned with BertSum. We use the Adam optimizer with $\\beta _1=0.9$, and $\\beta _2=0.999$). Our learning rate schedule follows BIBREF3 with warming-up ($ \\operatorname{\\operatorname{warmup}}=10,000$):"]},{"idx":7,"heading":"Fine-tuning Bert for Summarization ::: Abstractive Summarization","paragraphs":["We use a standard encoder-decoder framework for abstractive summarization BIBREF6. The encoder is the pretrained BertSum and the decoder is a 6-layered Transformer initialized randomly. It is conceivable that there is a mismatch between the encoder and the decoder, since the former is pretrained while the latter must be trained from scratch. This can make fine-tuning unstable; for example, the encoder might overfit the data while the decoder underfits, or vice versa. To circumvent this, we design a new fine-tuning schedule which separates the optimizers of the encoder and the decoder.","We use two Adam optimizers with $\\beta _1=0.9$ and $\\beta _2=0.999$ for the encoder and the decoder, respectively, each with different warmup-steps and learning rates:","where $\\tilde{lr}_{\\mathcal {E}}=2e^{-3}$, and $\\operatorname{\\operatorname{warmup}}_{\\mathcal {E}}=20,000$ for the encoder and $\\tilde{lr}_{\\mathcal {D}}=0.1$, and $\\operatorname{\\operatorname{warmup}}_{\\mathcal {D}}=10,000$ for the decoder. This is based on the assumption that the pretrained encoder should be fine-tuned with a smaller learning rate and smoother decay (so that the encoder can be trained with more accurate gradients when the decoder is becoming stable).","In addition, we propose a two-stage fine-tuning approach, where we first fine-tune the encoder on the extractive summarization task (Section SECREF8) and then fine-tune it on the abstractive summarization task (Section SECREF13). Previous work BIBREF4, BIBREF23 suggests that using extractive objectives can boost the performance of abstractive summarization. Also notice that this two-stage approach is conceptually very simple, the model can take advantage of information shared between these two tasks, without fundamentally changing its architecture. We name the default abstractive model BertSumAbs and the two-stage fine-tuned model BertSumExtAbs."]},{"idx":8,"heading":"Experimental Setup","paragraphs":["In this section, we describe the summarization datasets used in our experiments and discuss various implementation details."]},{"idx":9,"heading":"Experimental Setup ::: Summarization Datasets","paragraphs":["We evaluated our model on three benchmark datasets, namely the CNN/DailyMail news highlights dataset BIBREF24, the New York Times Annotated Corpus (NYT; BIBREF25), and XSum BIBREF22. These datasets represent different summary styles ranging from highlights to very brief one sentence summaries. The summaries also vary with respect to the type of rewriting operations they exemplify (e.g., some showcase more cut and paste operations while others are genuinely abstractive). Table TABREF12 presents statistics on these datasets (test set); example (gold-standard) summaries are provided in the supplementary material."]},{"idx":10,"heading":"Experimental Setup ::: Summarization Datasets ::: CNN/DailyMail","paragraphs":["contains news articles and associated highlights, i.e., a few bullet points giving a brief overview of the article. We used the standard splits of BIBREF24 for training, validation, and testing (90,266/1,220/1,093 CNN documents and 196,961/12,148/10,397 DailyMail documents). We did not anonymize entities. We first split sentences with the Stanford CoreNLP toolkit BIBREF26 and pre-processed the dataset following BIBREF6. Input documents were truncated to 512 tokens."]},{"idx":11,"heading":"Experimental Setup ::: Summarization Datasets ::: NYT","paragraphs":["contains 110,540 articles with abstractive summaries. Following BIBREF27, we split these into 100,834/9,706 training/test examples, based on the date of publication (the test set contains all articles published from January 1, 2007 onward). We used 4,000 examples from the training as validation set. We also followed their filtering procedure, documents with summaries less than 50 words were removed from the dataset. The filtered test set (NYT50) includes 3,452 examples. Sentences were split with the Stanford CoreNLP toolkit BIBREF26 and pre-processed following BIBREF27. Input documents were truncated to 800 tokens."]},{"idx":12,"heading":"Experimental Setup ::: Summarization Datasets ::: XSum","paragraphs":["contains 226,711 news articles accompanied with a one-sentence summary, answering the question “What is this article about?”. We used the splits of BIBREF22 for training, validation, and testing (204,045/11,332/11,334) and followed the pre-processing introduced in their work. Input documents were truncated to 512 tokens.","Aside from various statistics on the three datasets, Table TABREF12 also reports the proportion of novel bi-grams in gold summaries as a measure of their abstractiveness. We would expect models with extractive biases to perform better on datasets with (mostly) extractive summaries, and abstractive models to perform more rewrite operations on datasets with abstractive summaries. CNN/DailyMail and NYT are somewhat abstractive, while XSum is highly abstractive."]},{"idx":13,"heading":"Experimental Setup ::: Implementation Details","paragraphs":["For both extractive and abstractive settings, we used PyTorch, OpenNMT BIBREF28 and the `bert-base-uncased' version of Bert to implement BertSum. Both source and target texts were tokenized with Bert's subwords tokenizer."]},{"idx":14,"heading":"Experimental Setup ::: Implementation Details ::: Extractive Summarization","paragraphs":["All extractive models were trained for 50,000 steps on 3 GPUs (GTX 1080 Ti) with gradient accumulation every two steps. Model checkpoints were saved and evaluated on the validation set every 1,000 steps. We selected the top-3 checkpoints based on the evaluation loss on the validation set, and report the averaged results on the test set. We used a greedy algorithm similar to BIBREF7 to obtain an oracle summary for each document to train extractive models. The algorithm generates an oracle consisting of multiple sentences which maximize the ROUGE-2 score against the gold summary.","When predicting summaries for a new document, we first use the model to obtain the score for each sentence. We then rank these sentences by their scores from highest to lowest, and select the top-3 sentences as the summary.","During sentence selection we use Trigram Blocking to reduce redundancy BIBREF9. Given summary $S$ and candidate sentence $c$, we skip $c$ if there exists a trigram overlapping between $c$ and $S$. The intuition is similar to Maximal Marginal Relevance (MMR; BIBREF29); we wish to minimize the similarity between the sentence being considered and sentences which have been already selected as part of the summary."]},{"idx":15,"heading":"Experimental Setup ::: Implementation Details ::: Abstractive Summarization","paragraphs":["In all abstractive models, we applied dropout (with probability $0.1$) before all linear layers; label smoothing BIBREF30 with smoothing factor $0.1$ was also used. Our Transformer decoder has 768 hidden units and the hidden size for all feed-forward layers is 2,048. All models were trained for 200,000 steps on 4 GPUs (GTX 1080 Ti) with gradient accumulation every five steps. Model checkpoints were saved and evaluated on the validation set every 2,500 steps. We selected the top-3 checkpoints based on their evaluation loss on the validation set, and report the averaged results on the test set.","During decoding we used beam search (size 5), and tuned the $\\alpha $ for the length penalty BIBREF31 between $0.6$ and 1 on the validation set; we decode until an end-of-sequence token is emitted and repeated trigrams are blocked BIBREF9. It is worth noting that our decoder applies neither a copy nor a coverage mechanism BIBREF6, despite their popularity in abstractive summarization. This is mainly because we focus on building a minimum-requirements model and these mechanisms may introduce additional hyper-parameters to tune. Thanks to the subwords tokenizer, we also rarely observe issues with out-of-vocabulary words in the output; moreover, trigram-blocking produces diverse summaries managing to reduce repetitions."]},{"idx":16,"heading":"Results ::: Automatic Evaluation","paragraphs":["We evaluated summarization quality automatically using ROUGE BIBREF32. We report unigram and bigram overlap (ROUGE-1 and ROUGE-2) as a means of assessing informativeness and the longest common subsequence (ROUGE-L) as a means of assessing fluency. Table TABREF23 summarizes our results on the CNN/DailyMail dataset. The first block in the table includes the results of an extractive Oracle system as an upper bound. We also present the Lead-3 baseline (which simply selects the first three sentences in a document). The second block in the table includes various extractive models trained on the CNN/DailyMail dataset (see Section SECREF5 for an overview). For comparison to our own model, we also implemented a non-pretrained Transformer baseline (TransformerExt) which uses the same architecture as BertSumExt, but with fewer parameters. It is randomly initialized and only trained on the summarization task. TransformerExt has 6 layers, the hidden size is 512, and the feed-forward filter size is 2,048. The model was trained with same settings as in BIBREF3. The third block in Table TABREF23 highlights the performance of several abstractive models on the CNN/DailyMail dataset (see Section SECREF6 for an overview). We also include an abstractive Transformer baseline (TransformerAbs) which has the same decoder as our abstractive BertSum models; the encoder is a 6-layer Transformer with 768 hidden size and 2,048 feed-forward filter size. The fourth block reports results with fine-tuned Bert models: BertSumExt and its two variants (one without interval embeddings, and one with the large version of Bert), BertSumAbs, and BertSumExtAbs. Bert-based models outperform the Lead-3 baseline which is not a strawman; on the CNN/DailyMail corpus it is indeed superior to several extractive BIBREF7, BIBREF8, BIBREF19 and abstractive models BIBREF6. Bert models collectively outperform all previously proposed extractive and abstractive systems, only falling behind the Oracle upper bound. Among Bert variants, BertSumExt performs best which is not entirely surprising; CNN/DailyMail summaries are somewhat extractive and even abstractive models are prone to copying sentences from the source document when trained on this dataset BIBREF6. Perhaps unsurprisingly we observe that larger versions of Bert lead to performance improvements and that interval embeddings bring only slight gains. Table TABREF24 presents results on the NYT dataset. Following the evaluation protocol in BIBREF27, we use limited-length ROUGE Recall, where predicted summaries are truncated to the length of the gold summaries. Again, we report the performance of the Oracle upper bound and Lead-3 baseline. The second block in the table contains previously proposed extractive models as well as our own Transformer baseline. Compress BIBREF27 is an ILP-based model which combines compression and anaphoricity constraints. The third block includes abstractive models from the literature, and our Transformer baseline. Bert-based models are shown in the fourth block. Again, we observe that they outperform previously proposed approaches. On this dataset, abstractive Bert models generally perform better compared to BertSumExt, almost approaching Oracle performance.","Table TABREF26 summarizes our results on the XSum dataset. Recall that summaries in this dataset are highly abstractive (see Table TABREF12) consisting of a single sentence conveying the gist of the document. Extractive models here perform poorly as corroborated by the low performance of the Lead baseline (which simply selects the leading sentence from the document), and the Oracle (which selects a single-best sentence in each document) in Table TABREF26. As a result, we do not report results for extractive models on this dataset. The second block in Table TABREF26 presents the results of various abstractive models taken from BIBREF22 and also includes our own abstractive Transformer baseline. In the third block we show the results of our Bert summarizers which again are superior to all previously reported models (by a wide margin)."]},{"idx":17,"heading":"Results ::: Model Analysis ::: Learning Rates","paragraphs":["Recall that our abstractive model uses separate optimizers for the encoder and decoder. In Table TABREF27 we examine whether the combination of different learning rates ($\\tilde{lr}_{\\mathcal {E}}$ and $\\tilde{lr}_{\\mathcal {D}}$) is indeed beneficial. Specifically, we report model perplexity on the CNN/DailyMail validation set for varying encoder/decoder learning rates. We can see that the model performs best with $\\tilde{lr}_{\\mathcal {E}}=2e-3$ and $\\tilde{lr}_{\\mathcal {D}}=0.1$."]},{"idx":18,"heading":"Results ::: Model Analysis ::: Position of Extracted Sentences","paragraphs":["In addition to the evaluation based on ROUGE, we also analyzed in more detail the summaries produced by our model. For the extractive setting, we looked at the position (in the source document) of the sentences which were selected to appear in the summary. Figure FIGREF31 shows the proportion of selected summary sentences which appear in the source document at positions 1, 2, and so on. The analysis was conducted on the CNN/DailyMail dataset for Oracle summaries, and those produced by BertSumExt and the TransformerExt. We can see that Oracle summary sentences are fairly smoothly distributed across documents, while summaries created by TransformerExt mostly concentrate on the first document sentences. BertSumExt outputs are more similar to Oracle summaries, indicating that with the pretrained encoder, the model relies less on shallow position features, and learns deeper document representations."]},{"idx":19,"heading":"Results ::: Model Analysis ::: Novel N-grams","paragraphs":["We also analyzed the output of abstractive systems by calculating the proportion of novel n-grams that appear in the summaries but not in the source texts. The results are shown in Figure FIGREF33. In the CNN/DailyMail dataset, the proportion of novel n-grams in automatically generated summaries is much lower compared to reference summaries, but in XSum, this gap is much smaller. We also observe that on CNN/DailyMail, BertExtAbs produces less novel n-ngrams than BertAbs, which is not surprising. BertExtAbs is more biased towards selecting sentences from the source document since it is initially trained as an extractive model. The supplementary material includes examples of system output and additional ablation studies."]},{"idx":20,"heading":"Results ::: Human Evaluation","paragraphs":["In addition to automatic evaluation, we also evaluated system output by eliciting human judgments. We report experiments following a question-answering (QA) paradigm BIBREF33, BIBREF8 which quantifies the degree to which summarization models retain key information from the document. Under this paradigm, a set of questions is created based on the gold summary under the assumption that it highlights the most important document content. Participants are then asked to answer these questions by reading system summaries alone without access to the article. The more questions a system can answer, the better it is at summarizing the document as a whole. Moreover, we also assessed the overall quality of the summaries produced by abstractive systems which due to their ability to rewrite content may produce disfluent or ungrammatical output. Specifically, we followed the Best-Worst Scaling BIBREF34 method where participants were presented with the output of two systems (and the original document) and asked to decide which one was better according to the criteria of Informativeness, Fluency, and Succinctness.","Both types of evaluation were conducted on the Amazon Mechanical Turk platform. For the CNN/DailyMail and NYT datasets we used the same documents (20 in total) and questions from previous work BIBREF8, BIBREF18. For XSum, we randomly selected 20 documents (and their questions) from the release of BIBREF22. We elicited 3 responses per HIT. With regard to QA evaluation, we adopted the scoring mechanism from BIBREF33; correct answers were marked with a score of one, partially correct answers with 0.5, and zero otherwise. For quality-based evaluation, the rating of each system was computed as the percentage of times it was chosen as better minus the times it was selected as worse. Ratings thus range from -1 (worst) to 1 (best).","Results for extractive and abstractive systems are shown in Tables TABREF37 and TABREF38, respectively. We compared the best performing BertSum model in each setting (extractive or abstractive) against various state-of-the-art systems (whose output is publicly available), the Lead baseline, and the Gold standard as an upper bound. As shown in both tables participants overwhelmingly prefer the output of our model against comparison systems across datasets and evaluation paradigms. All differences between BertSum and comparison models are statistically significant ($p<0.05$), with the exception of TConvS2S (see Table TABREF38; XSum) in the QA evaluation setting."]},{"idx":21,"heading":"Conclusions","paragraphs":["In this paper, we showcased how pretrained Bert can be usefully applied in text summarization. We introduced a novel document-level encoder and proposed a general framework for both abstractive and extractive summarization. Experimental results across three datasets show that our model achieves state-of-the-art results across the board under automatic and human-based evaluation protocols. Although we mainly focused on document encoding for summarization, in the future, we would like to take advantage the capabilities of Bert for language generation."]},{"idx":22,"heading":"Acknowledgments","paragraphs":["This research is supported by a Google PhD Fellowship to the first author. We gratefully acknowledge the support of the European Research Council (Lapata, award number 681760, “Translating Multiple Modalities into Text”). We would also like to thank Shashi Narayan for providing us with the XSum dataset."]},{"idx":23,"heading":"Figures and Tables","paragraphs":["Figure 1: Architecture of the original BERT model (left) and BERTSUM (right). The sequence on top is the input document, followed by the summation of three kinds of embeddings for each token. The summed vectors are used as input embeddings to several bidirectional Transformer layers, generating contextual vectors for each token. BERTSUM extends BERT by inserting multiple [CLS] symbols to learn sentence representations and using interval segmentation embeddings (illustrated in red and green color) to distinguish multiple sentences.","Table 1: Comparison of summarization datasets: size of training, validation, and test sets and average document and summary length (in terms of words and sentences). The proportion of novel bi-grams that do not appear in source documents but do appear in the gold summaries quantifies corpus bias towards extractive methods.","Table 2: ROUGE F1 results on CNN/DailyMail test set (R1 and R2 are shorthands for unigram and bigram overlap; RL is the longest common subsequence). Results for comparison systems are taken from the authors’ respective papers or obtained on our data by running publicly released software.","Table 4: ROUGE F1 results on the XSum test set. Results for comparison systems are taken from the authors’ respective papers or obtained on our data by running publicly released software.","Table 3: ROUGE Recall results on NYT test set. Results for comparison systems are taken from the authors’ respective papers or obtained on our data by running publicly released software. Table cells are filled with — whenever results are not available.","Table 5: Model perplexity (CNN/DailyMail; validation set) under different combinations of encoder and decoder learning rates.","Figure 2: Proportion of extracted sentences according to their position in the original document.","Figure 3: Proportion of novel n-grams in model generated summaries.","Table 7: QA-based and ranking-based evaluation. Models with † are significantly different from BERTSUM (using a paired student t-test; p < 0.05). Table cells are filled with — whenever system output is not available. GOLD is not used in QA setting, and LEAD is not used in Rank evaluation.","Table 6: QA-based evaluation. Models with † are significantly different from BERTSUM (using a paired student t-test; p < 0.05). Table cells are filled with — whenever system output is not available."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1908.09355","title":"Patient Knowledge Distillation for BERT Model Compression","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Pre-trained language models such as BERT have proven to be highly effective for natural language processing (NLP) tasks. However, the high demand for computing resources in training such models hinders their application in practice. In order to alleviate this resource hunger in large-scale model training, we propose a Patient Knowledge Distillation approach to compress an original large model (teacher) into an equally-effective lightweight shallow network (student). Different from previous knowledge distillation methods, which only use the output from the last layer of the teacher network for distillation, our student model patiently learns from multiple intermediate layers of the teacher model for incremental knowledge extraction, following two strategies: ($i$) PKD-Last: learning from the last $k$ layers; and ($ii$) PKD-Skip: learning from every $k$ layers. These two patient distillation schemes enable the exploitation of rich information in the teacher's hidden layers, and encourage the student model to patiently learn from and imitate the teacher through a multi-layer distillation process. Empirically, this translates into improved results on multiple NLP tasks with significant gain in training efficiency, without sacrificing model accuracy."]},{"idx":1,"heading":"Introduction","paragraphs":["Language model pre-training has proven to be highly effective in learning universal language representations from large-scale unlabeled data. ELMo BIBREF0, GPT BIBREF1 and BERT BIBREF2 have achieved great success in many NLP tasks, such as sentiment classification BIBREF3, natural language inference BIBREF4, and question answering BIBREF5.","Despite its empirical success, BERT's computational efficiency is a widely recognized issue because of its large number of parameters. For example, the original BERT-Base model has 12 layers and 110 million parameters. Training from scratch typically takes four days on 4 to 16 Cloud TPUs. Even fine-tuning the pre-trained model with task-specific dataset may take several hours to finish one epoch. Thus, reducing computational costs for such models is crucial for their application in practice, where computational resources are limited.","Motivated by this, we investigate the redundancy issue of learned parameters in large-scale pre-trained models, and propose a new model compression approach, Patient Knowledge Distillation (Patient-KD), to compress original teacher (e.g., BERT) into a lightweight student model without performance sacrifice. In our approach, the teacher model outputs probability logits and predicts labels for the training samples (extendable to additional unannotated samples), and the student model learns from the teacher network to mimic the teacher's prediction.","Different from previous knowledge distillation methods BIBREF6, BIBREF7, BIBREF8, we adopt a patient learning mechanism: instead of learning parameters from only the last layer of the teacher, we encourage the student model to extract knowledge also from previous layers of the teacher network. We call this `Patient Knowledge Distillation'. This patient learner has the advantage of distilling rich information through the deep structure of the teacher network for multi-layer knowledge distillation.","We also propose two different strategies for the distillation process: ($i$) PKD-Last: the student learns from the last $k$ layers of the teacher, under the assumption that the top layers of the original network contain the most informative knowledge to teach the student; and ($ii$) PKD-Skip: the student learns from every $k$ layers of the teacher, suggesting that the lower layers of the teacher network also contain important information and should be passed along for incremental distillation.","We evaluate the proposed approach on several NLP tasks, including Sentiment Classification, Paraphrase Similarity Matching, Natural Language Inference, and Machine Reading Comprehension. Experiments on seven datasets across these four tasks demonstrate that the proposed Patient-KD approach achieves superior performance and better generalization than standard knowledge distillation methods BIBREF6, with significant gain in training efficiency and storage reduction while maintaining comparable model accuracy to original large models. To the authors' best knowledge, this is the first known effort for BERT model compression."]},{"idx":2,"heading":"Related Work ::: Language Model Pre-training","paragraphs":["Pre-training has been widely applied to universal language representation learning. Previous work can be divided into two main categories: ($i$) feature-based approach; ($ii$) fine-tuning approach.","Feature-based methods mainly focus on learning: ($i$) context-independent word representation (e.g., word2vec BIBREF9, GloVe BIBREF10, FastText BIBREF11); ($ii$) sentence-level representation (e.g., BIBREF12, BIBREF13, BIBREF14); and ($iii$) contextualized word representation (e.g., Cove BIBREF15, ELMo BIBREF0). Specifically, ELMo BIBREF0 learns high-quality, deep contextualized word representation using bidirectional language model, which can be directly plugged into standard NLU models for performance boosting.","On the other hand, fine-tuning approaches mainly pre-train a language model (e.g., GPT BIBREF1, BERT BIBREF2) on a large corpus with an unsupervised objective, and then fine-tune the model with in-domain labeled data for downstream applications BIBREF16, BIBREF17. Specifically, BERT is a large-scale language model consisting of multiple layers of Transformer blocks BIBREF18. BERT-Base has 12 layers of Transformer and 110 million parameters, while BERT-Large has 24 layers of Transformer and 330 million parameters. By pre-training via masked language modeling and next sentence prediction, BERT has achieved state-of-the-art performance on a wide-range of NLU tasks, such as the GLUE benchmark BIBREF19 and SQuAD BIBREF20.","However, these modern pre-trained language models contain millions of parameters, which hinders their application in practice where computational resource is limited. In this paper, we aim at addressing this critical and challenging problem, taking BERT as an example, i.e., how to compress a large BERT model into a shallower one without sacrificing performance. Besides, the proposed approach can also be applied to other large-scale pre-trained language models, such as recently proposed XLNet BIBREF21 and RoBERTa BIBREF22."]},{"idx":3,"heading":"Related Work ::: Model Compression & Knowledge Distillation","paragraphs":["Our focus is model compression, i.e., making deep neural networks more compact BIBREF23, BIBREF24. A similar line of work has focused on accelerating deep network inference at test time BIBREF25 and reducing model training time BIBREF26.","A conventional understanding is that a large number of connections (weights) is necessary for training deep networks BIBREF27, BIBREF28. However, once the network has been trained, there will be a high degree of parameter redundancy. Network pruning BIBREF29, BIBREF30, in which network connections are reduced or sparsified, is one common strategy for model compression. Another direction is weight quantization BIBREF31, BIBREF32, in which connection weights are constrained to a set of discrete values, allowing weights to be represented by fewer bits. However, most of these pruning and quantization approaches perform on convolutional networks. Only a few work are designed for rich structural information such as deep language models BIBREF33.","Knowledge distillation BIBREF6 aims to compress a network with a large set of parameters into a compact and fast-to-execute model. This can be achieved by training a compact model to imitate the soft output of a larger model. BIBREF34 further demonstrated that intermediate representations learned by the large model can serve as hints to improve the training process and the final performance of the compact model. BIBREF35 introduced techniques for efficiently transferring knowledge from an existing network to a deeper or wider network. More recently, BIBREF36 used knowledge from ensemble models to improve single model performance on NLU tasks. BIBREF37 tried knowledge distillation for multilingual translation. Different from the above efforts, we investigate the problem of compressing large-scale language models, and propose a novel patient knowledge distillation approach to effectively transferring knowledge from a teacher to a student model."]},{"idx":4,"heading":"Patient Knowledge Distillation","paragraphs":["In this section, we first introduce a vanilla knowledge distillation method for BERT compression (Section SECREF5), then present the proposed Patient Knowledge Distillation (Section SECREF12) in details."]},{"idx":5,"heading":"Patient Knowledge Distillation ::: Problem Definition","paragraphs":["The original large teacher network is represented by a function $f(\\mathbf {x};\\mathbf {\\theta })$, where $\\mathbf {x}$ is the input to the network, and $\\mathbf {\\theta }$ denotes the model parameters. The goal of knowledge distillation is to learn a new set of parameters $\\mathbf {\\theta }^{\\prime }$ for a shallower student network $g(\\mathbf {x};\\mathbf {\\theta }^{\\prime })$, such that the student network achieves similar performance to the teacher, with much lower computational cost. Our strategy is to force the student model to imitate outputs from the teacher model on the training dataset with a defined objective $L_{KD}$."]},{"idx":6,"heading":"Patient Knowledge Distillation ::: Distillation Objective","paragraphs":["In our setting, the teacher $f(\\mathbf {x};\\mathbf {\\theta })$ is defined as a deep bidirectional encoder, e.g., BERT, and the student $g(\\mathbf {x};\\mathbf {\\theta }^{\\prime })$ is a lightweight model with fewer layers. For simplicity, we use BERT$_k$ to denote a model with $k$ layers of Transformers. Following the original BERT paper BIBREF2, we also use BERT-Base and BERT-Large to denote BERT$_{12}$ and BERT$_{24}$, respectively.","Assume $\\lbrace \\mathbf {x}_i, \\mathbf {y}_i\\rbrace _{i=1}^N$ are $N$ training samples, where $\\mathbf {x}_i$ is the $i$-th input instance for BERT, and $\\mathbf {y}_i$ is the corresponding ground-truth label. BERT first computes a contextualized embedding $\\mathbf {h}_i = \\text{BERT} (\\mathbf {x}_i) \\in \\mathbb {R}^d$. Then, a softmax layer $\\hat{\\mathbf {y}}_i = P(\\mathbf {y}_i | \\mathbf {x}_i) = softmax(\\mathbf {W} \\mathbf {h}_i)$ for classification is applied to the embedding of BERT output, where $\\mathbf {W}$ is a weight matrix to be learned.","To apply knowledge distillation, first we need to train a teacher network. For example, to train a 12-layer BERT-Base as the teacher model, the learned parameters are denoted as:","where the superscript $t$ denotes parameters in the teacher model, $[N]$ denotes set $\\lbrace 1, 2, \\dots , N\\rbrace $, $L_{CE}^t$ denotes the cross-entropy loss for the teacher training, and $\\theta _{\\text{BERT}_{12}}$ denotes parameters of BERT$_{12}$.","The output probability for any given input $\\mathbf {x}_i$ can be formulated as:","where ${P}^t(\\cdot |\\cdot )$ denotes the probability output from the teacher. $\\hat{\\mathbf {y}}_i$ is fixed as soft labels, and $T$ is the temperature used in KD, which controls how much to rely on the teacher's soft predictions. A higher temperature produces a more diverse probability distribution over classes BIBREF6. Similarly, let $\\theta ^s$ denote parameters to be learned for the student model, and ${P}^s(\\cdot |\\cdot )$ denote the corresponding probability output from the student model. Thus, the distance between the teacher's prediction and the student's prediction can be defined as:","where $c$ is a class label and $C$ denotes the set of class labels.","Besides encouraging the student model to imitate the teacher's behavior, we can also fine-tune the student model on target tasks, where task-specific cross-entropy loss is included for model training:","Thus, the final objective function for knowledge distillation can be formulated as:","where $\\alpha $ is the hyper-parameter that balances the importance of the cross-entropy loss and the distillation loss."]},{"idx":7,"heading":"Patient Knowledge Distillation ::: Patient Teacher for Model Compression","paragraphs":["Using a weighted combination of ground-truth labels and soft predictions from the last layer of the teacher network, the student network can achieve comparable performance to the teacher model on the training set. However, with the number of epochs increasing, the student model learned with this vanilla KD framework quickly reaches saturation on the test set (see Figure FIGREF17 in Section SECREF4).","One hypothesis is that overfitting during knowledge distillation may lead to poor generalization. To mitigate this issue, instead of forcing the student to learn only from the logits of the last layer, we propose a “patient” teacher-student mechanism to distill knowledge from the teacher's intermediate layers as well. Specifically, we investigate two patient distillation strategies: ($i$) PKD-Skip: the student learns from every $k$ layers of the teacher (Figure FIGREF11: Left); and ($ii$) PKD-Last: the student learns from the last $k$ layers of the teacher (Figure FIGREF11: Right).","Learning from the hidden states of all the tokens is computationally expensive, and may introduce noise. In the original BERT implementation BIBREF2, prediction is performed by only using the output from the last layer's [CLS] token. In some variants of BERT, like SDNet BIBREF38, a weighted average of all layers' [CLS] embeddings is applied. In general, the final logit can be computed based on $\\mathbf {h}_{\\text{final}} = \\sum _{j \\in [k]} w_j \\mathbf {h}_j$, where $w_j$ could be either learned parameters or a pre-defined hyper-parameter, $\\mathbf {h}_j$ is the embedding of [CLS] from the hidden layer $j$, and $k$ is the number of hidden layers. Derived from this, if the compressed model can learn from the representation of [CLS] in the teacher's intermediate layers for any given input, it has the potential of gaining a generalization ability similar to the teacher model.","Motivated by this, in our Patient-KD framework, the student is cultivated to imitate the representations only for the [CLS] token in the intermediate layers, following the intuition aforementioned that the [CLS] token is important in predicting the final labels. For an input $\\mathbf {x}_i$, the outputs of the [CLS] tokens for all the layers are denoted as:","We denote the set of intermediate layers to distill knowledge from as $I_{pt}$. Take distilling from BERT$_{12}$ to BERT$_6$ as an example. For the PKD-Skip strategy, $I_{pt} = \\lbrace 2,4,6,8,10\\rbrace $; and for the PKD-Last strategy, $I_{pt} = \\lbrace 7,8,9,10,11\\rbrace $. Note that $k=5$ for both cases, because the output from the last layer (e.g., Layer 12 for BERT-Base) is omitted since its hidden states are connected to the softmax layer, which is already included in the KD loss defined in Eqn. (DISPLAY_FORM10). In general, for BERT student with $n$ layers, $k$ always equals to $n-1$.","The additional training loss introduced by the patient teacher is defined as the mean-square loss between the normalized hidden states:","where $M$ denotes the number of layers in the student network, $N$ is the number of training samples, and the superscripts $s$ and $t$ in $\\mathbf {h}$ indicate the student and the teacher model, respectively. Combined with the KD loss introduced in Section SECREF5, the final objective function can be formulated as:","where $\\beta $ is another hyper-parameter that weights the importance of the features for distillation in the intermediate layers."]},{"idx":8,"heading":"Experiments","paragraphs":["In this section, we describe our experiments on applying the proposed Patient-KD approach to four different NLP tasks. Details on the datasets and experimental results are provided in the following sub-sections."]},{"idx":9,"heading":"Experiments ::: Datasets","paragraphs":["We evaluate our proposed approach on Sentiment Classification, Paraphrase Similarity Matching, Natural Language Inference, and Machine Reading Comprehension tasks. For Sentiment Classification, we test on Stanford Sentiment Treebank (SST-2) BIBREF3. For Paraphrase Similarity Matching, we use Microsoft Research Paraphrase Corpus (MRPC) BIBREF39 and Quora Question Pairs (QQP) datasets. For Natural Language Inference, we evaluate on Multi-Genre Natural Language Inference (MNLI) BIBREF4, QNLI BIBREF20, and Recognizing Textual Entailment (RTE).","More specifically, SST-2 is a movie review dataset with binary annotations, where the binary label indicates positive and negative reviews. MRPC contains pairs of sentences and corresponding labels, which indicate the semantic equivalence relationship between each pair. QQP is designed to predict whether a pair of questions is duplicate or not, provided by a popular online question-answering website Quora. MNLI is a multi-domain NLI task for predicting whether a given premise-hypothesis pair is entailment, contradiction or neural. Its test and development datasets are further divided into in-domain (MNLI-m) and cross-domain (MNLI-mm) splits to evaluate the generality of tested models. QNLI is a task for predicting whether a question-answer pair is entailment or not. Finally, RTE is based on a series of textual entailment challenges, created by General Language Understanding Evaluation (GLUE) benchmark BIBREF19.","For the Machine Reading Comprehension task, we evaluate on RACE BIBREF5, a large-scale dataset collected from English exams, containing 25,137 passages and 87,866 questions. For each question, four candidate answers are provided, only one of which is correct. The dataset is further divided into RACE-M and RACE-H, containing exam questions for middle school and high school students."]},{"idx":10,"heading":"Experiments ::: Baselines and Training Details","paragraphs":["For experiments on the GLUE benchmark, since all the tasks can be considered as sentence (or sentence-pair) classification, we use the same architecture in the original BERT BIBREF2, and fine-tune each task independently.","For experiments on RACE, we denote the input passage as $P$, the question as $q$, and the four answers as $a_1, \\dots , a_4$. We first concatenate the tokens in $q$ and each $a_i$, and arrange the input of BERT as [CLS] $P$ [SEP] $q+a_i$ [SEP] for each input pair $(P, q+a_i)$, where [CLS] and [SEP] are the special tokens used in the original BERT. In this way, we can obtain a single logit value for each $a_i$. At last, a softmax layer is placed on top of these four logits to obtain the normalized probability of each answer $a_i$ being correct, which is then used to compute the cross-entropy loss for modeling training.","We fine-tune BERT-Base (denoted as BERT$_{12}$) as the teacher model to compute soft labels for each task independently, where the pretrained model weights are obtained from Google's official BERT's repo, and use 3 and 6 layers of Transformers as the student models (BERT$_{3}$ and BERT$_{6}$), respectively. We initialize BERT$_k$ with the first $k$ layers of parameters from pre-trained BERT-Base, where $k\\in \\lbrace 3, 6\\rbrace $. To validate the effectiveness of our proposed approach, we first conduct direct fine-tuning on each task without using any soft labels. In order to reduce the hyper-parameter search space, we fix the number of hidden units in the final softmax layer as 768, the batch size as 32, and the number of epochs as 4 for all the experiments, with a learning rate from {5e-5, 2e-5, 1e-5}. The model with the best validation accuracy is selected for each setting.","Besides direct fine-tuning, we further implement a vanilla KD method on all the tasks by optimizing the objective function in Eqn. (DISPLAY_FORM10). We set the temperature $T$ as {5, 10, 20}, $\\alpha = \\lbrace 0.2, 0.5, 0.7 \\rbrace $, and perform grid search over $T$, $\\alpha $ and learning rate, to select the model with the best validation accuracy. For our proposed Patient-KD approach, we conduct additional search over $\\beta $ from $\\lbrace 10, 100, 500, 1000\\rbrace $ on all the tasks. Since there are so many hyper-parameters to learn for Patient KD, we fix $\\alpha $ and $T$ to the values used in the model with the best performance from the vanilla KD experiments, and only search over $\\beta $ and learning rate."]},{"idx":11,"heading":"Experiments ::: Experimental Results","paragraphs":["We submitted our model predictions to the official GLUE evaluation server to obtain results on the test data. Results are summarized in Table TABREF16. Compared to direct fine-tuning and vanilla KD, our Patient-KD models with BERT$_3$ and BERT$_6$ students perform the best on almost all the tasks except MRPC. For MNLI-m and MNLI-mm, our 6-layer model improves 1.1% and 1.3% over fine-tune (FT) baselines; for QNLI and QQP, even though the gap between BERT$_6$-KD and BERT$_{12}$ teacher is relatively small, our approach still succeeded in improving over both FT and KD baselines and further closing the gap between the student and the teacher models.","Furthermore, in 5 tasks out of 7 (SST-2 (-2.3% compared to BERT-Base teacher), QQP (-0.1%), MNLI-m (-2.2%), MNLI-mm (-1.8%), and QNLI (-1.4%)), the proposed 6-layer student coached by the patient teacher achieved similar performance to the original BERT-Base, demonstrating the effectiveness of our approach. Interestingly, all those 5 tasks have more than 60k training samples, which indicates that our method tends to perform better when there is a large amount of training data.","For the QQP task, we can further reduce the model size to 3 layers, where BERT$_3$-PKD can still have a similar performance to the teacher model. The learning curves on the QNLI and MNLI datasets are provided in Figure FIGREF17. The student model learned with vanilla KD quickly saturated on the dev set, while the proposed Patient-KD keeps learning from the teacher and improving accuracy, only starting to plateau in a later stage.","For the MRPC dataset, one hypothesis for the reason on vanilla KD outperforming our model is that the lack of enough training samples may lead to overfitting on the dev set. To further investigate, we repeat the experiments three times and compute the average accuracy on the dev set. We observe that fine-tuning and vanilla KD have a mean dev accuracy of 82.23% and 82.84%, respectively. Our proposed method has a higher mean dev accuracy of 83.46%, hence indicating that our Patient-KD method slightly overfitted to the dev set of MRPC due to the small amount of training data. This can also be observed on the performance gap between teacher and student on RTE in Table TABREF28, which also has a small training set.","We further investigate the performance gain from two different patient teacher designs: PKD-Last vs. PKD-Skip. Results of both PKD variants on the GLUE benchmark (with BERT$_6$ as the student) are summarized in Table TABREF23. Although both strategies achieved improvement over the vanilla KD baseline (see Table TABREF16), PKD-Skip performs slightly better than PKD-Last. Presumably, this might be due to the fact that distilling information across every $k$ layers captures more diverse representations of richer semantics from low-level to high-level, while focusing on the last $k$ layers tends to capture relatively homogeneous semantic information.","Results on RACE are reported in Table TABREF25, which shows that the Vanilla KD method outperforms direct fine-tuning by 4.42%, and our proposed patient teacher achieves further 1.6% performance lift, which again demonstrates the effectiveness of Patient-KD."]},{"idx":12,"heading":"Experiments ::: Analysis of Model Efficiency","paragraphs":["We have demonstrated that the proposed Patient-KD method can effectively compress BERT$_{12}$ into BERT$_6$ models without performance sacrifice. In this section, we further investigate the efficiency of Patient-KD on storage saving and inference-time speedup. Parameter statistics and inference time are summarized in Table TABREF26. All the models share the same embedding layer with 24 millon parameters that map a 30k-word vocabulary to a 768-dimensional vector, which leads to 1.64 and 2.4 times of machine memory saving from BERT$_6$ and BERT$_3$, respectively.","To test the inference speed, we ran experiments on 105k samples from QNLI training set BIBREF20. Inference is performed on a single Titan RTX GPU with batch size set to 128, maximum sequence length set to 128, and FP16 activated. The inference time for the embedding layer is negligible compared to the Transformer layers. Results in Table TABREF26 show that the proposed Patient-KD approach achieves an almost linear speedup, 1.94 and 3.73 times for BERT$_6$ and BERT$_3$, respectively."]},{"idx":13,"heading":"Experiments ::: Does a Better Teacher Help?","paragraphs":["To evaluate the effectiveness of the teacher model in our Patient-KD framework, we conduct additional experiments to measure the difference between BERT-Base teacher and BERT-Large teacher for model compression.","Each Transformer layer in BERT-Large has 12.6 million parameters, which is much larger than the Transformer layer used in BERT-Base. For a compressed BERT model with 6 layers, BERT$_6$ with BERT-Base Transformer (denoted as BERT$_6$[Base]) has only 67.0 million parameters, while BERT$_6$ with BERT-Large Transformer (denoted as BERT$_6$[Large]) has 108.4 million parameters. Since the size of the [CLS] token embedding is different between BERT-Large and BERT-Base, we cannot directly compute the patient teacher loss (DISPLAY_FORM14) for BERT$_6$[Base] when BERT-Large is used as teacher. Hence, in the case where the teacher is BERT-Large and the student is BERT$_6$[Base], we only conduct experiments in the vanilla KD setting.","Results are summarized in Table TABREF28. When the teacher changes from BERT$_{12}$ to BERT$_{24}$ (i.e., Setting #1 vs. #2), there is not much difference between the students' performance. Specifically, BERT$_{12}$ teacher performs better on SST-2, QQP and QNLI, while BERT$_{24}$ performs better on MNLI-m, MNLI-mm and RTE. Presumably, distilling knowledge from a larger teacher requires a larger training dataset, thus better results are observed on MNLI-m and MNLI-mm.","We also report results on using BERT-Large as the teacher and BERT$_6$[Large] as the student. Interestingly, when comparing Setting #1 with #3, BERT$_6$[Large] performs much worse than BERT$_6$[Base] even though a better teacher is used in the former case. The BERT$_6$[Large] student also has 1.6 times more parameters than BERT$_6$[Base]. One intuition behind this is that the compression ratio for the BERT$_6$[Large] model is 4:1 (24:6), which is larger than the ratio used for the BERT$_6$[Base] model (2:1 (12:6)). The higher compression ratio renders it more challenging for the student model to absorb important weights.","When comparing Setting # 2 and #3, we observe that even when the same large teacher is used, BERT$_6$[Large] still performs worse than BERT$_6$[Base]. Presumably, this may be due to initialization mismatch. Ideally, we should pre-train BERT$_6$[Large] and BERT$_6$[Base] from scratch, and use the weights learned from the pre-training step for weight initialization in KD training. However, due to computational limits of training BERT$_6$ from scratch, we only initialize the student model with the first six layers of BERT$_{12}$ or BERT$_{24}$. Therefore, the first six layers of BERT$_{24}$ may not be able to capture high-level features, leading to worse KD performance.","Finally, when comparing Setting #3 vs. #4, where for setting #4 we use Patient-KD-Skip instead of vanilla KD, we observe a performance gain on almost all the tasks, which indicates Patient-KD is a generic approach independent of the selection of the teacher model (BERT$_{12}$ or BERT$_{24}$)."]},{"idx":14,"heading":"Conclusion","paragraphs":["In this paper, we propose a novel approach to compressing a large BERT model into a shallow one via Patient Knowledge Distillation. To fully utilize the rich information in deep structure of the teacher network, our Patient-KD approach encourages the student model to patiently learn from the teacher through a multi-layer distillation process. Extensive experiments over four NLP tasks demonstrate the effectiveness of our proposed model.","For future work, we plan to pre-train BERT from scratch to address the initialization mismatch issue, and potentially modify the proposed method such that it could also help during pre-training. Designing more sophisticated distance metrics for loss functions is another exploration direction. We will also investigate Patient-KD in more complex settings such as multi-task learning and meta learning."]},{"idx":15,"heading":"Figures and Tables","paragraphs":["Figure 1: Model architecture of the proposed Patient Knowledge Distillation approach to BERT model compression. (Left) PKD-Skip: the student network learns the teacher’s outputs in every 2 layers. (Right) PKDLast: the student learns the teacher’s outputs from the last 6 layers. Trm: Transformer.","Table 1: Results from the GLUE test server. The best results for 3-layer and 6-layer models are in-bold. Google’s submission results are obtained from official GLUE leaderboard. BERT12 (Teacher) is our own implementation of the BERT teacher model. FT represents direct fine-tuning on each dataset without using knowledge distillation. KD represents using a vanilla knowledge distillation method. And PKD represents our proposed Patient-KD-Skip approach. Results show that PKD-Skip outperforms the baselines on almost all the datasets except for MRPC. The numbers under each dataset indicate the corresponding number of training samples.","Figure 2: Accuracy on the training and dev sets of QNLI and MNLI datasets, by directly applying vanilla knowledge distillation (KD) and the proposed Patient-KD-Skip. The teacher and the student networks are BERT12 and BERT6, respectively. The student network learned with vanilla KD quickly saturates on the dev set, while the proposed Patient-KD starts to plateau only in a later stage.","Table 2: Performance comparison between PKD-Last and PKD-Skip on GLUE benchmark.","Table 3: Results on RACE test set. BERT12 (Leaderboard) denotes results extracted from the official leaderboard (http://www.qizhexie.com/ /data/RACE_leaderboard). BERT12 (Teacher) is our own implementation. Results of BERT3 are not included due to the large gap between the teacher and the BERT3 student.","Table 4: The number of parameters and inference time for BERT3, BERT6 and BERT12. Parameters in Transformers (Trm) grow linearly with the increase of layers. Note that the summation of # Param (Emb) and # Param (Trm) does not exactly equal to Total Params, because there is another softmax layer with 0.6M parameters.","Table 5: Performance comparison with different teacher and student models. BERT6[Base]/[Large] denotes a BERT6 model with a BERT-Base/Large Transformer in each layer. For PKD, we use the PKD-Skip architecture."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1908.09590","title":"Rethinking Attribute Representation and Injection for Sentiment Classification","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Text attributes, such as user and product information in product reviews, have been used to improve the performance of sentiment classification models. The de facto standard method is to incorporate them as additional biases in the attention mechanism, and more performance gains are achieved by extending the model architecture. In this paper, we show that the above method is the least effective way to represent and inject attributes. To demonstrate this hypothesis, unlike previous models with complicated architectures, we limit our base model to a simple BiLSTM with attention classifier, and instead focus on how and where the attributes should be incorporated in the model. We propose to represent attributes as chunk-wise importance weight matrices and consider four locations in the model (i.e., embedding, encoding, attention, classifier) to inject attributes. Experiments show that our proposed method achieves significant improvements over the standard approach and that attention mechanism is the worst location to inject attributes, contradicting prior work. We also outperform the state-of-the-art despite our use of a simple base model. Finally, we show that these representations transfer well to other tasks. Model implementation and datasets are released here: this https URL."]},{"idx":1,"heading":"Introduction","paragraphs":["The use of categorical attributes (e.g., user, topic, aspects) in the sentiment analysis community BIBREF0, BIBREF1, BIBREF2 is widespread. Prior to the deep learning era, these information were used as effective categorical features BIBREF3, BIBREF4, BIBREF5, BIBREF6 for the machine learning model. Recent work has used them to improve the overall performance BIBREF7, BIBREF8, interpretability BIBREF9, BIBREF10, and personalization BIBREF11 of neural network models in different tasks such as sentiment classification BIBREF12, review summarization BIBREF13, and text generation BIBREF8.","In particular, user and product information have been widely incorporated in sentiment classification models, especially since they are important metadata attributes found in review websites. BIBREF12 first showed significant accuracy increase of neural models when these information are used. Currently, the accepted standard method is to use them as additional biases when computing the weights $a$ in the attention mechanism, as introduced by BIBREF7 as:","where $u$ and $p$ are the user and product embeddings, and $h$ is a word encoding from BiLSTM. Since then, most of the subsequent work attempted to improve the model by extending the model architecture to be able to utilize external features BIBREF14, handle cold-start entities BIBREF9, and represent user and product separately BIBREF15.","Intuitively, however, this method is not the ideal method to represent and inject attributes because of two reasons. First, representing attributes as additional biases cannot model the relationship between the text and attributes. Rather, it only adds a user- and product-specific biases that are independent from the text when calculating the attention weights. Second, injecting the attributes in the attention mechanism means that user and product information are only used to customize how the model choose which words to focus on, as also shown empirically in previous work BIBREF7, BIBREF15. However, we argue that there are more intuitive locations to inject the attributes such as when contextualizing words to modify their sentiment intensity.","We propose to represent user and product information as weight matrices (i.e., $W$ in the equation above). Directly incorporating these attributes into $W$ leads to large increase in parameters and subsequently makes the model difficult to optimize. To mitigate these problems, we introduce chunk-wise importance weight matrices, which (1) uses a weight matrix smaller than $W$ by a chunk size factor, and (2) transforms these matrix into gates such that it corresponds to the relative importance of each neuron in $W$. We investigate the use of this method when injected to several locations in the base model: word embeddings, BiLSTM encoder, attention mechanism, and logistic classifier.","The results of our experiments can be summarized in three statements. First, our preliminary experiments show that doing bias-based attribute representation and attention-based injection is not an effective method to incorporate user and product information in sentiment classification models. Second, despite using only a simple BiLSTM with attention classifier, we significantly outperform previous state-of-the-art models that use more complicated architectures (e.g., models that use hierarchical models, external memory networks, etc.). Finally, we show that these attribute representations transfer well to other tasks such as product category classification and review headline generation."]},{"idx":2,"heading":"How and Where to Inject Attributes?","paragraphs":["In this section, we explore different ways on how to represent attributes and where in the model can we inject them."]},{"idx":3,"heading":"How and Where to Inject Attributes? ::: The Base Model","paragraphs":["The majority of this paper uses a base model that accepts a review $\\mathbf {x}=x_1,...,x_n$ as input and returns a sentiment $y$ as output, which we extend to also accept the corresponding user $u$ and product $p$ attributes as additional inputs. Different from previous work where models use complex architectures such as hierarchical LSTMs BIBREF7, BIBREF14 and external memory networks BIBREF16, BIBREF17, we aim to achieve improvements by only modifying how we represent and inject attributes. Thus, we use a simple classifier as our base model, which consists of four parts explained briefly as follows.","First, we embed $\\mathbf {x}$ using a word embedding matrix that returns word embeddings $x^{\\prime }_1,...,x^{\\prime }_n$. We subsequently apply a non-linear function to each word:","Second, we run a bidirectional LSTM BIBREF18 encoder to contextualize the words into $h_t=[\\overrightarrow{h}_t;\\overleftarrow{h}_t]$ based on their forward and backward neighbors. The forward and backward LSTM look similar, thus for brevity we only show the forward LSTM below:","Third, we pool the encodings $h_t$ into one document encoding $d$ using attention mechanism BIBREF19, where $v$ is a latent representation of informativeness BIBREF20:","Finally, we classify the document using a logistic classifier to get a predicted $y^{\\prime }$:","Training is done normally by minimizing the cross entropy loss."]},{"idx":4,"heading":"How and Where to Inject Attributes? ::: How: Attribute Representation","paragraphs":["Note that at each part of the model, we see similar non-linear functions, all using the same form, i.e. $g(f(x)) = g(Wx + b)$, where $f(x)$ is an affine transformation function of $x$, $g$ is a non-linear activation, $W$ and $b$ are weight matrix and bias parameters, respectively. Without extending the base model architecture, we can represent the attributes either as the weight matrix $W$ or as the bias $b$ to one of these functions by modifying them to accept $u$ and $p$ as inputs, i.e. $f(x,u,p)$."]},{"idx":5,"heading":"How and Where to Inject Attributes? ::: How: Attribute Representation ::: Bias-based","paragraphs":["The current accepted standard approach to represent the attributes is through the bias parameter $b$. Most of the previous work BIBREF7, BIBREF14, BIBREF9, BIBREF21 use Equation DISPLAY_FORM2 in the attention mechanism, which basically updates the original bias $b$ to $b^{\\prime } = W_u u + W_p p + b$. However, we argue that this is not the ideal way to incorporate attributes since it means we only add a user- and product-specific bias towards the goal of the function, without looking at the text. Figure FIGREF9 shows an intuitive example: When we represent user $u$ as a bias in the logistic classifier, in which it means that $u$ has a biased logits vector $b_u$ of classifying the text as a certain sentiment (e.g., $u$ tends to classify texts as three-star positive), shifting the final probability distribution regardless of what the text content may have been."]},{"idx":6,"heading":"How and Where to Inject Attributes? ::: How: Attribute Representation ::: Matrix-based","paragraphs":["A more intuitive way of representing attributes is through the weight matrix $W$. Specifically, given the attribute embeddings $u$ and $p$, we linearly transform their concatenation into a vector $w^{\\prime }$ of size $D_1*D_2$ where $D_1$ and $D_2$ are the dimensions of $W$. We then reshape $w^{\\prime }$ into $W^{\\prime }$ to get the same shape as $W$ and replace $W$ with $W^{\\prime }$:","Theoretically, this should perform better than bias-based representations since direct relationship between text and attributes are modeled. For example, following the example above, $W^{\\prime }x$ is a user-biased logits vector based on the document encoding $d$ (e.g., $u$ tends to classify texts as two-star positive when the text mentions that the dessert was sweet).","However, the model is burdened by a large number of parameters; matrix-based attribute representation increases the number of parameters by $|U|*|P|*D_1*D_2$, where $|U|$ and $|P|$ correspond to the number of users and products, respectively. This subsequently makes the weights difficult to optimize during training. Thus, directly incorporating attributes into the weight matrix may cause harm in the performance of the model."]},{"idx":7,"heading":"How and Where to Inject Attributes? ::: How: Attribute Representation ::: CHIM-based","paragraphs":["We introduce Chunk-wise Importance Matrix (CHIM) based representation, which improves over the matrix-based approach by mitigating the optimization problems mentioned above, using the following two tricks. First, instead of using a big weight matrix $W^{\\prime }$ of shape $(D_1, D_2)$, we use a chunked weight matrix $C$ of shape $(D_1/C_1, D_2/C_2)$ where $C_1$ and $C_2$ are chunk size factors. Second, we use the chunked weight matrix as importance gates that shrinks the weights close to zero when they are deemed unimportant. We show the CHIM-based representation method in Figure FIGREF16.","We start by linearly transforming the concatenated attributes into $c$. Then we reshape $c$ into $C$ with shape $(D_1/C_1, D_2/C_2)$. These operations are similar to Equations DISPLAY_FORM14 and . We then repeat this matrix $C_1*C_2$ times and concatenate them such that we create a matrix $W^{\\prime }$ of shape $(D_1, D_2)$. Finally, we use the sigmoid function $\\sigma $ to transform the matrix into gates that represent importance:","Finally we broadcast-multiply $W^{\\prime }$ with the original weight matrix $W$ to shrink the weights. The result is a sparse version of $W$, which can be seen as either a regularization step BIBREF22 where most weights are set close to zero, or a correction step BIBREF23 where the important gates are used to correct the weights. The use of multiple chunks regards CHIM as coarse-grained access control BIBREF24 where the use of different important gates for every node is unnecessary and expensive. The final function is shown below:","To summarize, chunking helps reduce the number of parameters while retaining the model performance, and importance matrix makes optimization easier during training, resulting to a performance improvement. We also tried alternative methods for importance matrix such as residual addition (i.e., $\\tanh (W^{\\prime }) + W$) introduced in BIBREF25, and low-rank adaptation methods BIBREF26, BIBREF27, but these did not improve the model performance."]},{"idx":8,"heading":"How and Where to Inject Attributes? ::: Where: Attribute Injection","paragraphs":["Using the approaches described above, we can inject attribute representation into four different parts of the model. This section describes what it means to inject attributes to a certain location and why previous work have been injecting them in the worst location (i.e., in the attention mechanism)."]},{"idx":9,"heading":"How and Where to Inject Attributes? ::: Where: Attribute Injection ::: In the attention mechanism","paragraphs":["Injecting attributes to the attention mechanism means that we bias the selection of more informative words during pooling. For example, in Figure FIGREF9, a user may find delicious drinks to be the most important aspect in a restaurant. Injection in the attention mechanism would bias the selection of words such as wine, smooth, and sweet to create the document encoding. This is the standard location in the model to inject the attributes, and several BIBREF7, BIBREF9 have shown how the injected attention mechanism selects different words when the given user or product is different.","We argue, however, that attention mechanism is not the best location to inject the attributes. This is because we cannot obtain user- or product-biased sentiment information from the representation. In the example above, although we may be able to select, with user bias, the words wine and sweet in the text, we do not know whether the user has a positive or negative sentiment towards these words (e.g., Does the user like wine? How about sweet wines? etc.). In contrast, the three other locations we discuss below use the attributes to modify how the model looks at sentiment at different levels of textual granularity."]},{"idx":10,"heading":"How and Where to Inject Attributes? ::: Where: Attribute Injection ::: In the word embedding","paragraphs":["Injecting attributes to the word embedding means that we bias the sentiment intensity of a word independent from its neighboring context. For example, if a user normally uses the words tasty and delicious with a less and more positive intensity, respectively, the corresponding attribute-injected word embeddings would come out less similar, despite both words being synonymous."]},{"idx":11,"heading":"How and Where to Inject Attributes? ::: Where: Attribute Injection ::: In the BiLSTM encoder","paragraphs":["Injecting attributes to the encoder means that we bias the contextualization of words based on their neighbors in the text. For example, if a user likes their cake sweet but their drink with no sugar, the attribute-injected encoder would give a positive signal to the encoding of sweet in the text “the cake was sweet” and a negative signal in the text “the drink was sweet”."]},{"idx":12,"heading":"How and Where to Inject Attributes? ::: Where: Attribute Injection ::: In the logistic classifier","paragraphs":["Injecting attributes to the classifier means that we bias the probability distribution of sentiment based on the final document encoding. If a user tends to classify the sentiment of reviews about sweet cakes as highly positive, then the model would give a high probability to highly positive sentiment classes for texts such as “the cake was sweet”."]},{"idx":13,"heading":"Experiments ::: General Setup","paragraphs":["We perform experiments on two tasks. The first task is Sentiment Classification, where we are tasked to classify the sentiment of a review text, given additionally the user and product information as attributes. The second task is Attribute Transfer, where we attempt to transfer the attribute encodings learned from the sentiment classification model to solve two other different tasks: (a) Product Category Classification, where we are tasked to classify the category of the product, and (b) Review Headline Generation, where we are tasked to generate the title of the review, given only both the user and product attribute encodings. Datasets, evaluation metrics, and competing models are different for each task and are described in their corresponding sections.","Unless otherwise stated, our models are implemented with the following settings. We set the dimensions of the word, user, and product vectors to 300. We use pre-trained GloVe embeddings BIBREF28 to initialize the word vectors. We also set the dimensions of the hidden state of BiLSTM to 300 (i.e., 150 dimensions for each of the forward/backward hidden state). The chunk size factors $C_1$ and $C_2$ are both set to 15. We use dropout BIBREF29 on all non-linear connections with a dropout rate of 0.1. We set the batch size to 32. Training is done via stochastic gradient descent over shuffled mini-batches with the Adadelta update rule BIBREF30 and with $l_2$ constraint BIBREF31 of 3. We perform early stopping using the development set. Training and experiments are done using an NVIDIA GeForce GTX 1080 Ti graphics card."]},{"idx":14,"heading":"Experiments ::: Sentiment Classification ::: Datasets and Evaluation","paragraphs":["We use the three widely used sentiment classification datasets with user and product information available: IMDB, Yelp 2013, and Yelp 2014 datasets. These datasets are curated by BIBREF12, where they ensured twenty-core for both users and products (i.e., users have at least twenty products and vice versa), split them into train, dev, and test sets with an 8:1:1 ratio, and tokenized and sentence-split using the Stanford CoreNLP BIBREF32. Dataset statistics are shown in Table TABREF20. Evaluation is done using two metrics: the accuracy which measures the overall sentiment classification performance, and RMSE which measures the divergence between predicted and ground truth classes."]},{"idx":15,"heading":"Experiments ::: Sentiment Classification ::: Comparisons of different attribute representation and injection methods","paragraphs":["To conduct a fair comparison among the different methods described in Section SECREF2, we compare these methods when applied to our base model using the development set of the datasets. Specifically, we use a smaller version of our base model (with dimensions set to 64) and incorporate the user and product attributes using nine different approaches: (1) bias-attention: the bias-based method injected to the attention mechanism, (2-5) the matrix-based method injected to four different locations (matrix-embedding, matrix-encoder, matrix-attention, matrix-classifier), and (6-9) the CHIM-based method injected to four different locations (CHIM-embedding, CHIM-encoder, CHIM-attention, CHIM-classifier). We then calculate the accuracy of each approach for all datasets.","Results are shown in Figure FIGREF25. The figure shows that bias-attention consistently performs poorly compared to other approaches. As expected, matrix-based representations perform the worst when injected to embeddings and encoder, however we can already see improvements over bias-attention when these representations are injected to attention and classifier. This is because the number of parameters used in the the weight matrices of attention and classifier are relatively smaller compared to those of embeddings and encoder, thus they are easier to optimize. The CHIM-based representations perform the best among other approaches, where CHIM-embedding garners the highest accuracy across datasets. Finally, even when using a better representation method, CHIM-attention consistently performs the worst among CHIM-based representations. This shows that attention mechanism is not the optimal location to inject attributes."]},{"idx":16,"heading":"Experiments ::: Sentiment Classification ::: Comparisons with models in the literature","paragraphs":["We also compare with models from previous work, listed below:","UPNN BIBREF12 uses a CNN classifier as base model and incorporates attributes as user- and product-specific weight parameters in the word embeddings and logistic classifier.","UPDMN BIBREF16 uses an LSTM classifier as base model and incorporates attributes as a separate deep memory network that uses other related documents as memory.","NSC BIBREF7 uses a hierarchical LSTM classifier as base model and incorporates attributes using the bias-attention method on both word- and sentence-level LSTMs.","DUPMN BIBREF17 also uses a hierarchical LSTM as base model and incorporates attributes as two separate deep memory network, one for each attribute.","PMA BIBREF14 is similar to NSC but uses external features such as the ranking preference method of a specific user.","HCSC BIBREF9 uses a combination of BiLSTM and CNN as base model, incorporates attributes using the bias-attention method, and also considers the existence of cold start entities.","CMA BIBREF15 uses a combination of LSTM and hierarchical attention classifier as base model, incorporates attributes using the bias-attention method, and does this separately for user and product.","Notice that most of these models, especially the later ones, use the bias-attention method to represent and inject attributes, but also employ a more complex model architecture to enjoy a boost in performance. Results are summarized in Table TABREF33. On all three datasets, our best results outperform all previous models based on accuracy and RMSE. Among our four models, CHIM-embedding performs the best in terms of accuracy, with performance increases of 2.4%, 1.3%, and 1.6% on IMDB, Yelp 2013, and Yelp 2014, respectively. CHIM-classifier performs the best in terms of RMSE, outperforming all other models on both Yelp 2013 and 2014 datasets. Among our models, CHIM-attention mechanism performs the worst, which shows similar results to our previous experiment (see Figure FIGREF25). We emphasize that our models use a simple BiLSTM as base model, and extensions to the base model (e.g., using multiple hierarchical LSTMs as in BIBREF21), as well as to other aspects (e.g., consideration of cold-start entities as in BIBREF9), are orthogonal to our proposed attribute representation and injection method. Thus, we expect a further increase in performance when these extensions are done."]},{"idx":17,"heading":"Experiments ::: Attribute Transfer","paragraphs":["In this section, we investigate whether it is possible to transfer the attribute encodings, learned from the sentiment classification model, to other tasks: product category classification and review headline generation. The experimental setup is as follows. First, we train a sentiment classification model using an attribute representation and injection method of choice to learn the attribute encodings. Then, we use these fixed encodings as input to the task-specific model."]},{"idx":18,"heading":"Experiments ::: Attribute Transfer ::: Dataset","paragraphs":["We collected a new dataset from Amazon, which includes the product category and the review headline, aside from the review text, the sentiment score, and the user and product attributes. Following BIBREF12, we ensured that both users and products are twenty-core, split them into train, dev, and test sets with an 8:1:1 ratio, and tokenized and sentence-split the text using Stanford CoreNLP BIBREF32. The final dataset contains 77,028 data points, with 1,728 users and 1,890 products. This is used as the sentiment classification dataset. To create the task-specific datasets, we split the dataset again such that no users and no products are seen in at least two different splits. That is, if user $u$ is found in the train set, then it should not be found in the dev and the test sets. We remove the user-product pairs that do not satistfy this condition. We then append the corresponding product category and review headline for each user-product pair. The final split contains 46,151 training, 711 development, and 840 test instances. It also contains two product categories: Music and Video DVD. The review headline is tokenized using SentencePiece with 10k vocabulary. The datasets are released here for reproducibility: https://github.com/rktamplayo/CHIM."]},{"idx":19,"heading":"Experiments ::: Attribute Transfer ::: Evaluation","paragraphs":["In this experiment, we compare five different attribute representation and injection methods: (1) the bias-attention method, and (2-5) the CHIM-based representation method injected to all four different locations in the model. We use the attribute encodings, which are learned from pre-training on the sentiment classification dataset, as input to the transfer tasks, in which they are fixed and not updated during training. As a baseline, we also show results when using encodings of randomly set weights. Moreover, we additionally show the majority class as additional baseline for product category classification. For the product category classification task, we use a logistic classifier as the classification model and accuracy as the evaluation metric. For the review headline generation task, we use an LSTM decoder as the generation model and perplexity as the evaluation metric."]},{"idx":20,"heading":"Experiments ::: Attribute Transfer ::: Results","paragraphs":["For the product category classification task, the results are reported in Table TABREF47. The table shows that representations learned from CHIM-based methods perform better than the random baseline. The best model, CHIM-encoder, achieves an increase of at least 3 points in accuracy compared to the baseline. This means that, interestingly, CHIM-based attribute representations have also learned information about the category of the product. In contrast, representations learned from the bias-attention method are not able to transfer well on this task, leading to worse results compared to the random and majority baseline. Moreover, CHIM-attention performs the worst among CHIM-based models, which further shows the ineffectiveness of injecting attributes to the attention mechanism.","Results for the review headline generation task are also shown in Table TABREF47. The table shows less promising results, where the best model, CHIM-encoder, achieves a decrease of 0.88 points in perplexity from the random encodings. Although this still means that some information has been transferred, one may argue that the gain is too small to be considered significant. However, it has been well perceived, that using only the user and product attributes to generate text is unreasonable, since we expect the model to generate coherent texts using only two vectors. This impossibility is also reported by BIBREF8 where they also used sentiment information, and BIBREF33 where they additionally used learned aspects and a short version of the text to be able to generate well-formed texts. Nevertheless, the results in this experiment agree to the results above regarding injecting attributes to the attention mechanism; bias-attention performs worse than the random baseline, and CHIM-attention performs the worst among CHIM-based models."]},{"idx":21,"heading":"Experiments ::: Where should attributes be injected?","paragraphs":["All our experiments unanimously show that (a) the bias-based attribute representation method is not the most optimal method, and (b) injecting attributes in the attention mechanism results to the worst performance among all locations in the model, regardless of the representation method used. The question “where is the best location to inject attributes?” remains unanswered, since different tasks and settings produce different best models. That is, CHIM-embedding achieves the best accuracy while CHIM-classifier achieves the best RMSE on sentiment classification. Moreover, CHIM-encoder produces the most transferable attribute encoding for both product category classification and review headline generation. The suggestion then is to conduct experiments on all locations and check which one is best for the task at hand.","Finally, we also investigate whether injecting in to more than one location would result to better performance. Specifically, we jointly inject in two different locations at once using CHIM, and do this for all possible pairs of locations. We use the smaller version of our base model and calculate the accuracies of different models using the development set of the Yelp 2013 dataset. Figure FIGREF49 shows a heatmap of the accuracies of jointly injected models, as well as singly injected models. Overall, the results are mixed and can be summarized into two statements. Firstly, injecting on the embedding and another location (aside from the attention mechanism) leads to a slight decrease in performance. Secondly and interestingly, injecting on the attention mechanism and another location always leads to the highest increase in performance, where CHIM-attention+embedding performs the best, outperforming CHIM-embedding. This shows that injecting in different locations might capture different information, and we leave this investigation for future work."]},{"idx":22,"heading":"Related Work ::: Attributes for Sentiment Classification","paragraphs":["Aside from user and product information, other attributes have been used for sentiment classification. Location-based BIBREF34 and time-based BIBREF35 attributes help contextualize the sentiment geographically and temporally. Latent attributes that are learned from another model have also been employed as additional features, such as latent topics from a topic model BIBREF36, latent aspects from an aspect extraction model BIBREF37, argumentation features BIBREF38, among others. Unfortunately, current benchmark datasets do not include these attributes, thus it is practically impossible to compare and use these attributes in our experiments. Nevertheless, the methods in this paper are not limited to only user and product attributes, but also to these other attributes as well, whenever available."]},{"idx":23,"heading":"Related Work ::: User/Product Attributes for NLP Tasks","paragraphs":["Incorporating user and product attributes to NLP models makes them more personalized and thus user satisfaction can be increased BIBREF39. Examples of other NLP tasks that use these attributes are text classification BIBREF27, language modeling BIBREF26, text generation BIBREF8, BIBREF33, review summarization BIBREF40, machine translation BIBREF41, and dialogue response generation BIBREF42. On these tasks, the usage of the bias-attention method is frequent since it is trivially easy and there have been no attempts to investigate different possible methods for attribute representation and injection. We expect this paper to serve as the first investigatory paper that contradicts to the positive results previous work have seen from the bias-attention method."]},{"idx":24,"heading":"Conclusions","paragraphs":["We showed that the current accepted standard for attribute representation and injection, i.e. bias-attention, which incorporates attributes as additional biases in the attention mechanism, is the least effective method. We proposed to represent attributes as chunk-wise importance weight matrices (CHIM) and showed that this representation method significantly outperforms the bias-attention method. Despite using a simple BiLSTM classifier as base model, CHIM significantly outperforms the current state-of-the-art models, even when those models use a more complex base model architecture. Furthermore, we conducted several experiments that conclude that injection to the attention mechanism, no matter which representation method is used, garners the worst performance. This result contradicts previously reported conclusions regarding attribute injection to the attention mechanism. Finally, we show promising results on transferring the attribute representations from sentiment classification, and use them to two different tasks such as product category classification and review headline generation."]},{"idx":25,"heading":"Acknowledgments","paragraphs":["We would like to thank the anonymous reviewers for their helpful feedback and suggestions. Reinald Kim Amplayo is grateful to be supported by a Google PhD Fellowship."]},{"idx":26,"heading":"Figures and Tables","paragraphs":["Figure 1: Illustrative examples of issues when representing attributes as biases and injecting them in the attention mechanism. The gray process icon indicates the model without incorporating attributes, while the same icon in green indicates the model customized for the green user.","Figure 2: CHIM-based attribute representation and injection to a non-linear funtion in the model.","Table 1: Statistics of the datasets used for the Sentiment Classification task.","Figure 3: Accuracies (y-axis) of different attribute representation (bias, matrix, CHIM) and injection (emb: embed, enc: encode, att: attend, cls: classify) approaches on the development set of the datasets.","Table 2: Sentiment classification results of competing models based on accuracy and RMSE metrics on the three datasets. Underlined values correspond to the best values for each block. Boldfaced values correspond to the best values across the board. 1uses additional external features, 2uses a method that considers cold-start entities, 3uses separate bias-attention for user and product.","Table 3: Accuracy (higher is better) and perplexity (lower is better) of competing models on the Amazon dataset for the transfer tasks on product category classification and review headline generation, respectively. Accuracy intervals are calculated by running the model 10 times. Performance worse than the random and majority baselines are colored red.","Figure 4: Heatmap of the accuracies of singly and jointly injected CHIM models. Values on each cell represents either the accuracy (for singly injected models) or the difference between the singly and doubly injected models per row."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1908.10084","title":"Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks","sections":[{"idx":0,"heading":"Abstract","paragraphs":["BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has set a new state-of-the-art performance on sentence-pair regression tasks like semantic textual similarity (STS). However, it requires that both sentences are fed into the network, which causes a massive computational overhead: Finding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations (~65 hours) with BERT. The construction of BERT makes it unsuitable for semantic similarity search as well as for unsupervised tasks like clustering. ::: In this publication, we present Sentence-BERT (SBERT), a modification of the pretrained BERT network that use siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT. ::: We evaluate SBERT and SRoBERTa on common STS tasks and transfer learning tasks, where it outperforms other state-of-the-art sentence embeddings methods."]},{"idx":1,"heading":"Introduction","paragraphs":["In this publication, we present Sentence-BERT (SBERT), a modification of the BERT network using siamese and triplet networks that is able to derive semantically meaningful sentence embeddings. This enables BERT to be used for certain new tasks, which up-to-now were not applicable for BERT. These tasks include large-scale semantic similarity comparison, clustering, and information retrieval via semantic search.","BERT set new state-of-the-art performance on various sentence classification and sentence-pair regression tasks. BERT uses a cross-encoder: Two sentences are passed to the transformer network and the target value is predicted. However, this setup is unsuitable for various pair regression tasks due to too many possible combinations. Finding in a collection of $n=10\\,000$ sentences the pair with the highest similarity requires with BERT $n\\cdot (n-1)/2=49\\,995\\,000$ inference computations. On a modern V100 GPU, this requires about 65 hours. Similar, finding which of the over 40 million existent questions of Quora is the most similar for a new question could be modeled as a pair-wise comparison with BERT, however, answering a single query would require over 50 hours.","A common method to address clustering and semantic search is to map each sentence to a vector space such that semantically similar sentences are close. Researchers have started to input individual sentences into BERT and to derive fixed-size sentence embeddings. The most commonly used approach is to average the BERT output layer (known as BERT embeddings) or by using the output of the first token (the [CLS] token). As we will show, this common practice yields rather bad sentence embeddings, often worse than averaging GloVe embeddings BIBREF2.","To alleviate this issue, we developed SBERT. The siamese network architecture enables that fixed-sized vectors for input sentences can be derived. Using a similarity measure like cosine-similarity or Manhatten / Euclidean distance, semantically similar sentences can be found. These similarity measures can be performed extremely efficient on modern hardware, allowing SBERT to be used for semantic similarity search as well as for clustering. The complexity for finding the most similar sentence pair in a collection of 10,000 sentences is reduced from 65 hours with BERT to the computation of 10,000 sentence embeddings (5 seconds with SBERT) and computing cosine-similarity (0.01 seconds). By using optimized index structures, finding the most similar Quora question can be reduced from 50 hours to a few milliseconds BIBREF3.","We fine-tune SBERT on NLI data, which creates sentence embeddings that significantly outperform other state-of-the-art sentence embedding methods like InferSent BIBREF4 and Universal Sentence Encoder BIBREF5. On seven Semantic Textual Similarity (STS) tasks, SBERT achieves an improvement of 11.7 points compared to InferSent and 5.5 points compared to Universal Sentence Encoder. On SentEval BIBREF6, an evaluation toolkit for sentence embeddings, we achieve an improvement of 2.1 and 2.6 points, respectively.","SBERT can be adapted to a specific task. It sets new state-of-the-art performance on a challenging argument similarity dataset BIBREF7 and on a triplet dataset to distinguish sentences from different sections of a Wikipedia article BIBREF8.","The paper is structured in the following way: Section SECREF3 presents SBERT, section SECREF4 evaluates SBERT on common STS tasks and on the challenging Argument Facet Similarity (AFS) corpus BIBREF7. Section SECREF5 evaluates SBERT on SentEval. In section SECREF6, we perform an ablation study to test some design aspect of SBERT. In section SECREF7, we compare the computational efficiency of SBERT sentence embeddings in contrast to other state-of-the-art sentence embedding methods."]},{"idx":2,"heading":"Related Work","paragraphs":["We first introduce BERT, then, we discuss state-of-the-art sentence embedding methods.","BERT BIBREF0 is a pre-trained transformer network BIBREF9, which set for various NLP tasks new state-of-the-art results, including question answering, sentence classification, and sentence-pair regression. The input for BERT for sentence-pair regression consists of the two sentences, separated by a special [SEP] token. Multi-head attention over 12 (base-model) or 24 layers (large-model) is applied and the output is passed to a simple regression function to derive the final label. Using this setup, BERT set a new state-of-the-art performance on the Semantic Textual Semilarity (STS) benchmark BIBREF10. RoBERTa BIBREF1 showed, that the performance of BERT can further improved by small adaptations to the pre-training process. We also tested XLNet BIBREF11, but it led in general to worse results than BERT.","A large disadvantage of the BERT network structure is that no independent sentence embeddings are computed, which makes it difficult to derive sentence embeddings from BERT. To bypass this limitations, researchers passed single sentences through BERT and then derive a fixed sized vector by either averaging the outputs (similar to average word embeddings) or by using the output of the special CLS token (for example: bertsentenceembeddings1,bertsentenceembeddings2,bertsentenceembeddings3). These two options are also provided by the popular bert-as-a-service-repository. Up to our knowledge, there is so far no evaluation if these methods lead to useful sentence embeddings.","Sentence embeddings are a well studied area with dozens of proposed methods. Skip-Thought BIBREF12 trains an encoder-decoder architecture to predict the surrounding sentences. InferSent BIBREF4 uses labeled data of the Stanford Natural Language Inference dataset BIBREF13 and the Multi-Genre NLI dataset BIBREF14 to train a siamese BiLSTM network with max-pooling over the output. Conneau et al. showed, that InferSent consistently outperforms unsupervised methods like SkipThought. Universal Sentence Encoder BIBREF5 trains a transformer network and augments unsupervised learning with training on SNLI. hill-etal-2016-learning showed, that the task on which sentence embeddings are trained significantly impacts their quality. Previous work BIBREF4, BIBREF5 found that the SNLI datasets are suitable for training sentence embeddings. yang-2018-learning presented a method to train on conversations from Reddit using siamese DAN and siamese transformer networks, which yielded good results on the STS benchmark dataset.","polyencoders addresses the run-time overhead of the cross-encoder from BERT and present a method (poly-encoders) to compute a score between $m$ context vectors and pre-computed candidate embeddings using attention. This idea works for finding the highest scoring sentence in a larger collection. However, poly-encoders have the drawback that the score function is not symmetric and the computational overhead is too large for use-cases like clustering, which would require $O(n^2)$ score computations.","Previous neural sentence embedding methods started the training from a random initialization. In this publication, we use the pre-trained BERT and RoBERTa network and only fine-tune it to yield useful sentence embeddings. This reduces significantly the needed training time: SBERT can be tuned in less than 20 minutes, while yielding better results than comparable sentence embedding methods."]},{"idx":3,"heading":"Model","paragraphs":["SBERT adds a pooling operation to the output of BERT / RoBERTa to derive a fixed sized sentence embedding. We experiment with three pooling strategies: Using the output of the CLS-token, computing the mean of all output vectors (MEAN-strategy), and computing a max-over-time of the output vectors (MAX-strategy). The default configuration is MEAN.","In order to fine-tune BERT / RoBERTa, we create siamese and triplet networks BIBREF15 to update the weights such that the produced sentence embeddings are semantically meaningful and can be compared with cosine-similarity.","The network structure depends on the available training data. We experiment with the following structures and objective functions.","Classification Objective Function. We concatenate the sentence embeddings $u$ and $v$ with the element-wise difference $|u-v|$ and multiply it with the trainable weight $W_t \\in \\mathbb {R}^{3n \\times k}$:","where $n$ is the dimension of the sentence embeddings and $k$ the number of labels. We optimize cross-entropy loss. This structure is depicted in Figure FIGREF4.","Regression Objective Function. The cosine-similarity between the two sentence embeddings $u$ and $v$ is computed (Figure FIGREF5). We use mean-squared-error loss as the objective function.","Triplet Objective Function. Given an anchor sentence $a$, a positive sentence $p$, and a negative sentence $n$, triplet loss tunes the network such that the distance between $a$ and $p$ is smaller than the distance between $a$ and $n$. Mathematically, we minimize the following loss function:","with $s_x$ the sentence embedding for $a$/$n$/$p$, $||\\cdot ||$ a distance metric and margin $\\epsilon $. Margin $\\epsilon $ ensures that $s_p$ is at least $\\epsilon $ closer to $s_a$ than $s_n$. As metric we use Euclidean distance and we set $\\epsilon =1$ in our experiments."]},{"idx":4,"heading":"Model ::: Training Details","paragraphs":["We train SBERT on the combination of the SNLI BIBREF13 and the Multi-Genre NLI BIBREF14 dataset. The SNLI is a collection of 570,000 sentence pairs annotated with the labels contradiction, eintailment, and neutral. MultiNLI contains 430,000 sentence pairs and covers a range of genres of spoken and written text. We fine-tune SBERT with a 3-way softmax-classifier objective function for one epoch. We used a batch-size of 16, Adam optimizer with learning rate $2\\mathrm {e}{-5}$, and a linear learning rate warm-up over 10% of the training data. Our default pooling strategy is MEAN."]},{"idx":5,"heading":"Evaluation - Semantic Textual Similarity","paragraphs":["We evaluate the performance of SBERT for common Semantic Textual Similarity (STS) tasks. State-of-the-art methods often learn a (complex) regression function that maps sentence embeddings to a similarity score. However, these regression functions work pair-wise and due to the combinatorial explosion those are often not scalable if the collection of sentences reaches a certain size. Instead, we always use cosine-similarity to compare the similarity between two sentence embeddings. We ran our experiments also with negative Manhatten and negative Euclidean distances as similarity measures, but the results for all approaches remained roughly the same."]},{"idx":6,"heading":"Evaluation - Semantic Textual Similarity ::: Unsupervised STS","paragraphs":["We evaluate the performance of SBERT for STS without using any STS specific training data. We use the STS tasks 2012 - 2016 BIBREF16, BIBREF17, BIBREF18, BIBREF19, BIBREF20, the STS benchmark BIBREF10, and the SICK-Relatedness dataset BIBREF21. These datasets provide labels between 0 and 5 on the semantic relatedness of sentence pairs. We showed in BIBREF22 that Pearson correlation is badly suited for STS. Instead, we compute the Spearman's rank correlation between the cosine-similarity of the sentence embeddings and the gold labels. The setup for the other sentence embedding methods is equivalent, the similarity is computed by cosine-similarity. The results are depicted in Table TABREF6.","The results shows that directly using the output of BERT leads to rather poor performances. Averaging the BERT embeddings achieves an average correlation of only 54.81, and using the CLS-token output only achieves an average correlation of 29.19. Both are worse than computing average GloVe embeddings.","Using the described siamese network structure and fine-tuning mechanism substantially improves the correlation, outperforming both InferSent and Universal Sentence Encoder substantially. The only dataset where SBERT performs worse than Universal Sentence Encoder is SICK-R. Universal Sentence Encoder was trained on various datasets, including news, question-answer pages and discussion forums, which appears to be more suitable to the data of SICK-R. In contrast, SBERT was pre-trained only on Wikipedia (via BERT) and on NLI data.","While RoBERTa was able to improve the performance for several supervised tasks, we only observe minor difference between SBERT and SRoBERTa for generating sentence embeddings."]},{"idx":7,"heading":"Evaluation - Semantic Textual Similarity ::: Supervised STS","paragraphs":["The STS benchmark (STSb) BIBREF10 provides is a popular dataset to evaluate supervised STS systems. The data includes 8,628 sentence pairs from the three categories captions, news, and forums. It is divided into train (5,749), dev (1,500) and test (1,379). BERT set a new state-of-the-art performance on this dataset by passing both sentences to the network and using a simple regression method for the output.","We use the training set to fine-tune SBERT using the regression objective function. At prediction time, we compute the cosine-similarity between the sentence embeddings. All systems are trained with 10 random seeds to counter variances BIBREF23.","The results are depicted in Table TABREF10. We experimented with two setups: Only training on STSb, and first training on NLI, then training on STSb. We observe that the later strategy leads to a slight improvement of 1-2 points. This two-step approach had an especially large impact for the BERT cross-encoder, which improved the performance by 3-4 points. We do not observe a significant difference between BERT and RoBERTa."]},{"idx":8,"heading":"Evaluation - Semantic Textual Similarity ::: Argument Facet Similarity","paragraphs":["We evaluate SBERT on the Argument Facet Similarity (AFS) corpus by MisraEW16. The AFS corpus annotated 6,000 sentential argument pairs from social media dialogs on three controversial topics: gun control, gay marriage, and death penalty. The data was annotated on a scale from 0 (“different topic\") to 5 (“completely equivalent\"). The similarity notion in the AFS corpus is fairly different to the similarity notion in the STS datasets from SemEval. STS data is usually descriptive, while AFS data are argumentative excerpts from dialogs. To be considered similar, arguments must not only make similar claims, but also provide a similar reasoning. Further, the lexical gap between the sentences in AFS is much larger. Hence, simple unsupervised methods as well as state-of-the-art STS systems perform badly on this dataset BIBREF24.","We evaluate SBERT on this dataset in two scenarios: 1) As proposed by Misra et al., we evaluate SBERT using 10-fold cross-validation. A draw-back of this evaluation setup is that it is not clear how well approaches generalize to different topics. Hence, 2) we evaluate SBERT in a cross-topic setup. Two topics serve for training and the approach is evaluated on the left-out topic. We repeat this for all three topics and average the results.","SBERT is fine-tuned using the Regression Objective Function. The similarity score is computed using cosine-similarity based on the sentence embeddings. We also provide the Pearson correlation $r$ to make the results comparable to Misra et al. However, we showed BIBREF22 that Pearson correlation has some serious drawbacks and should be avoided for comparing STS systems. The results are depicted in Table TABREF12.","Unsupervised methods like tf-idf, average GloVe embeddings or InferSent perform rather badly on this dataset with low scores. Training SBERT in the 10-fold cross-validation setup gives a performance that is nearly on-par with BERT.","However, in the cross-topic evaluation, we observe a performance drop of SBERT by about 7 points Spearman correlation. To be considered similar, arguments should address the same claims and provide the same reasoning. BERT is able to use attention to compare directly both sentences (e.g. word-by-word comparison), while SBERT must map individual sentences from an unseen topic to a vector space such that arguments with similar claims and reasons are close. This is a much more challenging task, which appears to require more than just two topics for training to work on-par with BERT."]},{"idx":9,"heading":"Evaluation - Semantic Textual Similarity ::: Wikipedia Sections Distinction","paragraphs":["ein-dor-etal-2018-learning use Wikipedia to create a thematically fine-grained train, dev and test set for sentence embeddings methods. Wikipedia articles are separated into distinct sections focusing on certain aspects. Dor et al. assume that sentences in the same section are thematically closer than sentences in different sections. They use this to create a large dataset of weakly labeled sentence triplets: The anchor and the positive example come from the same section, while the negative example comes from a different section of the same article. For example, from the Alice Arnold article: Anchor: Arnold joined the BBC Radio Drama Company in 1988., positive: Arnold gained media attention in May 2012., negative: Balding and Arnold are keen amateur golfers.","We use the dataset from Dor et al. We use the Triplet Objective, train SBERT for one epoch on the about 1.8 Million training triplets and evaluate it on the 222,957 test triplets. Test triplets are from a distinct set of Wikipedia articles. As evaluation metric, we use accuracy: Is the positive example closer to the anchor than the negative example?","Results are presented in Table TABREF14. Dor et al. fine-tuned a BiLSTM architecture with triplet loss to derive sentence embeddings for this dataset. As the table shows, SBERT clearly outperforms the BiLSTM approach by Dor et al."]},{"idx":10,"heading":"Evaluation - SentEval","paragraphs":["SentEval BIBREF6 is a popular toolkit to evaluate the quality of sentence embeddings. Sentence embeddings are used as features for a logistic regression classifier. The logistic regression classifier is trained on various tasks in a 10-fold cross-validation setup and the prediction accuracy is computed for the test-fold.","The purpose of SBERT sentence embeddings are not to be used for transfer learning for other tasks. Here, we think fine-tuning BERT as described by devlin2018bert for new tasks is the more suitable method, as it updates all layers of the BERT network. However, SentEval can still give an impression on the quality of our sentence embeddings for various tasks.","We compare the SBERT sentence embeddings to other sentence embeddings methods on the following seven SentEval transfer tasks:","MR: Sentiment prediction for movie reviews snippets on a five start scale BIBREF25.","CR: Sentiment prediction of customer product reviews BIBREF26.","SUBJ: Subjectivity prediction of sentences from movie reviews and plot summaries BIBREF27.","MPQA: Phrase level opinion polarity classification from newswire BIBREF28.","SST: Stanford Sentiment Treebank with binary labels BIBREF29.","TREC: Fine grained question-type classification from TREC BIBREF30.","MRPC: Microsoft Research Paraphrase Corpus from parallel news sources BIBREF31.","The results can be found in Table TABREF15. SBERT is able to achieve the best performance in 5 out of 7 tasks. The average performance increases by about 2 percentage points compared to InferSent as well as the Universal Sentence Encoder. Even though transfer learning is not the purpose of SBERT, it outperforms other state-of-the-art sentence embeddings methods on this task.","It appears that the sentence embeddings from SBERT capture well sentiment information: We observe large improvements for all sentiment tasks (MR, CR, and SST) from SentEval in comparison to InferSent and Universal Sentence Encoder.","The only dataset where SBERT is significantly worse than Universal Sentence Encoder is the TREC dataset. Universal Sentence Encoder was pre-trained on question-answering data, which appears to be beneficial for the question-type classification task of the TREC dataset.","Average BERT embeddings or using the CLS-token output from a BERT network achieved bad results for various STS tasks (Table TABREF6), worse than average GloVe embeddings. However, for SentEval, average BERT embeddings and the BERT CLS-token output achieves decent results (Table TABREF15), outperforming average GloVe embeddings. The reason for this are the different setups. For the STS tasks, we used cosine-similarity to estimate the similarities between sentence embeddings. Cosine-similarity treats all dimensions equally. In contrast, SentEval fits a logistic regression classifier to the sentence embeddings. This allows that certain dimensions can have higher or lower impact on the classification result.","We conclude that average BERT embeddings / CLS-token output from BERT return sentence embeddings that are infeasible to be used with cosine-similarity or with Manhatten / Euclidean distance. For transfer learning, they yield slightly worse results than InferSent or Universal Sentence Encoder. However, using the described fine-tuning setup with a siamese network structure on NLI datasets yields sentence embeddings that achieve a new state-of-the-art for the SentEval toolkit."]},{"idx":11,"heading":"Ablation Study","paragraphs":["We have demonstrated strong empirical results for the quality of SBERT sentence embeddings. In this section, we perform an ablation study of different aspects of SBERT in order to get a better understanding of their relative importance.","We evaluated different pooling strategies (MEAN, MAX, and CLS). For the classification objective function, we evaluate different concatenation methods. For each possible configuration, we train SBERT with 10 different random seeds and average the performances.","The objective function (classification vs. regression) depends on the annotated dataset. For the classification objective function, we train SBERT-base on the SNLI and the Multi-NLI dataset. For the regression objective function, we train on the training set of the STS benchmark dataset. Performances are measured on the development split of the STS benchmark dataset. Results are shown in Table TABREF23.","When trained with the classification objective function on NLI data, the pooling strategy has a rather minor impact. The impact of the concatenation mode is much larger. InferSent BIBREF4 and Universal Sentence Encoder BIBREF5 both use $(u, v, |u-v|, u*v)$ as input for a softmax classifier. However, in our architecture, adding the element-wise $u*v$ decreased the performance.","The most important component is the element-wise difference $|u-v|$. Note, that the concatenation mode is only relevant for training the softmax classifier. At inference, when predicting similarities for the STS benchmark dataset, only the sentence embeddings $u$ and $v$ are used in combination with cosine-similarity. The element-wise difference measures the distance between the dimensions of the two sentence embeddings, ensuring that similar pairs are closer and dissimilar pairs are further apart.","When trained with the regression objective function, we observe that the pooling strategy has a large impact. There, the MAX strategy perform significantly worse than MEAN or CLS-token strategy. This is in contrast to BIBREF4, who found it beneficial for the BiLSTM-layer of InferSent to use MAX instead of MEAN pooling."]},{"idx":12,"heading":"Computational Efficiency","paragraphs":["Sentence embeddings need potentially be computed for Millions of sentences, hence, a high computation speed is desired. In this section, we compare SBERT to average GloVe embeddings, InferSent BIBREF4, and Universal Sentence Encoder BIBREF5.","For our comparison we use the sentences from the STS benchmark BIBREF10. We compute average GloVe embeddings using a simple for-loop with python dictionary lookups and NumPy. InferSent is based on PyTorch. For Universal Sentence Encoder, we use the TensorFlow Hub version, which is based on TensorFlow. SBERT is based on PyTorch. For improved computation of sentence embeddings, we implemented a smart batching strategy: Sentences with similar lengths are grouped together and are only padded to the longest element in a mini-batch. This drastically reduces computational overhead from padding tokens.","Performances were measured on a server with Intel i7-5820K CPU @ 3.30GHz, Nvidia Tesla V100 GPU, CUDA 9.2 and cuDNN. The results are depicted in Table TABREF26.","On CPU, InferSent is about 65% faster than SBERT. This is due to the much simpler network architecture. InferSent uses a single BiLSTM layer, while BERT uses 12 stacked transformer layers. However, an advantage of transformer networks is the computational efficiency on GPUs. There, SBERT with smart batching is about 9% faster than InferSent and about 55% faster than Universal Sentence Encoder. Smart batching achieves a speed-up of 89% on CPU and 48% on GPU. Average GloVe embeddings is obviously by a large margin the fastest method to compute sentence embeddings."]},{"idx":13,"heading":"Conclusion","paragraphs":["We showed that BERT out-of-the-box maps sentences to a vector space that is rather unsuitable to be used with common similarity measures like cosine-similarity. The performance for seven STS tasks was below the performance of average GloVe embeddings.","To overcome this shortcoming, we presented Sentence-BERT (SBERT). SBERT fine-tunes BERT in a siamese / triplet network architecture. We evaluated the quality on various common benchmarks, where it could achieve a significant improvement over state-of-the-art sentence embeddings methods. Replacing BERT with RoBERTa did not yield a significant improvement in our experiments.","SBERT is computationally efficient. On a GPU, it is about 9% faster than InferSent and about 55% faster than Universal Sentence Encoder. SBERT can be used for tasks which are computationally not feasible to be modeled with BERT. For example, clustering of 10,000 sentences with hierarchical clustering requires with BERT about 65 hours, as around 50 Million sentence combinations must be computed. With SBERT, we were able to reduce the effort to about 5 seconds."]},{"idx":14,"heading":"Acknowledgments","paragraphs":["This work has been supported by the German Research Foundation through the German-Israeli Project Cooperation (DIP, grant DA 1600/1-1 and grant GU 798/17-1). It has been co-funded by the German Federal Ministry of Education and Research (BMBF) under the promotional references 03VP02540 (ArgumenText)."]},{"idx":15,"heading":"Figures and Tables","paragraphs":["Figure 1: SBERT architecture with classification objective function, e.g., for fine-tuning on SNLI dataset. The two BERT networks have tied weights (siamese network structure).","Figure 2: SBERT architecture at inference, for example, to compute similarity scores. This architecture is also used with the regression objective function.","Table 1: Spearman rank correlation ρ between the cosine similarity of sentence representations and the gold labels for various Textual Similarity (STS) tasks. Performance is reported by convention as ρ × 100. STS12-STS16: SemEval 2012-2016, STSb: STSbenchmark, SICK-R: SICK relatedness dataset.","Table 2: Evaluation on the STS benchmark test set. BERT systems were trained with 10 random seeds and 4 epochs. SBERT was fine-tuned on the STSb dataset, SBERT-NLI was pretrained on the NLI datasets, then fine-tuned on the STSb dataset.","Table 4: Evaluation on the Wikipedia section triplets dataset (Dor et al., 2018). SBERT trained with triplet loss for one epoch.","Table 3: Average Pearson correlation r and average Spearman’s rank correlation ρ on the Argument Facet Similarity (AFS) corpus (Misra et al., 2016). Misra et al. proposes 10-fold cross-validation. We additionally evaluate in a cross-topic scenario: Methods are trained on two topics, and are evaluated on the third topic.","Table 5: Evaluation of SBERT sentence embeddings using the SentEval toolkit. SentEval evaluates sentence embeddings on different sentence classification tasks by training a logistic regression classifier using the sentence embeddings as features. Scores are based on a 10-fold cross-validation.","Table 6: SBERT trained on NLI data with the classification objective function, on the STS benchmark (STSb) with the regression objective function. Configurations are evaluated on the development set of the STSb using cosine-similarity and Spearman’s rank correlation. For the concatenation methods, we only report scores with MEAN pooling strategy.","Table 7: Computation speed (sentences per second) of sentence embedding methods. Higher is better."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1909.00015","title":"Adaptively Sparse Transformers","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Attention mechanisms have become ubiquitous in NLP. Recent architectures, notably the Transformer, learn powerful context-aware word representations through layered, multi-headed attention. The multiple heads learn diverse types of word relationships. However, with standard softmax attention, all attention heads are dense, assigning a non-zero weight to all context words. In this work, we introduce the adaptively sparse Transformer, wherein attention heads have flexible, context-dependent sparsity patterns. This sparsity is accomplished by replacing softmax with $\\alpha$-entmax: a differentiable generalization of softmax that allows low-scoring words to receive precisely zero weight. Moreover, we derive a method to automatically learn the $\\alpha$ parameter -- which controls the shape and sparsity of $\\alpha$-entmax -- allowing attention heads to choose between focused or spread-out behavior. Our adaptively sparse Transformer improves interpretability and head diversity when compared to softmax Transformers on machine translation datasets. Findings of the quantitative and qualitative analysis of our approach include that heads in different layers learn different sparsity preferences and tend to be more diverse in their attention distributions than softmax Transformers. Furthermore, at no cost in accuracy, sparsity in attention heads helps to uncover different head specializations."]},{"idx":1,"heading":"Introduction","paragraphs":["The Transformer architecture BIBREF0 for deep neural networks has quickly risen to prominence in NLP through its efficiency and performance, leading to improvements in the state of the art of Neural Machine Translation BIBREF1, BIBREF2, as well as inspiring other powerful general-purpose models like BERT BIBREF3 and GPT-2 BIBREF4. At the heart of the Transformer lie multi-head attention mechanisms: each word is represented by multiple different weighted averages of its relevant context. As suggested by recent works on interpreting attention head roles, separate attention heads may learn to look for various relationships between tokens BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9.","The attention distribution of each head is predicted typically using the softmax normalizing transform. As a result, all context words have non-zero attention weight. Recent work on single attention architectures suggest that using sparse normalizing transforms in attention mechanisms such as sparsemax – which can yield exactly zero probabilities for irrelevant words – may improve performance and interpretability BIBREF12, BIBREF13, BIBREF14. Qualitative analysis of attention heads BIBREF0 suggests that, depending on what phenomena they capture, heads tend to favor flatter or more peaked distributions.","Recent works have proposed sparse Transformers BIBREF10 and adaptive span Transformers BIBREF11. However, the “sparsity\" of those models only limits the attention to a contiguous span of past tokens, while in this work we propose a highly adaptive Transformer model that is capable of attending to a sparse set of words that are not necessarily contiguous. Figure FIGREF1 shows the relationship of these methods with ours.","Our contributions are the following:","We introduce sparse attention into the Transformer architecture, showing that it eases interpretability and leads to slight accuracy gains.","We propose an adaptive version of sparse attention, where the shape of each attention head is learnable and can vary continuously and dynamically between the dense limit case of softmax and the sparse, piecewise-linear sparsemax case.","We make an extensive analysis of the added interpretability of these models, identifying both crisper examples of attention head behavior observed in previous work, as well as novel behaviors unraveled thanks to the sparsity and adaptivity of our proposed model."]},{"idx":2,"heading":"Background ::: The Transformer","paragraphs":["In NMT, the Transformer BIBREF0 is a sequence-to-sequence (seq2seq) model which maps an input sequence to an output sequence through hierarchical multi-head attention mechanisms, yielding a dynamic, context-dependent strategy for propagating information within and across sentences. It contrasts with previous seq2seq models, which usually rely either on costly gated recurrent operations BIBREF15, BIBREF16 or static convolutions BIBREF17.","Given $n$ query contexts and $m$ sequence items under consideration, attention mechanisms compute, for each query, a weighted representation of the items. The particular attention mechanism used in BIBREF0 is called scaled dot-product attention, and it is computed in the following way:","where $\\mathbf {Q} \\in \\mathbb {R}^{n \\times d}$ contains representations of the queries, $\\mathbf {K}, \\mathbf {V} \\in \\mathbb {R}^{m \\times d}$ are the keys and values of the items attended over, and $d$ is the dimensionality of these representations. The $\\mathbf {\\pi }$ mapping normalizes row-wise using softmax, $\\mathbf {\\pi }(\\mathbf {Z})_{ij} = \\operatornamewithlimits{\\mathsf {softmax}}(\\mathbf {z}_i)_j$, where","In words, the keys are used to compute a relevance score between each item and query. Then, normalized attention weights are computed using softmax, and these are used to weight the values of each item at each query context.","However, for complex tasks, different parts of a sequence may be relevant in different ways, motivating multi-head attention in Transformers. This is simply the application of Equation DISPLAY_FORM7 in parallel $H$ times, each with a different, learned linear transformation that allows specialization:","In the Transformer, there are three separate multi-head attention mechanisms for distinct purposes:","Encoder self-attention: builds rich, layered representations of each input word, by attending on the entire input sentence.","Context attention: selects a representative weighted average of the encodings of the input words, at each time step of the decoder.","Decoder self-attention: attends over the partial output sentence fragment produced so far.","Together, these mechanisms enable the contextualized flow of information between the input sentence and the sequential decoder."]},{"idx":3,"heading":"Background ::: Sparse Attention","paragraphs":["The softmax mapping (Equation DISPLAY_FORM8) is elementwise proportional to $\\exp $, therefore it can never assign a weight of exactly zero. Thus, unnecessary items are still taken into consideration to some extent. Since its output sums to one, this invariably means less weight is assigned to the relevant items, potentially harming performance and interpretability BIBREF18. This has motivated a line of research on learning networks with sparse mappings BIBREF19, BIBREF20, BIBREF21, BIBREF22. We focus on a recently-introduced flexible family of transformations, $\\alpha $-entmax BIBREF23, BIBREF14, defined as:","where $\\triangle ^d \\lbrace \\mathbf {p}\\in \\mathbb {R}^d:\\sum _{i} p_i = 1\\rbrace $ is the probability simplex, and, for $\\alpha \\ge 1$, $\\mathsf {H}^{\\textsc {T}}_\\alpha $ is the Tsallis continuous family of entropies BIBREF24:","This family contains the well-known Shannon and Gini entropies, corresponding to the cases $\\alpha =1$ and $\\alpha =2$, respectively.","Equation DISPLAY_FORM14 involves a convex optimization subproblem. Using the definition of $\\mathsf {H}^{\\textsc {T}}_\\alpha $, the optimality conditions may be used to derive the following form for the solution (Appendix SECREF83):","where $[\\cdot ]_+$ is the positive part (ReLU) function, $\\mathbf {1}$ denotes the vector of all ones, and $\\tau $ – which acts like a threshold – is the Lagrange multiplier corresponding to the $\\sum _i p_i=1$ constraint."]},{"idx":4,"heading":"Background ::: Sparse Attention ::: Properties of @!START@$\\alpha $@!END@-entmax.","paragraphs":["The appeal of $\\alpha $-entmax for attention rests on the following properties. For $\\alpha =1$ (i.e., when $\\mathsf {H}^{\\textsc {T}}_\\alpha $ becomes the Shannon entropy), it exactly recovers the softmax mapping (We provide a short derivation in Appendix SECREF89.). For all $\\alpha >1$ it permits sparse solutions, in stark contrast to softmax. In particular, for $\\alpha =2$, it recovers the sparsemax mapping BIBREF19, which is piecewise linear. In-between, as $\\alpha $ increases, the mapping continuously gets sparser as its curvature changes.","To compute the value of $\\alpha $-entmax, one must find the threshold $\\tau $ such that the r.h.s. in Equation DISPLAY_FORM16 sums to one. BIBREF23 propose a general bisection algorithm. BIBREF14 introduce a faster, exact algorithm for $\\alpha =1.5$, and enable using $\\mathop {\\mathsf {\\alpha }\\textnormal {-}\\mathsf {entmax }}$ with fixed $\\alpha $ within a neural network by showing that the $\\alpha $-entmax Jacobian w.r.t. $\\mathbf {z}$ for $\\mathbf {p}^\\star = \\mathop {\\mathsf {\\alpha }\\textnormal {-}\\mathsf {entmax }}(\\mathbf {z})$ is","Our work furthers the study of $\\alpha $-entmax by providing a derivation of the Jacobian w.r.t. the hyper-parameter $\\alpha $ (Section SECREF3), thereby allowing the shape and sparsity of the mapping to be learned automatically. This is particularly appealing in the context of multi-head attention mechanisms, where we shall show in Section SECREF35 that different heads tend to learn different sparsity behaviors."]},{"idx":5,"heading":"Adaptively Sparse Transformers with @!START@$\\alpha $@!END@-entmax","paragraphs":["We now propose a novel Transformer architecture wherein we simply replace softmax with $\\alpha $-entmax in the attention heads. Concretely, we replace the row normalization $\\mathbf {\\pi }$ in Equation DISPLAY_FORM7 by","This change leads to sparse attention weights, as long as $\\alpha >1$; in particular, $\\alpha =1.5$ is a sensible starting point BIBREF14."]},{"idx":6,"heading":"Adaptively Sparse Transformers with @!START@$\\alpha $@!END@-entmax ::: Different @!START@$\\alpha $@!END@ per head.","paragraphs":["Unlike LSTM-based seq2seq models, where $\\alpha $ can be more easily tuned by grid search, in a Transformer, there are many attention heads in multiple layers. Crucial to the power of such models, the different heads capture different linguistic phenomena, some of them isolating important words, others spreading out attention across phrases BIBREF0. This motivates using different, adaptive $\\alpha $ values for each attention head, such that some heads may learn to be sparser, and others may become closer to softmax. We propose doing so by treating the $\\alpha $ values as neural network parameters, optimized via stochastic gradients along with the other weights."]},{"idx":7,"heading":"Adaptively Sparse Transformers with @!START@$\\alpha $@!END@-entmax ::: Derivatives w.r.t. @!START@$\\alpha $@!END@.","paragraphs":["In order to optimize $\\alpha $ automatically via gradient methods, we must compute the Jacobian of the entmax output w.r.t. $\\alpha $. Since entmax is defined through an optimization problem, this is non-trivial and cannot be simply handled through automatic differentiation; it falls within the domain of argmin differentiation, an active research topic in optimization BIBREF25, BIBREF26.","One of our key contributions is the derivation of a closed-form expression for this Jacobian. The next proposition provides such an expression, enabling entmax layers with adaptive $\\alpha $. To the best of our knowledge, ours is the first neural network module that can automatically, continuously vary in shape away from softmax and toward sparse mappings like sparsemax.","Proposition 1 Let $\\mathbf {p}^\\star \\mathop {\\mathsf {\\alpha }\\textnormal {-}\\mathsf {entmax }}(\\mathbf {z})$ be the solution of Equation DISPLAY_FORM14. Denote the distribution $\\tilde{p}_i {(p_i^\\star )^{2 - \\alpha }}{ \\sum _j(p_j^\\star )^{2-\\alpha }}$ and let $h_i -p^\\star _i \\log p^\\star _i$. The $i$th component of the Jacobian $\\mathbf {g} \\frac{\\partial \\mathop {\\mathsf {\\alpha }\\textnormal {-}\\mathsf {entmax }}(\\mathbf {z})}{\\partial \\alpha }$ is","proof uses implicit function differentiation and is given in Appendix SECREF10.","Proposition UNKREF22 provides the remaining missing piece needed for training adaptively sparse Transformers. In the following section, we evaluate this strategy on neural machine translation, and analyze the behavior of the learned attention heads."]},{"idx":8,"heading":"Experiments","paragraphs":["We apply our adaptively sparse Transformers on four machine translation tasks. For comparison, a natural baseline is the standard Transformer architecture using the softmax transform in its multi-head attention mechanisms. We consider two other model variants in our experiments that make use of different normalizing transformations:","1.5-entmax: a Transformer with sparse entmax attention with fixed $\\alpha =1.5$ for all heads. This is a novel model, since 1.5-entmax had only been proposed for RNN-based NMT models BIBREF14, but never in Transformers, where attention modules are not just one single component of the seq2seq model but rather an integral part of all of the model components.","$\\alpha $-entmax: an adaptive Transformer with sparse entmax attention with a different, learned $\\alpha _{i,j}^t$ for each head.","The adaptive model has an additional scalar parameter per attention head per layer for each of the three attention mechanisms (encoder self-attention, context attention, and decoder self-attention), i.e.,","and we set $\\alpha _{i,j}^t = 1 + \\operatornamewithlimits{\\mathsf {sigmoid}}(a_{i,j}^t) \\in ]1, 2[$. All or some of the $\\alpha $ values can be tied if desired, but we keep them independent for analysis purposes."]},{"idx":9,"heading":"Experiments ::: Datasets.","paragraphs":["Our models were trained on 4 machine translation datasets of different training sizes:","[itemsep=.5ex,leftmargin=2ex]","IWSLT 2017 German $\\rightarrow $ English BIBREF27: 200K sentence pairs.","KFTT Japanese $\\rightarrow $ English BIBREF28: 300K sentence pairs.","WMT 2016 Romanian $\\rightarrow $ English BIBREF29: 600K sentence pairs.","WMT 2014 English $\\rightarrow $ German BIBREF30: 4.5M sentence pairs.","All of these datasets were preprocessed with byte-pair encoding BIBREF31, using joint segmentations of 32k merge operations."]},{"idx":10,"heading":"Experiments ::: Training.","paragraphs":["We follow the dimensions of the Transformer-Base model of BIBREF0: The number of layers is $L=6$ and number of heads is $H=8$ in the encoder self-attention, the context attention, and the decoder self-attention. We use a mini-batch size of 8192 tokens and warm up the learning rate linearly until 20k steps, after which it decays according to an inverse square root schedule. All models were trained until convergence of validation accuracy, and evaluation was done at each 10k steps for ro$\\rightarrow $en and en$\\rightarrow $de and at each 5k steps for de$\\rightarrow $en and ja$\\rightarrow $en. The end-to-end computational overhead of our methods, when compared to standard softmax, is relatively small; in training tokens per second, the models using $\\alpha $-entmax and $1.5$-entmax are, respectively, $75\\%$ and $90\\%$ the speed of the softmax model."]},{"idx":11,"heading":"Experiments ::: Results.","paragraphs":["We report test set tokenized BLEU BIBREF32 results in Table TABREF27. We can see that replacing softmax by entmax does not hurt performance in any of the datasets; indeed, sparse attention Transformers tend to have slightly higher BLEU, but their sparsity leads to a better potential for analysis. In the next section, we make use of this potential by exploring the learned internal mechanics of the self-attention heads."]},{"idx":12,"heading":"Analysis","paragraphs":["We conduct an analysis for the higher-resource dataset WMT 2014 English $\\rightarrow $ German of the attention in the sparse adaptive Transformer model ($\\alpha $-entmax) at multiple levels: we analyze high-level statistics as well as individual head behavior. Moreover, we make a qualitative analysis of the interpretability capabilities of our models."]},{"idx":13,"heading":"Analysis ::: High-Level Statistics ::: What kind of @!START@$\\alpha $@!END@ values are learned?","paragraphs":["Figure FIGREF37 shows the learning trajectories of the $\\alpha $ parameters of a selected subset of heads. We generally observe a tendency for the randomly-initialized $\\alpha $ parameters to decrease initially, suggesting that softmax-like behavior may be preferable while the model is still very uncertain. After around one thousand steps, some heads change direction and become sparser, perhaps as they become more confident and specialized. This shows that the initialization of $\\alpha $ does not predetermine its sparsity level or the role the head will have throughout. In particular, head 8 in the encoder self-attention layer 2 first drops to around $\\alpha =1.3$ before becoming one of the sparsest heads, with $\\alpha \\approx 2$.","The overall distribution of $\\alpha $ values at convergence can be seen in Figure FIGREF38. We can observe that the encoder self-attention blocks learn to concentrate the $\\alpha $ values in two modes: a very sparse one around $\\alpha \\rightarrow 2$, and a dense one between softmax and 1.5-entmax . However, the decoder self and context attention only learn to distribute these parameters in a single mode. We show next that this is reflected in the average density of attention weight vectors as well."]},{"idx":14,"heading":"Analysis ::: High-Level Statistics ::: Attention weight density when translating.","paragraphs":["For any $\\alpha >1$, it would still be possible for the weight matrices in Equation DISPLAY_FORM9 to learn re-scalings so as to make attention sparser or denser. To visualize the impact of adaptive $\\alpha $ values, we compare the empirical attention weight density (the average number of tokens receiving non-zero attention) within each module, against sparse Transformers with fixed $\\alpha =1.5$.","Figure FIGREF40 shows that, with fixed $\\alpha =1.5$, heads tend to be sparse and similarly-distributed in all three attention modules. With learned $\\alpha $, there are two notable changes: (i) a prominent mode corresponding to fully dense probabilities, showing that our models learn to combine sparse and dense attention, and (ii) a distinction between the encoder self-attention – whose background distribution tends toward extreme sparsity – and the other two modules, who exhibit more uniform background distributions. This suggests that perhaps entirely sparse Transformers are suboptimal.","The fact that the decoder seems to prefer denser attention distributions might be attributed to it being auto-regressive, only having access to past tokens and not the full sentence. We speculate that it might lose too much information if it assigned weights of zero to too many tokens in the self-attention, since there are fewer tokens to attend to in the first place.","Teasing this down into separate layers, Figure FIGREF41 shows the average (sorted) density of each head for each layer. We observe that $\\alpha $-entmax is able to learn different sparsity patterns at each layer, leading to more variance in individual head behavior, to clearly-identified dense and sparse heads, and overall to different tendencies compared to the fixed case of $\\alpha =1.5$."]},{"idx":15,"heading":"Analysis ::: High-Level Statistics ::: Head diversity.","paragraphs":["To measure the overall disagreement between attention heads, as a measure of head diversity, we use the following generalization of the Jensen-Shannon divergence:","where $\\mathbf {p}_j$ is the vector of attention weights assigned by head $j$ to each word in the sequence, and $\\mathsf {H}^\\textsc {S}$ is the Shannon entropy, base-adjusted based on the dimension of $\\mathbf {p}$ such that $JS \\le 1$. We average this measure over the entire validation set. The higher this metric is, the more the heads are taking different roles in the model.","Figure FIGREF44 shows that both sparse Transformer variants show more diversity than the traditional softmax one. Interestingly, diversity seems to peak in the middle layers of the encoder self-attention and context attention, while this is not the case for the decoder self-attention.","The statistics shown in this section can be found for the other language pairs in Appendix SECREF8."]},{"idx":16,"heading":"Analysis ::: Identifying Head Specializations","paragraphs":["Previous work pointed out some specific roles played by different heads in the softmax Transformer model BIBREF33, BIBREF5, BIBREF9. Identifying the specialization of a head can be done by observing the type of tokens or sequences that the head often assigns most of its attention weight; this is facilitated by sparsity."]},{"idx":17,"heading":"Analysis ::: Identifying Head Specializations ::: Positional heads.","paragraphs":["One particular type of head, as noted by BIBREF9, is the positional head. These heads tend to focus their attention on either the previous or next token in the sequence, thus obtaining representations of the neighborhood of the current time step. In Figure FIGREF47, we show attention plots for such heads, found for each of the studied models. The sparsity of our models allows these heads to be more confident in their representations, by assigning the whole probability distribution to a single token in the sequence. Concretely, we may measure a positional head's confidence as the average attention weight assigned to the previous token. The softmax model has three heads for position $-1$, with median confidence $93.5\\%$. The $1.5$-entmax model also has three heads for this position, with median confidence $94.4\\%$. The adaptive model has four heads, with median confidences $95.9\\%$, the lowest-confidence head being dense with $\\alpha =1.18$, while the highest-confidence head being sparse ($\\alpha =1.91$).","For position $+1$, the models each dedicate one head, with confidence around $95\\%$, slightly higher for entmax. The adaptive model sets $\\alpha =1.96$ for this head."]},{"idx":18,"heading":"Analysis ::: Identifying Head Specializations ::: BPE-merging head.","paragraphs":["Due to the sparsity of our models, we are able to identify other head specializations, easily identifying which heads should be further analysed. In Figure FIGREF51 we show one such head where the $\\alpha $ value is particularly high (in the encoder, layer 1, head 4 depicted in Figure FIGREF37). We found that this head most often looks at the current time step with high confidence, making it a positional head with offset 0. However, this head often spreads weight sparsely over 2-3 neighboring tokens, when the tokens are part of the same BPE cluster or hyphenated words. As this head is in the first layer, it provides a useful service to the higher layers by combining information evenly within some BPE clusters.","For each BPE cluster or cluster of hyphenated words, we computed a score between 0 and 1 that corresponds to the maximum attention mass assigned by any token to the rest of the tokens inside the cluster in order to quantify the BPE-merging capabilities of these heads. There are not any attention heads in the softmax model that are able to obtain a score over $80\\%$, while for $1.5$-entmax and $\\alpha $-entmax there are two heads in each ($83.3\\%$ and $85.6\\%$ for $1.5$-entmax and $88.5\\%$ and $89.8\\%$ for $\\alpha $-entmax)."]},{"idx":19,"heading":"Analysis ::: Identifying Head Specializations ::: Interrogation head.","paragraphs":["On the other hand, in Figure FIGREF52 we show a head for which our adaptively sparse model chose an $\\alpha $ close to 1, making it closer to softmax (also shown in encoder, layer 1, head 3 depicted in Figure FIGREF37). We observe that this head assigns a high probability to question marks at the end of the sentence in time steps where the current token is interrogative, thus making it an interrogation-detecting head. We also observe this type of heads in the other models, which we also depict in Figure FIGREF52. The average attention weight placed on the question mark when the current token is an interrogative word is $98.5\\%$ for softmax, $97.0\\%$ for $1.5$-entmax, and $99.5\\%$ for $\\alpha $-entmax.","Furthermore, we can examine sentences where some tendentially sparse heads become less so, thus identifying sources of ambiguity where the head is less confident in its prediction. An example is shown in Figure FIGREF55 where sparsity in the same head differs for sentences of similar length."]},{"idx":20,"heading":"Related Work ::: Sparse attention.","paragraphs":["Prior work has developed sparse attention mechanisms, including applications to NMT BIBREF19, BIBREF12, BIBREF20, BIBREF22, BIBREF34. BIBREF14 introduced the entmax function this work builds upon. In their work, there is a single attention mechanism which is controlled by a fixed $\\alpha $. In contrast, this is the first work to allow such attention mappings to dynamically adapt their curvature and sparsity, by automatically adjusting the continuous $\\alpha $ parameter. We also provide the first results using sparse attention in a Transformer model."]},{"idx":21,"heading":"Related Work ::: Fixed sparsity patterns.","paragraphs":["Recent research improves the scalability of Transformer-like networks through static, fixed sparsity patterns BIBREF10, BIBREF35. Our adaptively-sparse Transformer can dynamically select a sparsity pattern that finds relevant words regardless of their position (e.g., Figure FIGREF52). Moreover, the two strategies could be combined. In a concurrent line of research, BIBREF11 propose an adaptive attention span for Transformer language models. While their work has each head learn a different contiguous span of context tokens to attend to, our work finds different sparsity patterns in the same span. Interestingly, some of their findings mirror ours – we found that attention heads in the last layers tend to be denser on average when compared to the ones in the first layers, while their work has found that lower layers tend to have a shorter attention span compared to higher layers."]},{"idx":22,"heading":"Related Work ::: Transformer interpretability.","paragraphs":["The original Transformer paper BIBREF0 shows attention visualizations, from which some speculation can be made of the roles the several attention heads have. BIBREF7 study the syntactic abilities of the Transformer self-attention, while BIBREF6 extract dependency relations from the attention weights. BIBREF8 find that the self-attentions in BERT BIBREF3 follow a sequence of processes that resembles a classical NLP pipeline. Regarding redundancy of heads, BIBREF9 develop a method that is able to prune heads of the multi-head attention module and make an empirical study of the role that each head has in self-attention (positional, syntactic and rare words). BIBREF36 also aim to reduce head redundancy by adding a regularization term to the loss that maximizes head disagreement and obtain improved results. While not considering Transformer attentions, BIBREF18 show that traditional attention mechanisms do not necessarily improve interpretability since softmax attention is vulnerable to an adversarial attack leading to wildly different model predictions for the same attention weights. Sparse attention may mitigate these issues; however, our work focuses mostly on a more mechanical aspect of interpretation by analyzing head behavior, rather than on explanations for predictions."]},{"idx":23,"heading":"Conclusion and Future Work","paragraphs":["We contribute a novel strategy for adaptively sparse attention, and, in particular, for adaptively sparse Transformers. We present the first empirical analysis of Transformers with sparse attention mappings (i.e., entmax), showing potential in both translation accuracy as well as in model interpretability.","In particular, we analyzed how the attention heads in the proposed adaptively sparse Transformer can specialize more and with higher confidence. Our adaptivity strategy relies only on gradient-based optimization, side-stepping costly per-head hyper-parameter searches. Further speed-ups are possible by leveraging more parallelism in the bisection algorithm for computing $\\alpha $-entmax.","Finally, some of the automatically-learned behaviors of our adaptively sparse Transformers – for instance, the near-deterministic positional heads or the subword joining head – may provide new ideas for designing static variations of the Transformer."]},{"idx":24,"heading":"Acknowledgments","paragraphs":["This work was supported by the European Research Council (ERC StG DeepSPIN 758969), and by the Fundação para a Ciência e Tecnologia through contracts UID/EEA/50008/2019 and CMUPERI/TIC/0046/2014 (GoLocal). We are grateful to Ben Peters for the $\\alpha $-entmax code and Erick Fonseca, Marcos Treviso, Pedro Martins, and Tsvetomila Mihaylova for insightful group discussion. We thank Mathieu Blondel for the idea to learn $\\alpha $. We would also like to thank the anonymous reviewers for their helpful feedback.","Supplementary Material"]},{"idx":25,"heading":"Background ::: Regularized Fenchel-Young prediction functions","paragraphs":["Definition 1 (BIBREF23)","Let $\\Omega \\colon \\triangle ^d \\rightarrow {\\mathbb {R}}\\cup \\lbrace \\infty \\rbrace $ be a strictly convex regularization function. We define the prediction function $\\mathbf {\\pi }_{\\Omega }$ as"]},{"idx":26,"heading":"Background ::: Characterizing the @!START@$\\alpha $@!END@-entmax mapping","paragraphs":["Lemma 1 (BIBREF14) For any $\\mathbf {z}$, there exists a unique $\\tau ^\\star $ such that","Proof: From the definition of $\\mathop {\\mathsf {\\alpha }\\textnormal {-}\\mathsf {entmax }}$,","we may easily identify it with a regularized prediction function (Def. UNKREF81):","We first note that for all $\\mathbf {p}\\in \\triangle ^d$,","From the constant invariance and scaling properties of $\\mathbf {\\pi }_{\\Omega }$ BIBREF23,","Using BIBREF23, noting that $g^{\\prime }(t) = t^{\\alpha - 1}$ and $(g^{\\prime })^{-1}(u) = u^{{1}{\\alpha -1}}$, yields","Since $\\mathsf {H}^{\\textsc {T}}_\\alpha $ is strictly convex on the simplex, $\\mathop {\\mathsf {\\alpha }\\textnormal {-}\\mathsf {entmax }}$ has a unique solution $\\mathbf {p}^\\star $. Equation DISPLAY_FORM88 implicitly defines a one-to-one mapping between $\\mathbf {p}^\\star $ and $\\tau ^\\star $ as long as $\\mathbf {p}^\\star \\in \\triangle $, therefore $\\tau ^\\star $ is also unique."]},{"idx":27,"heading":"Background ::: Connections to softmax and sparsemax","paragraphs":["The Euclidean projection onto the simplex, sometimes referred to, in the context of neural attention, as sparsemax BIBREF19, is defined as","The solution can be characterized through the unique threshold $\\tau $ such that $\\sum _i \\operatornamewithlimits{\\mathsf {sparsemax}}(\\mathbf {z})_i = 1$ and BIBREF38","Thus, each coordinate of the sparsemax solution is a piecewise-linear function. Visibly, this expression is recovered when setting $\\alpha =2$ in the $\\alpha $-entmax expression (Equation DISPLAY_FORM85); for other values of $\\alpha $, the exponent induces curvature.","On the other hand, the well-known softmax is usually defined through the expression","which can be shown to be the unique solution of the optimization problem","where $\\mathsf {H}^\\textsc {S}(\\mathbf {p}) -\\sum _i p_i \\log p_i$ is the Shannon entropy. Indeed, setting the gradient to 0 yields the condition $\\log p_i = z_j - \\nu _i - \\tau - 1$, where $\\tau $ and $\\nu > 0$ are Lagrange multipliers for the simplex constraints $\\sum _i p_i = 1$ and $p_i \\ge 0$, respectively. Since the l.h.s. is only finite for $p_i>0$, we must have $\\nu _i=0$ for all $i$, by complementary slackness. Thus, the solution must have the form $p_i = {\\exp (z_i)}{Z}$, yielding Equation DISPLAY_FORM92."]},{"idx":28,"heading":"Jacobian of @!START@$\\alpha $@!END@-entmax w.r.t. the shape parameter @!START@$\\alpha $@!END@: Proof of Proposition @!START@UID22@!END@","paragraphs":["Recall that the entmax transformation is defined as:","where $\\alpha \\ge 1$ and $\\mathsf {H}^{\\textsc {T}}_{\\alpha }$ is the Tsallis entropy,","and $\\mathsf {H}^\\textsc {S}(\\mathbf {p}):= -\\sum _j p_j \\log p_j$ is the Shannon entropy.","In this section, we derive the Jacobian of $\\operatornamewithlimits{\\mathsf {entmax }}$ with respect to the scalar parameter $\\alpha $."]},{"idx":29,"heading":"Jacobian of @!START@$\\alpha $@!END@-entmax w.r.t. the shape parameter @!START@$\\alpha $@!END@: Proof of Proposition @!START@UID22@!END@ ::: General case of @!START@$\\alpha >1$@!END@","paragraphs":["From the KKT conditions associated with the optimization problem in Eq. DISPLAY_FORM85, we have that the solution $\\mathbf {p}^{\\star }$ has the following form, coordinate-wise:","where $\\tau ^{\\star }$ is a scalar Lagrange multiplier that ensures that $\\mathbf {p}^{\\star }$ normalizes to 1, i.e., it is defined implicitly by the condition:","For general values of $\\alpha $, Eq. DISPLAY_FORM98 lacks a closed form solution. This makes the computation of the Jacobian","non-trivial. Fortunately, we can use the technique of implicit differentiation to obtain this Jacobian.","The Jacobian exists almost everywhere, and the expressions we derive expressions yield a generalized Jacobian BIBREF37 at any non-differentiable points that may occur for certain ($\\alpha $, $\\mathbf {z}$) pairs. We begin by noting that $\\frac{\\partial p_i^{\\star }}{\\partial \\alpha } = 0$ if $p_i^{\\star } = 0$, because increasing $\\alpha $ keeps sparse coordinates sparse. Therefore we need to worry only about coordinates that are in the support of $\\mathbf {p}^\\star $. We will assume hereafter that the $i$th coordinate of $\\mathbf {p}^\\star $ is non-zero. We have:","We can see that this Jacobian depends on $\\frac{\\partial \\tau ^{\\star }}{\\partial \\alpha }$, which we now compute using implicit differentiation.","Let $\\mathcal {S} = \\lbrace i: p^\\star _i > 0 \\rbrace $). By differentiating both sides of Eq. DISPLAY_FORM98, re-using some of the steps in Eq. DISPLAY_FORM101, and recalling Eq. DISPLAY_FORM97, we get","from which we obtain:","Finally, plugging Eq. DISPLAY_FORM103 into Eq. DISPLAY_FORM101, we get:","where we denote by","The distribution $\\tilde{\\mathbf {p}}(\\alpha )$ can be interpreted as a “skewed” distribution obtained from $\\mathbf {p}^{\\star }$, which appears in the Jacobian of $\\mathop {\\mathsf {\\alpha }\\textnormal {-}\\mathsf {entmax }}(\\mathbf {z})$ w.r.t. $\\mathbf {z}$ as well BIBREF14."]},{"idx":30,"heading":"Jacobian of @!START@$\\alpha $@!END@-entmax w.r.t. the shape parameter @!START@$\\alpha $@!END@: Proof of Proposition @!START@UID22@!END@ ::: Solving the indetermination for @!START@$\\alpha =1$@!END@","paragraphs":["We can write Eq. DISPLAY_FORM104 as","When $\\alpha \\rightarrow 1^+$, we have $\\tilde{\\mathbf {p}}(\\alpha ) \\rightarrow \\mathbf {p}^{\\star }$, which leads to a $\\frac{0}{0}$ indetermination.","To solve this indetermination, we will need to apply L'Hôpital's rule twice. Let us first compute the derivative of $\\tilde{p}_i(\\alpha )$ with respect to $\\alpha $. We have","therefore","Differentiating the numerator and denominator in Eq. DISPLAY_FORM107, we get:","with","and","When $\\alpha \\rightarrow 1^+$, $B$ becomes again a $\\frac{0}{0}$ indetermination, which we can solve by applying again L'Hôpital's rule. Differentiating the numerator and denominator in Eq. DISPLAY_FORM112:","Finally, summing Eq. DISPLAY_FORM111 and Eq. DISPLAY_FORM113, we get"]},{"idx":31,"heading":"Jacobian of @!START@$\\alpha $@!END@-entmax w.r.t. the shape parameter @!START@$\\alpha $@!END@: Proof of Proposition @!START@UID22@!END@ ::: Summary","paragraphs":["To sum up, we have the following expression for the Jacobian of $\\mathop {\\mathsf {\\alpha }\\textnormal {-}\\mathsf {entmax }}$ with respect to $\\alpha $:"]},{"idx":32,"heading":"Figures and Tables","paragraphs":["Figure 1: Attention distributions of different selfattention heads for the time step of the token “over”, shown to compare our model to other related work. While the sparse Transformer (Child et al., 2019) and the adaptive span Transformer (Sukhbaatar et al., 2019) only attend to words within a contiguous span of the past tokens, our model is not only able to obtain different and not necessarily contiguous sparsity patterns for each attention head, but is also able to tune its support over which tokens to attend adaptively.","Table 1: Machine translation tokenized BLEU test results on IWSLT 2017 DE EN, KFTT JA EN, WMT 2016 RO EN and WMT 2014 EN DE, respectively.","Figure 2: Trajectories of α values for a subset of the heads during training. Initialized at random, most heads become denser in the beginning, before converging. This suggests that dense attention may be more beneficial while the network is still uncertain, being replaced by sparse attention afterwards.","Figure 4: Distribution of attention densities (average number of tokens receiving non-zero attention weight) for all attention heads and all validation sentences. When compared to 1.5-entmax, α-entmax distributes the sparsity in a more uniform manner, with a clear mode at fully dense attentions, corresponding to the heads with low α. In the softmax case, this distribution would lead to a single bar with density 1.","Figure 3: Distribution of learned α values per attention block. While the encoder self-attention has a bimodal distribution of values of α, the decoder self-attention and context attention have a single mode.","Figure 5: Head density per layer for fixed and learned α. Each line corresponds to an attention head; lower values mean that that attention head is sparser. Learned α has higher variance.","Figure 6: Jensen-Shannon Divergence between heads at each layer. Measures the disagreement between heads: the higher the value, the more the heads are disagreeing with each other in terms of where to attend. Models using sparse entmax have more diverse attention than the softmax baseline.","Figure 9: Interrogation-detecting heads in the three models. The top sentence is interrogative while the bottom one is declarative but includes the interrogative word “what”. In the top example, these interrogation heads assign a high probability to the question mark in the time step of the interrogative word (with ≥ 97.0% probability), while in the bottom example since there is no question mark, the same head does not assign a high probability to the last token in the sentence during the interrogative word time step. Surprisingly, this head prefers a low α = 1.05, as can be seen from the dense weights. This allows the head to identify the noun phrase “Armani Polo” better.","Figure 7: Self-attention from the most confidently previous-position head in each model. The learned parameter in the α-entmax model is α = 1.91. Quantitatively more confident, visual inspection confirms that the adaptive head behaves more consistently.","Figure 8: BPE-merging head (α = 1.91) discovered in the α-entmax model. Found in the first encoder layer, this head learns to discover some subword units and combine their information, leaving most words intact. It places 99.09% of its probability mass within the same BPE cluster as the current token: more than any head in any other model.","Figure 10: Example of two sentences of similar length where the same head (α = 1.33) exhibits different sparsity. The longer phrase in the example on the right “a sexually transmitted disease” is handled with higher confidence, leading to more sparsity.","Figure 11: Histograms of α values.","Figure 12: Histograms of head densities.","Figure 13: Jensen-Shannon divergence over layers.","Figure 14: Head densities over layers."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1909.01860","title":"Visual Question Answering using Deep Learning: A Survey and Performance Analysis","sections":[{"idx":0,"heading":"Abstract","paragraphs":["The Visual Question Answering (VQA) task combines challenges for processing data with both Visual and Linguistic processing, to answer basic `common sense' questions about given images. Given an image and a question in natural language, the VQA system tries to find the correct answer to it using visual elements of the image and inference gathered from textual questions. In this survey, we cover and discuss the recent datasets released in the VQA domain dealing with various types of question-formats and enabling robustness of the machine-learning models. Next, we discuss about new deep learning models that have shown promising results over the VQA datasets. At the end, we present and discuss some of the results computed by us over the vanilla VQA models, Stacked Attention Network and the VQA Challenge 2017 winner model. We also provide the detailed analysis along with the challenges and future research directions."]},{"idx":1,"heading":"Introduction","paragraphs":["Visual Question Answering (VQA) refers to a challenging task which lies at the intersection of image understanding and language processing. The VQA task has witnessed a significant progress in the recent years by the machine intelligence community. The aim of VQA is to develop a system to answer specific questions about an input image. The answer could be in any of the following forms: a word, a phrase, binary answer, multiple choice answer, or a fill in the blank answer. Agarwal et al. BIBREF0 presented a novel way of combining computer vision and natural language processing concepts of to achieve Visual Grounded Dialogue, a system mimicking the human understanding of the environment with the use of visual observation and language understanding.","The advancements in the field of deep learning have certainly helped to develop systems for the task of Image Question Answering. Krizhevsky et al BIBREF1 proposed the AlexNet model, which created a revolution in the computer vision domain. The paper introduced the concept of Convolution Neural Networks (CNN) to the mainstream computer vision application. Later many authors have worked on CNN, which has resulted in robust, deep learning models like VGGNet BIBREF2, Inception BIBREF3, ResNet BIBREF4, and etc. Similarly, the recent advancements in natural language processing area based on deep learning have improved the text understanding prforance as well. The first major algorithm in the context of text processing is considered to be the Recurrent Neural Networks (RNN) BIBREF5 which introduced the concept of prior context for time series based data. This architecture helped the growth of machine text understanding which gave new boundaries to machine translation, text classification and contextual understanding. Another major breakthrough in the domain was the introduction of Long-Short Term Memory (LSTM) architecture BIBREF6 which improvised over the RNN by introducing a context cell which stores the prior relevant information.","The vanilla VQA model BIBREF0 used a combination of VGGNet BIBREF2 and LSTM BIBREF6. This model has been revised over the years, employing newer architectures and mathematical formulations. Along with this, many authors have worked on producing datasets for eliminating bias, strengthening the performance of the model by robust question-answer pairs which try to cover the various types of questions, testing the visual and language understanding of the system. In this survey, first we cover major datasets published for validating the Visual Question Answering task, such as VQA dataset BIBREF0, DAQUAR BIBREF7, Visual7W BIBREF8 and most recent datasets up to 2019 include Tally-QA BIBREF9 and KVQA BIBREF10. Next, we discuss the state-of-the-art architectures designed for the task of Visual Question Answering such as Vanilla VQA BIBREF0, Stacked Attention Networks BIBREF11 and Pythia v1.0 BIBREF12. Next we present some of our computed results over the three architectures: vanilla VQA model BIBREF0, Stacked Attention Network (SAN) BIBREF11 and Teney et al. model BIBREF13. Finally, we discuss the observations and future directions."]},{"idx":2,"heading":"Datasets","paragraphs":["The major VQA datasets are summarized in Table TABREF2. We present the datasets below.","DAQUAR: DAQUAR stands for Dataset for Question Answering on Real World Images, released by Malinowski et al. BIBREF7. It is the first dataset released for the IQA task. The images are taken from NYU-Depth V2 dataset BIBREF17. The dataset is small with a total of 1449 images. The question bank includes 12468 question-answer pairs with 2483 unique questions. The questions have been generated by human annotations and confined within 9 question templates using annotations of the NYU-Depth dataset.","VQA Dataset: The Visual Question Answering (VQA) dataset BIBREF0 is one of the largest datasets collected from the MS-COCO BIBREF18 dataset. The VQA dataset contains at least 3 questions per image with 10 answers per question. The dataset contains 614,163 questions in the form of open-ended and multiple choice. In multiple choice questions, the answers can be classified as: 1) Correct Answer, 2) Plausible Answer, 3) Popular Answers and 4) Random Answers. Recently, VQA V2 dataset BIBREF0 is released with additional confusing images. The VQA sample images and questions are shown in Fig. SECREF2 in 1st row and 1st column.","Visual Madlibs: The Visual Madlibs dataset BIBREF15 presents a different form of template for the Image Question Answering task. One of the forms is the fill in the blanks type, where the system needs to supplement the words to complete the sentence and it mostly targets people, objects, appearances, activities and interactions. The Visual Madlibs samples are shown in Fig. SECREF2 in 1st row and 2nd column.","Visual7W: The Visual7W dataset BIBREF8 is also based on the MS-COCO dataset. It contains 47,300 COCO images with 327,939 question-answer pairs. The dataset also consists of 1,311,756 multiple choice questions and answers with 561,459 groundings. The dataset mainly deals with seven forms of questions (from where it derives its name): What, Where, When, Who, Why, How, and Which. It is majorly formed by two types of questions. The ‘telling’ questions are the ones which are text-based, giving a sort of description. The ‘pointing’ questions are the ones that begin with ‘Which,’ and have to be correctly identified by the bounding boxes among the group of plausible answers.","CLEVR: CLEVR BIBREF16 is a synthetic dataset to test the visual understanding of the VQA systems. The dataset is generated using three objects in each image, namely cylinder, sphere and cube. These objects are in two different sizes, two different materials and placed in eight different colors. The questions are also synthetically generated based on the objects placed in the image. The dataset also accompanies the ground-truth bounding boxes for each object in the image.","Tally-QA: Very recently, in 2019, the Tally-QA BIBREF9 dataset is proposed which is the largest dataset of object counting in the open-ended task. The dataset includes both simple and complex question types which can be seen in Fig. SECREF2. The dataset is quite large in numbers as well as it is 2.5 times the VQA dataset. The dataset contains 287,907 questions, 165,000 images and 19,000 complex questions. The Tally-QA samples are shown in Fig. SECREF2 in 2nd row and 1st column.","KVQA: The recent interest in common-sense questions has led to the development of world Knowledge based VQA dataset BIBREF10. The dataset contains questions targeting various categories of nouns and also require world knowledge to arrive at a solution. Questions in this dataset require multi-entity, multi-relation, and multi- hop reasoning over large Knowledge Graphs (KG) to arrive at an answer. The dataset contains 24,000 images with 183,100 question-answer pairs employing around 18K proper nouns. The KVQA samples are shown in Fig. SECREF2 in 2nd row and 2nd column."]},{"idx":3,"heading":"Deep Learning Based VQA Methods","paragraphs":["The emergence of deep-learning architectures have led to the development of the VQA systems. We discuss the state-of-the-art methods with an overview in Table TABREF6.","Vanilla VQA BIBREF0: Considered as a benchmark for deep learning methods, the vanilla VQA model uses CNN for feature extraction and LSTM or Recurrent networks for language processing. These features are combined using element-wise operations to a common feature, which is used to classify to one of the answers as shown in Fig. FIGREF4.","Stacked Attention Networks BIBREF11: This model introduced the attention using the softmax output of the intermediate question feature. The attention between the features are stacked which helps the model to focus on the important portion of the image.","Teney et al. Model BIBREF13: Teney et al. introduced the use of object detection on VQA models and won the VQA Challenge 2017. The model helps in narrowing down the features and apply better attention to images. The model employs the use of R-CNN architecture and showed significant performance in accuracy over other architectures.","Neural-Symbolic VQA BIBREF23: Specifically made for CLEVR dataset, this model leverages the question formation and image generation strategy of CLEVR. The images are converted to structured features and the question features are converted to their original root question strategy. This feature is used to filter out the required answer.","Focal Visual Text Attention (FVTA) BIBREF24: This model combines the sequence of image features generated by the network, text features of the image (or probable answers) and the question. It applies the attention based on the both text components, and finally classifies the features to answer the question. This model is better suited for the VQA in videos which has more use cases than images.","Pythia v1.0 BIBREF27: Pythia v1.0 is the award winning architecture for VQA Challenge 2018. The architecture is similar to Teney et al. BIBREF13 with reduced computations with element-wise multiplication, use of GloVe vectors BIBREF22, and ensemble of 30 models.","Differential Networks BIBREF19: This model uses the differences between forward propagation steps to reduce the noise and to learn the interdependency between features. Image features are extracted using Faster-RCNN BIBREF21. The differential modules BIBREF29 are used to refine the features in both text and images. GRU BIBREF30 is used for question feature extraction. Finally, it is combined with an attention module to classify the answers. The Differential Networks architecture is illustrated in Fig. FIGREF5."]},{"idx":4,"heading":"Experimental Results and Analysis","paragraphs":["The reported results for different methods over different datasets are summarized in Table TABREF2 and Table TABREF6. It can be observed that VQA dataset is very commonly used by different methods to test the performance. Other datasets like Visual7W, Tally-QA and KVQA are also very challenging and recent datasets. It can be also seen that the Pythia v1.0 is one of the recent methods performing very well over VQA dataset. The Differentail Network is the very recent method proposed for VQA task and shows very promising performance over different datasets.","As part of this survey, we also implemented different methods over different datasets and performed the experiments. We considered the following three models for our experiments, 1) the baseline Vanilla VQA model BIBREF0 which uses the VGG16 CNN architecture BIBREF2 and LSTMs BIBREF6, 2) the Stacked Attention Networks BIBREF11 architecture, and 3) the 2017 VQA challenge winner Teney et al. model BIBREF13. We considered the widely adapted datasets such as standard VQA dataset BIBREF0 and Visual7W dataset BIBREF8 for the experiments. We used the Adam Optimizer for all models with Cross-Entropy loss function. Each model is trained for 100 epochs for each dataset.","The experimental results are presented in Table TABREF7 in terms of the accuracy for three models over two datasets. In the experiments, we found that the Teney et al. BIBREF13 is the best performing model on both VQA and Visual7W Dataset. The accuracies obtained over the Teney et al. model are 67.23% and 65.82% over VQA and Visual7W datasets for the open-ended question-answering task, respectively. The above results re-affirmed that the Teney et al. model is the best performing model till 2018 which has been pushed by Pythia v1.0 BIBREF12, recently, where they have utilized the same model with more layers to boost the performance."]},{"idx":5,"heading":"Conclusion","paragraphs":["The Visual Question Answering has recently witnessed a great interest and development by the group of researchers and scientists from all around the world. The recent trends are observed in the area of developing more and more real life looking datasets by incorporating the real world type questions and answers. The recent trends are also seen in the area of development of sophisticated deep learning models by better utilizing the visual cues as well as textual cues by different means. The performance of even best model is still lagging and around 60-70% only. Thus, it is still an open problem to develop better deep learning models as well as more challenging datasets for VQA. Different strategies like object level details, segmentation masks, deeper models, sentiment of the question, etc. can be considered to develop the next generation VQA models."]},{"idx":6,"heading":"Figures and Tables","paragraphs":["Fig. 1. The timeline of major breakthrough in Visual Question Answering (VQA) in last 5 years, ranging from DAQUAR in 2014 to Differential Networks in 2019.","TABLE I OVERVIEW OF VQA DATASETS DESCRIBED IN THIS PAPER.","Fig. 3. Vanilla VQA Network Model [1].","Fig. 4. Differential Networks Model [20].","TABLE II OVERVIEW OF MODELS DESCRIBED IN THIS PAPER. THE PYTHIA V0.1 IS THE BEST PERFORMING MODEL OVER VQA DATASET.","TABLE III THE ACCURACIES OBTAINED USING VANILLA VQA [1], STACKED ATTENTION NETWORKS [12] AND TENEY ET AL. [14] MODELS WHEN TRAINED ON VQA [1] AND VISUAL7W [9] DATASETS."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1909.02027","title":"An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Task-oriented dialog systems need to know when a query falls outside their range of supported intents, but current text classification corpora only define label sets that cover every example. We introduce a new dataset that includes queries that are out-of-scope---i.e., queries that do not fall into any of the system's supported intents. This poses a new challenge because models cannot assume that every query at inference time belongs to a system-supported intent class. Our dataset also covers 150 intent classes over 10 domains, capturing the breadth that a production task-oriented agent must handle. We evaluate a range of benchmark classifiers on our dataset along with several different out-of-scope identification schemes. We find that while the classifiers perform well on in-scope intent classification, they struggle to identify out-of-scope queries. Our dataset and evaluation fill an important gap in the field, offering a way of more rigorously and realistically benchmarking text classification in task-driven dialog systems."]},{"idx":1,"heading":"Introduction","paragraphs":["Task-oriented dialog systems have become ubiquitous, providing a means for billions of people to interact with computers using natural language. Moreover, the recent influx of platforms and tools such as Google's DialogFlow or Amazon's Lex for building and deploying such systems makes them even more accessible to various industries and demographics across the globe.","Tools for developing such systems start by guiding developers to collect training data for intent classification: the task of identifying which of a fixed set of actions the user wishes to take based on their query. Relatively few public datasets exist for evaluating performance on this task, and those that do exist typically cover only a very small number of intents (e.g. BIBREF0, which has 7 intents). Furthermore, such resources do not facilitate analysis of out-of-scope queries: queries that users may reasonably make, but fall outside of the scope of the system-supported intents.","Figure FIGREF1 shows example query-response exchanges between a user and a task-driven dialog system for personal finance. In the first user-system exchange, the system correctly identifies the user's intent as an in-scope balance query. In the second and third exchanges, the user queries with out-of-scope inputs. In the second exchange, the system incorrectly identifies the query as in-scope and yields an unrelated response. In the third exchange, the system correctly classifies the user's query as out-of-scope, and yields a fallback response.","Out-of-scope queries are inevitable for a task-oriented dialog system, as most users will not be fully cognizant of the system's capabilities, which are limited by the fixed number of intent classes. Correctly identifying out-of-scope cases is thus crucial in deployed systems—both to avoid performing the wrong action and also to identify potential future directions for development. However, this problem has seen little attention in analyses and evaluations of intent classification systems.","This paper fills this gap by analyzing intent classification performance with a focus on out-of-scope handling. To do so, we constructed a new dataset with 23,700 queries that are short and unstructured, in the same style made by real users of task-oriented systems. The queries cover 150 intents, plus out-of-scope queries that do not fall within any of the 150 in-scope intents.","We evaluate a range of benchmark classifiers and out-of-scope handling methods on our dataset. BERT BIBREF1 yields the best in-scope accuracy, scoring 96% or above even when we limit the training data or introduce class imbalance. However, all methods struggle with identifying out-of-scope queries. Even when a large number of out-of-scope examples are provided for training, there is a major performance gap, with the best system scoring 66% out-of-scope recall. Our results show that while current models work on known classes, they have difficulty on out-of-scope queries, particularly when data is not plentiful. This dataset will enable future work to address this key gap in the research and development of dialog systems. All data introduced in this paper can be found at https://github.com/clinc/oos-eval."]},{"idx":2,"heading":"Dataset","paragraphs":["We introduce a new crowdsourced dataset of 23,700 queries, including 22,500 in-scope queries covering 150 intents, which can be grouped into 10 general domains. The dataset also includes 1,200 out-of-scope queries. Table TABREF2 shows examples of the data."]},{"idx":3,"heading":"Dataset ::: In-Scope Data Collection","paragraphs":["We defined the intents with guidance from queries collected using a scoping crowdsourcing task, which prompted crowd workers to provide questions and commands related to topic domains in the manner they would interact with an artificially intelligent assistant. We manually grouped data generated by scoping tasks into intents. To collect additional data for each intent, we used the rephrase and scenario crowdsourcing tasks proposed by BIBREF2. For each intent, there are 100 training queries, which is representative of what a team with a limited budget could gather while developing a task-driven dialog system. Along with the 100 training queries, there are 20 validation and 30 testing queries per intent."]},{"idx":4,"heading":"Dataset ::: Out-of-Scope Data Collection","paragraphs":["Out-of-scope queries were collected in two ways. First, using worker mistakes: queries written for one of the 150 intents that did not actually match any of the intents. Second, using scoping and scenario tasks with prompts based on topic areas found on Quora, Wikipedia, and elsewhere. To help ensure the richness of this additional out-of-scope data, each of these task prompts contributed to at most four queries. Since we use the same crowdsourcing method for collecting out-of-scope data, these queries are similar in style to their in-scope counterparts.","The out-of-scope data is difficult to collect, requiring expert knowledge of the in-scope intents to painstakingly ensure that no out-of-scope query sample is mistakenly labeled as in-scope (and vice versa). Indeed, roughly only 69% of queries collected with prompts targeting out-of-scope yielded out-of-scope queries. Of the 1,200 out-of-scope queries collected, 100 are used for validation and 100 are used for training, leaving 1,000 for testing."]},{"idx":5,"heading":"Dataset ::: Data Preprocessing and Partitioning","paragraphs":["For all queries collected, all tokens were down-cased, and all end-of-sentence punctuation was removed. Additionally, all duplicate queries were removed and replaced.","In an effort to reduce bias in the in-scope data, we placed all queries from a given crowd worker in a single split (train, validation, or test). This avoids the potential issue of similar queries from a crowd worker ending up in both the train and test sets, for instance, which would make the train and test distributions unrealistically similar. We note that this is a recommendation from concurrent work by BIBREF3. We also used this procedure for the out-of-scope set, except that we split the data into train/validation/test based on task prompt instead of worker."]},{"idx":6,"heading":"Dataset ::: Dataset Variants","paragraphs":["In addition to the full dataset, we consider three variations. First, Small, in which there are only 50 training queries per each in-scope intent, rather than 100. Second, Imbalanced, in which intents have either 25, 50, 75, or 100 training queries. Third, OOS+, in which there are 250 out-of-scope training examples, rather than 100. These are intended to represent production scenarios where data may be in limited or uneven supply."]},{"idx":7,"heading":"Benchmark Evaluation","paragraphs":["To quantify the challenges that our new dataset presents, we evaluated the performance of a range of classifier models and out-of-scope prediction schemes."]},{"idx":8,"heading":"Benchmark Evaluation ::: Classifier Models","paragraphs":["SVM: A linear support vector machine with bag-of-words sentence representations.","MLP: A multi-layer perceptron with USE embeddings BIBREF4 as input.","FastText: A shallow neural network that averages embeddings of n-grams BIBREF5.","CNN: A convolutional neural network with non-static word embeddings initialized with GloVe BIBREF6.","BERT: A neural network that is trained to predict elided words in text and then fine-tuned on our data BIBREF1.","Platforms: Several platforms exist for the development of task-oriented agents. We consider Google's DialogFlow and Rasa NLU with spacy-sklearn."]},{"idx":9,"heading":"Benchmark Evaluation ::: Out-of-Scope Prediction","paragraphs":["We use three baseline approaches for the task of predicting whether a query is out-of-scope: (1) oos-train, where we train an additional (i.e. 151st) intent on out-of-scope training data; (2) oos-threshold, where we use a threshold on the classifier's probability estimate; and (3) oos-binary, a two-stage process where we first classify a query as in- or out-of-scope, then classify it into one of the 150 intents if classified as in-scope.","To reduce the severity of the class imbalance between in-scope versus out-of-scope query samples (i.e., 15,000 versus 250 queries for OOS+), we investigate two strategies when using oos-binary: one where we undersample the in-scope data and train using 1,000 in-scope queries sampled evenly across all intents (versus 250 out-of-scope), and another where we augment the 250 OOS+ out-of-scope training queries with 14,750 sentences sampled from Wikipedia.","From a development point of view, the oos-train and oos-binary methods both require careful curation of an out-of-scope training set, and this set can be tailored to individual systems. The oos-threshold method is a more general decision rule that can be applied to any model that produces a probability. In our evaluation, the out-of-scope threshold was chosen to be the value which yielded the highest validation score across all intents, treating out-of-scope as its own intent."]},{"idx":10,"heading":"Benchmark Evaluation ::: Metrics","paragraphs":["We consider two performance metrics for all scenarios: (1) accuracy over the 150 intents, and (2) recall on out-of-scope queries. We use recall to evaluate out-of-scope since we are more interested in cases where such queries are predicted as in-scope, as this would mean a system gives the user a response that is completely wrong. Precision errors are less problematic as the fallback response will prompt the user to try again, or inform the user of the system's scope of supported domains."]},{"idx":11,"heading":"Results ::: Results with oos-train","paragraphs":["Table TABREF14 presents results for all models across the four variations of the dataset. First, BERT is consistently the best approach for in-scope, followed by MLP. Second, out-of-scope query performance is much lower than in-scope across all methods. Training on less data (Small and Imbalanced) yields models that perform slightly worse on in-scope queries. The trend is mostly the opposite when evaluating out-of-scope, where recall increases under the Small and Imbalanced training conditions. Under these two conditions, the size of the in-scope training set was decreased, while the number of out-of-scope training queries remained constant. This indicates that out-of-scope performance can be increased by increasing the relative number of out-of-scope training queries. We do just that in the OOS+ setting—where the models were trained on the full training set as well as 150 additional out-of-scope queries—and see that performance on out-of-scope increases substantially, yet still remains low relative to in-scope accuracy."]},{"idx":12,"heading":"Results ::: Results with oos-threshold","paragraphs":["In-scope accuracy using the oos-threshold approach is largely comparable to oos-train. Out-of-scope recall tends to be much higher on Full, but several models suffer greatly on the limited datasets. BERT and MLP are the top oos-threshold performers, and for several models the threshold approach provided erratic results, particularly FastText and Rasa."]},{"idx":13,"heading":"Results ::: Results with oos-binary","paragraphs":["Table TABREF19 compares classifier performance using the oos-binary scheme. In-scope accuracy suffers for all models using the undersampling scheme when compared to training on the full dataset using the oos-train and oos-threshold approaches shown in Table TABREF14. However, out-of-scope recall improves compared to oos-train on Full but not OOS+. Augmenting the out-of-scope training set appears to help improve both in-scope and out-of-scope performance compared to undersampling, but out-of-scope performance remains weak."]},{"idx":14,"heading":"Prior Work","paragraphs":["In most other analyses and datasets, the idea of out-of-scope data is not considered, and instead the output classes are intended to cover all possible queries (e.g., TREC BIBREF7). Recent work by BIBREF8 considers a similar problem they call out-of-distribution detection. They use other datasets or classes excluded during training to form the out-of-distribution samples. This means that the out-of-scope samples are from a small set of coherent classes that differ substantially from the in-distribution samples. Similar experiments were conducted for evaluating unknown intent discovery models in BIBREF9. In contrast, our out-of-scope queries cover a broad range of phenomena and are similar in style and often similar in topic to in-scope queries, representing things a user might say given partial knowledge of the capabilities of a system.","Table TABREF20 compares our dataset with other short-query intent classification datasets. The Snips BIBREF0 dataset and the dataset presented in BIBREF10 are the most similar to the in-scope part of our work, with the same type of conversational agent requests. Like our work, both of these datasets were bootstrapped using crowdsourcing. However, the Snips dataset has only a small number of intents and an enormous number of examples of each. Snips does present a low-data variation, with 70 training queries per intent, in which performance drops slightly. The dataset presented in BIBREF10 has a large number of intent classes, yet also contains a wide range of samples per intent class (ranging from 24 to 5,981 queries per intent, and so is not constrained in all cases).","BIBREF11 created datasets with constrained training data, but with very few intents, presenting a very different type of challenge. We also include the TREC query classification datasets BIBREF7, which have a large set of labels, but they describe the desired response type (e.g., distance, city, abbreviation) rather than the action intents we consider. Moreover, TREC contains only questions and no commands. Crucially, none of the other datasets summarized in Table TABREF20 offer a feasible way to evaluate out-of-scope performance.","The Dialog State Tracking Challenge (DSTC) datasets are another related resource. Specifically, DSTC 1 BIBREF12, DSTC 2 BIBREF13, and DSTC 3 BIBREF14 contain “chatbot style\" queries, but the datasets are focused on state tracking. Moreover, most if not all queries in these datasets are in-scope. In contrast, the focus of our analysis is on both in- and out-of-scope queries that challenge a virtual assistant to determine whether it can provide an acceptable response."]},{"idx":15,"heading":"Conclusion","paragraphs":["This paper analyzed intent classification and out-of-scope prediction methods with a new dataset consisting of carefully collected out-of-scope data. Our findings indicate that certain models like BERT perform better on in-scope classification, but all methods investigated struggle with identifying out-of-scope queries. Models that incorporate more out-of-scope training data tend to improve on out-of-scope performance, yet such data is expensive and difficult to generate. We believe our analysis and dataset will lead to developing better, more robust dialog systems.","All datasets introduced in this paper can be found at https://github.com/clinc/oos-eval."]},{"idx":16,"heading":"Figures and Tables","paragraphs":["Figure 1: Example exchanges between a user (blue, right side) and a task-driven dialog system for personal finance (grey, left side). The system correctly identi-","Table 1: Sample queries from our dataset. The out-of-scope queries are similar in style to the in-scope queries.","Table 2: Benchmark classifier results under each data condition using the oos-train (top half) and oos-threshold (bottom half) prediction methods.","Table 3: Results of oos-binary experiments on OOS+, where we compare performance of undersampling (under) and augmentation using sentences from Wikipedia (wiki aug). The wiki aug approach was too large for the DialogFlow and Rasa classifiers.","Table 4: Classification dataset properties. Ours has the broadest range of intents and specially collected out-ofscope queries. We consider “chatbot style” queries to be short, possibly unstructured questions and commands."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1909.03405","title":"Symmetric Regularization based BERT for Pair-wise Semantic Reasoning","sections":[{"idx":0,"heading":"Abstract","paragraphs":["The ability of semantic reasoning over the sentence pair is essential for many natural language understanding tasks, e.g., natural language inference and machine reading comprehension. A recent significant improvement in these tasks comes from BERT. As reported, the next sentence prediction (NSP) in BERT, which learns the contextual relationship between two sentences, is of great significance for downstream problems with sentence-pair input. Despite the effectiveness of NSP, we suggest that NSP still lacks the essential signal to distinguish between entailment and shallow correlation. To remedy this, we propose to augment the NSP task to a 3-class categorization task, which includes a category for previous sentence prediction (PSP). The involvement of PSP encourages the model to focus on the informative semantics to determine the sentence order, thereby improves the ability of semantic understanding. This simple modification yields remarkable improvement against vanilla BERT. To further incorporate the document-level information, the scope of NSP and PSP is expanded into a broader range, i.e., NSP and PSP also include close but nonsuccessive sentences, the noise of which is mitigated by the label-smoothing technique. Both qualitative and quantitative experimental results demonstrate the effectiveness of the proposed method. Our method consistently improves the performance on the NLI and MRC benchmarks, including the challenging HANS dataset~\\cite{hans}, suggesting that the document-level task is still promising for the pre-training."]},{"idx":1,"heading":"Introduction","paragraphs":["The ability of semantic reasoning is essential for advanced natural language understanding (NLU) systems. Many NLU tasks that take sentence pairs as input, such as natural language inference (NLI) and machine reading comprehension (MRC), heavily rely on the ability of sophisticated semantic reasoning. For instance, the NLI task aims to determine whether the hypothesis sentence (e.g., a woman is sleeping) can be inferred from the premise sentence (e.g., a woman is talking on the phone). This requires the model to read and understand sentence pairs to make the specific semantic inference.","Bidirectional Encoder Representations from Transformer (BERT) BIBREF1 has shown strong ability in semantic reasoning. It was recently proposed and obtained impressive results on many tasks, ranging from text classification, natural language inference, and machine reading comprehension. BERT achieves this by employing two objectives in the pre-training, i.e., the masked language modeling (Masked LM) and the next sentence prediction (NSP). Intuitively, the Masked LM task concerns word-level knowledge, and the NSP task captures the global document-level information. The goal of NSP is to identify whether an input sentence is next to another input sentence. From the ablation study BIBREF1, the NSP task is quite useful for the downstream NLI and MRC tasks (e.g., +3.5% absolute gain on the Question NLI (QNLI) BIBREF2 task).","Despite its usefulness, we suggest that BERT has not made full use of the document-level knowledge. The sentences in the negative samples used in NSP are randomly drawn from other documents. Therefore, to discriminate against these sentences, BERT is prone to aggregating the shallow semantic, e.g., topic, neglecting context clues useful for detailed reasoning. In other words, the canonical NSP task would encourage the model to recognize the correlation between sentences, rather than obtaining the ability of semantic entailment. This setting weakens the BERT model from learning specific semantic for inference. Another issue that renders NSP less effective is that BERT is order-sensitive. Performance degradation was observed on typical NLI tasks when the order of two input sentences are reversed during the BERT fine-tuning phase. It is reasonable as the NSP task can be roughly analogy to the NLI task when the input comes as (premise, hypothesis), considering the causal order among sentences. However, this identity between NSP and NLI is compromised when the sentences are swapped.","Based on these considerations, we propose a simple yet effective method, i.e., introducing a IsPrev category to the classification task, which is a symmetric label of IsNext of NSP. The input of samples with IsPrev is the reverse of those with IsNext label. The advantages of using this previous sentence prediction (PSP) are three folds. (1) Learning the contrast between NSP and PSP forces the model to extract more detailed semantic, thereby the model is more capable of discriminating the correlation and entailment. (2) NSP and PSP are symmetric. This symmetric regularization alleviates the influence of the order of the input pair. (3) Empirical results indicate that our method is beneficial for all the semantic reasoning tasks that take sentence pair as input.","In addition, to further incorporating the document-level knowledge, NSP and PSP are extended with non-successive sentences, where the label smoothing technique is adopted. The proposed method yields a considerable improvement in our experiments. We evaluate the ability of semantic reasoning on standard NLI and MRC benchmarks, including the challenging HANS dataset BIBREF0. Analytical work on the HANS dataset provides a more comprehensible perspective towards the proposed method. Furthermore, the results on the Chinese benchmarks are provided to demonstrate its generality.","In summary, this work makes the following contributions:","The supervision signal from the original NSP task is weak for semantic inference. Therefore, a novel method is proposed to remedy the asymmetric issue and enhance the reasoning ability.","Both empirical and analytical evaluations are provided on the NLI and MRC datasets, which verifies the effectiveness of using more document-level knowledge."]},{"idx":2,"heading":"Related Work ::: Pair-wise semantic reasoning","paragraphs":["Many NLU tasks seek to model the relationship between two sentences. Semantic reasoning is performed on the sentence pair for the task-specific inference. Pair-wise semantic reasoning tasks have drawn a lot of attention from the NLP community as they largely require the comprehension ability of the learning systems. Recently, the significant improvement on these benchmarks comes from the pre-training models, e.g., BERT, StructBERT BIBREF3, ERNIE BIBREF4, BIBREF5, RoBERTa BIBREF6 and XLNet BIBREF7. These models learn from unsupervised/self-supervised objectives and perform excellently in the downstream tasks. Among these models, BERT adopts NSP as one of the objectives in the pre-training and shows that the NSP task has a positive effect on the NLI and MRC tasks. Although the primary study of XLNet and RoBERTa suggests that NSP is ineffective when the model is trained with a large sequence length of 512, the effect of NSP on the NLI problems should still be emphasized. The inefficiency of NSP is likely because the expected context length will be halved for Masked LM when taking a sentence pair as the input. The models derived from BERT, e.g., StructBERT and ERNIE 1.0/2.0, aim to incorporating more knowledge by elaborating pre-training objectives. This work aims to enhance the NSP task and verifies whether document-level information is helpful for the pre-training. To probe whether our method achieves a better regularization ability, our approach is also evaluated on the HANS BIBREF0 dataset, which contains hard data samples constructed by three heuristics. Previous advanced models such as BERT fail on the HANS dataset, and the test accuracy can barely exceed 0% in the subset of test examples."]},{"idx":3,"heading":"Related Work ::: Unsupervised learning from document","paragraphs":["In recent years, many unsupervised pre-training methods have been proposed in the NLP fields to extract knowledge among sentences DBLP:conf/nips/KirosZSZUTF15,DBLP:conf/emnlp/ConneauKSBB17,DBLP:conf/iclr/LogeswaranL18,DBLP:journals/corr/abs-1903-09424. The prediction of surrounding sentences endows the model with the ability to model the sentence-level coherence. Skip-Thought BIBREF8 consists of an encoder and two decoders. When a sentence is given and encoded into a vector by the encoder, the decoders are trained to predict the next sentence and the previous sentence. The goal is to obtain a better sentence representation that is useful for reconstructing the surrounding context. Considering that the estimation of the likelihood of sequences is computationally expensive and time-consuming, the Quick-Thought method BIBREF9 simplifies this in a manner similar to sampled softmax BIBREF10, which classifies the input sentences between surrounding sentences and the other. Note that Quick-Thought does not distinguish between the previous and next sentence as it is functionally rotation invariant. However, BERT is order-dependent, and the discrimination can provide more supervision signal for semantic learning. InferSent BIBREF11 instead pre-trains the model in a manner of supervised learning. It uses a large-scale NLI dataset as the pre-training task to learn the sentence representation. In our work, we focus on designing a more effective document-level objective, extended from the NSP task. The proposed method will be described in the following section and validated by providing extensive experimental results in the experiment part."]},{"idx":4,"heading":"Method","paragraphs":["Our method follows the same input format and the model architecture with original BERT. The proposed method solely concerns the NSP task. The NSP task is a binary classification task, which takes two sentences (A and B) as input and determines whether B is the next sentence of A. Although it has been proven to be very effective for BERT, there are two major deficiencies. (1) Discrimination between IsNext and DiffDoc (the label of the sentences drawn from different documents via negative sampling) is semantically shallow as the signal of sentence order is absent. The correlation between two successive sentences could be obvious, due to, for example, lexical overlap or the conjunction used at the beginning of the second sentence. As reported BIBREF1, the final pre-trained model is able to achieve 97%-98% accuracy on the NSP task. (2) BERT is order-sensitive, i.e., $f_{\\text{BERT}}( \\texttt {A}, \\texttt {B}) \\ne f_{\\text{BERT}}(\\texttt {B}, \\texttt {A})$, while NSP is uni-directional. When the order of the input NLI pair is reversed, the performance will degrade. For instance, the accuracy decreases by about 0.5% on MNLI BIBREF12 and 0.4% on QNLI after swapping the sentences in our experiments .","Motivated by these problems, we propose to extend the NSP task with previous sentence prediction (PSP). Despite its simplicity, empirical results show that this is beneficial for downstream tasks, including both NLI and MRC tasks. To further incorporate the document-level information, the scope is also expanded to include more surrounding sentences, not just the adjacent. The method is briefly illustrated in Fig. FIGREF6."]},{"idx":5,"heading":"Method ::: Previous Sentence Prediction","paragraphs":["Learning to recognize the previous sentence enables the model to capture more compact context information. One would argue that IsPrev (the label of PSP) is redundant as it plays a similar role of IsNext (the label of NSP). In fact, Quick-Thought uses the sampled softmax to approximate the sentence likelihood estimation of Skip-Thought, and it actually does not differentiate between the previous and next sentences. However, we suggest the order discrimination is essential for BERT pre-training. Quick-Thought aims at extracting sentence embedding, and it uses a rotating symmetric function, which makes IsPrev redundant in Quick-Thought. In contrast, BERT is order-sensitive, and learning the symmetric regularization is rather necessary. Another advantage of PSP is to enhance document-level supervision. In order to tell the difference between NSP and PSP, the model has to extract the detailed semantic for inference."]},{"idx":6,"heading":"Method ::: Gathering More Document-level Information","paragraphs":["Beyond NSP and PSP, which enable the model to learn the short-term dependency between sentences, we also propose to expand the scope of discrimination task to further incorporate the document-level information.","Specifically, we also include the in-adjacent sentences in the sentence-pair classification task. The in-adjacent sentences next to the IsPrev and IsNext sentences are sampled, labeled as IsPrevInadj and IsNextInadj (cf. the bottom of Fig. FIGREF6). Note that these in-adjacent sentences will introduce much more training noise to the model. Therefore, the label smoothing technique is adopted to reduce the noise of these additional samples. It achieves this by relaxing our confidence on the labels, e.g., transforming the target probability from (1.0, 0.0) to (0.8, 0.2) in a binary classification problem.","In summary, when A is given, the pre-training example for each label is constructed as follows:","IsNext: Choosing the adjacent following sentence as B.","IsPrev: Choosing the adjacent previous sentence as B.","IsNextInadj: Choosing the in-adjacent following sentence as B. There is a sentence between A and B.","IsPrevInadj: Choosing the in-adjacent previous sentence as B. There is a sentence between A and B.","DiffDoc: Drawing B randomly from a different document."]},{"idx":7,"heading":"Experiment Settings","paragraphs":["This section gives detailed experiment settings. The method is evaluated on the BERTbase model, which has 12 layers, 12 self-attention heads with a hidden size of 768.","To accelerate the training speed, two-phase training BIBREF1 is adopted. The first phase uses a maximal sentence length of 128, and 512 for the second phase. The numbers of training steps of two phases are 50K and 40K for the BERTBase model. We used AdamW BIBREF13 optimizer with a learning rate of 1e-4, a $\\beta _1$ of 0.9, a $\\beta _2$ of 0.999 and a L2 weight decay rate of $0.01$. The first 10% of the total steps are used for learning rate warming up, followed by the linear decay schema. We used a dropout probability of 0.1 on all layers. The data used for pre-training is the same as BERT, i.e., English Wikipedia (2500M words) and BookCorpus (800M words) BIBREF14. For the Masked LM task, we followed the same masking rate and settings as in BERT.","We explore three method settings for comparison.","BERT-PN: The NSP task in BERT is replaced by a 3-class task with IsNext, IsPrev and DiffDoc. The label distribution is 1:1:1.","BERT-PN5cls: The NSP task in BERT is replaced by a 5-class task with two additional labels IsNextInadj, IsPrevInadj. The label distribution is 1:1:1:1:1.","BERT-PNsmth: It uses the same data with BERT-PN5cls, except that the IsPrevInadj (IsNextInadj) label is mapped to IsPrev (IsNext) with a label smoothing factor of 0.8.","BERT-PN is used to verify the feasibility of PSP. The comparison with BERT-PN5cls illustrates whether more document-level information helps. BERT-PNsmth, which is the label-smoothed version of BERT-PN5cls, is used to compare with BERT-PN5cls to see whether the noise reduction is necessary.","In the following, we first show that BERT is order-sensitive and the use of PSP remedies this problem. Then we provide experimental results on the NLI and MRC tasks to verify the effectiveness of the proposed method. At last, the proposed method is evaluated on several Chinese datasets."]},{"idx":8,"heading":"Order-invariant with PSP","paragraphs":["NSP in the pre-training is useful for NLI and MRC task BIBREF1. However, we suggested that BERT trained with NSP is order-sensitive, i.e., the performance of BERT depends on the order of the input sentence pair. To verify our assumption, a primary experiment was conducted. The order of the input pair of NLI samples is reversed in the fine-tuning phase, and other hyper-parameters and settings keep the same with the BERT paper. Table TABREF19 shows the accuracy on the validation set of the MNLI and QNLI datasets. For the BERTBase model, when the sentences are swapped, the accuracy decreases by 0.5% on the MNLI task and 0.4% on the QNLI task. These results confirm that BERT trained with NSP only is indeed affected by the input order. This phenomenon motivates us to make the NSP task symmetric. The results of BERT-PN verify that BERT-PN is order-invariant. When the input order is reversed, the performance of BERT-PN remains stable. These results indicate that our method is able to remedy the order-sensitivity problem."]},{"idx":9,"heading":"Results of NLI Tasks ::: GLUE","paragraphs":["A popular benchmark for evaluation of language understanding is GLUE BIBREF2, which is a collection of three NLI tasks (MNLI, QNLI and RTE), three semantic textual similarity (STS) tasks (QQP, STS-B and MRPC), two text classification (TC) tasks (SST-2 and CoLA). Although the method is motivated for pair-wise reasoning, the results of other problems are also listed.","Our implementation follows the same way that BERT performs in these tasks. The fine-tuning was conducted for 3 epochs for all the tasks, with a learning rate of 2e-5. The predictions were obtained by evaluating the training checkpoint with the best validation performance.","Table TABREF21 illustrates the experimental results, showing that our method is beneficial for all of NLI tasks. The improvement on the RTE dataset is significant, i.e., 4% absolute gain over the BERTBase. Besides NLI, our model also performs better than BERTBase in the STS task. The STS tasks are semantically similar to the NLI tasks, and hence able to take advantage of PSP as well. Actually, the proposed method has a positive effect whenever the input is a sentence pair. The improvements suggest that the PSP task encourages the model to learn more detailed semantics in the pre-training, which improves the model on the downstream learning tasks. Moreover, our method is surprisingly able to achieve slightly better results in the single-sentence problem. The improvement should be attributed to better semantic representation.","When comparing between PN and PN5cls, PN5cls achieves better results than PN. This indicates that including a broader range of the context is effective for improving inference ability. Considering that the representation of IsNext and IsNextInadj should be coherent, we propose BERTBase-PNsmth to mitigate this problem. PNsmth further improves the performance and obtains an averaged score of 81.0."]},{"idx":10,"heading":"Results of NLI Tasks ::: HANS","paragraphs":["Although BERT has shown its effectiveness in the NLI tasks. BIBREF0 pointed out that BERT is still vulnerable in the NLI task as it is prone to adopting fallible heuristics. Therefore, they released a dataset, called The Heuristic Analysis for NLI Systems (HANS), to probe whether the model learns inappropriate inductive bias from the training set. It is constructed by three heuristics, i.e., lexical overlap heuristic, sub-sequence heuristic, and constituent heuristic. The first heuristic assumes that a premise entails all hypotheses constructed from words in the premise, the second assumes that a premise entails all of its contiguous sub-sequences and the third assumes that a premise entails all complete sub-trees in its parse tree. BERT and other advanced models fail on this dataset and barely exceeds 0% accuracy in most cases BIBREF0.","Fig. FIGREF23 illustrates the accuracy of BERTBase and BERTBase-PNsmth on the HANS dataset. The evaluation is made upon the model trained on the MNLI dataset and the predicted neutral and contradiction labels are mapped into non-entailment. The BERTBase-PNsmth evidently outperforms the BERTBase with the non-entailment examples. For the non-entailment samples constructed using the lexical overlap heuristic, our model achieves 160% relative improvement over the BERTBase model. Some samples are constructed by swapping the entities in the sentence (e.g., The doctor saw the lawyer $\\nrightarrow $ The lawyer saw the doctor) and our method outperforms BERTBase by 20% in accuracy. We suggest that the Masked LM task can hardly model the relationship between two entities and NSP only is too semantically shallow to capture the precise meaning. However, the discrimination between NSP and PSP enhances the model to realize the role of entities in a given sentence. For example, to determine that A (X is beautiful) rather than $\\bar{\\texttt {A}}$ (Y is beautiful) is the previous sentence of B (Y loves X), the model have to recognize the relationship between X and Y. In contrast, when PSP is absent, NSP can be probably inferred by learning the occurrence between beautiful and loves, regardless of the sentence structure. The detailed performance of the proposed method on the HANS dataset is illustrated in Fig. FIGREF24. The definition of each heuristic rules can be found in BIBREF0."]},{"idx":11,"heading":"Results of MRC Tasks ::: SQuAD v1.1 and v2.0","paragraphs":["We also evaluate our method on the MRC tasks. The Stanford Question Answering Dataset (SQuAD v1.1) is a question answering (QA) dataset, which consists of 100K samples BIBREF15. Each data sample has a question and a corresponding Wikipedia passage that contains the answer. The goal is to extract the answer from the passage for the given question.","In the fine-tuning procedure, we follow the exact way the BERT performed. The output vectors are used to compute the score of tokens being start and end of the answer span. The valid span that has the maximum score is selected as the prediction. And similarly, the fine-tuning training was performed for 3 epochs with a learning rate of 3e-5.","Table TABREF26 demonstrates the results on the SQuAD v1.1 dataset. The comparison between BERTBase-PN and BERTBase indicates that the inclusion of the PSP subtask is beneficial (2.4% absolute improvement). When using BERTBase-PNsmth, another 0.3% increase in EM can be obtained. The experimental results on the SQuAD v2.0 BIBREF16 are also shown in Table. TABREF26. The SQuAD v2.0 differs from SQuAD v1.1 by allowing the question-paragraph pairs that have no answer. For SQuAD v2.0, our method also achieved about 4% absolute improvement in both EM and F1 against BERTBase."]},{"idx":12,"heading":"Results of MRC Tasks ::: RACE","paragraphs":["The ReAding Comprehension from Examinations (RACE) dataset BIBREF17 consists of 100K questions taken from English exams, and the answers are generated by human experts. This is one of the most challenging MRC datasets that require sophisticated reasoning.","In our implementation, the question, document, and option are concatenated as a single sequence, separated by [SEP] token. And each part is truncated by a maximal length of 40/432/40, respectively. The model computes for a concatenation a scalar as the score, which is then used in a softmax layer for the final prediction. The fine-tuning was conducted for 5 epochs, with a batch size of 32 and a learning rate of 5e-5. As shown in Table TABREF28, the proposed method significantly improve the performance on the RACE dataset. BERTBase-PN obtains 2.6% accuracy improvement, and BERTBase-PN5cls further brings 0.4% absolute gain.","The comparisons on the SQuAD v1.1, SQuAD v2.0, and RACE dataset demonstrate that the involvement of additional sentence and discourse information is not only beneficial for the NLI task but also the MRC task. This is reasonable as these tasks heavily rely on the global semantic understanding and sophisticated reasoning among sentences. And this ability can be effectively enhanced by our method."]},{"idx":13,"heading":"Results of Chinese NLP Tasks","paragraphs":["The experiments are also conducted on Chinese NLP tasks:","XNLI BIBREF19 a multi-lingual dataset. The data sample in XNLI is a sentence pair annotated with textual entailment. The Chinese part is used.","LCQMC BIBREF20 is a dataset for sequence matching. A binary label is annotated for a sentence pair in the dataset to indicate whether these two sentences have the same intention.","NLPCC-DBQA BIBREF21 formulates the domain-based question answering as a binary classification task. Each data sample is a question-sentence pair. The goal is to identify whether the sentence contains the answer to the question.","CMRC-2018 is the Chinese Machine Reading Comprehension dataset. Similar to SQuAD, the system needs to extract fragments from the text as the answer.","DRCD BIBREF22 is also a Chinese MRC data set. The data follows the format of SQuAD.","For Chinese NLP tasks, we pre-train the model using Chinese corpus. We collected textual data (10879M tokens in total) from the website, consisting of Hudong Baike data (6084M tokens) , Zhihu data(465M tokens) , Sohu News(3937M tokens) and Wikipedia data (393M tokens).","For the first 3 Chinese tasks, we follow the settings as in ERNIE BIBREF4. The experimental results are given in Table TABREF29. The proposed method is compared with four models, i.e., BERTBase BIBREF1, BERTBase with whole word masking BIBREF18, ERNIE BIBREF4 and ERNIE 2.0 BIBREF5. Our method achieves comparable or even better results against ERNIE 2.0 BIBREF5. Note that the Chinese ERNIE 2.0 is equipped with 5 different objectives and it uses more training data (14988M tokens in total) than ours. The results indicate that the proposed method is quite effective for the pair-wise semantic reasoning as simply including PSP can achieve the results on par with multiple objectives.","The results of CMRC-2018 and DRCD datasets are given in Table TABREF30. Since the CMRC-2018 competition does not release the test set, the comparison on the test set is absent. Our results are obtained using the open-sourced code of BERT-wwm . We keep the hyper-parameters the same with that in ERNIE BIBREF4, except that the batch size is 12 instead of 64 due to the memory limit. Under this setting, we achieved similar results of BERTBase in the BERT-wwm paper BIBREF18. However, this is worse than the results of BERTBase reported in the ERNIE 2.0 paper BIBREF5 by about 1% in F1. This suggests that our results are currently incomparable with ERNIE 2.0. Overall, the results in Table TABREF30 illustrate that our method is also effective for the Chinese QA tasks."]},{"idx":14,"heading":"Conclusion","paragraphs":["This paper aims to enrich the NSP task to provide more document-level information in the pre-training. Motivated by the in-symmetric property of NSP, we propose to differentiate between different sentence orders by including PSP. Despite the simplicity, extensive experiments demonstrate that the model obtains a better ability in pair-wise semantic reasoning. Our work suggests that the document-level objective is effective, at least for the BERTbase model. In the future, we will investigate the way to take advantages of both large-scale training and our method."]},{"idx":15,"heading":"Figures and Tables","paragraphs":["Figure 1: An illustration of the proposed method. B denotes the second input sentence. (1) Top: original NSP task. (2) Middle: 3-class categorization task with DiffDoc, IsNext and IsPrev. (3) Bottom: 3-class task, but with a wider scope of NSP and PSP. The in-adjacent sentences are assisted with a label smoothing technique to reduce the noise.","Table 1: The accuracy of BERT and BERT-PN on the validation set of the MNLI and QNLI dataset. P&H denotes that the input is (premise, hypothesis), which is the order used in BERT. The reported accuracy is the average after 5 runs.","Table 2: Results on the test set of GLUE benchmark. The performance was obtained by the official evaluation server. The number below each task is the number of training examples. The ”Average” column follows the setting in the BERT paper, which excludes the problematic WNLI task. F1 scores are reported for QQP and MRPC, Spearman correlations are reported for STS-B, and accuracy scores are reported for the other tasks. All the listed models are trained on the Wikipedia and the Book Corpus datasets. The results are the average of 5 runs.","Table 3: The performance of various BERT models finetuned on the SQuAD v1.1 and v2.0 dataset. EM means the percentage of exact match. The results of RoBERTa is the DOC-SENTENCES version retrieved from Table 2 in (Liu et al. 2019).","Figure 2: The accuracy on evaluation set of HANS. It has six sub-components, each defined by its correct label and the heuristic it addresses.","Figure 3: Performance on thirty detailed sub-components of the HANS evaluation set (30K instances). Each sub-component is defined by three heuristics, i.e., Lexical overlap, Sub-sequence and Constituent. For instance, in prefix “ln” , “l” denotes lexical overlap heuristic, “n” denotes the non-entailment label. The suffix means a specific syntactic rule, e.g., subject/object swap means in the hypothesis sentence, the subject and the object are swapped.","Table 4: The experimental results on test set of the RACE dataset. The results of RoBERTa is the DOC-SENTENCES version retrieved from Table 2 in (Liu et al. 2019). All the listed models are trained on the Wikipedia and the Book Corpus datasets.","Table 5: Comparison on the Chinese NLP tasks. All the models are of “base” size. The results of BERT, BERT-wwm are retrieved from literature (Cui et al. 2019), except the results of NLPCC-DBQA which is from ERNIE 2.0 (2019b). The results of ERNIE, ERNIE 2.0 are retrieved from literature (Sun et al. 2019a; 2019b). The best result and the average (in bracket) of 5 runs are reported. The number below the model denotes the number of tokens in the pre-training data.","Table 6: Results on the CMRC-2018 and DRCD datasets. Three BERTBase models are reported from our reproduction, BERTwwm paper (Cui et al. 2019) and ERNIE 2.0 paper (Sun et al. 2019b), respectively. The results of BERTBase-wwm are obtained from the paper (Cui et al. 2019). EM denotes the percentage of exact matching. The best result and the average (in bracket) of 5 runs are reported."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1909.04242","title":"Mitigating Annotation Artifacts in Natural Language Inference Datasets to Improve Cross-dataset Generalization Ability","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Natural language inference (NLI) aims at predicting the relationship between a given pair of premise and hypothesis. However, several works have found that there widely exists a bias pattern called annotation artifacts in NLI datasets, making it possible to identify the label only by looking at the hypothesis. This irregularity makes the evaluation results over-estimated and affects models' generalization ability. In this paper, we consider a more trust-worthy setting, i.e., cross-dataset evaluation. We explore the impacts of annotation artifacts in cross-dataset testing. Furthermore, we propose a training framework to mitigate the impacts of the bias pattern. Experimental results demonstrate that our methods can alleviate the negative effect of the artifacts and improve the generalization ability of models."]},{"idx":1,"heading":"Introduction","paragraphs":["Natural language inference (NLI) is a widely-studied problem in natural language processing. It aims at comparing a pair of sentences (i.e. a premise and a hypothesis), and inferring the relationship between them (i.e., entailment, neutral and contradiction). Large-scaled datasets like SNLI BIBREF0 and MultiNLI BIBREF1 have been created by crowd-sourcing and fertilized NLI research substantially.","However, several works BIBREF2, BIBREF3, BIBREF4 have pointed out that crowd-sourcing workers have brought a bias pattern named annotation artifacts in these NLI datasets. Such artifacts in hypotheses can reveal the labels and make it possible to predict the labels solely by looking at the hypotheses. For example, models trained on SNLI with only the hypotheses can achieve an accuracy of 67.0%, despite the always predicting the majority-class baseline is only 34.3% BIBREF2.","Classifiers trained on NLI datasets are supposed to make predictions by understanding the semantic relationships between given sentence pairs. However, it is shown that models are unintentionally utilizing the annotation artifacts BIBREF4, BIBREF2. If the evaluation is conducted under a similar distribution as the training data, e.g., with the given testing set, models will enjoy additional advantages, making the evaluation results over-estimated. On the other hand, if the bias pattern cannot be generalized to the real-world, it may introduce noise to models, thus hurting the generalization ability.","In this paper, we use cross-dataset testing to better assess models' generalization ability. We investigate the impacts of annotation artifacts in cross-dataset testing. Furthermore, we propose an easy-adopting debiasing training framework, which doesn't require any additional data or annotations, and apply it to the high-performing Densely Interactive Inference Network BIBREF5. Experiments show that our method can effectively mitigate the bias pattern and improve the cross-dataset generalization ability of models. To the best of our knowledge, our work is the first attempt to alleviate the annotation artifacts without any extra resources."]},{"idx":2,"heading":"Related Work","paragraphs":["Frequently-used NLI datasets such as SNLI and MultiNLI are created by crowd-sourcing BIBREF0, BIBREF1, during which they present workers a premise and ask them to produce three hypotheses corresponding to labels. As BIBREF2 pointed out, workers may adopt some specific annotation strategies and heuristics when authoring hypotheses to save efforts, which produces certain patterns called annotation artifacts in the data. Models' trained on such datasets are heavily affected by the bias pattern BIBREF2.","BIBREF4 further investigate models' robustness to the bias pattern using swapping operations. BIBREF6 demonstrate that the annotation artifacts widely exist among NLI datasets. They show that hypothesis-only-model, which refers to models trained and predict only with hypotheses, outperforms always predicting the majority-class in six of ten NLI datasets.","The emergence of the pattern can be due to selection bias BIBREF7, BIBREF8, BIBREF9 in the datasets preparing procedure. Several works BIBREF10, BIBREF11 investigate the bias problem in relation inference datasest. BIBREF12 investigate the selection bias embodied in the comparing relationships in six natural language sentence matching datasets and propose a debiasing training and evaluation framework."]},{"idx":3,"heading":"Making Artifacts Unpredictable","paragraphs":["Essentially speaking, the problem of the bias pattern is that the artifacts in hypotheses are distributed differently among labels, so balancing them across labels may be a good solution to alleviate the impacts BIBREF2.","Based on the idea proposed by BIBREF12, we demonstrate that we can make artifacts in biased datasets balanced across different classes by assigning specific weights for every sample. We refer the distribution of the acquired weighted dataset as artifact-balanced distribution. We consider a supervised NLI task, which is to predict the relationship label $y$ given a sentence pair $x$, and we denote the hypothesis in $x$ as $h$. Without loss of generality, we assume that the prior probability of different labels is equal, and then we have the following theorem.","Theorem 1 For any classifier $f=f(x, h)$, and for any loss function $\\Delta (f(x, h), y)$, if we use $w = \\frac{1}{P(y|h)}$ as weight for every sample during training, it's equivalent to training with the artifact-balanced distribution.","Detailed assumptions and the proof of the theorem is presented in Appendix SECREF6. With the theorem, we can simply use cross predictions to estimate $P(y|h)$ in origin datasets and use them as sample weights during training. The step-by-step procedure for artifact-balanced learning is presented in Algorithm 1.","However, it is difficult to precisely estimate the probability $P(y|h)$. A minor error might lead to a significant difference to the weight, especially when the probability is close to zero. Thus, in practice, we use $w = \\frac{1}{(1-\\epsilon )P(y|h) + \\epsilon }$ as sample weights during training in order to improve the robustness. We can find that as $\\epsilon $ increases, the weights tend to be uniform, indicating that the debiasing effect decreases as the smooth term grows. Moreover, in order to keep the prior probability $P(Y)$ unchanged, we normalize the sum of weights of the three labels to the same."]},{"idx":4,"heading":"Experimental Results","paragraphs":["In this section, we present the experimental results for cross-dataset testing of artifacts and artifact-balanced learning. We show that cross-dataset testing is less affected by annotation artifacts, while there are still some influences more or less in different datasets. We also demonstrate that our proposed framework can mitigate the bias and improve the generalization ability of models."]},{"idx":5,"heading":"Experimental Results ::: Evaluation Scheme ::: Cross-dataset Testing","paragraphs":["We utilize SNLI BIBREF0, MultiNLI BIBREF1, JOCI BIBREF13 and SICK BIBREF14 for cross-dataset testing.","SNLI and MultiNLI are prepared by Human Elicited, in which workers are given a context and asked to produce hypotheses corresponding to labels. SICK and JOCI are created by Human Judged, referring that hypotheses and premises are automatically paired while labels are generated by humans BIBREF6. In order to maximumly mitigate the impacts of annotation artifacts during evaluations, we train and validate models respectively on SNLI and MultiNLI and test on both SICK and JOCI. We also report models' performances on SNLI and MultiNLI.","As to SNLI, we use the same partition as BIBREF0. For MultiNLI, we separately use two origin validation sets (Matched and Mismatched) as the testing sets for convenience, and refer them as MMatch and MMismatch. We randomly select 10000 samples out of the origin training set for validation and use the rest for training. As to JOCI, we use the whole “B” subsets for testing, whose premises are from SNLI-train while hypotheses are generated based on world knowledge BIBREF13, and convert the score to NLI labels following BIBREF6. As to SICK, we use the whole dataset for testing."]},{"idx":6,"heading":"Experimental Results ::: Evaluation Scheme ::: Hard-Easy Testing","paragraphs":["To determine how biased the models are, we partition the testing set of SNLI and MMatch into two subsets: examples that the hypothesis-only model can be correctly classified as Easy and the rest as Hard as seen in BIBREF2. More detailed information is presented in Appendix SECREF14."]},{"idx":7,"heading":"Experimental Results ::: Experiment Setup","paragraphs":["We refer models trained only with hypotheses as hypothesis-only-model (Hyp), and models that utilize both premises and hypotheses as normal-model (Norm). We implement a simple LSTM model for Hyp and use DIIN BIBREF5 as Norm. We report AUC for Hyp and ACC for Norm. More details can be seen in Appendix SECREF15","We estimate $P(y|h)$ for SNLI and MultiNLI respectively using BERT BIBREF15 with 10-fold predictions. To investigate the impacts of smooth terms, we choose a series of smooth values and present the results. Considering models may jiggle during the training phase due to the varied scale of weights, we sample examples with probabilities proportional to the weights for every mini-batch instead of adding weights to the loss directly.","The evaluation results are reported in Table TABREF3."]},{"idx":8,"heading":"Experimental Results ::: Can Artifacts Generalize Across Datasets?","paragraphs":["Anotation Artifacts can be generalized across Human Elicited datasets. From the AUC of Hyp baseline trained with SNLI, we can see that the bias pattern of SNLI has a strong predictive ability in itself and the other two testing sets of Human Elicited. The behavior of those trained with MultiNLI is similar.","Anotation Artifacts of SNLI and MultiNLI can be generalized to SICK. Unexpectedly, it is shown that Hyp baseline can get $0.6250$ (AUC) trained with SNLI and $0.6079$ (AUC) with MultiNLI when tested on SICK, indicating that the bias pattern of SNLI and MultiNLI are predictive on SICK. The results imply that the bias pattern can even be generalized across datasets prepared by different methods.","Annotation Artifacts of SNLI are nearly neutral in JOCI, while MultiNLI is misleading. We find that AUC of Hyp baseline trained with SNLI is very close to $0.5$ on JOCI, indicating that JOCI is nearly neutral to artifacts in SNLI. However, when it comes to training with MultiNLI, the AUC of Hyp baseline is lower than $0.5$, indicating that the artifacts are misleading in JOCI."]},{"idx":9,"heading":"Experimental Results ::: Debiasing Results ::: Effectiveness of Debiasing","paragraphs":["Focusing on the results when smooth equals $0.01$ for SNLI and smooth equals $0.02$ for MultiNLI, we observe that the AUC of Hyp for all testing sets are approximately $0.5$, indicating Hyp's predictions are approximately equivalent to randomly guessing. Also, the gap between Hard and Easy for Norm significantly decreases comparing with the baseline. With the smooth, we can conclude that our method effectively alleviates the bias pattern.","With other smooth terms, our method still has more or less debiasing abilities. In those testing sets which are not neutral to the bias pattern, the AUC of Hyp always come closer to $0.5$ comparing with the baseline with whatever smooth values. Performances of Norm on Hard and Easy also come closer comparing with the baseline. Norm trained with SNLI even exceed baseline in Hard with most smooth terms.","From the results of Hyp, we can find a trend that the larger the smooth value is, the lower the level of debiasing is, while with a very small or even no smooth value, the AUC may be lower than $0.5$. As mentioned before, we owe this to the imperfect estimation of $P(y|h)$, and we can conclude that a proper smooth value is a prerequisite for the best debiasing effect."]},{"idx":10,"heading":"Experimental Results ::: Debiasing Results ::: Benefits of Debiasing","paragraphs":["Debiasing may improve models' generalization ability from two aspects: (1) Mitigate the misleading effect of annotation artifacts. (2) Improve models' semantic learning ability.","When the annotation artifacts of the training set cannot be generalized to the testing set, which should be more common in the real-world, predicting by artifacts may hurt models' performance. Centering on the results of JOCI, in which the bias pattern of MultiNLI is misleading, we find that Norm trained with MultiNLI outperforms baseline after debiasing with all smooth values tested.","Furthermore, debiasing can reduce models' dependence on the bias pattern during training, thus force models to better learn semantic information to make predictions. Norm trained with SNLI exceed baseline in JOCI with smooth terms $0.01$ and $0.1$. With larger smooth terms, Norm trained with both SNLI and MultiNLI exceeds baseline in SICK. Given the fact that JOCI is almost neutral to artifacts in SNLI, and the bias pattern of both SNLI and MultiNLI are even predictive in SICK, we owe these promotions to that our method improves models' semantic learning ability.","As to other testing sets like SNLI, MMatch and MMismatch, we notice that the performance of Norm always decreases compared with the baseline. As mentioned before, both SNLI and MultiNLI are prepared by Huamn Elicited, and their artifacts can be generalized across each other. We owe the drop to that the detrimental effect of mitigating the predictable bias pattern exceeds the beneficial effect of the improvement of semantic learning ability."]},{"idx":11,"heading":"Conclusion","paragraphs":["In this paper, we take a close look at the annotation artifacts in NLI datasets. We find that the bias pattern could be predictive or misleading in cross-dataset testing. Furthermore, we propose a debiasing framework and experiments demonstrate that it can effectively mitigate the impacts of the bias pattern and improve the cross-dataset generalization ability of models. However, it remains an open problem that how we should treat the annotation artifacts. We cannot assert whether the bias pattern should not exist at all or it is actually some kind of nature. We hope that our findings will encourage more explorations on reliable evaluation protocols for NLI models."]},{"idx":12,"heading":"Detailed Assumptions and Proof of Theorem @!START@UID1@!END@","paragraphs":["We make a few assumptions about an artifact-balanced distribution and how the biased datasets are generated from it, and demonstrate that we can train models fitting the artifact-balanced distribution using only the biased datasets.","We consider the domain of the artifact-balanced distribution ${D}$ as $\\mathcal {X} \\times \\mathcal {A} \\times \\mathcal {Y} \\times \\mathcal {S}$, in which $\\mathcal {X}$ is the input variable space, $\\mathcal {Y}$ is the label space, $\\mathcal {A}$ is the feature space of annotation artifacts in hypotheses, $\\mathcal {S}$ is the selection intention space. We assume that the biased distribution $\\widehat{{D}}$ of origin datasets can be generated from the artifact-balanced distribution by selecting samples with $S = Y$, i.e., the selection intention matches with the label. We use $P(\\cdot )$ to represent the probability on $\\widehat{{D}}$ and use $Q(\\cdot )$ for ${D}$.","We also make some assumptions about the artifact-balanced distribution. The first one is that the label is independent with the artifact in the hypothesis, defined as follows,","The second one is that the selection intention is independent with $X$ and $Y$ when the annotation artifact is given,","And we can prove the equivalence of training with weight $\\frac{1}{P(Y|A)}$ and fitting the artifact-balanced distribution. We first present an equation as follows,","Without loss of generality, we can assume $Q(Y=i)=\\frac{1}{3}~(i=0,1,2)$ and get that,","With the above derivation, we can prove the equivalence like following,","As $Q(S=Y)$ is just a constant, training with the loss is equivalent to fitting the artifact-balanced distribution. Given hypotheses variable H, the probability $P(Y|A)$ can be replaced by $P(Y|H)$ since the predictive ability of hypotheses totally comes from the annotation artifacts, and we can have $w=\\frac{1}{P(Y|H)}$ as weights during training."]},{"idx":13,"heading":"Experiment Setting ::: Hard-Easy Datasets Setting","paragraphs":["For SNLI, we use Hard released by BIBREF2. For MMatch, we manually partition the set using fastText BIBREF18. And we summarize the size of the datasets used in Hard-Easy Testing below."]},{"idx":14,"heading":"Experiment Setting ::: Experiment Setup","paragraphs":["For DIIN, we use settings same as BIBREF5 but do not use syntactical features. Priors of labels are normalized to be the same. For hypothesis-only-model, we implement a naïve model with one LSTM layer and a three-layer MLP behind, implemented with Keras and Tensorflow backend BIBREF16. We use the 300-dimensional GloVe embeddings trained on the Common Crawl 840B tokens dataset BIBREF19 and keep them fixed during training. Batch Normalization BIBREF17 are applied after every hidden layer in MLP and we use Dropout BIBREF20 with rate 0.5 after the last hidden layer. We use RMSPropBIBREF21 as optimizer and set the learning rate as 1e-3. We set the gradient clipping to 1.0 and the batch size to 256."]},{"idx":15,"heading":"Figures and Tables","paragraphs":["Table 1: Evaluation Results of Hyp and Norm. Baseline refers to the model trained and validated without using weights. Hard, Easy refers to the Hard-Easy Testing generated from the testing set corresponding to the Trainset column. Results of Hyp are the average numbers of five runs with different random initialization. We report AUC for Hyp and ACC for Norm. “*” indicates where normal-model are better than the baseline."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1909.06708","title":"Hint-Based Training for Non-Autoregressive Machine Translation","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Due to the unparallelizable nature of the autoregressive factorization, AutoRegressive Translation (ART) models have to generate tokens sequentially during decoding and thus suffer from high inference latency. Non-AutoRegressive Translation (NART) models were proposed to reduce the inference time, but could only achieve inferior translation accuracy. In this paper, we proposed a novel approach to leveraging the hints from hidden states and word alignments to help the training of NART models. The results achieve significant improvement over previous NART models for the WMT14 En-De and De-En datasets and are even comparable to a strong LSTM-based ART baseline but one order of magnitude faster in inference."]},{"idx":1,"heading":"Introduction","paragraphs":["Neural machine translation has attracted much attention in recent years BIBREF0, BIBREF1, BIBREF2, BIBREF3. Given a sentence $x=(x_1, \\dots ,x_{T_x})$ from the source language, the straight-forward way for translation is to generate the target words $y=(y_1, \\dots , y_{T_y})$ one by one from left to right. This is also known as the AutoRegressive Translation (ART) models, in which the joint probability is decomposed into a chain of conditional probabilities:","While the ART models have achieved great success in terms of translation quality, the time consumption during inference is still far away from satisfactory. During training, the predictions at different positions can be estimated in parallel since the ground truth pair $(x,y)$ is exposed to the model. However, during inference, the model has to generate tokens sequentially as $y_{<t}$ must be inferred on the fly. Such autoregressive behavior becomes the bottleneck of the computational time BIBREF4.","In order to speed up the inference process, a line of works begin to develop non-autoregressive translation models. These models break the autoregressive dependency by decomposing the joint probability with","The lost of autoregressive dependency largely hurt the consistency of the output sentences, increase the difficulty in the learning process and thus lead to a low quality translation. Previous works mainly focus on adding different components into the NART model to improve the expressiveness of the network structure to overcome the loss of autoregressive dependency BIBREF5, BIBREF6, BIBREF7. However, the computational overhead of new components will hurt the inference speed, contradicting with the goal of the NART models: to parallelize and speed up neural machine translation models.","To tackle this, we proposed a novel hint-based method for NART model training. We first investigate the causes of the poor performance of the NART model. Comparing with the ART model, we find that: (1) the positions where the NART model outputs incoherent tokens will have very high hidden states similarity; (2) the attention distributions of the NART model are more ambiguous than those of ART model. Therefore, we design two kinds of hints from the hidden states and attention distributions of the ART model to help the training of the NART model. The experimental results show that our model achieves significant improvement over the NART baseline models and is even comparable to a strong ART baseline in BIBREF4."]},{"idx":2,"heading":"Approach","paragraphs":["In this section, we first describe the observations on the ART and NART models, and then discuss what kinds of information can be used as hints to help the training of the NART model. We follow the network structure in BIBREF8, use a copy of the source sentence as decoder input, remove the attention masks in decoder self-attention layers and add a positional attention layer as suggested in BIBREF5. We provide a visualization of ART and NART models we used in Figure FIGREF11 and a detailed description of the model structure in Appendix."]},{"idx":3,"heading":"Approach ::: Observation: Illed States and Attentions","paragraphs":["According to the case study in BIBREF5, the translations of the NART models contain incoherent phrases (e.g. repetitive words) and miss meaningful tokens on the source side, while these patterns do not commonly appear in ART models. After some empirical study, we find two non-obvious facts that lead to this phenomenon.","First, we visualize the cosine similarities between decoder hidden states of a certain layer in both ART and NART models for sampled cases. Mathematically, for a set of hidden states $r_1, \\ldots , r_T$, the pairwise cosine similarity can be derived by $\\cos _{ij} = {\\left<r_i, r_j\\right>}/{(\\Vert r_i\\Vert \\cdot \\Vert r_j\\Vert )}.$ We then plot the heatmap of the resulting matrix $\\cos $. A typical example is shown in Figure FIGREF4, where the cosine similarities in the NART model are larger than those of the ART model, indicating that the hidden states across positions in the NART model are “similar”. Positions with highly-correlated hidden states tend to generate the same word and make the NART model output repetitive tokens, e.g., the yellow area on the top-left of Figure FIGREF4(b), while this does not happen in the ART model (Figure FIGREF4(a)). According to our statistics, 70% of the cosine similarities between hidden states in the ART model are less than 0.25, and 95% are less than 0.5.","Second, we visualize the encoder-decoder attentions for sampled cases, shown in Figure FIGREF6. Good attentions between the source and target sentences are usually considered to lead to accurate translation while poor ones may cause wrong output tokens BIBREF0. In Figure FIGREF6(b), the attentions of the ART model almost covers all source tokens, while the attentions of the NART model do not cover “farm” but with two “morning”. This directly makes the translation result worse in the NART model. These phenomena inspire us to use the intermediate hidden information in the ART model to guide the learning process of the NART model."]},{"idx":4,"heading":"Approach ::: Hints from the ART teacher Model","paragraphs":["Our study motivates us to leverage the intermediate hidden information from an ART model to improve the NART model. We focus on how to define hints from a well-trained ART teacher model and use it to guide the training process of a NART student model. We study layer-to-layer hints and assume both the teacher and student models have an $M$-layer encoder and an $N$-layer decoder, despite the difference in stacked components.","Without the loss of generality, we discuss our method on a given paired sentence $(x,y)$. In real experiments, losses are averaged over all training data. For the teacher model, we use $a_{t,l,h}^\\mathit {tr}$ as the encoder-to-decoder attention distribution of $h$-th head in the $l$-th decoder layer at position $t$, and use $r_{t,l}^\\mathit {tr}$ as the output of the $l$-th decoder layer after feed forward network at position $t$. Correspondingly, $a_{t,l,h}^\\mathit {st}$ and $r_{t,l}^\\mathit {st}$ are used for the student model. We propose a hint-based training framework that contains two kinds of hints:"]},{"idx":5,"heading":"Approach ::: Hints from the ART teacher Model ::: Hints from hidden states","paragraphs":["The discrepancy of hidden states motivates us to use hidden states of the ART model as a hint for the learning process of the NART model. One straight-forward method is to regularize the $L_1$ or $L_2$ distance between each pair of hidden states in ART and NART models. However, since the network components are quite different in ART and NART models, applying the straight-forward regression on hidden states hurts the learning process and fails. Therefore, we design a more implicit loss to help the student refrain from the incoherent translation results by acting towards the teacher in the hidden-state level:","where $d_\\mathit {st} = \\cos (r_{s, l}^\\mathit {st},r_{t, l}^\\mathit {st})$, $d_\\mathit {tr} = \\cos (r_{s, l}^\\mathit {tr},r_{t, l}^\\mathit {tr})$, and $\\phi $ is a penalty function. In particular, we let","where $-1\\le \\gamma _\\mathit {st}, \\gamma _\\mathit {tr}\\le 1$ are two thresholds controlling whether to penalize or not. We design this loss since we only want to penalize hidden states that are highly similar in the NART model, but not similar in the ART model. We have tested several choices of $-\\log (1-d_\\mathit {st})$, e.g., $\\exp (d_\\mathit {st})$, from which we find similar experimental results."]},{"idx":6,"heading":"Approach ::: Hints from the ART teacher Model ::: Hints from word alignments","paragraphs":["We observe that meaningful words in the source sentence are sometimes untranslated by the NART model, and the corresponding positions often suffer from ambiguous attention distributions. Therefore, we use the word alignment information from the ART model to help the training of the NART model.","In particular, we minimize KL-divergence between the per-head encoder-to-decoder attention distributions of the teacher and the student to encourage the student to have similar word alignments to the teacher model, i.e.","Our final training loss $\\mathcal {L}$ is a weighted sum of two parts stated above and the negative log-likelihood loss $\\mathcal {L}_\\mathit {nll}$ defined on bilingual sentence pair $(x, y)$, i.e.","where $\\lambda $ and $\\mu $ are hyperparameters controlling the weight of different loss terms."]},{"idx":7,"heading":"Experiments ::: Experimental Settings","paragraphs":["The evaluation is on two widely used public machine translation datasets: IWSLT14 German-to-English (De-En) BIBREF9, BIBREF1 and WMT14 English-to-German (En-De) dataset BIBREF4, BIBREF10. To compare with previous works, we also reverse WMT14 English-to-German dataset and obtain WMT14 German-to-English dataset.","We pretrain Transformer BIBREF8 as the teacher model on each dataset, which achieves 33.26/27.30/31.29 in terms of BLEU BIBREF11 in IWSLT14 De-En, WMT14 En-De and De-En test sets. The student model shares the same number of layers in encoder/decoder, size of hidden states/embeddings and number of heads as the teacher models (Figure FIGREF11). Following BIBREF5, BIBREF12, we replace the target sentences by the decoded output of the teacher models.","Hyperparameters ($\\gamma _\\mathit {st}, \\gamma _\\mathit {tr}, \\lambda , \\mu $) for hint-based learning are determined to make the scales of three loss components similar after initialization. We also employ label smoothing of value $\\epsilon _\\mathit {ls}=0.1$ BIBREF13 in all experiments. We use Adam optimizer and follow the setting in BIBREF8. Models for WMT14/IWSLT14 tasks are trained on 8/1 NVIDIA M40 GPUs respectively. The model is based on the open-sourced tensor2tensor BIBREF14. More settings can be found in Appendix."]},{"idx":8,"heading":"Experiments ::: Inference","paragraphs":["During training, $T_y$ does not need to be predicted as the target sentence is given. During testing, we have to predict the length of the target sentence for each source sentence. In many languages, the length of the target sentence can be roughly estimated from the length of the source sentence. We choose a simple method to avoid the computational overhead, which uses input length to determine target sentence length: $T_y = T_x + C$, where $C$ is a constant bias determined by the average length differences between the source and target training sentences. We can also predict the target length ranging from $[(T_x+C)-B, (T_x+C)+B]$, where $B$ is the halfwidth. By doing this, we can obtain multiple translation results with different lengths. Note that we choose this method only to show the effectiveness of our proposed method and a more advanced length estimation method can be used to further improve the performance.","Once we have multiple translation results, we additionally use our ART teacher model to evaluate each result and select the one that achieves the highest probability. As the evaluation is fully parallelizable (since it is identical to the parallel training of the ART model), this rescoring operation will not hurt the non-autoregressive property of the NART model."]},{"idx":9,"heading":"Experiments ::: Experimental Results","paragraphs":["We compare our model with several baselines, including three ART models, the fertility based (FT) NART model BIBREF5, the deterministic iterative refinement based (IR) NART model BIBREF6, and the Latent Transformer BIBREF7 which is not fully non-autoregressive by incorporating an autoregressive sub-module in the NART model architecture.","The results are shown in the Table TABREF15. Across different datasets, our method achieves significant improvements over previous non-autoregressive models. Specifically, our method outperforms fertility based NART model with 6.54/7.11 BLEU score improvements on WMT En-De and De-En tasks in similar settings and achieves comparable results with state-of-the-art LSTM-based model on WMT En-De task. Furthermore, our model achieves a speedup of 30.2 (output a single sentence) or 17.8 (teacher rescoring) times over the ART counterparts. Note that our speedups significantly outperform all previous works, because of our lighter design of the NART model: without any computationally expensive module trying to improve the expressiveness.","We also visualize the hidden state cosine similarities and attention distributions for the NART model with hint-based training, as shown in Figure FIGREF4(c) and FIGREF6(c). With hints from hidden states, the hidden states similarities of the NART model decrease in general, and especially for the positions where the original NART model outputs incoherent phrases. The attention distribution of the NART model after hint-based training is more similar to the ART teacher model and less ambiguous comparing to the NART model without hints.","According to our empirical analysis, the percentage of repetitive words drops from 8.3% to 6.5% by our proposed methods on the IWSLT14 De-En test set, which is a 20%+ reduction. This shows that our proposed method effectively improve the quality of the translation outputs. We also provide several case studies in Appendix.","Finally, we conduct an ablation study on IWSLT14 De-En task. As shown in Table TABREF18, the hints from word alignments provide an improvement of about 1.6 BLEU points, and the hints from hidden states improve the results by about 0.8 BLEU points. We also test these models on a subsampled set whose source sentence lengths are at least 40. Our model outperforms the baseline model by more than 3 BLEU points (20.63 v.s. 17.48)."]},{"idx":10,"heading":"Conclusion","paragraphs":["In this paper, we proposed to use hints from a well-trained ART model to enhance the training of NART models. Our results on WMT14 En-De and De-En significantly outperform previous NART baselines, with one order of magnitude faster in inference than ART models. In the future, we will focus on designing new architectures and training methods for NART models to achieve comparable accuracy as ART models."]},{"idx":11,"heading":"Acknowledgment","paragraphs":["This work is supported by National Key R&D Program of China (2018YFB1402600), NSFC (61573026) and BJNSF (L172037) and a grant from Microsoft Research Asia. We would like to thank the anonymous reviewers for their valuable comments on our paper."]},{"idx":12,"heading":"Figures and Tables","paragraphs":["Figure 1: Case study: the above three figures visualize the hidden state cosine similarities of different models. The axes correspond to the generated target tokens. Each pixel shows the cosine similarities cosij between the last layer hidden states of the i-th and j-th generated tokens, where the diagonal pixel will always be 1.0.","Figure 2: Case study: the above three figures visualize the encoder-decoder attention weights of different models. The x-axis and y-axis correspond to the source and generated target tokens respectively. The attention distribution is from a single head of the third layer encoder-decoder attention, which is the most informative one according to our observation. Each pixel shows attention weights αij between the i-th source token and j-th target token.","Figure 3: Hint-based training from ART model to NART model.","Table 1: Performance on WMT14 En-De, De-En and IWSLT14 De-En tasks. “/” means non-reportable.","Table 2: Ablation studies on IWSLT14 De-En. Results are BLEU scores without teacher rescoring.","Table 3: Cases on IWSLT14 De-En."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1909.07575","title":"Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translation","sections":[{"idx":0,"heading":"Abstract","paragraphs":["End-to-end speech translation, a hot topic in recent years, aims to translate a segment of audio into a specific language with an end-to-end model. Conventional approaches employ multi-task learning and pre-training methods for this task, but they suffer from the huge gap between pre-training and fine-tuning. To address these issues, we propose a Tandem Connectionist Encoding Network (TCEN) which bridges the gap by reusing all subnets in fine-tuning, keeping the roles of subnets consistent, and pre-training the attention module. Furthermore, we propose two simple but effective methods to guarantee the speech encoder outputs and the MT encoder inputs are consistent in terms of semantic representation and sequence length. Experimental results show that our model outperforms baselines 2.2 BLEU on a large benchmark dataset."]},{"idx":1,"heading":"Introduction","paragraphs":["Speech-to-Text translation (ST) is essential for a wide range of scenarios: for example in emergency calls, where agents have to respond emergent requests in a foreign language BIBREF0; or in online courses, where audiences and speakers use different languages BIBREF1. To tackle this problem, existing approaches can be categorized into cascaded method BIBREF2, BIBREF3, where a machine translation (MT) model translates outputs of an automatic speech recognition (ASR) system into target language, and end-to-end method BIBREF4, BIBREF5, where a single model learns acoustic frames to target word sequence mappings in one step towards the final objective of interest. Although the cascaded model remains the dominant approach due to its better performance, the end-to-end method becomes more and more popular because it has lower latency by avoiding inferences with two models and rectifies the error propagation in theory.","Since it is hard to obtain a large-scale ST dataset, multi-task learning BIBREF5, BIBREF6 and pre-training techniques BIBREF7 have been applied to end-to-end ST model to leverage large-scale datasets of ASR and MT. A common practice is to pre-train two encoder-decoder models for ASR and MT respectively, and then initialize the ST model with the encoder of the ASR model and the decoder of the MT model. Subsequently, the ST model is optimized with the multi-task learning by weighing the losses of ASR, MT, and ST. This approach, however, causes a huge gap between pre-training and fine-tuning, which are summarized into three folds:","Subnet Waste: The ST system just reuses the ASR encoder and the MT decoder, while discards other pre-trained subnets, such as the MT encoder. Consequently, valuable semantic information captured by the MT encoder cannot be inherited by the final ST system.","Role Mismatch: The speech encoder plays different roles in pre-training and fine-tuning. The encoder is a pure acoustic model in pre-training, while it has to extract semantic and linguistic features additionally in fine-tuning, which significantly increases the learning difficulty.","Non-pre-trained Attention Module: Previous work BIBREF6 trains attention modules for ASR, MT and ST respectively, hence, the attention module of ST does not benefit from the pre-training.","To address these issues, we propose a Tandem Connectionist Encoding Network (TCEN), which is able to reuse all subnets in pre-training, keep the roles of subnets consistent, and pre-train the attention module. Concretely, the TCEN consists of three components, a speech encoder, a text encoder, and a target text decoder. Different from the previous work that pre-trains an encoder-decoder based ASR model, we only pre-train an ASR encoder by optimizing the Connectionist Temporal Classification (CTC) BIBREF8 objective function. In this way, the additional decoder of ASR is not required while keeping the ability to read acoustic features into the source language space by the speech encoder. Besides, the text encoder and decoder can be pre-trained on a large MT dataset. After that, we employ common used multi-task learning method to jointly learn ASR, MT and ST tasks.","Compared to prior works, the encoder of TCEN is a concatenation of an ASR encoder and an MT encoder and our model does not have an ASR decoder, so the subnet waste issue is solved. Furthermore, the two encoders work at tandem, disentangling acoustic feature extraction and linguistic feature extraction, ensuring the role consistency between pre-training and fine-tuning. Moreover, we reuse the pre-trained MT attention module in ST, so we can leverage the alignment information learned in pre-training.","Since the text encoder consumes word embeddings of plausible texts in MT task but uses speech encoder outputs in ST task, another question is how one guarantees the speech encoder outputs are consistent with the word embeddings. We further modify our model to achieve semantic consistency and length consistency. Specifically, (1) the projection matrix at the CTC classification layer for ASR is shared with the word embedding matrix, ensuring that they are mapped to the same latent space, and (2) the length of the speech encoder output is proportional to the length of the input frame, so it is much longer than a natural sentence. To bridge the length gap, source sentences in MT are lengthened by adding word repetitions and blank tokens to mimic the CTC output sequences.","We conduct comprehensive experiments on the IWSLT18 speech translation benchmark BIBREF1, demonstrating the effectiveness of each component. Our model is significantly better than previous methods by 3.6 and 2.2 BLEU scores for the subword-level decoding and character-level decoding strategies, respectively.","Our contributions are three-folds: 1) we shed light on why previous ST models cannot sufficiently utilize the knowledge learned from the pre-training process; 2) we propose a new ST model, which alleviates shortcomings in existing methods; and 3) we empirically evaluate the proposed model on a large-scale public dataset."]},{"idx":2,"heading":"Background ::: Problem Formulation","paragraphs":["End-to-end speech translation aims to translate a piece of audio into a target-language translation in one step. The raw speech signals are usually converted to sequences of acoustic features, e.g. Mel filterbank features. Here, we define the speech feature sequence as $\\mathbf {x} = (x_1, \\cdots , x_{T_x})$.The transcription and translation sequences are denoted as $\\mathbf {y^{s}} = (y_1^{s}, \\cdots , y_{T_s}^{s})$, and $\\mathbf {y^{t}} = (y_1^{t}, \\cdots , y_{T_t}^{t})$ repectively. Each symbol in $\\mathbf {y^{s}}$ or $\\mathbf {y^{t}}$ is an integer index of the symbol in a vocabulary $V_{src}$ or $V_{trg}$ respectively (e.g. $y^s_i=k, k\\in [0, |V_{src}|-1]$). In this work, we suppose that an ASR dataset, an MT dataset, and a ST dataset are available, denoted as $\\mathcal {A} = \\lbrace (\\mathbf {x_i}, \\mathbf {y^{s}_i})\\rbrace _{i=0}^I$, $\\mathcal {M} =\\lbrace (\\mathbf {y^{s}_j}, \\mathbf {y^{t}_j})\\rbrace _{j=0}^J$ and $ \\mathcal {S} =\\lbrace (\\mathbf {x_l}, \\mathbf {y^{t}_l})\\rbrace _{l=0}^L$ respectively. Given a new piece of audio $\\mathbf {x}$, our goal is to learn an end to end model to generate a translation sentence $\\mathbf {y^{t}}$ without generating an intermediate result $\\mathbf {y^{s}}$."]},{"idx":3,"heading":"Background ::: Multi-Task Learning and Pre-training for ST","paragraphs":["To leverage large scale ASR and MT data, multi-task learning and pre-training techniques are widely employed to improve the ST system. As shown in Figure FIGREF4, there are three popular multi-task strategies for ST, including 1) one-to-many setting, in which a speech encoder is shared between ASR and ST tasks; 2) many-to-one setting in which a decoder is shared between MT and ST tasks; and 3) many-to-many setting where both the encoder and decoder are shared.","A many-to-many multi-task model contains two encoders as well as two decoders. It can be jointly trained on ASR, MT, and ST tasks. As the attention module is task-specific, three attentions are defined.","Usually, the size of $\\mathcal {A}$ and $\\mathcal {M}$ is much larger than $\\mathcal {S}$. Therefore, the common training practice is to pre-train the model on ASR and MT tasks and then fine-tune it with a multi-task learning manner. However, as aforementioned, this method suffers from subnet waste, role mismatch and non-pre-trained attention issues, which severely limits the end-to-end ST performance."]},{"idx":4,"heading":"Our method","paragraphs":["In this section, we first introduce the architecture of TCEN, which consists of two encoders connected in tandem, and one decoder with an attention module. Then we give the pre-training and fine-tuning strategy for TCEN. Finally, we propose our solutions for semantic and length inconsistency problems, which are caused by multi-task learning."]},{"idx":5,"heading":"Our method ::: TCEN Architecture","paragraphs":["Figure FIGREF5 sketches the overall architecture of TCEN, including a speech encoder $enc_s$, a text encoder $enc_t$ and a decoder $dec$ with an attention module $att$. During training, the $enc_s$ acts like an acoustic model which reads the input $\\mathbf {x}$ to word or subword representations $\\mathbf {h^s}$, then $enc_t$ learns high-level linguistic knowledge into hidden representations $\\mathbf {h^t}$. Finally, the $dec$ defines a distribution probability over target words. The advantage of our architecture is that two encoders disentangle acoustic feature extraction and linguistic feature extraction, making sure that valuable knowledge learned from ASR and MT tasks can be effectively leveraged for ST training. Besides, every module in pre-training can be utilized in fine-tuning, alleviating the subnet waste problem.","Follow BIBREF9 inaguma2018speech, we use CNN-BiLSTM architecture to build our model. Specifically, the input features $\\mathbf {x}$ are organized as a sequence of feature vectors in length $T_x$. Then, $\\mathbf {x}$ is passed into a stack of two convolutional layers followed by max-pooling:","where $\\mathbf {v}^{(l-1)}$ is feature maps in last layer and $\\mathbf {W}^{(l)}$ is the filter. The max-pooling layers downsample the sequence in length by a total factor of four. The down-sampled feature sequence is further fed into a stack of five bidirectional $d$-dimensional LSTM layers:","where $[;]$ denotes the vector concatenation. The final output representation from the speech encoder is denoted as $\\mathbf {h^s}=(h^s_1, \\cdots , h^s_{\\frac{T_x}{4}})$, where $h_i^s \\in \\mathbb {R}^d$.","The text encoder $enc_t$ consists of two bidirectional LSTM layers. In ST task, $enc_t$ accepts speech encoder output $\\mathbf {h}^s$ as input. While in MT, $enc_t$ consumes the word embedding representation $\\mathbf {e^s}$ derived from $\\mathbf {y^s}$, where each element $e^s_i$ is computed by choosing the $y_i^s$-th vector from the source embedding matrix $W_{E^s}$. The goal of $enc_t$ is to extract high-level linguistic features like syntactic features or semantic features from lower level subword representations $\\mathbf {h}^s$ or $\\mathbf {e}^s$. Since $\\mathbf {h}^s$ and $\\mathbf {e}^s$ belong to different latent space and have different lengths, there remain semantic and length inconsistency problems. We will provide our solutions in Section SECREF21. The output sequence of $enc_t$ is denoted as $\\mathbf {h}^t$.","The decoder is defined as two unidirectional LSTM layers with an additive attention $att$. It predicts target sequence $\\mathbf {y^{t}}$ by estimating conditional probability $P(\\mathbf {y^{t}}|\\mathbf {x})$:","Here, $z_k$ is the the hidden state of the deocder RNN at $k$ step and $c_k$ is a time-dependent context vector computed by the attention $att$."]},{"idx":6,"heading":"Our method ::: Training Procedure","paragraphs":["Following previous work, we split the training procedure to pre-training and fine-tuning stages. In pre-training stage, the speech encoder $enc_s$ is trained towards CTC objective using dataset $\\mathcal {A}$, while the text encoder $enc_t$ and the decoder $dec$ are trained on MT dataset $\\mathcal {M}$. In fine-tuning stage, we jointly train the model on ASR, MT, and ST tasks."]},{"idx":7,"heading":"Our method ::: Training Procedure ::: Pre-training","paragraphs":["To sufficiently utilize the large dataset $\\mathcal {A}$ and $\\mathcal {M}$, the model is pre-trained on CTC-based ASR task and MT task in the pre-training stage.","For ASR task, in order to get rid of the requirement for decoder and enable the $enc_s$ to generate subword representation, we leverage connectionist temporal classification (CTC) BIBREF8 loss to train the speech encoder.","Given an input $\\mathbf {x}$, $enc_s$ emits a sequence of hidden vectors $\\mathbf {h^s}$, then a softmax classification layer predicts a CTC path $\\mathbf {\\pi }$, where $\\pi _t \\in V_{src} \\cup $ {`-'} is the observing label at particular RNN step $t$, and `-' is the blank token representing no observed labels:","where $W_{ctc} \\in \\mathbb {R}^{d \\times (|V_{src}|+1)}$ is the weight matrix in the classification layer and $T$ is the total length of encoder RNN.","A legal CTC path $\\mathbf {\\pi }$ is a variation of the source transcription $\\mathbf {y}^s$ by allowing occurrences of blank tokens and repetitions, as shown in Table TABREF14. For each transcription $\\mathbf {y}$, there exist many legal CTC paths in length $T$. The CTC objective trains the model to maximize the probability of observing the golden sequence $\\mathbf {y}^s$, which is calculated by summing the probabilities of all possible legal paths:","where $\\Phi _T(y)$ is the set of all legal CTC paths for sequence $\\mathbf {y}$ with length $T$. The loss can be easily computed using forward-backward algorithm. More details about CTC are provided in supplementary material.","For MT task, we use the cross-entropy loss as the training objective. During training, $\\mathbf {y^s}$ is converted to embedding vectors $\\mathbf {e^s}$ through embedding layer $W_{E^s}$, then $enc_t$ consumes $\\mathbf {e^s}$ and pass the output $\\mathbf {h^t}$ to decoder. The objective function is defined as:"]},{"idx":8,"heading":"Our method ::: Training Procedure ::: Fine-tune","paragraphs":["In fine-tune stage, we jointly update the model on ASR, MT, and ST tasks. The training for ASR and MT follows the same process as it was in pre-training stage.","For ST task, the $enc_s$ reads the input $\\mathbf {x}$ and generates $\\mathbf {h^s}$, then $enc_t$ learns high-level linguistic knowledge into $\\mathbf {h^t}$. Finally, the $dec$ predicts the target sentence. The ST loss function is defined as:","Following the update strategy proposed by BIBREF11 luong2015multi, we allocate a different training ratio $\\alpha _i$ for each task. When switching between tasks, we select randomly a new task $i$ with probability $\\frac{\\alpha _i}{\\sum _{j}\\alpha _{j}}$."]},{"idx":9,"heading":"Our method ::: Subnet-Consistency","paragraphs":["Our model keeps role consistency between pre-training and fine-tuning by connecting two encoders for ST task. However, this leads to some new problems: 1) The text encoder consumes $\\mathbf {e^s}$ during MT training, while it accepts $\\mathbf {h^s}$ during ST training. However, $\\mathbf {e^s}$ and $\\mathbf {h^s}$ may not follow the same distribution, resulting in the semantic inconsistency. 2) Besides, the length of $\\mathbf {h^s}$ is not the same order of magnitude with the length of $\\mathbf {e^s}$, resulting in the length inconsistency.","In response to the above two challenges, we propose two countermeasures: 1) We share weights between CTC classification layer and source-end word embedding layer during training of ASR and MT, encouraging $\\mathbf {e^s}$ and $\\mathbf {h^s}$ in the same space. 2)We feed the text encoder source sentences in the format of CTC path, which are generated from a seq2seq model, making it more robust toward long inputs."]},{"idx":10,"heading":"Our method ::: Subnet-Consistency ::: Semantic Consistency","paragraphs":["As shown in Figure FIGREF5, during multi-task training, two different hidden features will be fed into the text encoder $enc_t$: the embedding representation $\\mathbf {e}^s$ in MT task, and the $enc_s$ output $\\mathbf {h^s}$ in ST task. Without any regularization, they may belong to different latent spaces. Due to the space gap, the $enc_t$ has to compromise between two tasks, limiting its performance on individual tasks.","To bridge the space gap, our idea is to pull $\\mathbf {h^s}$ into the latent space where $\\mathbf {e}^s$ belong. Specifically, we share the weight $W_{ctc}$ in CTC classification layer with the source embedding weights $W_{E^s}$, which means $W_{ctc} = W_{E^s}$. In this way, when predicting the CTC path $\\mathbf {\\pi }$, the probability of observing the particular label $w_i \\in V_{src}\\cup ${`-'} at time step $t$, $p(\\pi _t=w_i|\\mathbf {x})$, is computed by normalizing the product of hidden vector $h_t^s$ and the $i$-th vector in $W_{E^s}$:","The loss function closes the distance between $h^s_t$ and golden embedding vector, encouraging $\\mathbf {h}^s$ have the same distribution with $\\mathbf {e}^s$."]},{"idx":11,"heading":"Our method ::: Subnet-Consistency ::: Length Consistency","paragraphs":["Another existing problem is length inconsistency. The length of the sequence $\\mathbf {h^s}$ is proportional to the length of the input frame $\\mathbf {x}$, which is much longer than the length of $\\mathbf {e^s}$. To solve this problem, we train an RNN-based seq2seq model to transform normal source sentences to noisy sentences in CTC path format, and replace standard MT with denoising MT for multi-tasking.","Specifically, we first train a CTC ASR model based on dataset $\\mathcal {A} = \\lbrace (\\mathbf {x}_i, \\mathbf {y}^s_i)\\rbrace _{i=0}^{I}$, and generate a CTC-path $\\mathbf {\\pi }_i$ for each audio $\\mathbf {x}_i$ by greedy decoding. Then we define an operation $S(\\cdot )$, which converts a CTC path $\\mathbf {\\pi }$ to a sequence of the unique tokens $\\mathbf {u}$ and a sequence of repetition times for each token $\\mathbf {l}$, denoted as $S(\\mathbf {\\pi }) = (\\mathbf {u}, \\mathbf {l})$. Notably, the operation is reversible, meaning that $S^{-1} (\\mathbf {u}, \\mathbf {l})=\\mathbf {\\pi }$. We use the example $\\mathbf {\\pi _1}$ in Table TABREF14 and show the corresponding $\\mathbf {u}$ and $\\mathbf {l}$ in Table TABREF24.","Then we build a dataset $\\mathcal {P} = \\lbrace (\\mathbf {y^s}_i, \\mathbf {u}_i, \\mathbf {l}_i)\\rbrace _{i=0}^{I}$ by decoding all the audio pieces in $\\mathcal {A}$ and transform the resulting path by the operation $S(\\cdot )$. After that, we train a seq2seq model, as shown in Figure FIGREF25, which takes $ \\mathbf {y^s}_i$ as input and decodes $\\mathbf {u}_i, \\mathbf {l}_i$ as outputs. With the seq2seq model, a noisy MT dataset $\\mathcal {M}^{\\prime }=\\lbrace (\\mathbf {\\pi }_l, \\mathbf {y^t}_l)\\rbrace _{l=0}^{L}$ is obtained by converting every source sentence $\\mathbf {y^s}_i \\in \\mathcal {M}$ to $\\mathbf {\\pi _i}$, where $\\mathbf {\\pi }_i = S^{-1}(\\mathbf {u}_i, \\mathbf {l}_i)$. We did not use the standard seq2seq model which takes $\\mathbf {y^s}$ as input and generates $\\mathbf {\\pi }$ directly, since there are too many blank tokens `-' in $\\mathbf {\\pi }$ and the model tends to generate a long sequence with only blank tokens. During MT training, we randomly sample text pairs from $\\mathcal {M}^{\\prime }$ and $\\mathcal {M}$ according to a hyper-parameter $k$. After tuning on the validation set, about $30\\%$ pairs are sampled from $\\mathcal {M}^{\\prime }$. In this way, the $enc_t$ is more robust toward the longer inputs given by the $enc_s$."]},{"idx":12,"heading":"Experiments","paragraphs":["We conduct experiments on the IWSLT18 speech translation task BIBREF1. Since IWSLT participators use different data pre-processing methods, we reproduce several competitive baselines based on the ESPnet BIBREF12 for a fair comparison."]},{"idx":13,"heading":"Experiments ::: Dataset ::: Speech translation data:","paragraphs":["The organizer provides a speech translation corpus extracting from the TED talk (ST-TED), which consists of raw English wave files, English transcriptions, and aligned German translations. The corpus contains 272 hours of English speech with 171k segments. We split 2k segments from the corpus as dev set and tst2010, tst2013, tst2014, tst2015 are used as test sets.","Speech recognition data: Aside from ST-TED, TED-LIUM2 corpus BIBREF13 is provided as speech recognition data, which contains 207 hours of English speech and 93k transcript sentences.","Text translation data: We use transcription and translation pairs in the ST-TED corpus and WIT3 as in-domain MT data, which contains 130k and 200k sentence pairs respectively. WMT2018 is used as out-of-domain training data which consists of 41M sentence pairs.","Data preprocessing: For speech data, the utterances are segmented into multiple frames with a 25 ms window size and a 10 ms step size. Then we extract 80-channel log-Mel filter bank and 3-dimensional pitch features using Kaldi BIBREF14, resulting in 83-dimensional input features. We normalize them by the mean and the standard deviation on the whole training set. The utterances with more than 3000 frames are discarded. The transcripts in ST-TED are in true-case with punctuation while in TED-LIUM2, transcripts are in lower-case and unpunctuated. Thus, we lowercase all the sentences and remove the punctuation to keep consistent. To increase the amount of training data, we perform speed perturbation on the raw signals with speed factors 0.9 and 1.1. For the text translation data, sentences longer than 80 words or shorter than 10 words are removed. Besides, we discard pairs whose length ratio between source and target sentence is smaller than 0.5 or larger than 2.0. Word tokenization is performed using the Moses scripts and both English and German words are in lower-case.","We use two different sets of vocabulary for our experiments. For the subword experiments, both English and German vocabularies are generated using sentencepiece BIBREF15 with a fixed size of 5k tokens. BIBREF9 inaguma2018speech show that increasing the vocabulary size is not helpful for ST task. For the character experiments, both English and German sentences are represented in the character level.","For evaluation, we segment each audio with the LIUM SpkDiarization tool BIBREF16 and then perform MWER segmentation with RWTH toolkit BIBREF17. We use lowercase BLEU as evaluation metric."]},{"idx":14,"heading":"Experiments ::: Baseline Models and Implementation","paragraphs":["We compare our method with following baselines.","Vanilla ST baseline: The vanilla ST BIBREF9 has only a speech encoder and a decoder. It is trained from scratch on the ST-TED corpus.","Pre-training baselines: We conduct three pre-training baseline experiments: 1) encoder pre-training, in which the ST encoder is initialized from an ASR model; 2) decoder pre-training, in which the ST decoder is initialized from an MT model; and 3) encoder-decoder pre-training, where both the encoder and decoder are pre-trained. The ASR model has the same architecture with vanilla ST model, trained on the mixture of ST-TED and TED-LIUM2 corpus. The MT model has a text encoder and decoder with the same architecture of which in TCEN. It is first trained on WMT data (out-of-domain) and then fine-tuned on in-domain data.","Multi-task baselines: We also conduct three multi-task baseline experiments including one-to-many setting, many-to-one setting, and many-to-many setting. In the first two settings, we train the model with $\\alpha _{st}=0.75$ while $\\alpha _{asr}=0.25$ or $\\alpha _{mt}=0.25$. For many-to-many setting, we use $\\alpha _{st}=0.6, \\alpha _{asr}=0.2$ and $\\alpha _{mt}=0.2$.. For MT task, we use only in-domain data.","Many-to-many+pre-training: We train a many-to-many multi-task model where the encoders and decoders are derived from pre-trained ASR and MT models. Triangle+pre-train: BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 proposed a triangle multi-task strategy for speech translation. Their model solves the subnet waste issue by concatenating an ST decoder to an ASR encoder-decoder model. Notably, their ST decoder can consume representations from the speech encoder as well as the ASR decoder. For a fair comparison, the speech encoder and the ASR decoder are initialized from the pre-trained ASR model. The Triangle model is fine-tuned under their multi-task manner.","All our baselines as well as TCEN are implemented based on ESPnet BIBREF12, the RNN size is set as $d=1024$ for all models. We use a dropout of 0.3 for embeddings and encoders, and train using Adadelta with initial learning rate of 1.0 for a maximum of 10 epochs.","For training of TCEN, we set $\\alpha _{asr}=0.2$ and $\\alpha _{mt}=0.8$ in the pre-training stage, since the MT dataset is much larger than ASR dataset. For fine-tune, we use $\\alpha _{st}=0.6, \\alpha _{asr}=0.2$ and $\\alpha _{mt}=0.2$, same as the `many-to-many' baseline.","For testing, we select the model with the best accuracy on speech translation task on dev set. At inference time, we use a beam size of 10, and the beam scores include length normalization with a weight of 0.2."]},{"idx":15,"heading":"Experiments ::: Experimental Results","paragraphs":["Table TABREF29 shows the results on four test sets as well as the average performance. Our method significantly outperforms the strong `many-to-many+pretrain' baseline by 3.6 and 2.2 BLEU scores respectively, indicating the proposed method is very effective that substantially improves the translation quality. Besides, both pre-training and multi-task learning can improve translation quality, and the pre-training settings (2nd-4th rows) are more effective compared to multi-task settings (5th-8th rows). We observe a performance degradation in the `triangle+pretrain' baseline. Compared to our method, where the decoder receives higher-level syntactic and semantic linguistic knowledge extracted from text encoder, their ASR decoder can only provide lower word-level linguistic information. Besides, since their model lacks text encoder and the architecture of ST decoder is different from MT decoder, their model cannot utilize the large-scale MT data in all the training stages. Interestingly, we find that the char-level models outperform the subword-level models in all settings, especially in vanilla baseline. A similar phenomenon is observed by BIBREF6 berard2018end. A possible explanation is that learning the alignments between speech frames and subword units in another language is notoriously difficult. Our method can bring more gains in the subword setting since our model is good at learning the text-to-text alignment and the subword-level alignment is more helpful to the translation quality."]},{"idx":16,"heading":"Experiments ::: Discussion ::: Ablation Study","paragraphs":["To better understand the contribution of each component, we perform an ablation study on subword-level experiments. The results are shown in Table TABREF37. In `-MT noise' setting, we do not add noise to source sentences for MT. In `-weight sharing' setting, we use different parameters in CTC classification layer and source embedding layer. These two experiments prove that both weight sharing and using noisy MT input benefit to the final translation quality. Performance degrades more in `-weight sharing', indicating the semantic consistency contributes more to our model. In the `-pretrain' experiment, we remove the pre-training stage and directly update the model on three tasks, leading to a dramatic decrease on BLEU score, indicating the pre-training is an indispensable step for end-to-end ST."]},{"idx":17,"heading":"Experiments ::: Discussion ::: Learning Curve","paragraphs":["It is interesting to investigate why our method is superior to baselines. We find that TCEN achieves a higher final result owing to a better start-point in fine-tuning. Figure FIGREF39 provides learning curves of subword accuracy on validation set. The x-axis denotes the fine-tuning training steps. The vanilla model starts at a low accuracy, because its networks are not pre-trained on the ASR and MT data. The trends of our model and `many-to-many+pretrain' are similar, but our model outperforms it about five points in the whole fine-tuning process. It indicates that the gain comes from bridging the gap between pre-training and fine-tuning rather than a better fine-tuning process."]},{"idx":18,"heading":"Experiments ::: Discussion ::: Compared with a Cascaded System","paragraphs":["Table TABREF29 compares our model with end-to-end baselines. Here, we compare our model with cascaded systems. We build a cascaded system by combining the ASR model and MT model used in pre-training baseline. Word error rate (WER) of the ASR system and BLEU score of the MT system are reported in the supplementary material. In addition to a simple combination of the ASR and MT systems, we also re-segment the ASR outputs before feeding to the MT system, denoted as cascaded+re-seg. Specifically, we train a seq2seq model BIBREF19 on the MT dataset, where the source side is a no punctuation sentence and the target side is a natural sentence. After that, we use the seq2seq model to add sentence boundaries and punctuation on ASR outputs. Experimental results are shown in Table TABREF41. Our end-to-end model outperforms the simple cascaded model over 2 BLEU scores, and it achieves a comparable performance with the cascaded model combining with a sentence re-segment model."]},{"idx":19,"heading":"Related Work","paragraphs":["Early works conduct speech translation in a pipeline manner BIBREF2, BIBREF20, where the ASR output lattices are fed into an MT system to generate target sentences. HMM BIBREF21, DenseNet BIBREF22, TDNN BIBREF23 are commonly used ASR systems, while RNN with attention BIBREF19 and Transformer BIBREF10 are top choices for MT. To enhance the robustness of the NMT model towards ASR errors, BIBREF24 DBLP:conf/eacl/TsvetkovMD14 and BIBREF25 DBLP:conf/asru/ChenHHL17 propose to simulate the noise in training and inference.","To avoid error propagation and high latency issues, recent works propose translating the acoustic speech into text in target language without yielding the source transcription BIBREF4. Since ST data is scarce, pre-training BIBREF7, multi-task learning BIBREF4, BIBREF6, curriculum learning BIBREF26, attention-passing BIBREF27, and knowledge distillation BIBREF28, BIBREF29 strategies have been explored to utilize ASR data and MT data. Specifically, BIBREF5 DBLP:conf/interspeech/WeissCJWC17 show improvements of performance by training the ST model jointly with the ASR and the MT model. BIBREF6 berard2018end observe faster convergence and better results due to pre-training and multi-task learning on a larger dataset. BIBREF7 DBLP:conf/naacl/BansalKLLG19 show that pre-training a speech encoder on one language can improve ST quality on a different source language. All of them follow the traditional multi-task training strategies. BIBREF26 DBLP:journals/corr/abs-1802-06003 propose to use curriculum learning to improve ST performance on syntactically distant language pairs. To effectively leverage transcriptions in ST data, BIBREF18 DBLP:conf/naacl/AnastasopoulosC18 augment the multi-task model where the target decoder receives information from the source decoder and they show improvements on low-resource speech translation. Their model just consumes ASR and ST data, in contrast, our work sufficiently utilizes the large-scale MT data to capture the rich semantic knowledge. BIBREF30 DBLP:conf/icassp/JiaJMWCCALW19 use pre-trained MT and text-to-speech (TTS) synthesis models to convert weakly supervised data into ST pairs and demonstrate that an end-to-end MT model can be trained using only synthesised data."]},{"idx":20,"heading":"Conclusion","paragraphs":["This paper has investigated the end-to-end method for ST. It has discussed why there is a huge gap between pre-training and fine-tuning in previous methods. To alleviate these issues, we have proposed a method, which is capable of reusing every sub-net and keeping the role of sub-net consistent between pre-training and fine-tuning. Empirical studies have demonstrated that our model significantly outperforms baselines."]},{"idx":21,"heading":"Figures and Tables","paragraphs":["Figure 1: An illustration of multi-task learning for speech translation. Networks inherited from pre-trained models are labeled by rectangles.","Figure 2: The architecture of our model. The linear projection matrix in ASR is shared with the word embedding matrix in MT.","Table 1: An example of the comparison between the golden transcript and the predicted CTC paths given the corresponding speech. ‘-’ denotes the blank token and the following number represents repeat times.","Figure 3: The architecture of seq2seq model. It predicts the next token and its number of repetition at the same time.","Table 2: The CTC path π1 and corresponding unique tokens u and repetition times l, where S(π) = (u, l).","Table 3: BLEU score results on IWSLT-18. “Average” denotes it averages the results of the tst2010, tst2013, tst2014 and tst2015. We copy the numbers of vanilla model from https://github.com/espnet/espnet/blob/master/egs/iwslt18/st1/RESULTS. Since pre-training data is different, we run ESPnet code to obtain the numbers of pre-training and multi-task learning method, which are slightly higher than numbers in their report.","Table 4: Ablation study for subword-level experiments.","Table 5: BLEU comparison of cascaded results and our best end-to-end results. re-seg denotes the ASR outputs are resegmented before fed into the MT model.","Figure 4: Model learning curves in fine-tuning."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1909.09070","title":"Look, Read and Enrich - Learning from Scientific Figures and their Captions","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Compared to natural images, understanding scientific figures is particularly hard for machines. However, there is a valuable source of information in scientific literature that until now has remained untapped: the correspondence between a figure and its caption. In this paper we investigate what can be learnt by looking at a large number of figures and reading their captions, and introduce a figure-caption correspondence learning task that makes use of our observations. Training visual and language networks without supervision other than pairs of unconstrained figures and captions is shown to successfully solve this task. We also show that transferring lexical and semantic knowledge from a knowledge graph significantly enriches the resulting features. Finally, we demonstrate the positive impact of such features in other tasks involving scientific text and figures, like multi-modal classification and machine comprehension for question answering, outperforming supervised baselines and ad-hoc approaches."]},{"idx":1,"heading":"Introduction","paragraphs":["Scientific knowledge is heterogeneous and can present itself in many forms, including text, mathematical equations, figures and tables. Like many other manifestations of human thought, the scientific discourse usually adopts the form of a narrative, a scientific publication where related knowledge is presented in mutually supportive ways over different modalities. In the case of scientific figures, like charts, images and diagrams, these are usually accompanied by a text paragraph, a caption, that elaborates on the analysis otherwise visually represented.","In this paper, we make use of this observation and tap on the potential of learning from the enormous source of free supervision available in the scientific literature, with millions of figures and their captions. We build models that learn from the scientific discourse both visually and textually by simply looking at the figures and reading their explanatory captions, inspired in how humans learn by reading a scientific publication. To this purpose, we explore how multi-modal scientific knowledge can be learnt from the correspondence between figures and captions.","The main contributions of this paper are the following:","An unsupervised Figure-Caption Correspondence task (FCC) that jointly learns text and visual features useful to address a range of tasks involving scientific text and figures.","A method to enrich such features with semantic knowledge transferred from structured knowledge graphs (KG).","A study of the complexity of figure-caption correspondence compared to classical image-sentence matching.","A qualitative and quantitative analysis of the learnt text and visual features through transfer learning tasks.","A corpus of scientific figures and captions extracted from SN SciGraph and AI2 Semantic Scholar.","We present the FCC task in section SECREF3, including the network architecture, training protocol, and how adding pre-trained word and semantic embeddings can enrich the resulting text and visual features. In section SECREF4, we first introduce our datasets and evaluate the performance of our method in the task it was trained to solve, the correspondence between scientific figures and captions. Then, we relate our work to the state of the art in image-sentence matching and evaluate our approach in two challenging transfer learning tasks: caption and figure classification and multi-modal machine comprehension. In section SECREF5 we perform a qualitative study that illustrates how the FCC task leads to detailed textual and visual discrimination. Finally, in section SECREF6 we conclude the paper and advance future work."]},{"idx":2,"heading":"Related work","paragraphs":["Understanding natural images has been a major area of research in computer vision, with well established datasets like ImageNet BIBREF0, Flickr8K BIBREF1, Flickr30K BIBREF2 and COCO BIBREF3. However, reasoning with other visual representations like scientific figures and diagrams has not received the same attention yet and entails additional challenges: Scientific figures are more abstract and symbolic, their captions tend to be significantly longer and use specialized lexicon, and the relation between a scientific figure and its caption is unique, i.e. in a scientific publication there is only one caption that corresponds with one figure and vice versa.","The FCC task presented herein is a form of co-training BIBREF4 where there are two views of the data and each view provides complementary information. Similar two-branch neural architectures focus on image-sentence BIBREF5, BIBREF6 and audio-video BIBREF7 matching. Others like BIBREF8 learn common embeddings from images and text. However, in such cases one or both networks are typically pre-trained.","Focused on geometry, BIBREF9 maximize the agreement between text and visual data. In BIBREF10, the authors apply machine vision and natural language processing to extract data from figures and their associated text in bio-curation tasks. In BIBREF11, they parse diagram components and connectors as a Diagram Parse Graph (DPG), semantically interpret the DPG and use the model to answer diagram questions. While we rely on the correspondence between figures and captions, they train a specific classifier for each component and connector type and yet another model to ground the semantics of the DPG in each domain, like food webs or water cycles.","Knowledge fusion approaches like BIBREF12 investigate the potential of complementing KG embeddings with text and natural images by integrating information across the three modalities in a single latent representation. They assume pre-trained entity representations exist in each individual modality, e.g. the visual features encoding the image of a ball, the word embeddings associated to the token \"ball\", and the KG embeddings related to the ball entity, which are then stitched together. In contrast, FCC co-trains text and visual features from figures and their captions and supports the enrichment of such features with lexical and semantic knowledge transferred from a KG during the training of the FCC task."]},{"idx":3,"heading":"Figure-Caption Correspondence","paragraphs":["The main idea of our approach is to learn a correspondence task between scientific figures and their captions as they appear in a scientific publication. The information captured in the caption explains the corresponding figure in natural language, providing guidance to identify the key features of the figure and vice versa. By seeing a figure and reading the textual description in its caption we ultimately aim to learn representations that capture e.g. what it means that two plots are similar or what gravity looks like.","We leverage this observation to learn a figure-caption correspondence task. In essence, FCC is a binary classification task that receives a figure and a caption and determines whether they correspond or not. For training, the positive pairs are actual figures and their captions from a collection of scientific publications. Negative pairs are extracted from combinations of figures and any other randomly selected captions. The network is then made to learn text and visual features from scratch, without additional labelled data."]},{"idx":4,"heading":"Figure-Caption Correspondence ::: FCC Architecture and Model","paragraphs":["We propose a 2-branch neural architecture (figure FIGREF7) that has three main parts: the vision and language subnetworks, respectively extracting visual and text features, and a fusion subnetwork that takes the resulting features from the visual and text blocks and uses them to evaluate figure-caption correspondence.","The vision subnetwork follows a VGG-style BIBREF13 design, with 3x3 convolutional filters, 2x2 max-pooling layers with stride 2 and no padding. It contains 4 blocks of conv+conv+pool layers, where inside each block the two convolutional layers have the same number of filters, while consecutive blocks have doubling number of filters (64, 128, 256, 512). The input layer receives 224x224x3 images. The final layer produces a 512-D vector after 28x28 max-pooling. Each convolutional layer is followed by batch normalization BIBREF14 and ReLU layers. Based on BIBREF15, the language subnetwork has 3 convolutional blocks, each with 512 filters and a 5-element window size with ReLU activation. Each convolutional layer is followed by a 5-max pooling layer, except for the final layer, which produces a 512-D vector after 35-max pooling. The language subnetwork has a 300-D embeddings layer at the input, with a maximum sequence length of 1,000 tokens. The fusion subnetwork calculates the element-wise product of the 512-D visual and text feature vectors into a single vector $r$ to produce a 2-way classification output (correspond or not). It has two fully connected layers, with ReLU and an intermediate feature size of 128-D. The probability of each choice is the softmax of $r$, i.e. $\\hat{y} = softmax(r) \\in \\mathbb {R}^{2}$. During training, we minimize the negative log probability of the correct choice.","This architecture enables the FCC task to learn visual and text features from scratch in a completely unsupervised manner, just by observing the correspondence of figures and captions. Next, we extend it to enable the transfer of additional pre-trained information. Here, we focus on adding pre-trained embeddings on the language branch, and then back-propagate to the visual features during FCC training. Adding pre-trained visual features is also possible and indeed we also evaluate its impact in the FCC task in section SECREF14.","Let $V$ be a vocabulary of words from a collection of documents $D$. Also, let $L$ be their lemmas, i.e. base forms without morphological or conjugational variations, and $C$ the concepts (or senses) in a KG. Each word $w_k$ in $V$, e.g. made, has one lemma $l_k$ (make) and may be linked to one or more concepts $c_k$ in $C$ (create or produce something).","For each word $w_k$, the FCC task learns a d-D embedding $\\vec{w}_k$, which can be combined with pre-trained word ($\\vec{w^{\\prime }}_k$), lemma ($\\vec{l}_k$) and concept ($\\vec{c}_k$) embeddings to produce a single vector $\\vec{t}_k$. If no pre-trained knowledge is transferred from an external source, then $\\vec{t}_k=\\vec{w}_k$. Note that we previously lemmatize and disambiguate $D$ against the KG in order to select the right pre-trained lemma and concept embeddings for each particular occurrence of $w_k$. Equation DISPLAY_FORM8 shows the different combinations of learnt and pre-trained embeddings we consider: (a) learnt word embeddings only, (b) learnt and pre-trained word embeddings and (c) learnt word embeddings and pre-trained semantic embeddings, including both lemmas and concepts, in line with our recent findings presented in BIBREF16.","In our experiments, concatenation proved optimal to combine the embeddings learnt by the network and the pre-trained embeddings, compared to other methods like summation, multiplication, average or learning a task-specific weighting of the different representations as in BIBREF17. Since some words may not have associated pre-trained word, lemma or concept embeddings, we pad these sequences with $\\varnothing _W$, $\\varnothing _L$ and $\\varnothing _C$, which are never included in the vocabulary. The dimensionality of $\\vec{t}_k$ is fixed to 300, i.e. the size of each sub-vector in configurations $(a)$, $(b)$ and $(c)$ is 300, 150 and 100, respectively. In doing so, we aimed at limiting the number of trainable parameters and balance the contribution of each information source.","In its most basic form, i.e. configuration $(a)$, the FCC network has over 32M trainable parameters (28M in the language subnetwork, 4M in the vision subnetwork and 135K in the fusion subnetwork) and takes 12 hours to train on a single GPU Nvidia GeForce RTX 2080 Ti for a relatively small corpus (SN SciGraph, see section SECREF12). We used 10-fold cross validation, Adam optimization BIBREF18 with learning rate $10^{-4}$ and weight decay $10^{-5}$. The network was implemented in Keras and TensorFlow, with batch size 32. The number of positive and negative cases is balanced within the batches."]},{"idx":5,"heading":"Figure-Caption Correspondence ::: Semantic Embeddings","paragraphs":["We use HolE BIBREF19 and Vecsigrafo BIBREF16 to learn semantic embeddings. The latter extends the Swivel algorithm BIBREF20 to jointly learn word, lemma and concept embeddings on a corpus disambiguated against the KG, outperforming the previous state of the art in word and word-sense embeddings by co-training word, lemma and concept embeddings as opposed to training each individually. In contrast to Vecsigrafo, which requires both a text corpus and a KG, HolE follows a graph-based approach where embeddings are learnt exclusively from the KG. As section SECREF14 will show, this gives Vecsigrafo a certain advantage in the FCC task. Following up with the work presented in BIBREF16, our experiments focus on Sensigrafo, the KG underlying Expert System's Cogito NLP proprietary platform. Similar to WordNet, on which Vecsigrafo has also been successfully trained, Sensigrafo is a general-purpose KG with lexical and semantic information that contains over 300K concepts, 400K lemmas and 80 types of relations rendering 3M links. We use Cogito to disambiguate the text corpora prior to training Vecsigrafo. All the semantic (lemma and concept) embeddings produced with HolE or Vecsigrafo are 100-D."]},{"idx":6,"heading":"Results and Discussion","paragraphs":["In this section, first we evaluate the actual FCC task against two supervised baselines. Then, we situate our work in the more general image-sentence matching problem, showing empirical evidence of the additional complexity associated to the scientific domain and the figure-caption case compared to natural images. Next, we test the visual and text features learnt in the FCC task in two different transfer learning settings: classification of scientific figures and captions and multi-modal machine comprehension for question answering given a context of text, figures and images."]},{"idx":7,"heading":"Results and Discussion ::: Datasets","paragraphs":["We have used the following datasets for training and evaluation:","The Semantic Scholar corpus BIBREF21 (SemScholar) is a large dataset of scientific publications made available by AI2. From its 39M articles, we downloaded 3,3M PDFs (the rest were behind paywalls, did not have a link or it was broken) and extracted 12.5M figures and captions through PDFFigures2 BIBREF22. We randomly selected 500K papers to train the FCC task on their figures and captions and another 500K to train Vecsigrafo on the text of their titles and abstracts.","Springer Nature's SciGraph contains 7M scientific publications organized in 22 scientific fields or categories. Since SciGraph does not provide a link to the PDF of the publication, we selected the intersection with SemScholar, producing a smaller corpus of 80K papers (in addition to the 1M papers from SemScholar mentioned above) and 82K figures that we used for training certain FCC configurations and supervised baselines (section SECREF14).","The Textbook Question Answering corpus BIBREF23 includes 1,076 lessons and 26,260 multi-modal test questions from middle school science curricula. Its complexity and scope make it a challenging textual and visual question answering dataset.","Wikipedia. We used the January 2018 English Wikipedia dataset as one of the corpora on which to train Vecsigrafo. As opposed to SciGraph or SemScholar, specific of the scientific domain, Wikipedia is a source of general-purpose information.","Flickr30K and COCO, as image-sentence matching benchmarks."]},{"idx":8,"heading":"Results and Discussion ::: Figure-Caption Correspondence","paragraphs":["We evaluate our method in the task it was trained to solve: determining whether a figure and a caption correspond. We also compare the performance of the FCC task against two supervised baselines, training them on a classification task against the SciGraph taxonomy. For such baselines we first train the vision and language networks independently and then combine them. The feature extraction parts of both networks are the same as described in section SECREF6. On top of them, we attach a fully connected layer with 128 neurons and ReLU activation and a softmax layer, with as many neurons as target classes.","The direct combination baseline computes the figure-caption correspondence through the scalar product between the softmax outputs of both networks. If it exceeds a threshold, which we heuristically fixed on 0.325, the result is positive. The supervised pre-training baseline freezes the weights of the feature extraction trunks from the two trained networks, assembles them in the FCC architecture as shown in section SECREF6, and trains the FCC task on the fully connected layers. While direct combination provides a notion of the agreement between the two branches, supervised pre-training is the most similar supervised approach to our method.","Table TABREF15 shows the results of the FCC task and the supervised baselines. $FCC_k$ denotes the corpus and word representation used to train the FCC task. Acc$_{vgg}$ shows the accuracy after replacing our visual branch with pre-trained VGG16 features learnt on ImageNet. This provides an estimate of how specific of the scientific domain scientific figures and therefore the resulting visual features can be, compared to natural images. As the table shows, the results obtained using pre-trained visual features are clearly worse in general (only slightly better in $FCC_3$), suggesting that the visual information contained in scientific figures indeed differs from natural images.","We trained the FCC network on two different scientific corpora: SciGraph ($FCC_{1-5}$) and SemScholar ($FCC_{6-7}$). Both $FCC_1$ and $FCC_6$ learnt their own word representations without transfer of any pre-trained knowledge. Even in its most basic form our approach substantially improves over the supervised baselines, confirming that the visual and language branches learn from each other and also that figure-caption correspondence is an effective source of free supervision.","Adding pre-trained knowledge at the input layer of the language subnetwork provides an additional boost, particularly with lemma and concept embeddings from Vecsigrafo ($FCC_5$). Vecsigrafo clearly outperformed HolE ($FCC_3$), which was also beaten by pre-trained fastText BIBREF24 word embeddings ($FCC_2$) trained on SemScholar.","Since graph-based KG embedding approaches like HolE only generate embeddings of the artifacts explicitly contained in the KG, this may indicate that Sensigrafo, the KG used in this task, provides a partial coverage of the scientific domain, as could be expected since we are using an off-the-shelf version. Deeper inspection shows that HolE only covers 20% of the lemmas in the SciGraph vocabulary. On the other hand, Vecsigrafo, trained on the same KG, also captures lexical information from the text corpora it is trained on, Wikipedia or SemScholar, raising lemma coverage to 42% and 47%, respectively.","Although the size of Wikipedia is almost triple of our SemScholar corpus, training Vecsigrafo on the latter resulted in better FCC accuracy ($FCC_4$ vs. $FCC_5$), suggesting that domain relevance is more significant than sheer volume, in line with our previous findings in BIBREF25. Training FCC on SemScholar, much larger than SciGraph, further improves accuracy, as shown in $FCC_6$ and $FCC_7$."]},{"idx":9,"heading":"Results and Discussion ::: Image-Sentence Matching","paragraphs":["We put our FCC task in the context of the more general problem of image-sentence matching through a bidirectional retrieval task where images are sought given a text query and vice versa. While table TABREF20 focuses on natural images datasets (Flickr30K and COCO), table TABREF21 shows results on scientific datasets (SciGraph and SemScholar) rich in scientific figures and diagrams. The selected baselines (Embedding network, 2WayNet, VSE++ and DSVE-loc) report results obtained on the Flickr30K and COCO datasets, which we also include in table TABREF20. Performance is measured in recall at k ($Rk$), with k={1,5,10}. From the baselines, we successfully reproduced DSVE-loc, using the code made available by the authors, and trained it on SciGraph and SemScholar.","We trained the FCC task on all the datasets, both in a totally unsupervised way and with pre-trained semantic embeddings (indicated with subscript $vec$), and executed the bidirectional retrieval task using the resulting text and visual features. We also experimented with pre-trained VGG16 visual features extracted from ImageNet (subscript $vgg$), with more than 14 million hand-annotated images. Following common practice in image-sentence matching, our splits are 1,000 samples for test and the rest for training.","We can see a marked division between the results obtained on natural images datasets (table TABREF20) and those focused on scientific figures (table TABREF21). In the former case, VSE++ and DSVE-loc clearly beat all the other approaches. In contrast, our model performs poorly on such datasets although results are ameliorated when we use pre-trained visual features from ImageNet (\"Oursvgg\" and \"Oursvgg-vec\"). Interestingly, the situation reverts with the scientific datasets. While the recall of DSVE-loc drops dramatically in SciGraph, and even more in SemScholar, our approach shows the opposite behavior in both figure and caption retrieval. Using visual features enriched with pre-trained semantic embeddings from Vecsigrafo during training of the FCC task further improves recall in the bidirectional retrieval task. Compared to natural images, the additional complexity of scientific figures and their caption texts, which in addition are considerably longer (see table TABREF19), seems to have a clear impact in this regard.","Unlike in Flickr30K and COCO, replacing the FCC visual features with pre-trained ones from ImageNet brings us little benefit in SciGraph and even less in SemScholar, where the combination of FCC and Vecsigrafo (\"Oursvec\") obtains the best results across the board. This and the extremely poor performance of the best image-sentence matching baseline (DSVE-loc) in the scientific datasets shows evidence that dealing with scientific figures is considerably more complex than natural images. Indeed, the best results in figure-caption correspondence (\"Oursvec\" in SemScholar) are still far from the SoA in image-sentence matching (DSVE-loc in COCO)."]},{"idx":10,"heading":"Results and Discussion ::: Caption and Figure Classification","paragraphs":["We evaluate the language and visual representations emerging from FCC in the context of two classification tasks that aim to identify the scientific field an arbitrary text fragment (a caption) or a figure belong to, according to the SciGraph taxonomy. The latter is a particularly hard task due to the whimsical nature of the figures that appear in our corpus: figure and diagram layout is arbitrary; charts, e.g. bar and pie charts, are used to showcase data in any field from health to engineering; figures and natural images appear indistinctly, etc. Also, note that we only rely on the actual figure, not the text fragment where it is mentioned in the paper.","We pick the text and visual features that produced the best FCC results with and without pre-trained semantic embeddings (table TABREF15, $FCC_7$ and $FCC_6$, respectively) and use the language and vision subnetworks presented in section SECREF6 to train our classifiers on SciGraph in two different scenarios. First, we only fine tune the fully connected and softmax layers, freezing the text and visual weights (non-trainable in the table). Second, we fine tune all the parameters in both networks (trainable). In both cases, we compare against a baseline using the same networks initialized with random weights, without FCC training. In doing so, through the first, non-trainable scenario, we seek to quantify the information contributed by the FCC features, while training from scratch on the target corpus should provide an upper bound for figure and caption classification. Additionally, for figure classification, we select a baseline of frozen VGG16 weights trained on ImageNet. We train using 10-fold cross validation and Adam. For the caption classification task, we select learning rate $10^{-3}$ and batch size 128. In figure classification, we use learning rate $10^{-4}$, weight decay $10^{-5}$ and batch size 32.","The results in table TABREF23 show that our approach amply beats the baselines, including the upper bound (training from scratch on SciGraph). The delta is particularly noticeable in the non trainable case for both caption and figure classification and is considerably increased in \"Ours $FCC_7$\", which uses pre-trained semantic embeddings. This includes both the random and VGG baselines and illustrates again the additional complexity of analyzing scientific figures compared to natural images, even if the latter is trained on a considerably larger corpus like ImageNet. Fine tuning the whole networks on SciGraph further improves accuracies. In this case, \"Ours $FCC_6$\", which uses FCC features without additional pre-trained embeddings, slightly outperforms \"Ours $FCC_7$\", suggesting a larger margin to learn from the task-specific corpus. Note that both $FCC_6$ and $FCC_7$ were trained on SemScholar."]},{"idx":11,"heading":"Results and Discussion ::: Textbook Question Answering (TQA) for Multi-Modal Machine Comprehension","paragraphs":["We leverage the TQA dataset and the baselines in BIBREF23 to evaluate the features learnt by the FCC task in a multi-modal machine comprehension scenario. We study how our model, which was not originally trained for this task, performs against state of the art models specifically trained for diagram question answering and textual reading comprehension in a very challenging dataset. We also study how pre-trained semantic embeddings impact in the TQA task: first, by enriching the visual features learnt in the FCC task as shown in section SECREF6 and then by using pre-trained semantic embeddings to enrich word representations in the TQA corpus.","We focus on multiple-choice questions, 73% of the dataset. Table TABREF24 shows the performance of our model against the results reported in BIBREF23 for five TQA baselines: random, BiDAF (focused on text machine comprehension), text only ($TQA_1$, based on MemoryNet), text+image ($TQA_2$, VQA), and text+diagrams ($TQA_3$, DSDP-NET). We successfully reproduced the $TQA_1$ and $TQA_2$ architectures and adapted the latter. Then, we replaced the visual features in $TQA_2$ with those learnt by the FCC visual subnetwork both in a completely unsupervised way ($FCC_6$ in table TABREF15) and with pre-trained semantic embeddings ($FCC_7$), resulting in $TQA_4$ and $TQA_5$, respectively.","While $TQA_{1-5}$ used no pre-trained embeddings at all, $TQA_{6-10}$ were trained including pre-trained Vecsigrafo semantic embeddings. Unlike FCC, where we used concatenation to combine pre-trained lemma and concept embeddings with the word embeddings learnt by the task, element-wise addition worked best in the case of TQA.","Following the recommendations in BIBREF23, we pre-processed the TQA corpus to i) consider knowledge from previous lessons in the textbook in addition to the lesson of the question at hand and ii) address challenges like long question contexts with a large lexicon. In both text and diagram MC, applying the Pareto principle to reduce the maximum token sequence length in the text of each question, their answers and context improved accuracy considerably. This optimization allowed reducing the amount of text to consider for each question, improving the signal to noise ratio. Finally, we obtained the most relevant paragraphs for each question through tf-idf and trained the models using 10-fold cross validation, Adam, learning rate $10^{-2}$ and batch size 128. In text MC we also used 0.5 dropout and recurrent dropout in the LSTM layers.","Fitting multi-modal sources into a single memory, the use of visual FCC features clearly outperforms all the TQA baselines in diagram MC. Enhancing word representation with pre-trained semantic embeddings during training of the TQA task provides an additional boost that results in the highest accuracies for both text MC and diagram MC. These are significantly good results since, according to the TQA authors BIBREF23, most diagram questions in the TQA corpus would normally require a specific rich diagram parse, which we did not aim to provide."]},{"idx":12,"heading":"Qualitative Analysis","paragraphs":["We inspect the features learnt by our FCC task to gain a deeper understanding of the syntactic and semantic patterns captured for figure and caption representation. The findings reported herein are qualitatively consistent for all the FCC variations in table TABREF15.","Vision features. The analysis was carried out on an unconstrained variety of charts, diagrams and natural images from SciGraph, without filtering by figure type or scientific field. To obtain a representative sample of what the FCC network learns, we focus on the 512-D vector resulting from the last convolutional block before the fusion subnetwork. We pick the features with the most significant activation over the whole dataset and select the figures that activate them most. To this purpose, we prioritize those with higher maximum activation against the average activation.","Figure FIGREF27 shows a selection of 6 visual features with the 4 figures that activate each feature more significantly and their activation heatmaps. Only figures are used as input, no text. As can be seen, the vision subnetwork has automatically learnt, without explicit supervision, to recognize different types of diagrams, charts and content, such as (from left to right) whisker plots, western blots (a technique used to identify proteins in a tissue sample), multi-image comparison diagrams, multi-modal data visualization charts (e.g. western plots vs. bar charts), line plots, and text within the figures. Furthermore, as shown by the heatmaps, our model discriminates the key elements associated to the figures that most activate each feature: the actual whiskers, the blots, the borders of each image under comparison, the blots and their complementary bar charts, as well as the line plots and the correspondence between them and the values in the x and y axes. Also, see (right-most column) how a feature discriminates text inserted in the figure, regardless of the remaining elements that may appear and the connections between them. This shows evidence of how the visual features learnt by the FCC task support the parsing of complex scientific diagrams.","We also estimated a notion of semantic specificity based on the concepts of a KG. For each visual feature, we aggregated the captions of the figures that most activate it and used Cogito to disambiguate the Sensigrafo concepts that appear in them. Then, we estimated how important each concept is to each feature by calculating its tf-idf. Finally, we averaged the resulting values to obtain a consolidated semantic specificity score per feature.","The scores of the features in figure FIGREF27 range between 0.42 and 0.65, which is consistently higher than average (0.4). This seems to indicate a correlation between activation and the semantic specificity of each visual feature. For example, the heatmaps of the figures related to the feature with the lowest tf-idf (left-most column) highlights a particular visual pattern, i.e. the whiskers, that may spread over many, possibly unrelated domains. On the other hand, the feature with the highest score (second column) focuses on a type of diagrams, western blots, almost exclusive of protein and genetic studies. Others, like the feature illustrated by the figures in the fifth column, capture the semantics of a specific type of 2D charts relating two magnitudes x and y. Analyzing their captions with Cogito, we see that concepts like e.g. isochronal and exponential functions are mentioned. If we look at the second and four top-most figures in the column, we can see that such concepts are also visually depicted in the figures, suggesting that the FCC task has learnt to recognize them both from the text and visually.","Text features. Similar to the visual case, we selected the features from the last block of the language subnetwork with the highest activation. For visualization purposes, we picked the figures corresponding to the captions in SciGraph that most activate such features (figure FIGREF28). No visual information is used.","Several distinct patterns emerge from the text. The text feature in the first column seems to focus on genetics and histochemistry, including terms like western blots or immunostaining and variations like immunoblot-s/ted/ting. Interestingly, it also seems to have learnt some type of is-a relations (western blot is a type of immunoblot). The second feature focuses on variations of the term radiograph, e.g. radiograph-y/s. The third feature specializes in text related to curve plots involving several statistic analysis, e.g. Real-time PCR, one-way ANOVA or Gaussian distribution. Sometimes (fourth figure from top) the caption does not mention the plot directly, but focuses on the analysis instead, e.g. \"the data presented here are mean values of duplicate experiments\", indicating transfer of knowledge from the visual part during training. The fourth feature extracts citations and models named after prominent scientists, e.g. Evans function (first and fourth figure), Manley (1992) (second), and Aliev-Panfilov model (third). The fifth feature extracts chromatography terminology, e.g. 3D surface plot, photomicrograph or color map and, finally, the right-most feature focuses on different types of named diagrams, like flow charts and state diagrams, e.g. phylogenetic trees.","All the captions show a strong semantic correspondence with their associated figures. Figure FIGREF29 shows the activation heatmaps for two sample captions, calculated on the embeddings layer of the language subnetwork. The upper one corresponds to the fourth column left-right and third figure top-down in figure FIGREF28. Its caption reads: \"The Aliev-Panfilov model with $\\alpha =0.01$...The phase portrait depicts trajectories for distinct initial values $\\varphi _0$ and $r_0$...\". Below, (first column, fourth figure in figure FIGREF28): \"Relative protein levels of ubiquitin-protein conjugates in M. quadriceps...A representative immunoblot specific to ubiquitin...\". Consistently with our analysis, activation focuses on the most relevant tokens for each text feature: \"Aliev-Panfilov model\" and \"immunoblot\", respectively."]},{"idx":13,"heading":"Conclusions","paragraphs":["There is a wealth of knowledge in scientific literature and only a fraction of it is text. However, understanding scientific figures is a challenging task for machines, which is beyond their ability to process natural images. In this paper, we provide empirical evidence of this and show that co-training text and visual features from a large corpus of scientific figures and their captions in a correspondence task (FCC) is an effective, flexible and elegant unsupervised means towards overcoming such complexity. We show how such features can be significantly improved by enriching them with additional knowledge sources and, particularly, structured KGs. We prove the benefits of our approach against supervised baselines and in different transfer learning tasks, including text and visual classification and multi-modal machine comprehension applied to question answering, with results generally beyond the state of the art. In the future, it will be interesting to further the study of the interplay between the semantic concepts explicitly represented in different KGs, contextualized embeddings e.g. from SciBERT BIBREF31, and the text and visual features learnt in the FCC task. We also plan to continue to charter the knowledge captured in such features and to pursue the optimization and practical application of our approach."]},{"idx":14,"heading":"Acknowledgments","paragraphs":["The research reported in this paper is supported by the EU Horizon 2020 programme, under grants European Language Grid-825627 and Co-inform-770302."]},{"idx":15,"heading":"Figures and Tables","paragraphs":["Figure 1: Proposed 2-branch architecture of the FCC task.","Table 1: FCC and supervised baselines results (% accuracy).","Table 2: Caption length: natural images vs scientific datasets.","Table 3: Bidirectional retrieval. FCC vs. image-sentence matching baselines (%recall@k). Natural images datasets.","Table 4: Bidirectional retrieval. FCC vs. image-sentence matching baselines (%recall@k). Scientific datasets.","Table 5: Caption and figure classification (%accuracy)","Table 6: TQA results (% accuracy). FCC vs. random, BiDAF, MemoryNet, VQA and DSDP-NET baselines.","Figure 2: Selected visual features and activation heatmaps. The top row labels the dominant pattern for each feature.","Figure 3: Selected text features. Top row labels the dominant pattern for each text feature.","Figure 4: Sample caption activation heatmaps. Darker means higher activation."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1909.11687","title":"Extreme Language Model Compression with Optimal Subwords and Shared Projections","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Pre-trained deep neural network language models such as ELMo, GPT, BERT and XLNet have recently achieved state-of-the-art performance on a variety of language understanding tasks. However, their size makes them impractical for a number of scenarios, especially on mobile and edge devices. In particular, the input word embedding matrix accounts for a significant proportion of the model's memory footprint, due to the large input vocabulary and embedding dimensions. Knowledge distillation techniques have had success at compressing large neural network models, but they are ineffective at yielding student models with vocabularies different from the original teacher models. We introduce a novel knowledge distillation technique for training a student model with a significantly smaller vocabulary as well as lower embedding and hidden state dimensions. Specifically, we employ a dual-training mechanism that trains the teacher and student models simultaneously to obtain optimal word embeddings for the student vocabulary. We combine this approach with learning shared projection matrices that transfer layer-wise knowledge from the teacher model to the student model. Our method is able to compress the BERT_BASE model by more than 60x, with only a minor drop in downstream task metrics, resulting in a language model with a footprint of under 7MB. Experimental results also demonstrate higher compression efficiency and accuracy when compared with other state-of-the-art compression techniques."]},{"idx":1,"heading":"Introduction","paragraphs":["Recently, contextual-aware language models such as ELMo BIBREF0, GPT BIBREF1, BERT BIBREF2 and XLNet BIBREF3 have shown to greatly outperform traditional word embedding models including Word2Vec BIBREF4 and GloVe BIBREF5 in a variety of NLP tasks. These pre-trained language models, when fine-tuned on downstream language understanding tasks such as sentiment classification BIBREF6, natural language inference BIBREF7 and reading comprehension BIBREF8, BIBREF9, have achieved state-of-the-art performance. However, the large number of parameters in these models, often above hundreds of millions, makes it impossible to host them on resource-constrained tasks such as doing real-time inference on mobile and edge devices.","Besides utilizing model quantization techniques BIBREF10, BIBREF11 which aim to reduce the floating-point accuracy of the parameters, significant recent research has focused on knowledge distillation BIBREF12, BIBREF13 techniques. Here, the goal is to train a small-footprint student model by borrowing knowledge, such as through a soft predicted label distribution, from a larger pre-trained teacher model.","However, a significant bottleneck that has been overlooked by previous efforts is the input vocabulary size and its corresponding word embedding matrix, often accounting for a significant proportion of all model parameters. For instance, the embedding table of the BERTBASE model, comprising over 30K WordPiece tokens BIBREF14, accounts for over $21\\%$ of the model size. While there has been existing work on reducing NLP model vocabulary sizes BIBREF15, distillation techniques cannot utilize these, since they require the student and teacher models to share the same vocabulary and output space. This profoundly limits their potential to further reduce model sizes.","We present two novel ideas to improve the effectiveness of knowledge distillation, in particular for BERT, with the focus on bringing down model sizes to as much as a few mega-bytes. Our model is among the first to propose to use a significantly smaller vocabulary for the student model learned during distillation. In addition, instead of distilling solely on the teacher model's final-layer outputs, our model leverages layer-wise teacher model parameters to directly optimize the parameters of the corresponding layers in the student model. Specifically, our contributions are:","[leftmargin=*]","Dual Training: Our teacher and student models have different vocabularies and incompatible tokenizations for the same sequence. To address this during distillation, we feed the teacher model a mix of teacher vocabulary-tokenized and student vocabulary-tokenized words within a single sequence. Coupled with the masked language modeling task, this encourages an implicit alignment of the teacher and student WordPiece embeddings, since the student vocabulary embedding may be used as context to predict a word tokenized by the teacher vocabulary and vice versa.","Shared Variable Projections: To minimize the loss of information from reducing the hidden state dimension, we introduce a separate loss to align the teacher and student models' trainable variables. This allows for more direct layer-wise transfer of knowledge to the student model.","Using the combination of dual training and shared variable projections, we train a 12-layer highly-compressed student BERT model, achieving a maximum compression ratio of $\\sim $61.94x (with 48 dimension size) compared to the teacher BERTBASE model. We conduct experiments for measuring both generalized language modeling perspective and for downstream tasks, demonstrating competitive performance with high compression ratios for both families of tasks."]},{"idx":2,"heading":"Related Work","paragraphs":["Research in neural network model compression has been concomitant with the rise in popularity of neural networks themselves, since these models have often been memory-intensive for the hardware of their time. Work in model compression for NLP applications falls broadly into four categories: matrix approximation, parameter pruning/sharing, weight quantization and knowledge distillation.","A family of approaches BIBREF16, BIBREF17 seeks to compress the matrix parameters of the models by low-rank approximation i.e. the full-rank matrix parameter is approximated using multiple low-rank matrices, thereby reducing the effective number of model parameters. Another line of work explores parameter pruning and sharing-based methods BIBREF18, BIBREF19, BIBREF20, BIBREF21, which explore the redundancy in model parameters and try to remove redundant weights as well as neurons, for a variety of neural network architectures.","Model weight quantization techniques BIBREF22, BIBREF23, BIBREF24, BIBREF25 focus on mapping model weights to lower-precision integers and floating-point numbers. These can be especially effective with hardware supporting efficient low-precision calculations. More recently, BIBREF26 apply quantization to BERT-based transformer models.","Knowledge distillation BIBREF12, BIBREF13 differs from the other discussed approaches: the smaller student model may be parametrized differently from the bigger teacher model, affording more modeling freedom. Teaching a student model to match the soft output label distributions from a larger model alongside the hard ground-truth distribution works well for many tasks, such as machine translation BIBREF27 and language modeling BIBREF28.","Not limited to the teacher model outputs, some approaches perform knowledge distillation via attention transfer BIBREF29, or via feature maps or intermediate model outputs BIBREF30, BIBREF31, BIBREF32. More relevant to current work, BIBREF33 and BIBREF34 employ variants of these techniques to BERT model compression by reducing the number of transformer layers. However, as explained before, these approaches are not immediately applicable to our setting due to incompatible teacher and student model vocabularies, and do not focus sufficiently on the embedding matrix size BIBREF35."]},{"idx":3,"heading":"Methodology","paragraphs":["Our knowledge distillation approach is centered around reducing the number of WordPiece tokens in the model vocabulary. In this section, we first discuss the rationale behind this reduction and the challenges it introduces, followed by our techniques, namely dual training and shared projection."]},{"idx":4,"heading":"Methodology ::: Optimal Subword Embeddings via Knowledge Distillation","paragraphs":["We follow the general knowledge distillation paradigm of training a small student model from a large teacher model. Our teacher model is a 12-layer uncased BERTBASE, trained with 30522 WordPiece tokens and 768-dimensional embeddings and hidden states. We denote the teacher model parameters by $\\theta _{t}$. Our student model consists of an equal number of transformer layers with parameters denoted by $\\theta _{s}$, but with a smaller vocabulary as well as embedding/hidden dimensions, illustrated in Figure FIGREF4. Using the same WordPiece algorithm and training corpus as BERT, we obtain a vocabulary of 4928 WordPieces, which we use for the student model.","WordPiece tokens BIBREF14 are sub-word units obtained by applying a greedy segmentation algorithm to the training corpus: a desired number (say, $D$) of WordPieces are chosen such that the segmented corpus is minimal in the number of WordPieces used. A cursory look at both vocabularies reveals that $93.9\\%$ of the WordPieces in the student vocabulary also exist in the teacher vocabulary, suggesting room for a reduction in the WordPiece vocabulary size from 30K tokens.","Since we seek to train a general-purpose student language model, we elect to reuse the teacher model's original training objective to optimize the student model, i.e., masked language modeling and next sentence prediction, before any fine-tuning. In the former task, words in context are randomly masked, and the language model needs to predict those words given the masked context. In the latter task, given a pair of sentences, the language model predicts whether the pair is consistent.","However, since the student vocabulary is not a complete subset of the teacher vocabulary, the two vocabularies may tokenize the same words differently. As a result, the outputs of the teacher and student model for the masked language modeling task may not align. Even with the high overlap between the two vocabularies, the need to train the student embedding from scratch, and the change in embedding dimension precludes existing knowledge distillation techniques, which rely on the alignment of both models' output spaces. As a result, we explore two alternative approaches that enable implicit transfer of knowledge to the student model, which we describe below."]},{"idx":5,"heading":"Methodology ::: Dual Training","paragraphs":["During distillation, for a given training sequence input to the teacher model, we propose to mix the teacher and student vocabularies by randomly selecting (with a probability $p_{DT}$, a hyperparameter) tokens from the sequence to segment using the student vocabulary, with the other tokens segmented using the teacher vocabulary. As illustrated in Figure FIGREF4, given the input context [ `I', `like', `machine', `learning'], the words `I' and `machine' are segmented using the teacher vocabulary (in green), while `like' and `learning' are segmented using the student vocabulary (in blue). Similar to cross-lingual training in BIBREF36, this encourages alignment of the representations for the same word as per the teacher and student vocabularies. This is effected through the masked language modeling task: the model now needs to learn to predict words from the student vocabulary using context words segmented using the teacher vocabulary, and vice versa.","The expectation is that the student embeddings can be learned effectively this way from the teacher embeddings as well as model parameters $\\theta _{t}$. Note that we perform dual training only for the teacher model inputs: the student model receives words segmented exclusively using the student vocabulary. Also, during masked language modeling, the model uses different softmax layers for the teacher and the student vocabularies depending on which one was used to segment the word in question."]},{"idx":6,"heading":"Methodology ::: Shared Projections","paragraphs":["Relying solely on teacher model outputs to train the student model may not generalize well BIBREF34. Therefore, some approaches utilize and try to align the student model's intermediate predictions to those of the teacher BIBREF30. In our setting, however, since the student and teacher model output spaces are not identical, intermediate model outputs may prove hard to align. Instead, we seek to directly minimize the loss of information from the teacher model parameters $\\theta _{t}$ to the student parameters $\\theta _{s}$ with smaller dimensions. We achieve this by projecting the model parameters into the same space, to encourage alignment.","More specifically, as in Figure FIGREF4, we project each trainable variable in $\\theta _{t}$ to the same shape as the corresponding variable in $\\theta _{s}$. For example, for all the trainable variables $\\theta _{t}$ with shape $768\\times 768$, we learn two projection matrices $\\mathbf {U}\\in \\mathbb {R}^{d\\times 768}$ and $\\mathbf {V}\\in \\mathbb {R}^{768\\times d}$ to project them into the corresponding space of the student model variable $\\theta _{t}^\\prime $, where $d$ is the student model's hidden dimension. $\\mathbf {U}$ and $\\mathbf {V}$ are common to all BERT model parameters of that dimensionality; in addition, $\\mathbf {U}$ and $\\mathbf {V}$ are not needed for fine-tuning or inference after distillation. In order to align the student variable and the teacher variable's projection, we introduce a separate mean square error loss defined in Equation DISPLAY_FORM7, where $\\downarrow $ stands for down projection (since the projection is to a lower dimension).","The above loss function aligns the trainable variables in the student space. Alternatively, we can project trainable variables in $\\theta _{s}$ to the same shape as in $\\theta _{t}$. This way, the loss function in Equation DISPLAY_FORM8, ($\\uparrow $ denotes up projection) can compare the trainable variables in the teacher space."]},{"idx":7,"heading":"Methodology ::: Optimization Objective","paragraphs":["Our final loss function includes, in addition to an optional projection loss, masked language modeling cross-entropy losses for the student as well as the teacher models, since the teacher model is trained with dual-vocabulary inputs and is not static. $P(y_i=c|\\theta _{s})$ and $P(y_i=c|\\theta _{t})$ denote the student and teacher model prediction probabilities for class c respectively, and $\\mathbb {1}$ denotes an indicator function. Equations DISPLAY_FORM10 and below define the final loss $L_{final}$, where $\\epsilon $ is a hyperparameter."]},{"idx":8,"heading":"Experiments","paragraphs":["To evaluate our knowledge distillation approach, we design two classes of experiments. First, we evaluate the distilled student language models using the masked word prediction task on an unseen evaluation corpus, for an explicit evaluation of the language model. Second, we fine-tune the language model by adding a task-specific affine layer on top of the student language model outputs, on a suite of downstream sentence and sentence pair classification tasks. This is meant to be an implicit evaluation of the quality of the representations learned by the student language model. We describe these experiments, along with details on training, implementation and our baselines below."]},{"idx":9,"heading":"Experiments ::: Language Model Training","paragraphs":["During the distillation of the teacher BERT model to train the student BERT language model, we utilize the same corpus as was used to train the teacher i.e. BooksCorpus BIBREF37 and English Wikipedia, with whitespaces used tokenize the text into words. We only use the masked language modeling task to calculate the overall distillation loss from Section SECREF6, since the next sentence prediction loss hurt performance slightly. Dual training is enabled for teacher model inputs, with $p_{DT}$, the probability of segmenting a teacher model input word using the student vocabulary, set to 0.5. For experiments including shared projection, the projection matrices $\\mathbf {U}$ and $\\mathbf {V}$ utilized Xavier initialization BIBREF38. The loss weight coefficient $\\epsilon $ is set to 1 after tuning.","It is worth noting that in contrast to a number of existing approaches, we directly distill the teacher BERT language model, not yet fine-tuned on a downstream task, to obtain a student language model that is task-agnostic. For downstream tasks, we fine-tune this distilled student language model.","Distillation is carried out on Cloud TPUs in a 4x4 pod configuration (32 TPU cores overall). We optimized the loss using LAMB BIBREF39 for 250K steps, with a learning rate of 0.00125 and batch size of 4096. Depending on the student model dimension, training took between 2-4 days."]},{"idx":10,"heading":"Experiments ::: Models and Baselines","paragraphs":["We evaluate three variants of our distilled student models: with only dual training of the teacher and student vocabularies (DualTrain) and with dual training along with down-projection (DualTrain + SharedProjDown) or up-projection (DualTrain + SharedProjUp) of the teacher model parameters. For each of these configurations, we train student models with embedding and hidden dimensions 48, 96 and 192, for 9 total variants, each using a compact 5K-WordPiece vocabulary. Table TABREF14 presents some statistics on these models' sizes: our smallest model contains two orders of magnitude fewer parameters, and requires only 1% floating-point operations when compared to the BERTBASE model.","For the language modeling evaluation, we also evaluate a baseline without knowledge distillation (termed NoKD), with a model parameterized identically to the distilled student models but trained directly on the teacher model objective from scratch. For downstream tasks, we compare with NoKD as well as Patient Knowledge Distillation (PKD) from BIBREF34, who distill the 12-layer BERTBASE model into 3 and 6-layer BERT models by using the teacher model's hidden states."]},{"idx":11,"heading":"Experiments ::: Evaluation Tasks and Datasets","paragraphs":["For explicit evaluation of the generalized language perspective of the distilled student language models, we use the Reddit dataset BIBREF40 to measure word mask prediction accuracy of the student models, since the language used on Reddit is different from that in the training corpora. The dataset is preprocessed similarly to the training corpora, except we do not need to tokenize it using the teacher vocabulary, since we only run and evaluate the student models.","For implicit evaluation on downstream language understanding tasks, we fine-tune and evaluate the distilled student models on three tasks from the GLUE benchmark BIBREF41:","[leftmargin=*,topsep=0pt]","Stanford Sentiment Treebank (SST-2) BIBREF6, a two-way sentence sentiment classification task with 67K training instances,","Microsoft Research Paraphrase Corpus (MRPC) BIBREF42, a two-way sentence pair classification task to identify paraphrases, with 3.7K training instances, and","Multi-Genre Natural Language Inference (MNLI) BIBREF7, a three-way sentence pair classification task with 393K training instances, to identify premise-hypothesis relations. There are separate development and test sets for genre-matched and genre-mismatched premise-hypothesis pairs; we tune our models solely on the genre-matched development set.","For all downstream task evaluations, we fine-tune for 10 epochs using LAMB with a learning rate of 0.0002 and batch size of 32. Since our language models are trained with a maximum sequence length of 128 tokens, we do not evaluate on reading comprehension datasets such as SQuAD BIBREF8 or RACE BIBREF9, which require models supporting longer sequences."]},{"idx":12,"heading":"Results","paragraphs":["Table TABREF20 contains masked word prediction accuracy figures for the different models and the NoKD baseline. We observe that dual training significantly improves over the baseline for all model dimensions, and that both shared projection losses added to dual training further improve the word prediction accuracy. It is interesting to note that for all model dimensions, SharedProjUp projecting into the teacher space outperforms SharedProjDown, significantly so for dimension 48. Expectedly, there is a noticeable performance drop going from 192 to 96 to 48-dimensional hidden state models. Note that because of the differing teacher and student model vocabularies, masked word prediction accuracy for the teacher BERTBASE model is not directly comparable with the student models.","Table TABREF21 shows results on the downstream language understanding tasks, as well as model sizes, for our approaches, the BERTBASE teacher model, and the PKD and NoKD baselines. We note that models trained with our proposed approaches perform strongly and consistently improve upon the identically parametrized NoKD baselines, indicating that the dual training and shared projection techniques are effective, without incurring significant losses against the BERTBASE teacher model. Comparing with the PKD baseline, our 192-dimensional models, achieving a higher compression rate than either of the PKD models, perform better than the 3-layer PKD baseline and are competitive with the larger 6-layer baseline on task accuracy while being nearly 5 times as small.","Another observation we make is that the performance drop from 192-dimensional to 96-dimensional models is minimal (less than 2% for most tasks). For the MRPC task, in fact, the 96-dimensional model trained with dual training achieves an accuracy of 80.5%, which is higher than even the PKD 6-layer baseline with nearly 12 times as many parameters. Finally, our highly-compressed 48-dimensional models also perform respectably: the best 48-dimensional models are in a similar performance bracket as the 3-layer PKD model, a model 25 times larger by memory footprint."]},{"idx":13,"heading":"Discussion","paragraphs":["Shared projections and model performance: We see that for downstream task performance, dual training still consistently improves upon the direct fine-tuning approach for virtually all experiments. The effect of shared variable projection, however, is less pronounced, with consistent improvements visible only for MRPC and for the 48-dimensional models i.e. the smallest dataset and models respectively in our experiments. This aligns with our intuition for variable projection as a more direct way to provide a training signal from the teacher model internals, which can assume more importance for a low-data or small-model scenario. However, for larger models and more data, the linear projection of parameters may be reducing the degrees of freedom available to the model, since linear projection is a fairly simple function to align the teacher and student parameter spaces.","A related comparison of interest is between up-projection and down-projection of the model variables: we note up-projection does visibly better on the language modeling task and slightly better on the downstream tasks. The parameters of a well-trained teacher model represent a high-quality local minimum in the teacher space, which may be easier to search for during up-projection.","Vocabulary size tradeoffs: Issues with input vocabulary size are peculiar to problems in natural language processing: they do not always apply to other areas such as computer vision, where a small fixed number of symbols can encode most inputs. There has been some work on reducing input vocabulary sizes for NLP, but typically not targeting model compression. One concern with reducing the vocabularies of NLP models is it pushes the average tokenized sequence lengths up, making model training harder. In this work, however, we consider classification tasks on shorter texts, which are not as affected by input sequence lengths as, say, tasks such as machine translation are. Furthermore, many real-world applications revolve around short text inputs, which is why a better trade-off between vocabulary size and sequence lengths may be worthwhile for such applications.","Order of distillation and fine-tuning: Most of the existing work on distilling language models such as BERT and reporting results on downstream tasks, including some of the baselines in this work, first fine-tune a teacher model on the downstream tasks, and then distill this model. Our goal in this work, however, is to explore the limits to which BERT's language modeling capacity itself, and how much of it is driven by its large WordPiece vocabulary. We leave experiments on distilling fine-tuned teacher models, potentially yielding better results on downstream tasks, to future work."]},{"idx":14,"heading":"Conclusion","paragraphs":["We proposed two novel ideas to improve the effectiveness of knowledge distillation for BERT, focusing on using a significantly smaller vocabulary, as well as smaller embedding and hidden dimensions for the student BERT language models. Our dual training mechanism encourages implicit alignment of the teacher and student WordPiece embeddings, and shared variable projection allows for the faster and direct layer-wise transfer of knowledge to the student BERT model. Combining the two techniques, we trained a series of highly-compressed 12-layer student BERT models. Experiments on these models, to evaluate both generalized language perspective and four standardized downstream tasks, demonstrate the effectiveness of our proposed methods on both model accuracy and compression efficiency.","One future direction of interest is to combine our approach with existing work to reduce the number of layers in the student models and explore other approaches such as low-rank matrix factorization to transfer model parameters from the teacher space to the student space. In addition, taking into account the frequency distribution of the WordPiece tokens while training embeddings may help optimize the model size further."]},{"idx":15,"heading":"Figures and Tables","paragraphs":["Figure 1: Knowledge Distillation on BERT with smaller student vocabulary. (Left) A pre-trained teacher BERT model with default BERT parameters (e.g., 30K vocab, 768 hidden state dimension). (Right) A student BERT model trained from scratch with smaller vocab (5K) and hidden state dimension (e.g., 48). During distillation, the teacher model randomly selects a vocabulary to segment each input word. The red and green square nexts to the transformer layers indicate trainable parameters for both the student and teacher models - note that our student models have smaller model dimensions. The projection matrices U and V, shown as having representative shapes, are shared across all layers for model parameters that have the same dimensions.","Table 1: A summary of our student models’ sizes compared to BERTBASE. #Params indicates the number of parameters in the student model, model size is measured in megabytes, and FLOPS ratio measures the relative ratio of floating point operations required for inference on the model.","Table 2: Masked language modeling task accuracy for the distilled student models and a fine-tunefrom-scratch baseline. We observe consistently better performance for our proposed approaches.","Table 3: Results of the distilled models, the teacher model and baselines on the downstream language understanding task test sets, obtained from the GLUE server, along with the size parameters and compression ratios of the respective models compared to the teacher BERTBASE. MNLI-m and MNLI-mm refer to the genre-matched and genre-mismatched test sets for MNLI."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1909.13466","title":"Regressing Word and Sentence Embeddings for Regularization of Neural Machine Translation","sections":[{"idx":0,"heading":"Abstract","paragraphs":["In recent years, neural machine translation (NMT) has become the dominant approach in automated translation. However, like many other deep learning approaches, NMT suffers from overfitting when the amount of training data is limited. This is a serious issue for low-resource language pairs and many specialized translation domains that are inherently limited in the amount of available supervised data. For this reason, in this paper we propose regressing word (ReWE) and sentence (ReSE) embeddings at training time as a way to regularize NMT models and improve their generalization. During training, our models are trained to jointly predict categorical (words in the vocabulary) and continuous (word and sentence embeddings) outputs. An extensive set of experiments over four language pairs of variable training set size has showed that ReWE and ReSE can outperform strong state-of-the-art baseline models, with an improvement that is larger for smaller training sets (e.g., up to +5:15 BLEU points in Basque-English translation). Visualizations of the decoder's output space show that the proposed regularizers improve the clustering of unique words, facilitating correct predictions. In a final experiment on unsupervised NMT, we show that ReWE and ReSE are also able to improve the quality of machine translation when no parallel data are available."]},{"idx":1,"heading":"Introduction","paragraphs":["Machine translation (MT) is a field of natural language processing (NLP) focussing on the automatic translation of sentences from a source language to a target language. In recent years, the field has been progressing quickly mainly thanks to the advances in deep learning and the advent of neural machine translation (NMT). The first NMT model was presented in 2014 by Sutskever et al. BIBREF0 and consisted of a plain encoder-decoder architecture based on recurrent neural networks (RNNs). In the following years, a series of improvements has led to major performance increases, including the attention mechanism (a word-aligment model between words in the source and target sentences) BIBREF1, BIBREF2 and the transformer (a non-recurrent neural network that offers an alternative to RNNs and makes NMT highly parallelizable) BIBREF3. As a result, NMT models have rapidly outperformed traditional approaches such as phrase-based statistical machine translation (PBSMT) BIBREF4 in challenging translation contexts (e.g., the WMT conference series). Nowadays, the majority of commercial MT systems utilise NMT in some form.","However, NMT systems are not exempt from limitations. The main is their tendence to overfit the training set due to their large number of parameters. This issue is common to many other tasks that use deep learning models and it is caused to a large extent by the way these models are trained: maximum likelihood estimation (MLE). As pointed out by Elbayad et al. BIBREF5, in the case of machine translation, MLE has two clear shortcomings that contribute to overfitting:","Single ground-truth reference: Usually, NMT models are trained with translation examples that have a single reference translation in the target language. MLE tries to give all the probability to the words of the ground-truth reference and zero to all others. Nevertheless, a translation that uses different words from the reference (e.g. paraphrase sentences, synonyms) can be equally correct. Standard MLE training is not able to leverage this type of information since it treats every word other than the ground truth as completely incorrect.","Exposure biasBIBREF6: NMT models are trained with “teacher forcing”, which means that the previous word from the reference sentence is given as input to the decoder for the prediction of the next. This is done to speed up training convergence and avoid prediction drift. However, at test time, due to the fact that the reference is not available, the model has to rely on its own predictions and the performance can be drastically lower.","Both these limitations can be mitigated with sufficient training data. In theory, MLE could achieve optimal performance with infinite training data, but in practice this is impossible as the available resources are always limited. In particular, when the training data are scarce such as in low-resource language pairs or specific translation domains, NMT models display a modest performance, and other traditional approaches (e.g., PBSMT)BIBREF7 often obtain better accuracies. As such, generalization of NMT systems still calls for significant improvement.","In our recent work BIBREF8, we have proposed a novel regularization technique that is based on co-predicting words and their embeddings (“regressing word embeddings”, or ReWE for short). ReWE is a module added to the decoder of a sequence-to-sequence model so that, during training, the model is trained to jointly predict the next word in the translation (categorical value) and its pre-trained word embedding (continuous value). This approach can leverage the contextual information embedded in pre-trained word vectors to achieve more accurate translations at test time. ReWE has been showed to be very effective over low/medium size training sets BIBREF8. In this paper, we extend this idea to its natural counterpart: sentence embedding. We propose regressing sentence embeddings (ReSE) as an additional regularization method to further improve the accuracy of the translations. ReSE uses a self-attention mechanism to infer a fixed-dimensional sentence vector for the target sentence. During training, the model is trained to regress this inferred vector towards the pre-trained sentence embedding of the ground-truth sentence. The main contributions of this paper are:","The proposal of a new regularization technique for NMT based on sentence embeddings (ReSE).","Extensive experimentation over four language pairs of different dataset sizes (from small to large) with both word and sentence regularization. We show that using both ReWE and ReSE can outperform strong state-of-the-art baselines based on long short-term memory networks (LSTMs) and transformers.","Insights on how ReWE and ReSE help to improve NMT models. Our analysis shows that these regularizers improve the organization of the decoder's output vector space, likely facilitating correct word classification.","Further experimentation of the regularizer on unsupervised machine translation, showing that it can improve the quality of the translations even in the absence of parallel training data.","The rest of this paper is organized as follows. Section SECREF2 presents and discusses the related work. Section SECREF3 describes the model used as baseline while Section SECREF4 presents the proposed regularization techniques, ReWE and ReSE. Section SECREF5 describes the experiments and analyzes the experimental results. Finally, Section SECREF6 concludes the paper."]},{"idx":2,"heading":"Related Work","paragraphs":["The related work is organized over the three main research subareas that have motivated this work: regularization techniques, word and sentence embeddings and unsupervised NMT."]},{"idx":3,"heading":"Related Work ::: Regularization Techniques","paragraphs":["In recent years, the research community has dedicated much attention to the problem of overfitting in deep neural models. Several regularization approaches have been proposed in turn such as dropout BIBREF9, BIBREF10, data augmentation BIBREF11 and multi-task learning BIBREF12, BIBREF13. Their common aim is to encourage the model to learn parameters that allow for better generalization.","In NMT, too, mitigating overfitting has been the focus of much research. As mentioned above, the two, main acknowledged problems are the single ground-truth reference and the exposure bias. For the former, Fadee et al. BIBREF11 have proposed augmenting the training data with synthetically-generated sentence pairs containing rare words. The intuition is that the model will be able to see the vocabulary's words in more varied contexts during training. Kudo BIBREF14 has proposed using variable word segmentations to improve the model's robustness, achieving notable improvements in low-resource languages and out-of-domain settings. Another line of work has focused on “smoothing” the output probability distribution over the target vocabulary BIBREF5, BIBREF15. These approaches use token-level and sentence-level reward functions that push the model to distribute the output probability mass over words other than the ground-truth reference. Similarly, Ma et al. BIBREF16 have added a bag-of-words term to the training objective, assuming that the set of correct translations share similar bag-of-word vectors.","There has also been extensive work on addressing the exposure bias problem. An approach that has proved effective is the incorporation of predictions in the training, via either imitation learning BIBREF17, BIBREF18, BIBREF19 or reinforcement learning BIBREF20, BIBREF21. Another approach, that is computationally more efficient, leverages scheduled sampling to obtain a stochastic mixture of words from the reference and the predictions BIBREF6. In turn, Wu et al. BIBREF22 have proposed a soft alignment algorithm to alleviate the missmatches between the reference translations and the predictions obtained with scheduled sampling; and Zhang et al.BIBREF23 have introduced two regularization terms based on the Kullback-Leibler (KL) divergence to improve the agreement of sentences predicted from left-to-right and right-to-left."]},{"idx":4,"heading":"Related Work ::: Word and Sentence Embeddings","paragraphs":["Word vectors or word embeddings BIBREF24, BIBREF25, BIBREF26 are ubiquitous in NLP since they provide effective input features for deep learning models. Recently, contextual word vectors such as ELMo BIBREF27, BERT BIBREF28 and the OpenAI transformer BIBREF29 have led to remarkable performance improvements in several language understanding tasks. Additionally, researchers have focused on developing embeddings for entire sentences and documents as they may facilitate several textual classification tasks BIBREF30, BIBREF31, BIBREF32, BIBREF33.","In NMT models, word embeddings play an important role as input of both the encoder and the decoder. A recent paper has shown that contextual word embeddings provide effective input features for both stages BIBREF34. However, very little research has been devoted to using word embeddings as targets. Kumar and Tsvetkov BIBREF35 have removed the typical output softmax layer, forcing the decoder to generate continuous outputs. At inference time, they use a nearest-neighbour search in the word embedding space to select the word to predict. Their model allows for significantly faster training while performing on par with state-of-the-art models. Our approach differs from BIBREF35 in that our decoder generates continuous outputs in parallel with the standard softmax layer, and only during training to provide regularization. At inference time, the continuous output is ignored and prediction operates as in a standard NMT model. To the best of our knowledge, our model is the first to use embeddings as targets for regularization, and at both word and sentence level."]},{"idx":5,"heading":"Related Work ::: Unsupervised NMT","paragraphs":["The amount of available parallel, human-annotated corpora for training NMT systems is at times very scarce. This is the case of many low-resource languages and specialized translation domains (e.g., health care). Consequently, there has been a growing interest in developing unsupervised NMT models BIBREF36, BIBREF37, BIBREF38 which do not require annotated data for training. Such models learn to translate by only using monolingual corpora, and even though their accuracy is still well below that of their supervised counterparts, they have started to reach interesting levels. The architecture of unsupervised NMT systems differs from that of supervised systems in that it combines translation in both directions (source-to-target and target-to-source). Typically, a single encoder is used to encode sentences from both languages, and a separate decoder generates the translations in each language. The training of such systems follows three stages: 1) building a bilingual dictionary and word embedding space, 2) training two monolingual language models as denoising autoencoders BIBREF39, and 3) converting the unsupervised problem into a weakly-supervised one by use of back-translations BIBREF40. For more details on unsupervised NMT systems, we refer the reader to the original papers BIBREF36, BIBREF37, BIBREF38.","In this paper, we explore using the proposed regularization approach also for unsupervised NMT. Unsupervised NMT models still require very large amounts of monolingual data for training, and often such amounts are not available. Therefore, these models, too, are expected to benefit from improved regularization."]},{"idx":6,"heading":"The Baseline NMT model","paragraphs":["In this section, we describe the NMT model that has been used as the basis for the proposed regularizer. It is a neural encoder-decoder architecture with attention BIBREF1 that can be regarded as a strong baseline as it incorporates both LSTMs and transformers as modules. Let us assume that $\\textbf {x}:\\lbrace x_1 \\dots x_n\\rbrace $ is the source sentence with $n$ tokens and $\\textbf {y}:\\lbrace y_1 \\dots y_m\\rbrace $ is the target translated sentence with $m$ tokens. First, the words in the source sentence are encoded into their word embeddings by an embedding layer:","and then the source sentence is encoded by a sequential module into its hidden vectors, ${\\textbf {h}_1 \\dots \\textbf {h}_n}$:","Next, for each decoding step $j=1 \\ldots m$, an attention network provides a context vector $\\textbf {c}_j$ as a weighted average of all the encoded vectors, $\\textbf {h}_1 \\dots \\textbf {h}_n$, conditional on the decoder output at the previous step, $\\textbf {s}_{j-1}$ (Eq. DISPLAY_FORM17). For this network, we have used the attention mechanism of Badhdanau et al.BIBREF1.","Given the context vector, $\\textbf {c}_j$, the decoder output at the previous step, $\\textbf {s}_{j-1}$, and the word embedding of the previous word in the target sentence, $\\textbf {y}^{e}_{j}$ (Eq. DISPLAY_FORM18), the decoder generates vector $\\textbf {s}_j$ (Eq. DISPLAY_FORM19). This vector is later transformed into a larger vector of the same size as the target vocabulary via learned parameters $\\textbf {W}$, $\\textbf {b}$ and a softmax layer (Eq. DISPLAY_FORM20). The resulting vector, $\\textbf {p}_j$, is the inferred probability distribution over the target vocabulary at decoding step $j$. Fig. FIGREF12 depicts the full architecture of the baseline model.","The model is trained by minimizing the negative log-likelihood (NLL) which can be expressed as:","where the probability of ground-truth word ${y}_j$ has been noted as $\\textbf {p}_{j}({y}_{j})$. Minimizing the NLL is equivalent to MLE and results in assigning maximum probability to the words in the reference translation, $y_j, j=1 \\ldots m$. The training objective is minimized with standard backpropagation over the training data, and at inference time the model uses beam search for decoding."]},{"idx":7,"heading":"Regressing word and sentence embeddings","paragraphs":["As mentioned in the introduction, MLE suffers from some limitations when training a neural machine translation system. To alleviate these shortcomings, in our recent paper BIBREF8 we have proposed a new regularization method based on regressing word embeddings. In this paper, we extend this idea to sentence embeddings."]},{"idx":8,"heading":"Regressing word and sentence embeddings ::: ReWE","paragraphs":["Pre-trained word embeddings are trained on large monolingual corpora by measuring the co-occurences of words in text windows (“contexts”). Words that occur in similar contexts are assumed to have similar meaning, and hence, similar vectors in the embedding space. Our goal with ReWE is to incorporate the information embedded in the word vector in the loss function to encourage model regularization.","In order to generate continuous vector representations as outputs, we have added a ReWE block to the NMT baseline (Fig. FIGREF14). At each decoding step, the ReWE block receives the hidden vector from the decoder, $\\textbf {s}_j$, as input and outputs another vector, $\\textbf {e}_j$, of the same size of the pre-trained word embeddings:","where $\\textbf {W}_1$, $\\textbf {W}_2$, $\\textbf {b}_1$ and $\\textbf {b}_2$ are the learnable parameters of a two-layer feed-forward network with a Rectified Linear Unit (ReLU) as activation function between the layers. Vector $\\textbf {e}_j$ aims to reproduce the word embedding of the target word, and thus the distributional properties (or co-occurrences) of its contexts. During training, the model is guided to regress the predicted vector, $\\textbf {e}_j$, towards the word embedding of the ground-truth word, $\\textbf {y}^{e}_j$. This is achieved by using a loss function that computes the distance between $\\textbf {e}_j$ and $\\textbf {y}^{e}_j$ (Eq. DISPLAY_FORM24). Previous work BIBREF8 has showed that the cosine distance is empirically an effective distance between word embeddings and has thus been adopted as loss. This loss and the original NLL loss are combined together with a tunable hyper-parameter, $\\lambda $ (Eq. DISPLAY_FORM25). Therefore, the model is trained to jointly predict both a categorical and a continuous representation of the words. Even though the system is performing a single task, this setting could also be interpreted as a form of multi-task learning with different representations of the same targets.","The word vectors of both the source ($\\textbf {x}^{e}$) and target ($\\textbf {y}^{e}$) vocabularies are initialized with pre-trained embeddings, but updated during training. At inference time, we ignore the outputs of the ReWE block and we perform translation using only the categorical prediction."]},{"idx":9,"heading":"Regressing word and sentence embeddings ::: ReSE","paragraphs":["Sentence vectors, too, have been extensively used as input representations in many NLP tasks such as text classification, paraphrase detection, natural language inference and question answering. The intuition behind them is very similar to that of word embeddings: sentences with similar meanings are expected to be close to each other in vector space. Many off-the-shelf sentence embedders are currently available and they can be easily integrated in deep learning models. Based on similar assumptions to the case of word embeddings, we have hypothesized that an NMT model could also benefit from a regularization term based on regressing sentence embeddings (the ReSE block in Fig. FIGREF14).","The main difference of ReSE compared to ReWE is that there has to be a single regressed vector per sentence rather than one per word. Thus, ReSE first uses a self-attention mechanism to learn a weighted average of the decoder's hidden vectors, $\\textbf {s}_1 \\dots \\textbf {s}_m$:","where the $\\alpha _j$ attention weights are obtained from Eqs. DISPLAY_FORM28 and DISPLAY_FORM29, and $\\textbf {U}_1$ and $\\textbf {U}_2$ are learnable parameters. Then, a two-layered neural network similar to ReWE's predicts the sentence vector, $\\textbf {r}$ (Eq. DISPLAY_FORM30). Parameters $\\textbf {W}_3$, $\\textbf {W}_4$, $\\textbf {b}_3$ and $\\textbf {b}_4$ are also learned during training.","Similarly to ReWE, a loss function computes the cosine distance between the predicted sentence vector, $\\textbf {r}$, and the sentence vector inferred with the off-the-shelf sentence embedder, $\\textbf {y}^r$ (Eq. DISPLAY_FORM31). This loss is added to the previous objective as an extra term with an additional, tunable hyper-parameter, $\\beta $:","Since the number of sentences is significantly lower than that of the words, $\\beta $ typically needs to be higher than $\\lambda $. Nevertheless, we tune it blindly using the validation set. The reference sentence embedding, $\\textbf {y}^{r}$, can be inferred with any off-the-shelf pre-trained embedder. At inference time, the model solely relies on the categorical prediction and ignores the predicted word and sentence vectors."]},{"idx":10,"heading":"Experiments","paragraphs":["We have carried out an ample range of experiments to probe the performance of the proposed regularization approaches. This section describes the datasets, the models and the hyper-parameters used, and presents and discusses all results."]},{"idx":11,"heading":"Experiments ::: Datasets","paragraphs":["Four different language pairs have been selected for the experiments. The datasets' size varies from tens of thousands to millions of sentences to test the regularizers' ability to improve translation over a range of low-resource and high-resource language pairs.","De-En: The German-English dataset (de-en) has been taken from the WMT18 news translation shared task. The training set contains over 5M sentence pairs collected from the Europarl, CommonCrawl and Newscommentary parallel corpora. As validation and test sets, we have used the newstest2017 and the newstest2018 datasets, respectively. We consider this dataset as a high-resource case.","En-Fr: The English-French dataset (en-fr) has been sourced from the IWSLT 2016 translation shared task. This corpus contains translations of TED talks of very diverse topics. The training data provided by the organizers consist of $219,777$ translations which allow us to categorize this dataset as low/medium-resource. Following Denkowski and Neubig BIBREF41, the validation set has been formed by merging the 2013 and 2014 test sets from the same shared task, and the test set has been formed with the 2015 and 2016 test sets.","Cs-En: The Czech-English dataset (cs-en) is also from the IWSLT 2016 TED talks translation task. However, this dataset is approximately half the size of en-fr as its training set consists of $114,243$ sentence pairs. Again following Denkowski and Neubig BIBREF41), the validation set has been formed by merging the 2012 and 2013 test sets, and the test set by merging the 2015 and 2016 test sets. We regard this dataset as low-resource.","Eu-En: The Basque-English dataset (eu-en) has been collected from the WMT16 IT-domain translation shared task. This is the smallest dataset, with only $89,413$ sentence pairs in the training set. However, only $2,000$ sentences in the training set have been translated by human annotators. The remaining sentence pairs are translations of IT-domain short phrases and Wikipedia titles. Therefore, we consider this dataset as extremely low-resource. It must be said that translations in the IT domain are somehow easier than in the news domain, as this domain is very specific and the wording of the sentences are less varied. For this dataset, we have used the validation and test sets ($1,000$ sentences each) provided in the shared task.","All the datasets have been pre-processed with moses-tokenizer. Additionally, words have been split into subword units using byte pair encoding (BPE) BIBREF42. For the BPE merge operations parameter, we have used $32,000$ (the default value) for all the datasets, except for eu-en where we have set it to $8,000$ since this dataset is much smaller. Experiments have been performed at both word and subword level since morphologically-rich languages such as German, Czech and Basque can benefit greatly from operating the NMT model at subword level."]},{"idx":12,"heading":"Experiments ::: Model Training and Hyper-Parameter Selection","paragraphs":["To implement ReWE and ReSE, we have modified the popular OpenNMT open-source toolkit BIBREF43. Two variants of the standard OpenNMT model have been used as baselines: the LSTM and the transformer, described hereafter.","LSTM: A strong NMT baseline was prepared by following the indications given by Denkowski and Neubig BIBREF41. The model uses a bidirectional LSTM BIBREF44 for the encoder and a unidirectional LSTM for the decoder, with two layers each. The size of the word embeddings was set to 300d and that of the sentence embeddings to 512d. The sizes of the hidden vectors of both LSTMs and of the attention network were set to 1024d. In turn, the LSTM's dropout rate was set to $0.2$ and the training batch size was set to 40 sentences. As optimizer, we have used Adam BIBREF45 with a learning rate of $0.001$. During training, the learning rate was halved with simulated annealing upon convergence of the perplexity over the validation set, which was evaluated every $25,000$ training sentences. Training was stopped after halving the learning rate 5 times.","Transformer: The transformer network BIBREF3 has somehow become the de-facto neural network for the encoder and decoder of NMT pipelines thanks to its strong empirical accuracy and highly-parallelizable training. For this reason, we have used it as another baseline for our model. For its hyper-parameters, we have used the default values set by the developers of OpenNMT. Both the encoder and the decoder are formed by a 6-layer network. The sizes of the word embeddings, the hidden vectors and the attention network have all been set to either 300d or 512d, depending on the best results over the validation set. The head count has been set correspondingly to either 6 or 8, and the dropout rate to $0.2$ as for the LSTM. The model was also optimized using Adam, but with a much higher learning rate of 1 (OpenAI default). For this model, we have not used simulated annealing since some preliminary experiments showed that it did penalize performance. The batch size used was $4,096$ and $1,024$ words, again selected based on the accuracy over the validation set. Training was stopped upon convergence in perplexity over the validation set, which was evaluated at every epoch.","In addition, the word embeddings for both models were initialized with pre-trained fastText embeddings BIBREF26. For the 300d word embeddings, we have used the word embeddings available on the official fastText website. For the 512d embeddings and the subword units, we have trained our own pre-trained vectors using the fastText embedder with a large monolingual corpora from Wikipedia and the training data. Both models have used the same sentence embeddings which have been computed with the Universal Sentence Encoder (USE). However, the USE is only available for English, so we have only been able to use ReSE with the datasets where English is the target language (i.e., de-en, cs-en and eu-en). When using BPE, the subwords of every sentence have been merged back into words before passing them to the USE. The BLEU score for the BPE models has also been computed after post-processing the subwords back into words. Finally, hyper-parameters $\\lambda $ and $\\beta $ have been tuned only once for all datasets by using the en-fr validation set. This was done in order to save the significant computational time that would have been required by further hyper-parameter exploration. However, in the de-en case the initial results were far from the state of the art and we therefore repeated the selection with its own validation set. For all experiments, we have used an Intel Xeon E5-2680 v4 with an NVidia GPU card Quadro P5000. On this machine, the training time of the transformer has been approximately an order of magnitude larger than that of the LSTM."]},{"idx":13,"heading":"Experiments ::: Results","paragraphs":["We have carried out a number of experiments with both baselines. The scores reported are an average of the BLEU scores (in percentage points, or pp) BIBREF46 over the test sets of 5 independently trained models. Table TABREF44 shows the results over the en-fr dataset. In this case, the models with ReWE have outperformed the LSTM and transformer baselines consistently. The LSTM did not benefit from using BPE, but the transformer+ReWE with BPE reached $36.30$ BLEU pp (a $+0.99$ pp improvement over the best model without ReWE). For this dataset we did not use ReSE because French was the target language.","Table TABREF45 reports the results over the cs-en dataset. Also in this case, all the models with ReWE have improved over the corresponding baselines. The LSTM+ReWE has achieved the best results ($23.72$ BLEU pp; an improvement of $+1.16$ pp over the best model without ReWE). This language pair has also benefited more from the BPE pre-processing, likely because Czech is a morphologically-rich language. For this dataset, it was possible to use ReSE in combination with ReWE, with an improvement for the LSTM at word level ($+0.14$ BLEU pp), but not for the remaining cases. We had also initially tried to use ReSE without ReWE (i.e., $\\lambda =0$), but the results were not encouraging and we did not continue with this line of experiments.","For the eu-en dataset (Table TABREF46), the results show that, again, ReWE outperforms the baselines by a large margin. Moreover, ReWE+ReSE has been able to improve the results even further ($+3.15$ BLEU pp when using BPE and $+5.15$ BLEU pp at word level over the corresponding baselines). Basque is, too, a morphologically-rich language and using BPE has proved very beneficial ($+4.27$ BLEU pp over the best word-level model). As noted before, the eu-en dataset is very low-resource (less than $100,000$ sentence pairs) and it is more likely that the baseline models generalize poorly. Consequently, regularizers such as ReWE and ReSE are more helpful, with larger margins of improvement with respect to the baselines. On a separate note, the transformer has unexpectedly performed well below the LSTM on this dataset, and especially so with BPE. We speculate that it may be more sensitive than the LSTM to the dataset's much smaller size, or in need of more refined hyper-parameter tuning.","Finally, Table TABREF47 shows the results over the de-en dataset that we categorize as high-resource (5M+ sentence pairs). For this dataset, we have only been able to perform experiments with the LSTM due to the exceedingly long training times of the transformer. At word level, both ReWE and ReWE+ReSE have been able to outperform the baseline, although the margins of improvement have been smaller than for the other language pairs ($+0.42$ and $+0.48$ BLEU pp, respectively). However, when using BPE both ReWE and ReWE+ReSE have performed slightly below the baseline ($-0.37$ and $-0.12$ points BLEU pp, respectively). This shows that when the training data are abundant, ReWE or ReSE may not be beneficial. To probe this further, we have repeated these experiments by training the models over subsets of the training set of increasing size (200K, 500K, 1M, and 2M sentence pairs). Fig. FIGREF57 shows the BLEU scores achieved by the baseline and the regularized models for the different training data sizes. The plot clearly shows that the performance margin increases as the training data size decreases, as expected from a regularized model.","Table TABREF54 shows two examples of the translations made by the different LSTM models for eu-en and cs-en. A qualitative analysis of these examples shows that both ReWE and ReWE+ReSE have improved the quality of these translations. In the eu-en example, ReWE has correctly translated “File tab”; and ReSE has correctly added “click Create”. In the cs-en example, the model with ReWE has picked the correct subject “they”, and only the model with ReWE and ReSE has correctly translated “students” and captured the opening phrase “What was...about this...”."]},{"idx":14,"heading":"Experiments ::: Understanding ReWE and ReSE","paragraphs":["The quantitative experiments have proven that ReWE and ReSE can act as effective regularizers for low- and medium-resource NMT. Yet, it would be very interesting to understand how do they influence the training to achieve improved models. For that purpose, we have conducted an exploration of the values of the hidden vectors on the decoder end ($\\textbf {s}_j$, Eq. DISPLAY_FORM19). These values are the “feature space” used by the final classification block (a linear transformation and a softmax) to generate the class probabilities and can provide insights on the model. For this reason, we have considered the cs-en test set and stored all the $\\textbf {s}_j$ vectors with their respective word predictions. Then, we have used t-SNE BIBREF47 to reduce the dimensionality of the $\\textbf {s}_j$ vectors to a visualizable 2d. Finally, we have chosen a particular word (architecture) as the center of the visualization, and plotted all the vectors within a chosen neighborhood of this center word (Fig. FIGREF58). To avoid cluttering the figure, we have not superimposed the predicted words to the vectors, but only used a different color for each distinct word. The center word in the two subfigures (a: baseline; b: baseline+ReWE) is the same (architecture) and from the same source sentence, so the visualized regions are comparable. The visualizations also display all other predicted instances of word architecture in the neighborhood.","These visualizations show two interesting behaviors: 1) from eye judgment, the points predicted by the ReWE model seem more uniformly spread out; 2) instances of the same words have $\\textbf {s}_j$ vectors that are close to each other. For instance, several instances of word architecture are close to each other in Fig. FIGREF58 while a single instance appears in Fig. FIGREF58. The overall observation is that the ReWE regularizer leads to a vector space that is easier to discriminate, i.e. find class boundaries for, facilitating the final word prediction. In order to confirm this observation, we have computed various clustering indexes over the clusters formed by the vectors with identical predicted word. As indexes, we have used the silhouette and the Davies-Bouldin indexes that are two well-known unsupervised metrics for clustering. The silhouette index ranges from -1 to +1, where values closer to 1 mean that the clusters are compact and well separated. The Davies-Bouldin index is an unbounded nonnegative value, with values closer to 0 meaning better clustering. Table TABREF62 shows the values of these clustering indexes over the entire cs-en test set for the LSTM models. As the table shows, the models with ReWE and ReWE+ReSE have reported the best values. This confirms that applying ReWE and ReSE has a positive impact on the decoder's hidden space, ultimately justifying the increase in word classification accuracy.","For further exploration, we have created another visualization of the $\\textbf {s}$ vectors and their predictions over a smaller neighborhood (Fig. FIGREF63). The same word (architecture) has been used as the center word of the plot. Then, we have “vibrated” each of the $\\textbf {s}_j$ vector by small increments (between 0.05 and 8 units) in each of their dimensions, creating several new synthetic instances of $\\textbf {s}$ vectors which are very close to the original ones. These synthetic vectors have then been decoded with the trained NMT model to obtain their predicted words. Finally, we have used t-SNE to reduce the dimensionality to 2d, and visualized all the vectors and their predictions in a small neighborhood ($\\pm 10$ units) around the center word. Fig. FIGREF63 shows that, with the ReWE model, all the $\\textbf {s}$ vectors surrounding the center word predict the same word (architecture). Conversely, with the baseline, the surrounding points predict different words (power, force, world). This is additional evidence that the $\\textbf {s}$ space is evened out by the use of the proposed regularizer."]},{"idx":15,"heading":"Experiments ::: Unsupervised NMT","paragraphs":["Finally, we have also experimented with the use of ReWE and ReWE+ReSE for an unsupervised NMT task. For this experiment, we have used the open-source model provided by Lample et al. BIBREF36 which is currently the state of the art for unsupervised NMT, and also adopted its default hyper-parameters and pre-processing steps which include 4-layer transformers for the encoder and both decoders, and BPE subword learning. The experiments have been performed using the WMT14 English-French test set for testing in both language directions (en-fr and fr-en), and the monolingual data from that year's shared task for training.","As described in Section SECREF13, an unsupervised NMT model contains two decoders to be able to translate into both languages. The model is trained by iterating over two alternate steps: 1) training using the decoders as monolingual, de-noising language models (e.g., en-en, fr-fr), and 2) training using back-translations (e.g., en-fr-en, fr-en-fr). Each step requires an objective function, which is usually an NLL loss. Moreover, each step is performed in both directions (en$\\rightarrow $fr and fr$\\rightarrow $en), which means that an unsupervised NMT model uses a total of four different objective functions. Potentially, the regularizers could be applied to each of them. However, the pre-trained USE sentence embeddings are only available in English, not in French, and for this reason we have limited our experiments to ReWE alone. In addition, the initial results have showed that ReWE is actually detrimental in the de-noising language model step, so we have limited its use to both language directions in the back-translation step, with the hyper-parameter, $\\lambda $, tuned over the validation set ($\\lambda =0.2$).","To probe the effectiveness of the regularized model, Fig. FIGREF67 shows the results over the test set from the different models trained with increasing amounts of monolingual data (50K, 500K, 1M, 2M, 5M and 10M sentences in each language). The model trained using ReWE has been able to consistently outperform the baseline in both language directions. The trend we had observed in the supervised case has applied to these experiments, too: the performance margin has been larger for smaller training data sizes. For example, in the en-fr direction the margin has been $+1.74$ BLEU points with 50K training sentences, but it has reduced to $+0.44$ BLEU points when training with 10M sentences. Again, this behavior is in line with the regularizing nature of the proposed regressive objectives."]},{"idx":16,"heading":"Conclusion","paragraphs":["In this paper, we have proposed regressing continuous representations of words and sentences (ReWE and ReSE, respectively) as novel regularization techniques for improving the generalization of NMT models. Extensive experiments over four different language pairs of different training data size (from 89K to 5M sentence pairs) have shown that both ReWE and ReWE+ReSE have improved the performance of NMT models, particularly in low- and medium-resource cases, for increases in BLEU score up to $5.15$ percentage points. In addition, we have presented a detailed analysis showing how the proposed regularization modifies the decoder's output space, enhancing the clustering of the vectors associated with unique words. Finally, we have showed that the regularized models have also outperformed the baselines in experiments on unsupervised NMT. As future work, we plan to explore how the categorical and continuous predictions from our model could be jointly utilized to further improve the quality of the translations."]},{"idx":17,"heading":"Acknowledgment","paragraphs":["The authors would like to thank the RoZetta Institute (formerly CMCRC) for providing financial support to this research.","[]Inigo Jauregi Unanue received the BEng degree in telecommunication systems from University of Navarra, Donostia-San Sebastian, Spain, in 2016. From 2014 to 2016, he was a research assistant at Centro de Estudio e Investigaciones Tecnicas (CEIT). Since 2016, he is a natural language processing and machine learning researcher at the RoZetta Institute (former CMCRC) in Sydney, Australia. Additionally, he is in the last year of his PhD at University of Technology Sydney, Australia. His research interests are machine learning, natural language processing and information theory.","[]Ehsan Zare Borzeshi received the PhD degree from University of Technology Sydney, Australia, in 2013. He is currently a Senior Data & Applied Scientist with Microsoft CSE (Commercial Software Engineering). He has previously held appointments as a senior researcher at the University of Newcastle, University of Technology Sydney, and the RoZetta Institute (formerly CMCRC) in Sydney. He has also been a Visiting Scholar with the University of Central Florida, Orlando, FL, USA. His current research interests include big data, deep learning and natural language processing where he has many publications.","[]Massimo Piccardi (SM'05) received the MEng and PhD degrees from the University of Bologna, Bologna, Italy, in 1991 and 1995, respectively. He is currently a Full Professor of computer systems with University of Technology Sydney, Australia. His research interests include natural language processing, computer vision and pattern recognition and he has co-authored over 150 papers in these areas. Prof. Piccardi is a Senior Member of the IEEE, a member of its Computer and Systems, Man, and Cybernetics Societies, and a member of the International Association for Pattern Recognition. He presently serves as an Associate Editor for the IEEE Transactions on Big Data."]},{"idx":18,"heading":"Figures and Tables","paragraphs":["Fig. 1: Baseline NMT model. (Left) The encoder receives the input sentence and generates a context vector cj for each decoding step using an attention mechanism. (Right) The decoder generates one-by-one the output vectors pj , which represent the probability distribution over the target vocabulary. During training yj is a token from the ground truth sentence, but during inference the model uses its own predictions.","Fig. 2: Full model: Baseline + ReWE + ReSE. (Left) The encoder with the attention mechanism generates vectos cj in the same way as the baseline system. (Right) The decoder generates one-by-one the output vectors pj , which represent the probability distribution over the target vocabulary, and ej , which is a continuous word vector. Additionally, the model can also generate another continuous vector, r, which represents the sentence embedding.","TABLE I: BLEU scores over the En-Fr test set. The reported results are the average of 5 independent runs.","TABLE III: BLEU scores over the Eu-En test set. The reported results are the average of 5 independent runs.","TABLE II: BLEU scores over the Cs-En test set. The reported results are the average of 5 independent runs.","TABLE IV: BLEU scores over the De-En test set. The reported results are the average of 5 independent runs.","TABLE V: Translation examples. Example 1: Eu-En and Example 2: Cs-En.","Fig. 3: BLEU scores over the De-En test set for models trained with training sets of different size.","Fig. 4: Visualization of the sj vectors from the decoder for a subset of the cs-en test set. Please refer to Section V-D for explanations. This figure should be viewed in color.","TABLE VI: Clustering indexes of the LSTM models over the cs-en test set. The reported results are the average of 5 independent runs.","Fig. 5: Visualization of the sj vectors in a smaller neighborhood of the center word.","Fig. 6: BLEU scores over the test set. The reported results are the average of 5 independent runs.. The red line represents the baseline model and the blue line is the baseline + ReWE."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1910.01108","title":"DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter","sections":[{"idx":0,"heading":"Abstract","paragraphs":["As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging. In this work, we propose a method to pre-train a smaller general-purpose language representation model, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger counterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage the inductive biases learned by larger models during pre-training, we introduce a triple loss combining language modeling, distillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train and we demonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative on-device study."]},{"idx":1,"heading":"Introduction","paragraphs":["The last two years have seen the rise of Transfer Learning approaches in Natural Language Processing (NLP) with large-scale pre-trained language models becoming a basic tool in many NLP tasks BIBREF0, BIBREF1, BIBREF2. While these models lead to significant improvement, they often have several hundred million parameters and current research on pre-trained models indicates that training even larger models still leads to better performances on downstream tasks.","The trend toward bigger models raises several concerns. First is the environmental cost of exponentially scaling these models' computational requirements as mentioned in BIBREF3, BIBREF4. Second, while operating these models on-device in real-time has the potential to enable novel and interesting language processing applications, the growing computational and memory requirements of these models may hamper wide adoption.","In this paper, we show that it is possible to reach similar performances on many downstream-tasks using much smaller language models pre-trained with knowledge distillation, resulting in models that are lighter and faster at inference time, while also requiring a smaller computational training budget. Our general-purpose pre-trained models can be fine-tuned with good performances on several downstream tasks, keeping the flexibility of larger models. We also show that our compressed models are small enough to run on the edge, e.g. on mobile devices.","Using a triple loss, we show that a 40% smaller Transformer (BIBREF5) pre-trained through distillation via the supervision of a bigger Transformer language model can achieve similar performance on a variety of downstream tasks, while being 60% faster at inference time. Further ablation studies indicate that all the components of the triple loss are important for best performances.","We have made the trained weights available along with the training code in the Transformers library from HuggingFace BIBREF6."]},{"idx":2,"heading":"Knowledge distillation","paragraphs":["Knowledge distillation BIBREF7, BIBREF8 is a compression technique in which a compact model - the student - is trained to reproduce the behaviour of a larger model - the teacher - or an ensemble of models.","In supervised learning, a classification model is generally trained to predict an instance class by maximizing the estimated probability of gold labels. A standard training objective thus involves minimizing the cross-entropy between the model's predicted distribution and the one-hot empirical distribution of training labels. A model performing well on the training set will predict an output distribution with high probability on the correct class and with near-zero probabilities on other classes. But some of these \"near-zero\" probabilities are larger than others and reflect, in part, the generalization capabilities of the model and how well it will perform on the test set.","Training loss The student is trained with a distillation loss over the soft target probabilities of the teacher: $L_{ce} = \\sum _i t_i * \\log (s_i)$ where $t_i$ (resp. $s_i$) is a probability estimated by the teacher (resp. the student). This objective results in a rich training signal by leveraging the full teacher distribution. Following BIBREF8 we used a softmax-temperature: $p_i = \\frac{\\exp (z_i / T)}{\\sum _j \\exp (z_j / T)}$ where $T$ controls the smoothness of the output distribution and $z_i$ is the model score for the class $i$. The same temperature $T$ is applied to the student and the teacher at training time, while at inference, $T$ is set to 1 to recover a standard softmax.","The final training objective is a linear combination of the distillation loss $L_{ce}$ with the supervised training loss, in our case the masked language modeling loss $L_{mlm}$ BIBREF0. We found it beneficial to add a cosine embedding loss ($L_{cos}$) which will tend to align the directions of the student and teacher hidden states vectors."]},{"idx":3,"heading":"DistilBERT: a distilled version of BERT","paragraphs":["Student architecture In the present work, the student - DistilBERT - has the same general architecture as BERT. The token-type embeddings and the pooler are removed while the number of layers is reduced by a factor of 2. Most of the operations used in the Transformer architecture (linear layer and layer normalisation) are highly optimized in modern linear algebra frameworks and our investigations showed that variations on the last dimension of the tensor (hidden size dimension) have a smaller impact on computation efficiency (for a fixed parameters budget) than variations on other factors like the number of layers. Thus we focus on reducing the number of layers.","Student initialization In addition to the previously described optimization and architectural choices, an important element in our training procedure is to find the right initialization for the sub-network to converge. Taking advantage of the common dimensionality between teacher and student networks, we initialize the student from the teacher by taking one layer out of two.","Distillation We applied best practices for training BERT model recently proposed in BIBREF2. As such, DistilBERT is distilled on very large batches leveraging gradient accumulation (up to 4K examples per batch) using dynamic masking and without the next sentence prediction objective.","Data and compute power We train DistilBERT on the same corpus as the original BERT model: a concatenation of English Wikipedia and Toronto Book Corpus BIBREF9. DistilBERT was trained on 8 16GB V100 GPUs for approximately 90 hours. For the sake of comparison, the RoBERTa model BIBREF2 required 1 day of training on 1024 32GB V100."]},{"idx":4,"heading":"Experiments","paragraphs":["General Language Understanding We assess the language understanding and generalization capabilities of DistilBERT on the General Language Understanding Evaluation (GLUE) benchmark BIBREF10, a collection of 9 datasets for evaluating natural language understanding systems. We report scores on the development sets for each task by fine-tuning DistilBERT without the use of ensembling or multi-tasking scheme for fine-tuning (which are mostly orthogonal to the present work). We compare the results to the baseline provided by the authors of GLUE: an ELMo (BIBREF11) encoder followed by two BiLSTMs.","The results on each of the 9 tasks are showed on Table TABREF6 along with the macro-score (average of individual scores). Among the 9 tasks, DistilBERT is always on par or improving over the ELMo baseline (up to 20 points of accuracy on STS-B). DistilBERT also compares surprisingly well to BERT, retaining 97% of the performance with 40% fewer parameters."]},{"idx":5,"heading":"Experiments ::: Downstream task benchmark","paragraphs":["Downstream tasks We further study the performances of DistilBERT on several downstream tasks under efficient inference constraints: a classification task (IMDb sentiment classification - BIBREF13) and a question answering task (SQuAD v1.1 - BIBREF14).","As shown in Table TABREF8, DistilBERT is only 0.6% point behind BERT in test accuracy on the IMDb benchmark while being 40% smaller. On SQuAD, DistilBERT is within 3.5 points of the full BERT.","We also studied whether we could add another step of distillation during the adaptation phase by fine-tuning DistilBERT on SQuAD using a BERT model previously fine-tuned on SQuAD as a teacher for an additional term in the loss (knowledge distillation). In this setting, there are thus two successive steps of distillation, one during the pre-training phase and one during the adaptation phase. In this case, we were able to reach interesting performances given the size of the model: 86.9 F1 and 79.1 EM, i.e. within 2 points of the full model.","Size and inference speed","To further investigate the speed-up/size trade-off of DistilBERT, we compare (in Table TABREF8) the number of parameters of each model along with the inference time needed to do a full pass on the STS-B development set on CPU (Intel Xeon E5-2690 v3 Haswell @2.9GHz) using a batch size of 1. DistilBERT has 40% fewer parameters than BERT and is 60% faster than BERT.","On device computation We studied whether DistilBERT could be used for on-the-edge applications by building a mobile application for question answering. We compare the average inference time on a recent smartphone (iPhone 7 Plus) against our previously trained question answering model based on BERT-base. Excluding the tokenization step, DistilBERT is 71% faster than BERT, and the whole model weighs 207 MB (which could be further reduced with quantization). Our code is available."]},{"idx":6,"heading":"Experiments ::: Ablation study","paragraphs":["In this section, we investigate the influence of various components of the triple loss and the student initialization on the performances of the distilled model. We report the macro-score on GLUE. Table TABREF11 presents the deltas with the full triple loss: removing the Masked Language Modeling loss has little impact while the two distillation losses account for a large portion of the performance."]},{"idx":7,"heading":"Related work","paragraphs":["Task-specific distillation Most of the prior works focus on building task-specific distillation setups. BIBREF15 transfer fine-tune classification model BERT to an LSTM-based classifier. BIBREF16 distill BERT model fine-tuned on SQuAD in a smaller Transformer model previously initialized from BERT. In the present work, we found it beneficial to use a general-purpose pre-training distillation rather than a task-specific distillation. BIBREF17 use the original pretraining objective to train smaller student, then fine-tuned via distillation. As shown in the ablation study, we found it beneficial to leverage the teacher's knowledge to pre-train with additional distillation signal.","Multi-distillation BIBREF18 combine the knowledge of an ensemble of teachers using multi-task learning to regularize the distillation. The authors apply Multi-Task Knowledge Distillation to learn a compact question answering model from a set of large question answering models. An application of multi-distillation is multi-linguality: BIBREF19 adopts a similar approach to us by pre-training a multilingual model from scratch solely through distillation. However, as shown in the ablation study, leveraging the teacher's knowledge with initialization and additional losses leads to substantial gains.","Other compression techniques have been studied to compress large models. Recent developments in weights pruning reveal that it is possible to remove some heads in the self-attention at test time without significantly degrading the performance BIBREF20. Some layers can be reduced to one head. A separate line of study leverages quantization to derive smaller models (BIBREF21). Pruning and quantization are orthogonal to the present work."]},{"idx":8,"heading":"Conclusion and future work","paragraphs":["We introduced DistilBERT, a general-purpose pre-trained version of BERT, 40% smaller, 60% faster, that retains 97% of the language understanding capabilities. We showed that a general-purpose language model can be successfully trained with distillation and analyzed the various components with an ablation study. We further demonstrated that DistilBERT is a compelling option for edge applications."]},{"idx":9,"heading":"Figures and Tables","paragraphs":["Table 1: DistilBERT retains 97% of BERT performance. Comparison on the dev sets of the GLUE benchmark. ELMo results as reported by the authors. BERT and DistilBERT results are the medians of 5 runs with different seeds.","Table 2: DistilBERT yields to comparable performance on downstream tasks. Comparison on downstream tasks: IMDb (test accuracy) and SQuAD 1.1 (EM/F1 on dev set). D: with a second step of distillation during fine-tuning.","Table 4: Ablation study. Variations are relative to the model trained with triple loss and teacher weights initialization."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1910.07973","title":"Universal Text Representation from BERT: An Empirical Study","sections":[{"idx":0,"heading":"Abstract","paragraphs":["We present a systematic investigation of layer-wise BERT activations for general-purpose text representations to understand what linguistic information they capture and how transferable they are across different tasks. Sentence-level embeddings are evaluated against two state-of-the-art models on downstream and probing tasks from SentEval, while passage-level embeddings are evaluated on four question-answering (QA) datasets under a learning-to-rank problem setting. Embeddings from the pre-trained BERT model perform poorly in semantic similarity and sentence surface information probing tasks. Fine-tuning BERT on natural language inference data greatly improves the quality of the embeddings. Combining embeddings from different BERT layers can further boost performance. BERT embeddings outperform BM25 baseline significantly on factoid QA datasets at the passage level, but fail to perform better than BM25 on non-factoid datasets. For all QA datasets, there is a gap between embedding-based method and in-domain fine-tuned BERT (we report new state-of-the-art results on two datasets), which suggests deep interactions between question and answer pairs are critical for those hard tasks."]},{"idx":1,"heading":"Introduction","paragraphs":["Universal text representations are important for many NLP tasks as modern deep learning models are becoming more and more data-hungry and computationally expensive. On one hand, most research and industry tasks face data sparsity problem due to the high cost of annotation. Universal text representations can mitigate this problem to a certain extent by performing implicit transfer learning among tasks. On the other hand, modern deep learning models with millions of parameters are expensive to train and host, while models using text representation as the building blocks can achieve similar performance with much fewer tunable parameters. The pre-computed text embeddings can also help decrease model latency dramatically at inference time.","Since the introduction of pre-trained word embeddings such as word2vec BIBREF0 and GloVe BIBREF1, a lot of efforts have been devoted to developing universal sentence embeddings. Initial attempts at learning sentence representation using unsupervised approaches did not yield satisfactory performance. Recent work BIBREF2 has shown that models trained in supervised fashion on datasets like Stanford Natural Language Inference (SNLI) corpus BIBREF3 can consistently outperform unsupervised methods like SkipThought vectors BIBREF4. More recently, Universal Sentence Encoder BIBREF5 equipped with the Transformer BIBREF6 as the encoder, co-trained on a large amount of unsupervised training data and SNLI corpus, has demonstrated surprisingly good performance with minimal amounts of supervised training data for a transfer task.","BERT BIBREF7, one of the latest models that leverage heavily on language model pre-training, has achieved state-of-the-art performance in many natural language understanding tasks ranging from sequence and sequence pair classification to question answering. The fact that pre-trained BERT can be easily fine-tuned with just one additional output layer to create a state-of-the-art model for a wide range of tasks suggests that BERT representations are potential universal text embeddings.","Passages that consist of multiple sentences are coherent units of natural languages that convey information at a pragmatic or discourse level. While there are many models for generating and evaluating sentence embeddings, there hasn't been a lot of work on passage level embedding generation and evaluation.","In this paper, we conducted an empirical study of layer-wise activations of BERT as general-purpose text embeddings. We want to understand to what extent does the BERT representation capture syntactic and semantic information. The sentence-level embeddings are evaluated on downstream and probing tasks using the SentEval toolkit BIBREF8, while the passage-level encodings are evaluated on four passage-level QA datasets (both factoid and non-factoid) under a learning-to-rank setting. Different methods of combining query embeddings with passage-level answer embeddings are examined."]},{"idx":2,"heading":"BERT Sentence Embedding","paragraphs":["We use the SentEval toolkit to evaluate the quality of sentence representations from BERT activations. The evaluation encompasses a variety of downstream and probing tasks. Downstream tasks include text classification, natural language inference, paraphrase detection, and semantic similarity. Probing tasks use single sentence embedding as input, are designed to probe sentence-level linguistic phenomena, from superficial properties of sentences to syntactic information to semantic acceptability. For details about the tasks, please refer to BIBREF8 and BIBREF9. We compare the BERT embeddings against two state-of-the-art sentence embeddings, Universal Sentence Encoder BIBREF5, InferSent BIBREF2, and a baseline of averaging GloVe word embeddings.","Effect of Encoder Layer: We compare the performance of embeddings extracted from different encoder layers of a pre-trained BERT using bert-as-service BIBREF10. Since we are interested in the linguistic information encoded in the embeddings, we only add a logistic regression layer on top of the embeddings for each classification task. The results of using [CLS] token activations as embeddings are presented in Figure FIGREF1. The raw values are provided in the Appendix. In the heatmap, the raw values of metrics are normalized by the best performance of a particular task from all the models we evaluated including BERT. The tasks in the figure are grouped by task category. For example, all semantic similarity related tasks are placed at the top of the figure.","As can be seen from the figure, embeddings from top layers generally perform better than lower layers. However, for certain semantic probing tasks such as tense classification, subject, and object number classifications, middle layer embeddings perform the best. Intuitively, embeddings from top layer should be more biased towards the target of BERT pre-training tasks, while bottom layer embeddings should be close to the word embeddings. We observed a higher correlation in performance between bottom layer embeddings and GloVe embeddings than embeddings from other layers. Overall, pre-trained BERT embeddings perform well in text classification and syntactic probing tasks. The biggest limitation lies in the semantic similarity and sentence surface information probing tasks, where we observed a big gap between BERT and other state-of-the-art models.","Effect of Pooling Methods: We examined different methods of extracting BERT hidden state activations. The pooling methods we evaluated include: CLS-pooling (the hidden state corresponding to the [CLS] token), SEP-pooling (the hidden state corresponding to the [SEP] token), Mean-pooling (the average of the hidden state of the encoding layer on the time axis), and Max-pooling (the maximum of the hidden state of the encoding layer on the time axis). To eliminate the layer-wise effects, we averaged the performance of each pooling method over different layers. The results are summarized in Table TABREF2, where the score for each task category is calculated by averaging the normalized values for the tasks within each category. Although the activations of [CLS] token hidden states are often used in fine-tuning BERT for classification tasks, Mean-pooling of hidden states performs the best in all task categories among all the pooling methods.","Pre-trained vs. Fine-tuned BERT: All the models we considered in this paper benefit from supervised training on natural language inference datasets. In this section, we compare the performance of embeddings from pre-trained BERT and fine-tuned BERT. Two natural language inference datasets, MNLI BIBREF11 and SNLI, were considered in the experiment. Inspired by the fact that embeddings from different layers excel in different tasks, we also conducted experiments by concatenating embeddings from multiple layers. The results are presented in Table TABREF3, and the raw values are provided in the Appendix.","As we can see from the table, embeddings from pre-trained BERT are good at capturing sentence-level syntactic information and semantic information, but poor at semantic similarity tasks and surface information tasks. Our findings are consistent with BIBREF12 work on assessing BERT's syntactic abilities. Fine-tuning on natural language inference datasets improves the quality of sentence embedding, especially on semantic similarity tasks and entailment tasks. Combining embeddings from two layers can further boost the performance on sentence surface and syntactic information probing tasks. Experiments were also conducted by combining embeddings from multiple layers. However, there is no significant and consistent improvement over pooling just from two layers. Adding multi-layer perceptron (MLP) instead of logistic regression layer on top of the embeddings also provides no significant changes in performance, which suggests that most linguistic properties can be extracted with just a linear readout of the embeddings. Our best model is the combination of embeddings from the top and bottom layer of the BERT fine-tuned on SNLI dataset."]},{"idx":3,"heading":"BERT Passage Embedding","paragraphs":["In this section, we evaluate BERT embeddings at passage level on question-answering datasets under a learning-to-rank problem setting.","Datasets: We experimented on four datasets: (1) WikiPassageQA BIBREF13, (2) InsuranceQA (version 1.0) BIBREF14, (3) Quasar-t BIBREF15, and (4) SearchQA BIBREF16. They cover both factoid and non-factoid QA and different average passage length. The statistics of the four datasets are provided in the Appendix. To generate passage-level question-answering data from Quasart-t and SearchQA, we used the retrieved passages for each question from OpenQA, and generated question-passage relevance label based on whether the ground truth answer is contained in the passage.","Experiment Setting: We use the same pooling methods as in the sentence embedding experiment to extract passage embeddings, and make sure that the passage length is within BERT's maximum sequence length. Different methods of combining query embeddings with answer passage embeddings were explored including: cosine similarity (no trainable parameter), bilinear function, concatenation, and $(u, v, u * v, |u - v|)$ where $u$ and $v$ are query embedding and answer embedding, respectively. A logistic regression layer or an MLP layer is added on top of the embeddings to output a ranking score. We apply the pairwise rank hinge loss $l(q, +a, -a; \\theta ) = max\\lbrace 0, - S(q, +a; \\theta )+S(q, -a; \\theta )\\rbrace $ to every tuple of $(query, +answer, -answer)$. Ranking metrics such as MRR (mean reciprocal rank), MAP (mean average precision), Precision@K and Recall@K are used to measure the performance. We compared BERT passage embeddings against the baseline of BM25, other state-of-the-art models, and a fine-tuned BERT on in-domain supervised data which serves as the upper bound. For in-domain BERT fine-tuning, we feed the hidden state of the [CLS] token from the top layer into a two-layer MLP which outputs a relevance score between the question and candidate answer passage. We fine-tune all BERT parameters except the word embedding layers.","Results: The comparison between BERT embeddings and other models is presented in Table TABREF5. Overall, in-domain fine-tuned BERT delivers the best performance. We report new state-of-the-art results on WikiPassageQA ($33\\%$ improvement in MAP) and InsuranceQA (version 1.0) ($3.6\\%$ improvement in P@1) by supervised fine-tuning BERT using pairwise rank hinge loss. When evaluated on non-factoid QA datasets, there is a big gap between BERT embeddings and the fully fine-tuned BERT, which suggests that deep interactions between questions and answers are critical to the task. However, the gap is much smaller for factoid QA datasets. Since non-factoid QA depends more on content matching rather than vocabulary matching, the results are kind of expected. Similar to BERT for sentence embeddings, mean-pooling and combining the top and bottom layer embeddings lead to better performance, and $(u, v, u * v, |u - v|)$ shows the strongest results among other interaction schemes. Different from sentence-level embeddings, fine-tuning BERT on SNLI doesn't lead to significant improvement, which suggests possible domain mismatch between SNLI and the QA datasets. MLP layer usually provided a 1-2 percent boost in performance compared to the logistic regression layer. For WikiPassageQA, BERT embeddings perform comparably as BM25 baseline. For InsuranceQA, BERT embeddings outperform a strong representation-based matching model DSSM BIBREF18, but still far behind the state-of-the-art interaction-based model SUBMULT+NN BIBREF17 and fully fine-tuned BERT. On factoid datasets (Quasar-t and SearchQA), BERT embeddings outperform BM25 baseline significantly."]},{"idx":4,"heading":"Conclusion","paragraphs":["In this paper, we conducted an empirical investigation of BERT activations as universal text embeddings. We show that sentence embeddings from BERT perform strongly on SentEval tasks, and combining embeddings from the top and bottom layers of BERT fine-tuned on SNLI provides the best performance. At passage-level, we evaluated BERT embeddings on four QA datasets. Models based on BERT passage embeddings outperform BM25 baseline significantly on factoid QA datasets but fail to perform better than BM25 on non-factoid datasets. We observed a big gap between embedding-based models and in-domain the fully fine-tuned BERT on QA datasets. Future research is needed to better model the interactions between pairs of text embeddings."]},{"idx":5,"heading":"Figures and Tables","paragraphs":["Figure 1: Sentence embedding performance of [CLS] token activation from different layers of BERT. Color value of 1 corresponds to the best performance on a given task. Numbers on the x-axis represent the pooling layer with -1 being the top encoder layer, -12 being the bottom layer.","Table 1: Comparison of pooling methods","Table 2: Comparison across models. PT stands for pre-trained BERT. MNLI and SNLI stand for BERT fine-tuned on MNLI, SNLI, representatively. Letters in parentheses represent BERT pooling layers. “t” means top layer, “b” means bottom layer. Mean-pooling is used for all BERT embeddings. Logistic regression layer is added on top of the embeddings.","Table 3: Results of BERT passage-level embeddings on question-answering datasets"]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1910.10487","title":"Memory-Augmented Recurrent Networks for Dialogue Coherence","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Recent dialogue approaches operate by reading each word in a conversation history, and aggregating accrued dialogue information into a single state. This fixed-size vector is not expandable and must maintain a consistent format over time. Other recent approaches exploit an attention mechanism to extract useful information from past conversational utterances, but this introduces an increased computational complexity. In this work, we explore the use of the Neural Turing Machine (NTM) to provide a more permanent and flexible storage mechanism for maintaining dialogue coherence. Specifically, we introduce two separate dialogue architectures based on this NTM design. The first design features a sequence-to-sequence architecture with two separate NTM modules, one for each participant in the conversation. The second memory architecture incorporates a single NTM module, which stores parallel context information for both speakers. This second design also replaces the sequence-to-sequence architecture with a neural language model, to allow for longer context of the NTM and greater understanding of the dialogue history. We report perplexity performance for both models, and compare them to existing baselines."]},{"idx":1,"heading":"Introduction","paragraphs":["Recently, chit-chat dialogue models have achieved improved performance in modelling a variety of conversational domains, including movie subtitles, Twitter chats and help forums BIBREF0, BIBREF1, BIBREF2, BIBREF3. These neural systems were used to model conversational dialogue via training on large chit-chat datasets such as the OpenSubtitles corpus, which contains generic dialogue conversations from movies BIBREF4. The datasets used do not have an explicit dialogue state to be modelled BIBREF5, but rather require the agent to learn the nuances of natural language in the context of casual peer-to-peer interaction.","Many recent chit-chat systems BIBREF2, BIBREF3 attempt to introduce increased diversity into model responses. However, dialogue systems have also been known to suffer from a lack of coherence BIBREF0. Given an input message history, systems often have difficulty tracking important information such as professions and names BIBREF0. It would be of benefit to create a system which extracts relevant features from the input that indicate which responses would be most appropriate, and conditions on this stored information to select the appropriate response.","A major problem with existing recurrent neural network (RNN) architectures is that these systems aggregate all input tokens into a state vector, which is passed to a decoder for generation of the final response, or in the case of a neural probabilistic language model BIBREF6, the state at each time step is used to predict the next token in the sequence. Ideally the size of the state should expand with the number of input tokens and should not lose important information about the input. However, RNN states are typically fixed sized, and for any chosen state size, there exists an input sequence length for which the RNN would not be able to store all relevant details for a final response. In addition, the RNN state undergoes constant transformation at each computational step. This makes it difficult to maintain a persistent storage of information that remains constant over many time steps.","The introduction of attention mechanisms BIBREF7 has sparked a change in the current design of RNN architectures. Instead of relying fully on a fixed-sized state vector, an attention mechanism allows each decoder word prediction step to extract relevant information from past states through a key-value query mechanism. However, this mechanism connects every input token with all preceeding ones via a computational step, increasing the complexity of the calculation to $O(N^2)$ for an input sequence size N. In the ideal case, the mapping of input conversation history to output response would have a computational complexity of $O(N)$. For this reason, it is desirable to have an information retrieval system that is both scale-able, but not proportional to input length.","We study the impact of accessible memory on response coherence by constructing a memory-augmented dialogue system. The motivation is that it would be beneficial to store details of the conversational history in a more permanent memory structure, instead of being captured inside a fixed-sized RNN hidden state. Our proposed system is able to both read and write to a persistent memory module after reading each input utterance. As such, it has access to a stable representation of the input message history when formulating a final response. We explore two distinct memory architectures with different properties, and compare their differences and benefits. We evaluate our proposed memory systems using perplexity evaluation, and compare them to competitive baselines."]},{"idx":2,"heading":"Recent Work","paragraphs":["vinyals2015neural train a sequence-to-sequence LSTM-based dialogue model on messages from an IT help-desk chat service, as well as the OpenSubtitles corpus, which contains subtitles from popular movies. This model was able to answer philosophical questions and performed well with common sense reasoning. Similarly, serban2016building train a hierarchical LSTM architecture (HRED) on the MovieTriples dataset, which contains examples of the form (utterance #1, utterance #2, utterance #3). However, this dataset is small and does not have conversations of larger length. They show that using a context recurrent neural network (RNN) to read representations at the utterance-level allows for a more top-down perspective on the dialogue history. Finally, serban2017hierarchical build a dialogue system which injects diversity into output responses (VHRED) through the use of a latent variable for variational inference BIBREF3. They argue that the injection of information from the latent variables during inference increases response coherence without degrading response quality. They train the full system on the Twitter Dialogue corpus, which contains generic multi-turn conversations from public Twitter accounts. They also train on the Ubuntu Dialogue Corpus, a collection of multi-turn vocabulary-rich conversations extracted from Ubuntu chat logs. du2018variational adapt from the VHRED architecture by increasing the influence of the latent variables on the output utterance. In this work, a backwards RNN carries information from future timesteps to present ones, such that a backward state contains a summary of all future utterances the model is required to generate. The authors constrain this backward state at each time step to be a latent variable, and minimize the KL loss to restrict information flow. At inference, all backward state latent variables are sampled from and decoded to the output response. The authors interpret the sampling of the latent variables as a \"plan\" of what to generate next.","bowman2015generating observe that latent variables can sometimes degrade, where the system chooses not to store information in the variable and does not condition on it when producing the output. bowman2015generating introduce a process called KL-annealing which slowly increases the KL divergence loss component over the course of training. However, BIBREF8 claim that KL annealing is not enough, and introduce utterance dropout to force the model to rely on information stored in the latent variable during response generation. They apply this system to conversational modelling.","Other attempts to increase diversity focus on selecting diverse responses after the model is trained. li2015diversity introduce a modification of beam search. Beam search attempts to find the highest probability response to a given input by producing a tree of possible responses and \"pruning\" branches that have the lowest probability. The top K highest probability responses are returned, of which the highest is selected as the output response. li2015diversity observe that beam search tends to select certain families of responses that temporarily have higher probability. To combat this, a discount factor of probabilities is added to responses that come from the same parent response candidate. This encourages selecting responses that are different from one another when searching for the highest probability target.","While coherence and diversity remain the primary focus of model dialogue architectures, many have tried to incorporate additional capabilities. zhou2017mojitalk introduce emotion into generated utterances by creating a large-scale fine-grained emotion dialogue dataset that uses tagged emojis to classify utterance sentiment. Then they train a conditional variational autoencoder (CVAE) to generate responses given an input emotion. Along this line of research, li2016persona use Reddit users as a source of persona, and learn individual persona embeddings per user. The system then conditions on these embeddings to generate a response while maintaining coherence specific to the given user. pandey2018exemplar expand the context of an existing dialogue model by extracting input responses from the training set that are most similar to the current input. These \"exemplar\" responses are then conditioned on to use as reference for final response generation. In another attempt to add context, young2018augmenting utilize a relational database to extract specific entity relations that are relevant for the current input. These relations provide more context for the dialogue model and allows it to respond to the user with information it did not observe in the training set.","Ideally, NLP models should have the ability to use and update information processed in the past. For dialogue generation, this ability is particularly important, because dialogue involves exchange of information in discourse, and all responses depend on what has been mentioned in the past. RNNs introduce \"memory\" by adding an output of one time step to their input in a future time step. Theoretically, properly trained RNNs are Turing-complete, but in reality vanilla RNNs often do not perform well due to the gradient vanishing problem. Gated RNNs such as LSTM and GRU introduces cell state, which can be understood as memory controlled by trainable logic gates. Gated RNNs do not suffer from the vanishing gradient problem as much, and indeed outperform vanilla RNNs in various NLP tasks. This is likely because the vanilla RNN state vector undergoes a linear transformation at each step, which can be difficult to control. In contrast, gated RNNs typically both control the flow of information, and ensure only elemnt-wise operations occur on the state, which allow gradients to pass more easily. However, they too fail in some basic memorization tasks such as copying and associative recall. A major issue is when the cell state gets updated, previous memories are forever erased. As a result, Gated RNNs can not model long-term dependencies well.","In recent years, there have been proposals to use memory neural networks to capture long-term information. A memory module is defined as an external component of the neural network system, and it is theoretically unlimited in capacity. weston2014memory propose a sequence prediction method using a memory with content-based addressing. In their implementation for the bAbI task BIBREF9 for example, their model encodes and sequentially saves words from text in memory slots. When a question about the text is asked, the model uses content-based addressing to retrieve memories relevant to the question, in order to generate answers. They use the k-best memory slots, where k is a relative small number (1 or 2 in their paper). sukhbaatar2015end propose an end-to-end neural network model, which uses content-based addressing to access multiple memory layers. This model has been implemented in a relatively simple goal-oriented dialogue system (restaurant booking) and has decent performance BIBREF10.","DBLP:journals/corr/GravesWD14 further develop the addressing mechanism and make old memory slots dynamically update-able. The model read heads access information from all the memory slots at once using soft addressing. The write heads, on the other hand, have the ability to modify memory slots. The content-based addressing serves to locate relevant information from memory, while another location-based addressing is also used, to achieve slot shifting, interpolation of address from the previous step, and so on. As a result, the memory management is much more complex than the previously proposed memory neural networks. This system is known as the Neural Turing Machine (NTM).","Other NTM variants have also been proposed recently. DBLP:journals/corr/ZhangYZ15 propose structured memory architectures for NTMs, and argue they could alleviate overfitting and increase predictive accuracy. DBLP:journals/nature/GravesWRHDGCGRA16 propose a memory access mechanism on top of NTM, which they call the Differentiable Neural Computer (DNC). DNC can store the transitions between memory locations it accesses, and thus can model some structured data. DBLP:journals/corr/GulcehreCCB16 proposed a Dynamic Neural Turing Machine (D-NTM) model, which allows more addressing mechanisms, such as multi-step addressing. DBLP:journals/corr/GulcehreCB17 further simplified the algorithm, so a single trainable matrix is used to get locations for read and write. Both models separate the address section from the content section of memory.","The Global Context Layer BIBREF11 independently proposes the idea of address-content separation, noting that the content-based addressing in the canonical NTM model is difficult to train. A crucial difference between GCL and these models is that they use input “content” to compute keys. In GCL, the addressing mechanism fully depends on the entity representations, which are provided by the context encoding layers and not computed by the GCL controller. Addressing then involves matching the input entities and the entities in memory. Such an approach is desirable for tasks like event temporal relation classification, entity co-reference and so on. GCL also simplified the location-based addressing proposed in NTM. For example, there is no interpolation between current addressing and previous addressing.","Other than NTM-based approaches, there are recent models that use an attention mechanism over either input or external memory. For instance, the Pointer Networks BIBREF12 uses attention over input timesteps. However, it has no power to rewrite information for later use, since they have no “memory” except for the RNN states. The Dynamic Memory Networks BIBREF13 have an “episodic memory” module which can be updated at each timestep. However, the memory is a vector (“episode”) without internal structure, and the attention mechanism only works on inputs, just as in Pointer Networks. The GCL model and other NTM-based models have a memory with multiple slots, and the addressing function dictates writing and reading to/from certain slots in the memory"]},{"idx":3,"heading":"Dual-NTM Seq2Seq Dialogue Architecture","paragraphs":["As a preliminary approach, we implement a dialogue generation system with segment-level memory manipulation. Segment-level memory refers to memory of sub-sentence level, which often corresponds to entity mentions, event mentions, and proper names, etc. We use NTM as the memory module, because it is more or less a default choice before specialized mechanisms are developed. Details of NTMs can be found in DBLP:journals/corr/GravesWD14.","As in the baseline model, the encoder and decoder each has an Gated Recurrent Unit (GRU) inside. A GRU is a type of recurrent neural networks that coordinates forgetting and write of information, to make sure they don't both occur simultaneously. This is accomplished via an \"update gate.\" A GRU architecture processes a list of inputs in sequence, and is described by the following equations:","For each input $x_t$ and previous state $h_{t-1}$, the GRU produces the next state $h_t$ given learned weights $W_z$, $W_r$ and $W$. $z_t$ denotes the update gate. The encoder GRU in this memory architecture reads a token at each time step, and encodes a context representation $c$ at the end of the input sequence. In addition to that, the memory enhanced model implements two Neural Turing Machines (NTMs). Each of them is for one speaker in the conversation, since the Ubuntu dataset has two speakers in every conversation. Every turn in a dialogue is divided in 4 “segments\". If a turn has 20 tokens, for example, a segment contains 5 tokens. The output of the GRU is written to the NTM at the end of every segment. It does not output anything useful here, but the internal memory is being updated each time. When the dialogue switches to next turn, the current NTM pauses and the other NTM starts to work in the same way. When an NTM pauses, its internal memory retains, so as soon as the dialogue moves to its turn again, it continues to read and update its internal memory.","Equation DISPLAY_FORM6 shows how one NTM updates. $T$ denotes the length of one turn, and $s$ is the output of the encoder GRU. $n=1,2,3,4$ represents the four time steps when the NTM updates in one turn of the conversation.","The two NTMs can be interpreted as two external memories tracking each speaker's utterances. When one speaker needs to make a response at the end of the conversation, he needs to refer to both speakers' history to make sure the response is coherent with respect to context. This allows for separate tracking of each participant, while also consolidating their representations.","The decoder GRU works the same way as the baseline model. Each time it reads a token, from either the true response or the generated response, depending on whether teacher force training is used. This token and the context representation $c$ generated by the encoder GRU are both used as input to the decoder GRU.","However, now the two NTMs also participate in token generation. At every time step, output of the decoder GRU is fed into the two NTMs, and outputs of the two NTMs are used together to make predictions.","In the equation above, $\\mathrm {FC}$ represents a fully connected layer, and $\\widehat{y_t}$ is the predicted vector.","From now on, we refer to this system as the D-NTMS (Dual-NTM Seq2Seq) system."]},{"idx":4,"heading":"NTM Language Model Dialogue Architecture","paragraphs":["In this section we introduce a somewhat simpler, but more effective memory module architecture. In contrast to the previous D-NTMS architecture, we combine the encoder-decoder architecture of the sequence to sequence GRU into a single language model. This combination entails the model predicting all tokens in the dialogue history in sequence. This change in setup exploits the property that the response is in essence drawn from the same distribution as all previous utterances, and so should not be treated any differently. This language model variant learns to predict all utterances in the dialogue history, and thus treats the response as just another utterance to predict. This setup may also help the model learn the flow of conversation from beginning to end.","With a neural language model predicting tokens, it is then necessary to insert reads and writes from a Neural Turing Machine. In this architecture, we only use one NTM. This change is motivated by the possibility that the speaker NTMs from the previous architecture may have difficulty exchanging information, and thus cannot adequately represent each utterance in the context of the previous one. We follow an identical setup as before and split the dialogue history into segments. A GRU processes each segment in sequence. Between each segment, the output GRU state is used to query and write to the NTM module to store and retrieve relevant information about the context history so far. This information is conditioned on for all subsequent tokens in the next segment, in order to exploit this information to make more informed predictions. Lastly, the GRU NTM has an internal LSTM controller which guides the read and writes to and from the memory section. Reads are facilitated via content-based addressing, where a cosine similarity mechanism selects entries that most resemble the query. The Neural Turing Machine utilized can be found as an existing Github implementation.","In further investigations, we refer to this model as the NTM-LM system."]},{"idx":5,"heading":"Baselines","paragraphs":["As a reliable baseline, we will evaluate a vanilla sequence-to-sequence GRU dialogue architecture, with the same hyper-parameters as our chosen model. We refer this this baseline as Seq2Seq. In addition, we report results for a vanilla GRU language model (LM). Finally, we include a more recent baseline, the Hierarchical Encoder-Decoder (HRED) system which is trained for the same number of epochs, same batch size, and with the same encoder and decoder size as the Seq2Seq baseline . As previously mentioned, we refer to our first proposed memory architecture as D-NTMS and to our second memory architecture as NTM-LM."]},{"idx":6,"heading":"Evaluation","paragraphs":["To evaluate the performance of each dialogue baseline against the proposed models, we use the Ubuntu Dialogue Corpus BIBREF14, chosen for its rich vocabulary size, diversity of responses, and dependence of each utterance on previous ones (coherence required). We perform perplexity evaluation using a held-out validation set. The results are reported in Table TABREF3. Perplexity is reported per word. For reference, a randomly-initialized model would receive a perplexity of 50,000 for our chosen vocabulary size. We also report generated examples from the model, shown in Table TABREF15."]},{"idx":7,"heading":"Results","paragraphs":["See Table TABREF3 for details on model and baseline perplexity. To begin, it is worth noting that all of the above architectures were trained in a similar environment, with the exception of HRED, which was trained using an existing Github implementation implementation. Overall, the NTM-LM architecture performed the best of all model architectures, whereas the sequence-to-sequence architecture performed the worst. The proposed NTM-LM outperformed the DNTM-S architecture.","After one epoch of training, the perplexity evaluated on the validation set was 68.50 for the proposed memory-augmented NTM-LM architecture. This is a 0.68 perplexity improvement over the vanilla language model without the NTM augmentation."]},{"idx":8,"heading":"Discussion","paragraphs":["Overall, the HRED baseline was top performing among all tested architectures. This baseline breaks up utterances in a conversation and reads them separately, producing a hierarchical view which likely promotes coherence at a high level.","Now we will discuss the memory-augmented D-NTMS architecture. The memory-augmented architecture improved performance above the baseline sequence-to-sequence architecture. As such, it is likely that the memory modules were able to store valuable information about the conversation, and were able to draw on that information during the decoder phase. One drawback of the memory enhanced model is that training was significantly slower. For this reason, model simplification is required in the future to make it more practical. In addition, the NTM has a lot of parameters and some of them may be redundant or damaging. In the DNTM-S system, we may not need to access the NTM at each step of decoding either. Instead, it can be accessed in some intervals of time steps, and the output is used for all steps within the interval.","The best performing model was the NTM-LM architecture. While the model received the best performance in perplexity, it demonstrated only a one-point improvement over the existing language model architecture. While in state-of-the-art comparisons a one point difference can be significant, it does indicate that the proposed NTM addition to the language model only contributed a small improvement. It is possible that the additional NTM module was too difficult to train, or that the NTM module injected noise into the input of the GRU such that training became difficult. It is still surprising that the NTM was not put to better use, for performance gains. It is possible the model has not been appropriately tuned.","Another consideration of the NTM-LM architecture is that it takes a significant amount of time to train. Similar to the D-NTMS, the NTM memory module requires a sizeable amount of computational steps to both retrieve a query response from available memory slots, and also to write to a new or existing slot using existing write weights. This must be repeated for each segment. Another source of slowdown with regard to computation is the fact that the intermittent NTM reads and writes force the input utterance into segments, as illustrated in Figure FIGREF2. This splitting of token processing steps requires additional overhead to maintain, and it may discourage parallel computation of different GRU input segments simultaneously. This problem is not theoretical, and may be solved using future optimizations of a chosen deep learning framework. For Pytorch, we observed a slowdown for a segmented dialogue history versus a complete history.","Of all models, the HRED architecture utilized pre-trained GloVe vectors as an initialization for its input word embedding matrix. This feature likely improved performance of the HRED in comparison to other systems, such as the vanilla sequence-to-sequence. However, in separate experiments, GloVe vectors only managed a 5% coverage of all words in the vocabulary. This low number is likely due to the fact that the Ubuntu Dialogues corpus contains heavy terminology from the Ubuntu operating system and user packages. In addition, the Ubuntu conversations contain a significant amount of typos and grammar errors, further complicating analysis. Context-dependent embeddings such as ElMo BIBREF15 may help alleviate this issue, as character-level RNNs can better deal with typos and detect sub word-level elements such morphemes.","Due to time requirements, there were no targeted evaluations of memory coherence other than perplexity, which evaluates overall coherence of the conversation. This form of specific evaluation may be achievable through a synethetic dataset of responses, for example, \"What is your profession? I am a doctor.</s>What do you do for work?</s>I am a doctor.\" This sort of example would require direct storage of the profession of a given speaker. However, the Ubuntu Dialogue corpus contains complicated utterances in a specific domain, and thus does not lend well to synthesized utterances from a simpler conversational domain. In addition, synthetic conversations like the one above do not sound overly natural, as a human speaker does not normally repeat a query for information after they have already asked for it. In that sense, it is difficult to directly evaluate dialogue coherence.","Not reported in this paper was a separate implementation of the language model that achieved better results (62 perplexity). While this was the best performing model, it was written in a different environment than the language model reported here or the NTM-LM model. As such, comparing the NTM-LM to this value would be misleading. Since the NTM-LM is an augmentation of the existing LM language model implementation, we report perplexity results from that implementation instead for fair comparison. In that implementation, the addition of the NTM memory model improved performance. For completeness, we report the existence of the outperforming language model here."]},{"idx":9,"heading":"Conclusion","paragraphs":["We establish memory modules as a valid means of storing relevant information for dialogue coherence, and show improved performance when compared to the sequence-to-sequence baseline and vanilla language model. We establish that augmenting these baseline architectures with NTM memory modules can provide a moderate bump in performance, at the cost of slower training speeds. The memory-augmented architectures described above should be modified for increased computational speed and a reduced number of parameters, in order to make each memory architecture more feasible to incorporate into future dialogue designs.","In future work, the memory module could be applied to other domains such as summary generation. While memory modules are able to capture neural vectors of information, they may not easily capture specific words for later use. A possible future approach might combine memory module architectures with pointer softmax networks BIBREF16 to allow memory models to store information about which words from previous utterances of the conversation to use in future responses."]},{"idx":10,"heading":"Appendix ::: Preprocessing","paragraphs":["We construct a vocabulary of size 50,000 (pruning less frequent tokens) from the chosen Ubuntu Dialogues Corpus, and represent all missing tokens using a special unknown symbol <unk>. When processing conversations for input into a sequence-to-sequence based model, we split each conversation history into history and response, where response is the final utterance. To clarify, all utterances in each conversation history are separated by a special <","s>symbol. A maximum of 170 tokens are allocated for the input history and 30 tokens are allocated for the maximum output response.","When inputting conversation dialogues into a language model-based implementation, the entire conversation history is kept intact, and is formatted for a maximum conversation length of 200 tokens. As for all maximum lengths specified here, an utterance which exceeds the maximum length is pruned, and extra tokens are not included in the perplexity calculation. This is likely not an issue, as perplexity calculations are per-word and include the end of sequence token."]},{"idx":11,"heading":"Appendix ::: Training/Parameters","paragraphs":["All models were trained using the Adam optimizer BIBREF23 and updated using a learning rate of 0.0001. All models used a batch size of 32, acceptable for the computational resources available. We develop all models within the deep learning framework Pytorch. To keep computation feasible, we train all models for one epoch. For reference, the NTM-LM architecture took over two and a half days of training for one epoch with the parameters specified."]},{"idx":12,"heading":"Appendix ::: Layer Dimensions","paragraphs":["In our preliminary experiment, each of the NTMs in the D-NTMS architecture were chosen to have 1 read head and 1 write head. The number of memory slots is 20. The capacity of each slot is 512, the same as the decoder GRU state dimensionality. Each has an LSTM controller, and the size is chosen to be 512 as well. These parameters are consistent for the NTM-LM architecture as well.","All sequence-to-sequence models utilized a GRU encoder size of 200, with a decoder GRU size of 400. All language models used a decoder of size 400. The encoder hidden size of the HRED model was set to 400 hidden units. The input embedding size to all models is 200, with only the HRED architecture randomly initializing these embeddings with pre-trained GloVe vectors. The sequence-to-sequence architecture learns separate input word embeddings for encoder and decoder.","Each Neural Turing Machine uses 8 heads for reading and writing, with each head having a size of 64 hidden units In the case of the NTM-LM architecture, 32 memory slots are available for storage by the model. When breaking GPU computation to read and write from the NTM, we break the input conversation into segments of size 20 with NTM communication in-between segments. In contrast, the D-NTMS architecture uses a segment size of 5, and breaks up the conversation in utterances which fit in each segment."]},{"idx":13,"heading":"Figures and Tables","paragraphs":["Figure 1: Memory-augmented dialogue architecture with dual NTMs (D-NTMS). GRU encoders read each utterance in the conversation in segments. After reading each segment, a write is made to the corresponding Neural Turing Machine memory module (NTM). Two NTMs are designated, one for each speaker in the conversation (two speakers total). The resulting NTMs are read from and their predictions are used to output the final response prediction.","Figure 2: Proposed single-NTM language model dialogue system (NTM-LM). The input dialogue history is broken into segments and each is processed by a GRU language model in sequence. At the end of each segment, the GRU state is used to read from and write to the persistent Neural Turing Machine (NTM).","Table 1: Word-level perplexity evaluation on proposed model and two selected baselines.","Figure 3: Example loss values for both training and validation datasets, over the course of model training. Displayed for NTM-LM model specifically, but similar loss curves were observed for all models.","Table 2: Samples decoded with random sampling from the best-performing NTM-LM architecture. First column shows the message history, with the second column showing a model response. Due to the nature of the Ubuntu Dialogue Corpus, the terminology is complex."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1910.12795","title":"Learning Data Manipulation for Augmentation and Weighting","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Manipulating data, such as weighting data examples or augmenting with new instances, has been increasingly used to improve model training. Previous work has studied various rule- or learning-based approaches designed for specific types of data manipulation. In this work, we propose a new method that supports learning different manipulation schemes with the same gradient-based algorithm. Our approach builds upon a recent connection of supervised learning and reinforcement learning (RL), and adapts an off-the-shelf reward learning algorithm from RL for joint data manipulation learning and model training. Different parameterization of the ``data reward'' function instantiates different manipulation schemes. We showcase data augmentation that learns a text transformation network, and data weighting that dynamically adapts the data sample importance. Experiments show the resulting algorithms significantly improve the image and text classification performance in low data regime and class-imbalance problems."]},{"idx":1,"heading":"Introduction","paragraphs":["The performance of machines often crucially depend on the amount and quality of the data used for training. It has become increasingly ubiquitous to manipulate data to improve learning, especially in low data regime or in presence of low-quality datasets (e.g., imbalanced labels). For example, data augmentation applies label-preserving transformations on original data points to expand the data size; data weighting assigns an importance weight to each instance to adapt its effect on learning; and data synthesis generates entire artificial examples. Different types of manipulation can be suitable for different application settings.","Common data manipulation methods are usually designed manually, e.g., augmenting by flipping an image or replacing a word with synonyms, and weighting with inverse class frequency or loss values BIBREF0, BIBREF1. Recent work has studied automated approaches, such as learning the composition of augmentation operators with reinforcement learning BIBREF2, BIBREF3, deriving sample weights adaptively from a validation set via meta learning BIBREF4, or learning a weighting network by inducing a curriculum BIBREF5. These learning-based approaches have alleviated the engineering burden and produced impressive results. However, the algorithms are usually designed specifically for certain types of manipulation (e.g., either augmentation or weighting) and thus have limited application scope in practice.","In this work, we propose a new approach that enables learning for different manipulation schemes with the same single algorithm. Our approach draws inspiration from the recent work BIBREF6 that shows equivalence between the data in supervised learning and the reward function in reinforcement learning. We thus adapt an off-the-shelf reward learning algorithm BIBREF7 to the supervised setting for automated data manipulation. The marriage of the two paradigms results in a simple yet general algorithm, where various manipulation schemes are reduced to different parameterization of the data reward. Free parameters of manipulation are learned jointly with the target model through efficient gradient descent on validation examples. We demonstrate instantiations of the approach for automatically fine-tuning an augmentation network and learning data weights, respectively.","We conduct extensive experiments on text and image classification in challenging situations of very limited data and imbalanced labels. Both augmentation and weighting by our approach significantly improve over strong base models, even though the models are initialized with large-scale pretrained networks such as BERT BIBREF8 for text and ResNet BIBREF9 for images. Our approach, besides its generality, also outperforms a variety of dedicated rule- and learning-based methods for either augmentation or weighting, respectively. Lastly, we observe that the two types of manipulation tend to excel in different contexts: augmentation shows superiority over weighting with a small amount of data available, while weighting is better at addressing class imbalance problems.","The way we derive the manipulation algorithm represents a general means of problem solving through algorithm extrapolation between learning paradigms, which we discuss more in section SECREF6."]},{"idx":2,"heading":"Related Work","paragraphs":["Rich types of data manipulation have been increasingly used in modern machine learning pipelines. Previous work each has typically focused on a particular manipulation type. Data augmentation that perturbs examples without changing the labels is widely used especially in vision BIBREF10, BIBREF11 and speech BIBREF12, BIBREF13 domains. Common heuristic-based methods on images include cropping, mirroring, rotation BIBREF11, and so forth. Recent work has developed automated augmentation approaches BIBREF3, BIBREF2, BIBREF14, BIBREF15, BIBREF16. BIBREF17 additionally use large-scale unlabeled data. BIBREF3, BIBREF2 learn to induce the composition of data transformation operators. Instead of treating data augmentation as a policy in reinforcement learning BIBREF3, we formulate manipulation as a reward function and use efficient stochastic gradient descent to learn the manipulation parameters. Text data augmentation has also achieved impressive success, such as contextual augmentation BIBREF18, BIBREF19, back-translation BIBREF20, and manual approaches BIBREF21, BIBREF22. In addition to perturbing the input text as in classification tasks, text generation problems expose opportunities to adding noise also in the output text, such as BIBREF23, BIBREF24. Recent work BIBREF6 shows output nosing in sequence generation can be treated as an intermediate approach in between supervised learning and reinforcement learning, and developed a new sequence learning algorithm that interpolates between the spectrum of existing algorithms. We instantiate our approach for text contextual augmentation as in BIBREF18, BIBREF19, but enhance the previous work by additionally fine-tuning the augmentation network jointly with the target model.","Data weighting has been used in various algorithms, such as AdaBoost BIBREF0, self-paced learning BIBREF25, hard-example mining BIBREF26, and others BIBREF27, BIBREF28. These algorithms largely define sample weights based on training loss. Recent work BIBREF5, BIBREF29 learns a separate network to predict sample weights. Of particular relevance to our work is BIBREF4 which induces sample weights using a validation set. The data weighting mechanism instantiated by our framework has a key difference in that samples weights are treated as parameters that are updated iteratively, instead of re-estimated from scratch at each step. We show improved performance of our approach. Besides, our data manipulation approach is derived based on a different perspective of reward learning, instead of meta-learning as in BIBREF4.","Another popular type of data manipulation involves data synthesis, which creates entire artificial samples from scratch. GAN-based approaches have achieved impressive results for synthesizing conditional image data BIBREF30, BIBREF31. In the text domain, controllable text generation BIBREF32 presents a way of co-training the data generator and classifier in a cyclic manner within a joint VAE BIBREF33 and wake-sleep BIBREF34 framework. It is interesting to explore the instantiation of the present approach for adaptive data synthesis in the future."]},{"idx":3,"heading":"Background","paragraphs":["We first present the relevant work upon which our automated data manipulation is built. This section also establishes the notations used throughout the paper.","Let $\\mathbf {x}$ denote the input and $y$ the output. For example, in text classification, $\\mathbf {x}$ can be a sentence and $y$ is the sentence label. Denote the model of interest as $p_\\theta (y|\\mathbf {x})$, where $\\mathbf {\\theta }$ is the model parameters to be learned. In supervised setting, given a set of training examples $\\mathcal {D}=\\lbrace (\\mathbf {x}^*, y^*)\\rbrace $, we learn the model by maximizing the data log-likelihood."]},{"idx":4,"heading":"Background ::: Equivalence between Data and Reward","paragraphs":["The recent work BIBREF6 introduced a unifying perspective of reformulating maximum likelihood supervised learning as a special instance of a policy optimization framework. In this perspective, data examples providing supervision signals are equivalent to a specialized reward function. Since the original framework BIBREF6 was derived for sequence generation problems, here we present a slightly adapted formulation for our context of data manipulation.","To connect the maximum likelihood supervised learning with policy optimization, consider the model $p_\\theta (y|\\mathbf {x})$ as a policy that takes “action” $y$ given the “state” $\\mathbf {x}$. Let $R(\\mathbf {x}, y |\\mathcal {D})\\in \\mathbb {R}$ denote a reward function, and $p(\\mathbf {x})$ be the empirical data distribution which is known given $\\mathcal {D}$. Further assume a variational distribution $q(\\mathbf {x}, y)$ that factorizes as $q(\\mathbf {x},y)=p(\\mathbf {x})q(y|\\mathbf {x})$. A variational policy optimization objective is then written as:","where $\\text{KL}(\\cdot \\Vert \\cdot )$ is the Kullback–Leibler divergence; $\\text{H}(\\cdot )$ is the Shannon entropy; and $\\alpha ,\\beta >0$ are balancing weights. The objective is in the same form with the RL-as-inference formalism of policy optimization BIBREF35, BIBREF36, BIBREF37. Intuitively, the objective maximizes the expected reward under $q$, and enforces the model $p_\\theta $ to stay close to $q$, with a maximum entropy regularization over $q$. The problem is solved with an EM procedure that optimizes $q$ and $\\mathbf {\\theta }$ alternatingly:","where $Z$ is the normalization term. With the established framework, it is easy to show that the above optimization procedure reduces to maximum likelihood learning by taking $\\alpha \\rightarrow 0, \\beta =1$, and the reward function:","That is, a sample $(\\mathbf {x}, y)$ receives a unit reward only when it matches a training example in the dataset, while the reward is negative infinite in all other cases. To make the equivalence to maximum likelihood learning clearer, note that the above M-step now reduces to","where the joint distribution $p(\\mathbf {x})\\exp \\lbrace R_\\delta \\rbrace /Z$ equals the empirical data distribution, which means the M-step is in fact maximizing the data log-likelihood of the model $p_\\theta $."]},{"idx":5,"heading":"Background ::: Gradient-based Reward Learning","paragraphs":["There is a rich line of research on learning the reward in reinforcement learning. Of particular interest to this work is BIBREF7 which learns a parametric intrinsic reward that additively transforms the original task reward (a.k.a extrinsic reward) to improve the policy optimization. For consistency of notations with above, formally, let $p_\\theta (y|\\mathbf {x})$ be a policy where $y$ is an action and $\\mathbf {x}$ is a state. Let $R_\\phi ^{in}$ be the intrinsic reward with parameters $\\mathbf {\\phi }$. In each iteration, the policy parameter $\\mathbf {\\theta }$ is updated to maximize the joint rewards, through:","where $\\mathcal {L}^{ex+in}$ is the expectation of the sum of extrinsic and intrinsic rewards; and $\\gamma $ is the step size. The equation shows $\\mathbf {\\theta }^{\\prime }$ depends on $\\mathbf {\\phi }$, thus we can write as $\\mathbf {\\theta }^{\\prime }=\\mathbf {\\theta }^{\\prime }(\\mathbf {\\phi })$.","The next step is to optimize the intrinsic reward parameters $\\mathbf {\\phi }$. Recall that the ultimate measure of the performance of a policy is the value of extrinsic reward it achieves. Therefore, a good intrinsic reward is supposed to, when the policy is trained with it, increase the eventual extrinsic reward. The update to $\\mathbf {\\phi }$ is then written as:","That is, we want the expected extrinsic reward $\\mathcal {L}^{ex}(\\mathbf {\\theta }^{\\prime })$ of the new policy $\\mathbf {\\theta }^{\\prime }$ to be maximized. Since $\\mathbf {\\theta }^{\\prime }$ is a function of $\\mathbf {\\phi }$, we can directly backpropagate the gradient through $\\mathbf {\\theta }^{\\prime }$ to $\\mathbf {\\phi }$."]},{"idx":6,"heading":"Learning Data Manipulation ::: Method ::: Parameterizing Data Manipulation","paragraphs":["We now develop our approach of learning data manipulation, through a novel marriage of supervised learning and the above reward learning. Specifically, from the policy optimization perspective, due to the $\\delta $-function reward (Eq.DISPLAY_FORM4), the standard maximum likelihood learning is restricted to use only the exact training examples $\\mathcal {D}$ in a uniform way. A natural idea of enabling data manipulation is to relax the strong restrictions of the $\\delta $-function reward and instead use a relaxed reward $R_\\phi (\\mathbf {x}, y | \\mathcal {D})$ with parameters $\\mathbf {\\phi }$. The relaxed reward can be parameterized in various ways, resulting in different types of manipulation. For example, when a sample $(\\mathbf {x}, y)$ matches a data instance, instead of returning constant 1 by $R_\\delta $, the new $R_\\phi $ can return varying reward values depending on the matched instance, resulting in a data weighting scheme. Alternatively, $R_\\phi $ can return a valid reward even when $\\mathbf {x}$ matches a data example only in part, or $(\\mathbf {x}, y)$ is an entire new sample not in $\\mathcal {D}$, which in effect makes data augmentation and data synthesis, respectively, in which cases $\\mathbf {\\phi }$ is either a data transformer or a generator. In the next section, we demonstrate two particular parameterizations for data augmentation and weighting, respectively.","We thus have shown that the diverse types of manipulation all boil down to a parameterized data reward $R_\\phi $. Such an concise, uniform formulation of data manipulation has the advantage that, once we devise a method of learning the manipulation parameters $\\mathbf {\\phi }$, the resulting algorithm can directly be applied to automate any manipulation type. We present a learning algorithm next."]},{"idx":7,"heading":"Learning Data Manipulation ::: Method ::: Learning Manipulation Parameters","paragraphs":["To learn the parameters $\\mathbf {\\phi }$ in the manipulation reward $R_\\phi (\\mathbf {x}, y | \\mathcal {D})$, we could in principle adopt any off-the-shelf reward learning algorithm in the literature. In this work, we draw inspiration from the above gradient-based reward learning (section SECREF3) due to its simplicity and efficiency. Briefly, the objective of $\\mathbf {\\phi }$ is to maximize the ultimate measure of the performance of model $p_\\theta (\\mathbf {y}|\\mathbf {x})$, which, in the context of supervised learning, is the model performance on a held-out validation set.","The algorithm optimizes $\\mathbf {\\theta }$ and $\\mathbf {\\phi }$ alternatingly, corresponding to Eq.(DISPLAY_FORM7) and Eq.(DISPLAY_FORM8), respectively. More concretely, in each iteration, we first update the model parameters $\\mathbf {\\theta }$ in analogue to Eq.(DISPLAY_FORM7) which optimizes intrinsic reward-enriched objective. Here, we optimize the log-likelihood of the training set enriched with data manipulation. That is, we replace $R_\\delta $ with $R_\\phi $ in Eq.(DISPLAY_FORM5), and obtain the augmented M-step:","By noticing that the new $\\mathbf {\\theta }^{\\prime }$ depends on $\\mathbf {\\phi }$, we can write $\\mathbf {\\theta }^{\\prime }$ as a function of $\\mathbf {\\phi }$, namely, $\\mathbf {\\theta }^{\\prime }=\\mathbf {\\theta }^{\\prime }(\\mathbf {\\phi })$. The practical implementation of the above update depends on the actual parameterization of manipulation $R_\\phi $, which we discuss in more details in the next section.","The next step is to optimize $\\mathbf {\\phi }$ in terms of the model validation performance, in analogue to Eq.(DISPLAY_FORM8). Formally, let $\\mathcal {D}^v$ be the validation set of data examples. The update is then:","where, since $\\mathbf {\\theta }^{\\prime }$ is a function of $\\mathbf {\\phi }$, the gradient is backpropagated to $\\mathbf {\\phi }$ through $\\mathbf {\\theta }^{\\prime }(\\mathbf {\\phi })$. Taking data weighting for example where $\\mathbf {\\phi }$ is the training sample weights (more details in section SECREF15), the update is to optimize the weights of training samples so that the model performs best on the validation set.","The resulting algorithm is summarized in Algorithm FIGREF11. Figure FIGREF11 illustrates the computation flow. Learning the manipulation parameters effectively uses a held-out validation set. We show in our experiments that a very small set of validation examples (e.g., 2 labels per class) is enough to significantly improve the model performance in low data regime.","It is worth noting that some previous work has also leveraged validation examples, such as learning data augmentation with policy gradient BIBREF3 or inducing data weights with meta-learning BIBREF4. Our approach is inspired from a distinct paradigm of (intrinsic) reward learning. In contrast to BIBREF3 that treats data augmentation as a policy, we instead formulate manipulation as a reward function and enable efficient stochastic gradient updates. Our approach is also more broadly applicable to diverse data manipulation types than BIBREF4, BIBREF3."]},{"idx":8,"heading":"Learning Data Manipulation ::: Instantiations: Augmentation & Weighting","paragraphs":["As a case study, we show two parameterizations of $R_\\phi $ which instantiate distinct data manipulation schemes. The first example learns augmentation for text data, a domain that has been less studied in the literature compared to vision and speech BIBREF18, BIBREF38. The second instance focuses on automated data weighting, which is applicable to any data domains."]},{"idx":9,"heading":"Learning Data Manipulation ::: Instantiations: Augmentation & Weighting ::: Fine-tuning Text Augmentation","paragraphs":["The recent work BIBREF18, BIBREF19 developed a novel contextual augmentation approach for text data, in which a powerful pretrained language model (LM), such as BERT BIBREF8, is used to generate substitutions of words in a sentence. Specifically, given an observed sentence $\\mathbf {x}^*$, the method first randomly masks out a few words. The masked sentence is then fed to BERT which fills the masked positions with new words. To preserve the original sentence class, the BERT LM is retrofitted as a label-conditional model, and trained on the task training examples. The resulting model is then fixed and used to augment data during the training of target model. We denote the augmentation distribution as $g_{\\phi _0}(\\mathbf {x}|\\mathbf {x}^*, \\mathbf {y}^*)$, where $\\mathbf {\\phi }_0$ is the fixed BERT LM parameters.","The above process has two drawbacks. First, the LM is fixed after fitting to the task data. In the subsequent phase of training the target model, the LM augments data without knowing the state of the target model, which can lead to sub-optimal results. Second, in the cases where the task dataset is small, the LM can be insufficiently trained for preserving the labels faithfully, resulting in noisy augmented samples.","To address the difficulties, it is beneficial to apply the proposed learning data manipulation algorithm to additionally fine-tune the LM jointly with target model training. As discussed in section SECREF4, this reduces to properly parameterizing the data reward function:","That is, a sample $(\\mathbf {x}, y)$ receives a unit reward when $y$ is the true label and $\\mathbf {x}$ is the augmented sample by the LM (instead of the exact original data $\\mathbf {x}^*$). Plugging the reward into Eq.(DISPLAY_FORM13), we obtain the data-augmented update for the model parameters:","That is, we pick an example from the training set, and use the LM to create augmented samples, which are then used to update the target model. Regarding the update of augmentation parameters $\\mathbf {\\phi }$ (Eq.DISPLAY_FORM14), since text samples are discrete, to enable efficient gradient propagation through $\\mathbf {\\theta }^{\\prime }$ to $\\mathbf {\\phi }$, we use a gumbel-softmax approximation BIBREF39 to $\\mathbf {x}$ when sampling substitution words from the LM."]},{"idx":10,"heading":"Learning Data Manipulation ::: Instantiations: Augmentation & Weighting ::: Learning Data Weights","paragraphs":["We now demonstrate the instantiation of data weighting. We aim to assign an importance weight to each training example to adapt its effect on model training. We automate the process by learning the data weights. This is achieved by parameterizing $R_\\phi $ as:","where $\\phi _i\\in \\mathbb {R}$ is the weight associated with the $i$th example. Plugging $R^{w}_\\phi $ into Eq.(DISPLAY_FORM13), we obtain the weighted update for the model $\\mathbf {\\theta }$:","In practice, when minibatch stochastic optimization is used, we approximate the weighted sampling by taking the softmax over the weights of only the minibatch examples. The data weights $\\mathbf {\\phi }$ are updated with Eq.(DISPLAY_FORM14). It is worth noting that the previous work BIBREF4 similarly derives data weights based on their gradient directions on a validation set. Our algorithm differs in that the data weights are parameters maintained and updated throughout the training, instead of re-estimated from scratch in each iteration. Experiments show the parametric treatment achieves superior performance in various settings. There are alternative parameterizations of $R_\\phi $ other than Eq.(DISPLAY_FORM20). For example, replacing $\\phi _i$ in Eq.(DISPLAY_FORM20) with $\\log \\phi _i$ in effect changes the softmax normalization in Eq.(DISPLAY_FORM21) to linear normalization, which is used in BIBREF4."]},{"idx":11,"heading":"Experiments","paragraphs":["We empirically validate the proposed data manipulation approach through extensive experiments on learning augmentation and weighting. We study both text and image classification, in two difficult settings of low data regime and imbalanced labels."]},{"idx":12,"heading":"Experiments ::: Experimental Setup","paragraphs":["Base Models. We choose strong pretrained networks as our base models for both text and image classification. Specifically, on text data, we use the BERT (base, uncased) model BIBREF8; while on image data, we use ResNet-34 BIBREF9 pretrained on ImageNet. We show that, even with the large-scale pretraining, data manipulation can still be very helpful to boost the model performance on downstream tasks. Since our approach uses validation sets for manipulation parameter learning, for a fair comparison with the base model, we train the base model in two ways. The first is to train the model on the training sets as usual and select the best step using the validation sets; the second is to train on the merged training and validation sets for a fixed number of steps. The step number is set to the average number of steps selected in the first method. We report the results of both methods.","Comparison Methods. We compare our approach with a variety of previous methods that were designed for specific manipulation schemes: (1) For text data augmentation, we compare with the latest model-based augmentation BIBREF19 which uses a fixed conditional BERT language model for word substitution (section SECREF15). As with base models, we also tried fitting the augmentatin model to both the training data and the joint training-validation data, and did not observe significant difference. Following BIBREF19, we also study a conventional approach that replaces words with their synonyms using WordNet BIBREF40. (2) For data weighting, we compare with the state-of-the-art approach BIBREF4 that dynamically re-estimates sample weights in each iteration based on the validation set gradient directions. We follow BIBREF4 and also evaluate the commonly-used proportion method that weights data by inverse class frequency.","Training. For both the BERT classifier and the augmentation model (which is also based on BERT), we use Adam optimization with an initial learning rate of 4e-5. For ResNets, we use SGD optimization with a learning rate of 1e-3. For text data augmentation, we augment each minibatch by generating two or three samples for each data points (each with 1, 2 or 3 substitutions), and use both the samples and the original data to train the model. For data weighting, to avoid exploding value, we update the weight of each data point in a minibatch by decaying the previous weight value with a factor of 0.1 and then adding the gradient. All experiments were implemented with PyTorch (pytorch.org) and were performed on a Linux machine with 4 GTX 1080Ti GPUs and 64GB RAM. All reported results are averaged over 15 runs $\\pm $ one standard deviation."]},{"idx":13,"heading":"Experiments ::: Low Data Regime","paragraphs":["We study the problem where only very few labeled examples for each class are available. Both of our augmentation and weighting boost base model performance, and are superior to respective comparison methods. We also observe that augmentation performs better than weighting in the low-data setting."]},{"idx":14,"heading":"Experiments ::: Low Data Regime ::: Setup","paragraphs":["For text classification, we use the popular benchmark datasets, including SST-5 for 5-class sentence sentiment BIBREF41, IMDB for binary movie review sentiment BIBREF42, and TREC for 6-class question types BIBREF43. We subsample a small training set on each task by randomly picking 40 instances for each class. We further create small validation sets, i.e., 2 instances per class for SST-5, and 5 instances per class for IMDB and TREC, respectively. The reason we use slightly more validation examples on IMDB and TREC is that the model can easily achieve 100% validation accuracy if the validation sets are too small. Thus, the SST-5 task has 210 labeled examples in total, while IMDB has 90 labels and TREC has 270. Such extremely small datasets pose significant challenges for learning deep neural networks. Since the manipulation parameters are trained using the small validation sets, to avoid possible overfitting we restrict the training to small number (e.g., 5 or 10) of epochs. For image classification, we similarly create a small subset of the CIFAR10 data, which includes 40 instances per class for training, and 2 instances per class for validation."]},{"idx":15,"heading":"Experiments ::: Low Data Regime ::: Results","paragraphs":["Table TABREF26 shows the manipulation results on text classification. For data augmentation, our approach significantly improves over the base model on all the three datasets. Besides, compared to both the conventional synonym substitution and the approach that keeps the augmentation network fixed, our adaptive method that fine-tunes the augmentation network jointly with model training achieves superior results. Indeed, the heuristic-based synonym approach can sometimes harm the model performance (e.g., SST-5 and IMDB), as also observed in previous work BIBREF19, BIBREF18. This can be because the heuristic rules do not fit the task or datasets well. In contrast, learning-based augmentation has the advantage of adaptively generating useful samples to improve model training.","Table TABREF26 also shows the data weighting results. Our weight learning consistently improves over the base model and the latest weighting method BIBREF4. In particular, instead of re-estimating sample weights from scratch in each iteration BIBREF4, our approach treats the weights as manipulation parameters maintained throughout the training. We speculate that the parametric treatment can adapt weights more smoothly and provide historical information, which is beneficial in the small-data context.","It is interesting to see from Table TABREF26 that our augmentation method consistently outperforms the weighting method, showing that data augmentation can be a more suitable technique than data weighting for manipulating small-size data. Our approach provides the generality to instantiate diverse manipulation types and learn with the same single procedure.","To investigate the augmentation model and how the fine-tuning affects the augmentation results, we show in Figure TABREF27 the top-5 most probable word substitutions predicted by the augmentation model for two masked tokens, respectively. Comparing the results of epoch 1 and epoch 3, we can see the augmentation model evolves and dynamically adjusts the augmentation behavior as the training proceeds. Through fine-tuning, the model seems to make substitutions that are more coherent with the conditioning label and relevant to the original words (e.g., replacing the word “striking” with “bland” in epoch 1 v.s. “charming” in epoch 3).","Table TABREF27 shows the data weighting results on image classification. We evaluate two settings with the ResNet-34 base model being initialized randomly or with pretrained weights, respectively. Our data weighting consistently improves over the base model and BIBREF4 regardless of the initialization."]},{"idx":16,"heading":"Experiments ::: Imbalanced Labels","paragraphs":["We next study a different problem setting where the training data of different classes are imbalanced. We show the data weighting approach greatly improves the classification performance. It is also observed that, the LM data augmentation approach, which performs well in the low-data setting, fails on the class-imbalance problems."]},{"idx":17,"heading":"Experiments ::: Imbalanced Labels ::: Setup","paragraphs":["Though the methods are broadly applicable to multi-way classification problems, here we only study binary classification tasks for simplicity. For text classification, we use the SST-2 sentiment analysis benchmark BIBREF41; while for image, we select class 1 and 2 from CIFAR10 for binary classification. We use the same processing on both datasets to build the class-imbalance setting. Specifically, we randomly select 1,000 training instances of class 2, and vary the number of class-1 instances in $\\lbrace 20, 50, 100\\rbrace $. For each dataset, we use 10 validation examples in each class. Trained models are evaluated on the full binary-class test set."]},{"idx":18,"heading":"Experiments ::: Imbalanced Labels ::: Results","paragraphs":["Table TABREF29 shows the classification results on SST-2 with varying imbalance ratios. We can see our data weighting performs best across all settings. In particular, the improvement over the base model increases as the data gets more imbalanced, ranging from around 6 accuracy points on 100:1000 to over 20 accuracy points on 20:1000. Our method is again consistently better than BIBREF4, validating that the parametric treatment is beneficial. The proportion-based data weighting provides only limited improvement, showing the advantage of adaptive data weighting. The base model trained on the joint training-validation data for fixed steps fails to perform well, partly due to the lack of a proper mechanism for selecting steps.","Table TABREF30 shows the results on imbalanced CIFAR10 classification. Similarly, our method outperforms other comparison approaches. In contrast, the fixed proportion-based method sometimes harms the performance as in the 50:1000 and 100:1000 settings.","We also tested the text augmentation LM on the SST-2 imbalanced data. Interestingly, the augmentation tends to hinder model training and yields accuracy of around 50% (random guess). This is because the augmentation LM is first fit to the imbalanced data, which makes label preservation inaccurate and introduces lots of noise during augmentation. Though a more carefully designed augmentation mechanism can potentially help with imbalanced classification (e.g., augmenting only the rare classes), the above observation further shows that the varying data manipulation schemes have different applicable scopes. Our approach is thus favorable as the single algorithm can be instantiated to learn different schemes."]},{"idx":19,"heading":"Discussions: Algorithm Extrapolation between Learning Paradigms","paragraphs":["Conclusions. We have developed a new method of learning different data manipulation schemes with the same single algorithm. Different manipulation schemes reduce to just different parameterization of the data reward function. The manipulation parameters are trained jointly with the target model parameters. We instantiate the algorithm for data augmentation and weighting, and show improved performance over strong base models and previous manipulation methods. We are excited to explore more types of manipulations such as data synthesis, and in particular study the combination of different manipulation schemes.","The proposed method builds upon the connections between supervised learning and reinforcement learning (RL) BIBREF6 through which we extrapolate an off-the-shelf reward learning algorithm in the RL literature to the supervised setting. The way we obtained the manipulation algorithm represents a general means of innovating problem solutions based on unifying formalisms of different learning paradigms. Specifically, a unifying formalism not only offers new understandings of the seemingly distinct paradigms, but also allows us to systematically apply solutions to problems in one paradigm to similar problems in another. Previous work along this line has made fruitful results in other domains. For example, an extended formulation of BIBREF6 that connects RL and posterior regularization (PR) BIBREF44, BIBREF45 has enabled to similarly export a reward learning algorithm to the context of PR for learning structured knowledge BIBREF46. By establishing a uniform abstration of GANs BIBREF47 and VAEs BIBREF33, BIBREF48 exchange techniques between the two families and get improved generative modeling. Other work in the similar spirit includes BIBREF49, BIBREF50, BIBREF51.","By extrapolating algorithms between paradigms, one can go beyond crafting new algorithms from scratch as in most existing studies, which often requires deep expertise and yields unique solutions in a dedicated context. Instead, innovation becomes easier by importing rich ideas from other paradigms, and is repeatable as a new algorithm can be methodically extrapolated to multiple different contexts."]},{"idx":20,"heading":"Figures and Tables","paragraphs":["Figure 1: Algorithm Computation. Blue arrows denote learning model θ. Red arrows denote learning manipulation φ. Solid arrows denote forward pass. Dashed arrows denote backward pass and parameter updates.","Table 1: Accuracy of Data Manipulation on Text Classification. All results are averaged over 15 runs ± one standard deviation. The numbers in parentheses next to the dataset names indicate the size of the datasets. For example, (40+2) denotes 40 training instances and 2 validation instances per class.","Figure 2: Words predicted with the highest probabilities by the augmentation LM. Two tokens “striking” and “grey” are masked for substitution. The boxes in respective colors list the predicted words after training epoch 1 and 3, respectively. E.g., “stunning” is the most probable substitution for “striking” in epoch 1.","Table 3: Accuracy of Data Weighting on Imbalanced SST-2. The first row shows the number of training examples in each of the two classes.","Table 4: Accuracy of Data Weighting on Imbalanced CIFAR10. The first row shows the number of training examples in each of the two classes."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1910.14589","title":"Machine Translation of Restaurant Reviews: New Corpus for Domain Adaptation and Robustness","sections":[{"idx":0,"heading":"Abstract","paragraphs":["We share a French-English parallel corpus of Foursquare restaurant reviews (this https URL), and define a new task to encourage research on Neural Machine Translation robustness and domain adaptation, in a real-world scenario where better-quality MT would be greatly beneficial. We discuss the challenges of such user-generated content, and train good baseline models that build upon the latest techniques for MT robustness. We also perform an extensive evaluation (automatic and human) that shows significant improvements over existing online systems. Finally, we propose task-specific metrics based on sentiment analysis or translation accuracy of domain-specific polysemous words."]},{"idx":1,"heading":"Introduction","paragraphs":["Very detailed information about social venues such as restaurants is available from user-generated reviews in applications like Google Maps, TripAdvisor or Foursquare. Most of these reviews are written in the local language and are not directly exploitable by foreign visitors: an analysis of the Foursquare database shows that, in Paris, only 49% of the restaurants have at least one review in English. It can be much worse for other cities and languages (e.g., only 1% of Seoul restaurants for a French-only speaker).","Machine Translation of such user-generated content can improve the situation and make the data available for direct display or for downstream NLP tasks (e.g., cross-lingual information retrieval, sentiment analysis, spam or fake review detection), provided its quality is sufficient.","We asked professionals to translate 11.5k French Foursquare reviews (18k sentences) to English. We believe that this resource will be valuable to the community for training and evaluating MT systems addressing challenges posed by user-generated content, which we discuss in detail in this paper.","We conduct extensive experiments and combine techniques that seek to solve these challenges (e.g., factored case, noise generation, domain adaptation with tags) on top of a strong Transformer baseline. In addition to BLEU evaluation and human evaluation, we use targeted metrics that measure how well polysemous words are translated, or how well sentiments expressed in the original review can still be recovered from its translation."]},{"idx":2,"heading":"Related work","paragraphs":["Translating restaurant reviews written by casual customers presents several difficulties for NMT, in particular robustness to non-standard language and adaptation to a specific style or domain (see Section SECREF7 for details).","Concerning robustness to noisy user generated content, BIBREF0 stress differences with traditional domain adaptation problems, and propose a typology of errors, many of which we also detected in the Foursquare data. They also released a dataset (MTNT), whose sources were selected from a social media (Reddit) on the basis of being especially noisy (see Appendix for a comparison with Foursquare). These sources were then translated by humans to produce a parallel corpus that can be used to engineer more robust NMT systems and to evaluate them. This corpus was the basis of the WMT 2019 Robustness Task BIBREF1, in which BIBREF2 ranked first. We use the same set of robustness and domain adaptation techniques, which we study more in depth and apply to our review translation task.","BIBREF3, BIBREF4 and BIBREF5 propose to improve robustness by training models on data-augmented corpora, containing noisy sources obtained by random word or character deletions, insertions, substitutions or swaps. Recently, BIBREF6 proposed to use a similar technique along with noise generation through replacement of a clean source by one obtained by back-translation.","We employ several well-known domain adaptation techniques: back-translation of large monolingual corpora close to the domain BIBREF7, BIBREF8, fine-tuning with in-domain parallel data BIBREF9, BIBREF10, BIBREF11, domain tags for knowledge transfer between domains BIBREF12, BIBREF2.","Addressing the technical issues of robustness and adaptation of an NMT system is decisive for real-world deployment, but evaluation is also critical. This aspect is stressed by BIBREF13 (NMT of curated hotel descriptions), who point out that automatic metrics like BLEU tend to neglect semantic differences that have a small textual footprint, but may be seriously misleading in practice, for instance by interpreting available parking as if it meant free parking. To mitigate this, we conduct additional evaluations of our models: human evaluation, translation accuracy of polysemous words, and indirect evaluation with sentiment analysis."]},{"idx":3,"heading":"Task description","paragraphs":["We present a new task of restaurant review translation, which combines domain adaptation and robustness challenges."]},{"idx":4,"heading":"Task description ::: Corpus description","paragraphs":["We sampled 11.5k French reviews from Foursquare, mostly in the food category, split them into 18k sentences, and grouped them into train, valid and test sets (see Table TABREF6). The French reviews contain on average 1.5 sentences and 17.9 words. Then, we hired eight professional translators to translate them to English. Two of them created the training set by post-editing (PE) the outputs of baseline NMT systems. The other six translated the valid and test sets from scratch. They were asked to translate (or post-edit) the reviews sentence-by-sentence (to avoid any alignment problem), but they could see the full context. We manually filtered the test set to remove translations that were not satisfactory. The full reviews and additional metadata (e.g., location and type of the restaurant) are also available as part of this resource, to encourage research on contextual machine translation.","Foursquare-HT was translated from scratch by the same translators who post-edited Foursquare-PE. While we did not use it in this work, it can be used as extra training or development data. We also release a human translation of the French-language test set (668 sentences) of the Aspect-Based Sentiment Analysis task at SemEval 2016 BIBREF14."]},{"idx":5,"heading":"Task description ::: Challenges","paragraphs":["Translating restaurant reviews presents two main difficulties compared to common tasks in MT. First, the reviews are written in a casual style, close to spoken language. Some liberty is taken w.r.t. spelling, grammar, and punctuation. Slang is also very frequent. MT should be robust to these variations. Second, they generally are reactions, by clients of a restaurant, about its food quality, service or atmosphere, with specific words relating to these aspects or sentiments. These require some degree of domain adaptation. The table above illustrates these issues, with outputs from an online MT system. Examples of full reviews from Foursquare-PE along with metadata are shown in Appendix.","Examples 1 and 2 fall into the robustness category: 1 is an extreme form of SMS-like, quasi-phonetic, language (et quand j'ai vu ça); 2 is a literal transcription of a long-vowel phonetic stress (trop $\\rightarrow $ trooop). Example 3 falls into the domain category: in a restaurant context, cadre typically refers to the setting. Examples 4 and 5 involve both robustness and domain adaptation: pété un cable is a non-compositional slang expression and garçon is not a boy in this domain; nickel is slang for great, très is missing an accent, and pâtes is misspelled as pattes, which is another French word.","Regarding robustness, we found many of the same errors listed by BIBREF0 as noise in social media text: SMS language (é qd g vu sa), typos and phonetic spelling (pattes), repeated letters (trooop, merciiii), slang (nickel, bof, mdr), missing or wrong accents (tres), emoticons (`:-)') and emojis, missing punctuation, wrong or non-standard capitalization (lowercase proper names, capitalized words for emphasis). Regarding domain aspects, there are polysemous words with typical specific meaning carte $\\rightarrow $ map, menu; cadre $\\rightarrow $ frame, executive, setting), idiomatic expressions (à tomber par terre $\\rightarrow $ to die for), and venue-related named entities (La Boîte à Sardines)."]},{"idx":6,"heading":"Robustness to noise","paragraphs":["We propose solutions for dealing with non-standard case, emoticons, emojis and other issues."]},{"idx":7,"heading":"Robustness to noise ::: Rare character placeholder","paragraphs":["We segment our training data into subwords with BPE BIBREF15, implemented in SentencePiece BIBREF16. BPE can deal with rare or unseen words by splitting them into more frequent subwords, but cannot deal with unseen characters. While this is not a problem in most tasks, Foursquare contains many emojis, and sometimes symbols in other scripts (e.g., Arabic). Unicode now defines around 3k emojis, most of which are likely to be out-of-vocabulary.","We replace rare characters on both sides of the training corpus by a placeholder (<x>). A model trained on this data is typically able to copy the placeholder at the correct position. Then, at inference time, we replace the output tokens <x> by the rare source-side characters, in the same order. This approach is similar to that of BIBREF18, who used the attention mechanism to replace UNK symbols with the aligned word in the source. BIBREF2 used the same technique to deal with emojis in the WMT robustness task."]},{"idx":8,"heading":"Robustness to noise ::: Capital letters","paragraphs":["As shown in Table TABREF11, capital letters are another source of confusion. HONTE and honte are considered as two different words. The former is out-of-vocabulary and is split very aggressively by BPE. This causes the MT model to hallucinate."]},{"idx":9,"heading":"Robustness to noise ::: Capital letters ::: Lowercasing","paragraphs":["A solution is to lowercase the input, both at training and at test time. However, when doing so, some information may be lost (e.g., named entities, acronyms, emphasis) which may result in lower translation quality."]},{"idx":10,"heading":"Robustness to noise ::: Capital letters ::: Factored translation","paragraphs":["BIBREF13 do factored machine translation BIBREF19, BIBREF20 where a word and its case are split in two different features. For instance, HONTE becomes honte + upper.","We implement this with two embedding matrices, one for words and one for case, and represent a token as the sum of the embeddings of its factors. For the target side, we follow BIBREF20 and have two softmax operations. We first predict the word in its lowercase form and then predict its case. The embeddings of the case and word are then summed and used as input for the next decoder step."]},{"idx":11,"heading":"Robustness to noise ::: Capital letters ::: Inline casing","paragraphs":["BIBREF2 propose another approach, inline casing, which does not require any change in the model. We insert the case as a regular token into the sequence right after the word. Special tokens <U>, <L> and <T> (upper, lower and title) are used for this purpose and appended to the vocabulary. Contrary to the previous solution, there is only one embedding matrix and one softmax.","In practice, words are assumed to be lowercase by default and the <L> tokens are dropped to keep the factored sequences as short as possible. “Best fries EVER\" becomes “best <T> _f ries _ever <U>\". Like BIBREF2, we force SentencePiece to split mixed-case words like MacDonalds into single-case subwords (Mac and Donalds)."]},{"idx":12,"heading":"Robustness to noise ::: Capital letters ::: Synthetic case noise","paragraphs":["Another solution that we experiment with (see Section SECREF6) is to inject noise on the source side of the training data by changing random source words to upper (5% chance), title (10%) or lower case (20%)."]},{"idx":13,"heading":"Robustness to noise ::: Natural noise","paragraphs":["One way to make an NMT system more robust is to train it with some of the most common errors that can be found in the in-domain data. Like BIBREF2, we detect the errors that occur naturally in the in-domain data and then apply them to our training corpus, while respecting their natural distribution. We call this “natural noise generation” in opposition to what is done in BIBREF3, BIBREF4, BIBREF6 or in Section SECREF10, where the noise is more synthetic."]},{"idx":14,"heading":"Robustness to noise ::: Natural noise ::: Detecting errors","paragraphs":["We compile a general-purpose French lexicon as a transducer, implemented to be traversed with extended edit distance flags, similar to BIBREF21. Whenever a word is not found in the lexicon (which means that it is a potential spelling mistake), we look for a French word in the lexicon within a maximum edit distance of 2, with the following set of edit operations:","We apply the transducer to the French monolingual Foursquare data (close to 1M sentences) to detect and count noisy variants of known French words. This step produces a dictionary mapping the correct spelling to the list of observed errors and their respective frequencies.","In addition to automatically extracted spelling errors, we extract a set of common abbreviations from BIBREF22 and we manually identify a list of common errors in French:"]},{"idx":15,"heading":"Robustness to noise ::: Natural noise ::: Generating errors","paragraphs":["With this dictionary, describing the real error distribution in Foursquare text, we take our large out-of-domain training corpus, and randomly replace source-side words with one of their variants (rules 1 to 6), while respecting the frequency of this variant in the real data. We also manually define regular expressions to randomly apply rules 7 to 11 (e.g., \"er \"$\\rightarrow $\"é \").","We obtain a noisy parallel corpus (which we use instead of the “clean” training data), where about 30% of all source sentences have been modified, as shown below:"]},{"idx":16,"heading":"Domain Adaptation","paragraphs":["To adapt our models to the restaurant review domain we apply the following types of techniques: back-translation of in-domain English data, fine-tuning with small amounts of in-domain parallel data, and domain tags."]},{"idx":17,"heading":"Domain Adaptation ::: Back-translation","paragraphs":["Back-translation (BT) is a popular technique for domain adaptation when large amounts of in-domain monolingual data are available BIBREF7, BIBREF8. While our in-domain parallel corpus is small (12k pairs), Foursquare contains millions of English-language reviews. Thus, we train an NMT model in the reverse direction (EN$\\rightarrow $FR) and translate all the Foursquare English reviews to French. This gives a large synthetic parallel corpus.","This in-domain data is concatenated to the out-of-domain parallel data and used for training.","BIBREF8 show that doing back-translation with sampling instead of beam search brings large improvements due to increased diversity. Following this work, we test several settings:","We use a temperature of $T=\\frac{1}{0.9}$ to avoid the extremely noisy output obtained with $T=1$ and strike a balance between quality and diversity."]},{"idx":18,"heading":"Domain Adaptation ::: Fine-tuning","paragraphs":["When small amounts of in-domain parallel data are available, fine-tuning (FT) is often the preferred solution for domain adaptation BIBREF9, BIBREF10. It consists in training a model on out-of-domain data, and then continuing its training for a few epochs on the in-domain data only."]},{"idx":19,"heading":"Domain Adaptation ::: Corpus tags","paragraphs":["BIBREF12 propose a technique for multi-domain NMT, which consists in inserting a token in each source sequence specifying its domain. The system can learn the particularities of multiple domains (e.g., polysemous words that have a different meaning depending on the domain), which we can control at test time by manually setting the tag. BIBREF23 also use tags to control politeness in the model's output.","As our corpus (see Section SECREF28) is not clearly divided into domains, we apply the same technique as BIBREF12 but use corpus tags (each sub-corpus has its own tag: TED, Paracrawl, etc.) which we add to each source sequence. Like in BIBREF2, the Foursquare post-edited and back-translated data also get their own tags (PE and BT). Figure FIGREF27 gives an example where using the PE corpus tag at test time helps the model pick a more adequate translation."]},{"idx":20,"heading":"Experiments ::: Training data","paragraphs":["After some initial work with the WMT 2014 data, we built a new training corpus named UGC (User Generated Content), closer to our domain, by combining: Multi UN, OpenSubtitles, Wikipedia, Books, Tatoeba, TED talks, ParaCrawl and Gourmet (See Table TABREF31). UGC does not include Common Crawl (which contains many misaligned sentences and caused hallucinations), but it includes OpenSubtitles BIBREF24 (spoken-language, possibly closer to Foursquare). We observed an improvement of more than 1 BLEU on newstest2014 when switching to UGC, and almost 6 BLEU on Foursquare-valid."]},{"idx":21,"heading":"Experiments ::: Pre-processing","paragraphs":["We use langid.py BIBREF25 to filter sentence pairs from UGC. We also remove duplicate sentence pairs, and lines longer than 175 words or with a length ratio greater than $1.5$ (see Table TABREF31). Then we apply SentencePiece and our rare character handling strategy (Section SECREF8). We use a joined BPE model of size 32k, trained on the concatenation of both sides of the corpus, and set SentencePiece's vocabulary threshold to 100. Finally, unless stated otherwise, we always use the inline casing approach (see Section SECREF10)."]},{"idx":22,"heading":"Experiments ::: Model and settings","paragraphs":["For all experiments, we use the Transformer Big BIBREF26 as implemented in Fairseq, with the hyperparameters of BIBREF27. Training is done on 8 GPUs, with accumulated gradients over 10 batches BIBREF27, and a max batch size of 3500 tokens per GPU. We train for 20 epochs, while saving a checkpoint every 2500 updates ($\\approx \\frac{2}{5}$ epoch on UGC) and average the 5 best checkpoints according to their perplexity on a validation set (a held-out subset of UGC).","For fine-tuning, we use a fixed learning rate, and a total batch size of 3500 tokens (training on a single GPU without delayed updates). To avoid overfitting on Foursquare-PE, we do early stopping according to perplexity on Foursquare-valid. For each fine-tuned model we test all 16 combinations of dropout in $\\lbrace 0.1,0.2,0.3,0.4\\rbrace $ and learning rate in $\\lbrace 1, 2, 5, 10\\rbrace \\times 10^{-5}$. We keep the model with the best perplexity on Foursquare-valid."]},{"idx":23,"heading":"Experiments ::: Evaluation methodology","paragraphs":["During our work, we used BLEU BIBREF28 on newstest[2012, 2013] to ensure that our models stayed good on a more general domain, and on Foursquare-valid to measure performance on the Foursquare domain.","For sake of brevity, we only give the final BLEU scores on newstest2014 and Foursquare-test. Scores on Foursquare-valid, and MTNT-test (for comparison with BIBREF0, BIBREF2) are given in Appendix. We evaluate “detokenized” MT outputs against raw references using SacreBLEU BIBREF29.","In addition to BLEU, we do an indirect evaluation on an Aspect-Based Sentiment Analysis (ABSA) task, a human evaluation, and a task-related evaluation based on polysemous words."]},{"idx":24,"heading":"Experiments ::: BLEU evaluation ::: Capital letters","paragraphs":["Table TABREF41 compares the case handling techniques presented in Section SECREF10. To better evaluate the robustness of our models to changes of case, we built 3 synthetic test sets from Foursquare-test, with the same target, but all source words in upper, lower or title case.","Inline and factored case perform equally well, significantly better than the default (cased) model, especially on all-uppercase inputs. Lowercasing the source is a good option, but gives a slightly lower score on regular Foursquare-test. Finally, synthetic case noise added to the source gives surprisingly good results. It could also be combined with factored or inline case."]},{"idx":25,"heading":"Experiments ::: BLEU evaluation ::: Natural noise","paragraphs":["Table TABREF44 compares the baseline “inline case” model with the same model augmented with natural noise (Section SECREF17). Performance is the same on Foursquare-test, but significantly better on newstest2014 artificially augmented with Foursquare-like noise."]},{"idx":26,"heading":"Experiments ::: BLEU evaluation ::: Domain adaptation","paragraphs":["Table TABREF46 shows the results of the back-translation (BT) techniques. Surprisingly, BT with beam search (BT-B) deteriorates BLEU scores on Foursquare-test, while BT with sampling gives a consistent improvement. BLEU scores on newstest2014 are not significantly impacted, suggesting that BT can be used for domain adaptation without hurting quality on other domains.","Table TABREF47 compares the domain adaptation techniques presented in Section SECREF5. We observe that:","Concatenating the small Foursquare-PE corpus to the 50M general domain corpus does not help much, unless using corpus tags.","Foursquare-PE + tags is not as good as fine-tuning with Foursquare-PE. However, fine-tuned models get slightly worse results on news.","Back-translation combined with tags gives a large boost. The BT tag should not be used at test time, as it degrades results.","Using no tag at test time works fine, even though all training sentences had tags.","As shown in Table TABREF54, these techniques can be combined to achieve the best results. The natural noise does not have a significant effect on BLEU scores. Back-translation combined with fine-tuning gives the best performance on Foursquare (+4.5 BLEU vs UGC). However, using tags instead of fine-tuning strikes a better balance between general domain and in-domain performance."]},{"idx":27,"heading":"Experiments ::: Targeted evaluation","paragraphs":["In this section we propose two metrics that target specific aspects of translation adequacy: translation accuracy of domain-specific polysemous words and Aspect-Based Sentiment Analysis performance on MT outputs."]},{"idx":28,"heading":"Experiments ::: Targeted evaluation ::: Translation of polysemous words","paragraphs":["We propose to count polysemous words specific to our domain, similarly to BIBREF31, to measure the degree of domain adaptation. TER between the translation hypotheses and the post-edited references in Foursquare-PE reveals the most common substitutions (e.g., “card” is often replaced with “menu”, suggesting that “card” is a common mistranslation of the polysemous word “carte”). We filter this list manually to only keep words that are polysemous and that have a high frequency in the test set. Table TABREF58 gives the 3 most frequent ones.","Table TABREF59 shows the accuracy of our models when translating these words. We see that the domain-adapted model is better at translating domain-specific polysemous words."]},{"idx":29,"heading":"Experiments ::: Targeted evaluation ::: Indirect evaluation with sentiment analysis","paragraphs":["We also measure adequacy by how well the translation preserves the polarity of the sentence regarding various aspects. To evaluate this, we perform an indirect evaluation on the SemEval 2016 Aspect-Based Sentiment Analysis (ABSA) task BIBREF14. We use our internal ABSA systems trained on English or French SemEval 2016 data. The evaluation is done on the SemEval 2016 French test set: either the original version (ABSA French), or its translation (ABSA English). As shown in Table TABREF61, translations obtained with domain-adapted models lead to significantly better scores on ABSA than the generic models."]},{"idx":30,"heading":"Experiments ::: Human Evaluation","paragraphs":["We conduct a human evaluation to confirm the observations with BLEU and to overcome some of the limitations of this metric.","We select 4 MT models for evaluation (see Table TABREF63) and show their 4 outputs at once, sentence-by-sentence, to human judges, who are asked to rank them given the French source sentence in context (with the full review). For each pair of models, we count the number of wins, ties and losses, and apply the Wilcoxon signed-rank test.","We took the first 300 test sentences to create 6 tasks of 50 sentences each. Then we asked bilingual colleagues to rank the output of 4 models by their translation quality. They were asked to do one or more of these tasks. The judge did not know about the list of models, nor the model that produced any given translation. We got 12 answers. The inter-judge Kappa coefficient ranged from 0.29 to 0.63, with an average of 0.47, which is a good value given the difficulty of the task. Table TABREF63 gives the results of the evaluation, which confirm our observations with BLEU.","We also did a larger-scale monolingual evaluation using Amazon Mechanical Turk (see Appendix), which lead to similar conclusions."]},{"idx":31,"heading":"Conclusion","paragraphs":["We presented a new parallel corpus of user reviews of restaurants, which we think will be valuable to the community. We proposed combinations of multiple techniques for robustness and domain adaptation, which address particular challenges of this new task. We also performed an extensive evaluation to measure the improvements brought by these techniques.","According to BLEU, the best single technique for domain adaptation is fine-tuning. Corpus tags also achieve good results, without degrading performance on a general domain. Back-translation helps, but only with sampling or tags. The robustness techniques (natural noise, factored case, rare character placeholder) do not improve BLEU.","While our models are promising, they still show serious errors when applied to user-generated content: missing negations, hallucinations, unrecognized named entities, insensitivity to context. This suggests that this task is far from solved.","We hope that this corpus, our natural noise dictionary, model outputs and human rankings will help better understand and address these problems. We also plan to investigate these problems on lower resource languages, where we expect the task to be even harder."]},{"idx":32,"heading":"Figures and Tables","paragraphs":["Table 1: Foursquare splits. Foursquare-PE is the training set. Foursquare-HT is not used in this work.","Table 2: Capital letters break NMT. BPE segmentation and translation of capitalized or lowercase input.","Figure 1: Example of ambiguous source sentence, where using corpus tags helps the model pick a more adequate translation.","Table 3: Size of the WMT and UGC training corpora (after filtering).","Table 4: Robustness to capital letters (see Section 4.2). Foursquare-test’s source side has been set to upper, lower or title case. The first column is case sensitive BLEU on Foursquare-test. “LC to cased” always gets the same scores because it is invariant to source case.","Table 5: Baseline model with or without natural noise (see Section 4.3). Noised news is the same type of noise, artificially applied to newstest2014.","Table 6: Comparison of different back-translation schemes (see Section 5.1). ⊕ denotes the concatenation of several training corpora.","Table 8: Combination of several robustness or domain adaptation techniques. At test time, we don’t use any tag on news, and use the PE tag on Foursquaretest (when applicable). BT: back-translation. PE: Foursquare-PE. FT: fine-tuning with Foursquare-PE. ⊕: concatenation.","Table 7: Domain adaptation with Foursquare-PE finetuning (FT) or corpus tags. The “tag” column represents the corpus tag used at test time (if any).","Table 9: French polysemous words found in Foursquare, and translation candidates in English. The most frequent meanings in Foursquare are underlined.","Table 10: Number of correct translations for difficult polysemous words in Foursquare-test by different models. The first row is the number of source sentences that contain this word. Other domain-adapted models (e.g., “UGC + FT” or “UGC ⊕ BT”) also get ≈ 99% accuracy.","Table 12: In-house human evaluation (“ ” means better with p ≤ 0.05). The 4 models Baseline, GT, Tags and Tags + noise correspond respectively to rows 2 (UGC with inline case), 3 (Google Translate), 6 (Combination of BT, PE and tags) and 8 (Same as 6 with natural noise) in Table 8.","Table 11: Indirect evaluation with Aspect-Based Sentiment Analysis (accuracy in %). ABSA French: ABSA model trained on French data and applied to the SemEval 2016 French test set; ABSA English: trained on English data and applied to human translations of the test set; ABSA English on MT outputs: applied to MT outputs instead of human translations.","Table 13: Noise comparison between Foursquare-test and MTNT-test (Michel and Neubig, 2018). Emojis, all-uppercase words (not counting acronyms) and spelling + grammar mistakes (according to MS Word) per 100 tokens.","Table 15: Large-scale Human Evaluation on Amazon Mechanical Turk (“ ” means p ≤ 0.01). The 4 models Baseline, GT, Tags and Tags + noise correspond respectively to rows 2 (UGC with inline case), 3 (Google Translate), 6 (Combination of BT, PE and tags) and 8 (Same as 6 with natural noise) in Table 8.","Table 14: Comparison of our models against the winner of the WMT 2019 Robustness Task on the MTNT test set (similar robustness challenges but different domain). We also give cased BLEU of our models on Foursquare-valid. Results on Foursquare-test are shown in the paper.","Table 17: Examples of challenging examples from Foursquare-PE. We show the full reviews with sentence delimiters (<s>) and metadata. The words that contain typos or that could cause trouble to a regular NMT model are shown in bold red.","Table 20: Examples of sentences from Foursquare-test with polysemous words (in bold red), where domain adaptation helps (with Foursquare-PE fine-tuning and back-translation).","Table 18: Examples of sentences from Foursquare-test with capitalized words, where default (cased) MT gets the translation wrong and inline case helps.","Table 19: Examples of sentences from Foursquare-test with noisy spelling (in bold red), where training with source-side natural noise helps.","Table 21: Examples of bad translations by our best model (Noise ⊕ BT ⊕ PE + tags). All examples are from Foursquare-test, except for the last one, which is from SemEval."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1910.14599","title":"Adversarial NLI: A New Benchmark for Natural Language Understanding","sections":[{"idx":0,"heading":"Abstract","paragraphs":["We introduce a new large-scale NLI benchmark dataset, collected via an iterative, adversarial human-and-model-in-the-loop procedure. We show that training models on this new dataset leads to state-of-the-art performance on a variety of popular NLI benchmarks, while posing a more difficult challenge with its new test set. Our analysis sheds light on the shortcomings of current state-of-the-art models, and shows that non-expert annotators are successful at finding their weaknesses. The data collection method can be applied in a never-ending learning scenario, becoming a moving target for NLU, rather than a static benchmark that will quickly saturate."]},{"idx":1,"heading":"Introduction","paragraphs":["Progress in AI has been driven by, among other things, the development of challenging large-scale benchmarks like ImageNet BIBREF0 in computer vision, and SNLI BIBREF1, SQuAD BIBREF2, and others in natural language processing (NLP). Recently, for natural language understanding (NLU) in particular, the focus has shifted to combined benchmarks like SentEval BIBREF3 and GLUE BIBREF4, which track model performance on multiple tasks and provide a unified platform for analysis.","With the rapid pace of advancement in AI, however, NLU benchmarks struggle to keep up with model improvement. Whereas it took around 15 years to achieve “near-human performance” on MNIST BIBREF5, BIBREF6, BIBREF7 and approximately 7 years to surpass humans on ImageNet BIBREF8, BIBREF0, BIBREF9, the GLUE benchmark did not last as long as we would have hoped after the advent of BERT BIBREF10, and rapidly had to be extended into SuperGLUE BIBREF11. This raises an important question: Can we collect a large benchmark dataset that can last longer?","The speed with which benchmarks become obsolete raises another important question: are current NLU models genuinely as good as their high performance on benchmarks suggests? A growing body of evidence shows that state-of-the-art models learn to exploit spurious statistical patterns in datasets BIBREF12, BIBREF13, BIBREF14, BIBREF15, BIBREF16, BIBREF17, instead of learning meaning in the flexible and generalizable way that humans do. Given this, human annotators—be they seasoned NLP researchers or non-experts—might easily be able to construct examples that expose model brittleness.","We propose an iterative, adversarial human-and-model-in-the-loop solution for NLU dataset collection that addresses both benchmark longevity and robustness issues. In the first stage, human annotators devise examples that our current best models cannot determine the correct label for. These resulting hard examples—which should expose additional model weaknesses—can be added to the training set and used to train a stronger model. We then subject the strengthened model to human interference and collect more weaknesses over several rounds. After each round, we both train a new model, and set aside a new test set. The process can be iteratively repeated in a never-ending learning BIBREF18 setting, with the model getting stronger and the test set getting harder in each new round.This process yields a “moving post” dynamic target for NLU systems, rather than a static benchmark that will eventually saturate.","Our approach draws inspiration from recent efforts that gamify collaborative training of machine learning agents over multiple rounds BIBREF19 and pit “builders” against “breakers” to learn better models BIBREF20. Recently, Dinan2019build showed that a similar approach can be used to make dialogue safety classifiers more robust. Here, we focus on natural language inference (NLI), arguably the most canonical task in NLU. We collected three rounds of data, and call our new dataset Adversarial NLI (ANLI).","Our contributions are as follows: 1) We introduce a novel human-and-model-in-the-loop dataset, currently consisting of three rounds that progressively increase in difficulty and complexity, that includes annotator-provided explanations. 2) We show that training models on this new dataset leads to state-of-the-art performance on a variety of popular NLI benchmarks. 3) We provide a detailed analysis of the collected data that sheds light on the shortcomings of current models, categorizes the data by inference type to examine weaknesses, and demonstrates good performance on NLI stress tests. The ANLI dataset is available at github.com/facebookresearch/anli/. A demo of the annotation procedure can be viewed at adversarialnli.com."]},{"idx":2,"heading":"Dataset collection","paragraphs":["The primary aim of this work is to create a new large-scale NLI benchmark on which current state-of-the-art models fail. This constitutes a new target for the field to work towards, and can elucidate model capabilities and limitations. As noted, however, static benchmarks do not last very long these days. If continuously deployed, the data collection procedure we introduce here can pose a dynamic challenge that allows for never-ending learning."]},{"idx":3,"heading":"Dataset collection ::: HAMLET","paragraphs":["To paraphrase the great bard BIBREF21, there is something rotten in the state of the art. We propose Human-And-Model-in-the-Loop Entailment Training (HAMLET), a training procedure to automatically mitigate problems with current dataset collection procedures (see Figure FIGREF1).","In our setup, our starting point is a base model, trained on NLI data. Rather than employing automated adversarial methods, here the model's “adversary” is a human annotator. Given a context (also often called a “premise” in NLI), and a desired target label, we ask the human writer to provide a hypothesis that fools the model into misclassifying the label. One can think of the writer as a “white hat” hacker, trying to identify vulnerabilities in the system. For each human-generated example that is misclassified, we also ask the writer to provide a reason why they believe it was misclassified.","For examples that the model misclassified, it is necessary to verify that they are actually correct —i.e., that the given context-hypothesis pairs genuinely have their specified target label. The best way to do this is to have them checked by another human. Hence, we provide the example to human verifiers. If two human verifiers agree with the writer, the example is considered a good example. If they disagree, we ask a third human verifier to break the tie. If there is still disagreement between the writer and the verifiers, the example is discarded. Occasionally, verifiers will overrule the original label of the writer.","Once data collection for the current round is finished, we construct a new training set from the collected data, with accompanying development and test sets. While the training set includes correctly classified examples, the development and tests sets are built solely from them. The test set was further restricted so as to: 1) include pairs from “exclusive” annotators that are never included in the training data; and 2) be balanced by label classes (and genres, where applicable). We subsequently train a new model on this and other existing data, and repeat the procedure three times."]},{"idx":4,"heading":"Dataset collection ::: Annotation details","paragraphs":["We employed crowdsourced workers from Mechanical Turk with qualifications. We collected hypotheses via the ParlAI framework. Annotators are presented with a context and a target label—either `entailment', `contradiction', or `neutral'—and asked to write a hypothesis that corresponds to the label. We phrase the label classes as “definitely correct”, “definitely incorrect”, or “neither definitely correct nor definitely incorrect” given the context, to make the task easier to grasp. Submitted hypotheses are given to the model to make a prediction for the context-hypothesis pair. The probability of each label is returned to the worker as feedback. If the model predicts the label incorrectly, the job is complete. If not, the worker continues to write hypotheses for the given (context, target-label) pair until the model predicts the label incorrectly or the number of tries exceeds a threshold (5 tries in the first round, 10 tries thereafter). To encourage workers, payments increased as rounds became harder. For hypotheses that the model predicted the incorrect label for, but were verified by other humans, we paid an additional bonus on top of the standard rate."]},{"idx":5,"heading":"Dataset collection ::: Round 1","paragraphs":["For the first round, we used a BERT-Large model BIBREF10 trained on a concatenation of SNLI BIBREF1 and MNLI BIBREF22, and selected the best-performing model we could train as the starting point for our dataset collection procedure. For Round 1 contexts, we randomly sampled short multi-sentence passages from Wikipedia (of 250-600 characters) from the manually curated HotpotQA training set BIBREF23. Contexts are either ground-truth contexts from that dataset, or they are Wikipedia passages retrieved using TF-IDF BIBREF24 based on a HotpotQA question."]},{"idx":6,"heading":"Dataset collection ::: Round 2","paragraphs":["For the second round, we used a more powerful RoBERTa model BIBREF25 trained on SNLI, MNLI, an NLI-version of FEVER BIBREF26, and the training data from the previous round (A1). After a hyperparameter search, we selected the model with the best performance on the A1 development set. Then, using the hyperparameters selected from this search, we created a final set of models by training several models with different random seeds. During annotation, we constructed an ensemble by randomly picking a model from the model set as the adversary each turn. This helps us avoid annotators exploiting vulnerabilities in one single model. A new non-overlapping set of contexts was again constructed from Wikipedia via HotpotQA using the same method as Round 1."]},{"idx":7,"heading":"Dataset collection ::: Round 3","paragraphs":["For the third round, we selected a more diverse set of contexts, in order to explore robustness under domain transfer. In addition to contexts from Wikipedia for Round 3, we also included contexts from the following domains: News (extracted from Common Crawl), fiction (extracted from BIBREF27, and BIBREF28), formal spoken text (excerpted from court and presidential debate transcripts in the Manually Annotated Sub-Corpus (MASC) of the Open American National Corpus), and causal or procedural text, which describes sequences of events or actions, extracted from WikiHow. Finally, we also collected annotations using the longer contexts present in the GLUE RTE training data, which came from the RTE5 dataset BIBREF29. We trained an even stronger RoBERTa model by adding the training set from the second round (A2) to the training data."]},{"idx":8,"heading":"Dataset collection ::: Comparing with other datasets","paragraphs":["The ANLI dataset improves upon previous work in several ways. First, and most obviously, the dataset is collected to be more difficult than previous datasets, by design. Second, it remedies a problem with SNLI, namely that its contexts (or premises) are very short, because they were selected from the image captioning domain. We believe longer contexts should naturally lead to harder examples, and so we constructed ANLI contexts from longer, multi-sentence source material.","Following previous observations that models might exploit spurious biases in NLI hypotheses, BIBREF12, BIBREF13, we conduct a study of the performance of hypothesis-only models on our dataset. We show that such models perform poorly on our test sets.","With respect to data generation with naïve annotators, Geva2019taskorannotator noted that models might pick up on annotator bias, modelling the annotators themselves rather than capturing the intended reasoning phenomenon. To counter this, we selected a subset of annotators (i.e., the “exclusive” workers) whose data would only be included in the test set. This enables us to avoid overfitting to the writing style biases of particular annotators, and also to determine how much individual annotator bias is present for the main portion of the data. Examples from each round of dataset collection are provided in Table TABREF2.","Furthermore, our dataset poses new challenges to the community that were less relevant for previous work, such as: can we improve performance online without having to train a new model from scratch every round, how can we overcome catastrophic forgetting, how do we deal with mixed model biases, etc. Because the training set includes examples that the model got right but were not verified, it might be noisy, posing filtering as an additional interesting problem."]},{"idx":9,"heading":"Dataset statistics","paragraphs":["The dataset statistics can be found in Table TABREF7. The number of examples we collected increases per round, starting with approximately 19k examples for Round 1, to around 47k examples for Round 2, to over 103k examples for Round 3. We collected more data for later rounds not only because that data is likely to be more interesting, but also simply because the base model is better and so annotation took longer to collect good, verified correct examples of model vulnerabilities.","For each round, we report the model error rate, both on verified and unverified examples. The unverified model error rate captures the percentage of examples where the model disagreed with the writer's target label, but where we are not (yet) sure if the example is correct. The verified model error rate is the percentage of model errors from example pairs that other annotators were able to confirm the correct label for. Note that this error rate represents a straightforward way to evaluate model quality: the lower the model error rate—assuming constant annotator quality and context-difficulty—the better the model.","We observe that model error rates decrease as we progress through rounds. In Round 3, where we included a more diverse range of contexts from various domains, the overall error rate went slightly up compared to the preceding round, but for Wikipedia contexts the error rate decreased substantially. While for the first round roughly 1 in every 5 examples were verified model errors, this quickly dropped over consecutive rounds, and the overall model error rate is less than 1 in 10. On the one hand, this is impressive, and shows how far we have come with just three rounds. On the other hand, it shows that we still have a long way to go if even untrained annotators can fool ensembles of state-of-the-art models with relative ease.","Table TABREF7 also reports the average number of “tries”, i.e., attempts made for each context until a model error was found (or the number of possible tries is exceeded), and the average time this took (in seconds). Again, these metrics represent a useful way to evaluate model quality. We observe that the average tries and average time per verified error both go up as we progress through the rounds. The numbers clearly demonstrate that the rounds are getting increasingly more difficult."]},{"idx":10,"heading":"Results","paragraphs":["Table TABREF13 reports the main results. In addition to BERT BIBREF10 and RoBERTa BIBREF25, we also include XLNet BIBREF30 as an example of a strong, but different, model architecture. We show test set performance on the ANLI test sets per round, the total ANLI test set, and the exclusive test subset (examples from test-set-exclusive workers). We also show accuracy on the SNLI test set and the MNLI development (for the purpose of comparing between different model configurations across table rows) set. In what follows, we briefly discuss our observations."]},{"idx":11,"heading":"Results ::: Base model performance is low.","paragraphs":["Notice that the base model for each round performs very poorly on that round's test set. This is the expected outcome: For round 1, the base model gets the entire test set wrong, by design. For rounds 2 and 3, we used an ensemble, so performance is not necessarily zero. However, as it turns out, performance still falls well below chance, indicating that workers did not find vulnerabilities specific to a single model, but generally applicable ones for that model class."]},{"idx":12,"heading":"Results ::: Rounds become increasingly more difficult.","paragraphs":["As already foreshadowed by the dataset statistics, round 3 is more difficult (yields lower performance) than round 2, and round 2 is more difficult than round 1. This is true for all model architectures."]},{"idx":13,"heading":"Results ::: Training on more rounds improves robustness.","paragraphs":["Generally, our results indicate that training on more rounds improves model performance. This is true for all model architectures. Simply training on more “normal NLI” data would not help a model be robust to adversarial attacks, but our data actively helps mitigate these."]},{"idx":14,"heading":"Results ::: RoBERTa achieves state-of-the-art performance...","paragraphs":["We obtain state of the art performance on both SNLI and MNLI with the RoBERTa model finetuned on our new data The RoBERTa paper BIBREF25 reports a score of $90.2$ for both MNLI-matched and -mismatched dev, while we obtain $91.0$ and $90.7$. The state of the art on SNLI is currently held by MT-DNN BIBREF31, which reports $91.6$ compared to our $92.9$."]},{"idx":15,"heading":"Results ::: ...but is outperformed when it is base model.","paragraphs":["However, the base (RoBERTa) models for rounds 2 and 3 are outperformed by both BERT and XLNet. This shows that annotators have managed to write examples that RoBERTa generally struggles with, and more training data alone cannot easily mitigated these shortcomings. It also implies that BERT, XLNet, and RoBERTa all have different weaknesses, possibly as a function of their training data (BERT, XLNet and RoBERTa were trained on very different data sets, which might or might not have contained information relevant to the weaknesses)—an additional round with a wider model variety would thus be interesting to investigate as a next step."]},{"idx":16,"heading":"Results ::: Continuously augmenting training data does not downgrade performance.","paragraphs":["Even though ANLI training data is different from SNLI and MNLI, adding this data to the training set does not harm performance on those tasks. Furthermore, as Table TABREF21 shows, training only on ANLI is transferable to SNLI and MNLI, but not vice versa. This suggests that methods could successfully be applied for many more consecutive rounds."]},{"idx":17,"heading":"Results ::: Exclusive test subset difference is small.","paragraphs":["In order to avoid the possibility that models might pick up on annotator-specific artifacts, a concern raised by Geva2019taskorannotator, we included an exclusive test subset with examples from annotators never seen in the training data. We find that the differences between this exclusive subset and the test set are small, indicating that our models do not over-rely on individual annotator's writing styles."]},{"idx":18,"heading":"Results ::: Hypothesis-only results","paragraphs":["For SNLI and MNLI, concerns have been raised about the propensity of models to pick up on spurious artifacts that are present just in the hypotheses BIBREF12, BIBREF13. To study this in the context of our results and task difficulty, we compare models trained on (context, hypothesis) pairs to models trained only on the hypothesis (marked $H$). Table TABREF21 reports results on the three rounds of ANLI, as well as SNLI and MNLI. The table shows some interesting take-aways:"]},{"idx":19,"heading":"Results ::: Hypothesis-only results ::: Hypothesis-only models perform poorly on ANLI.","paragraphs":["We corroborate that hypothesis-only models obtain good performance on SNLI and MNLI. Performance of such models on ANLI is substantially lower, and decreases with more rounds."]},{"idx":20,"heading":"Results ::: Hypothesis-only results ::: RoBERTa does not outperform hypothesis-only on rounds 2 and 3.","paragraphs":["On the two rounds where RoBERTa was used as the base model, its performance is not much better than the hypothesis-only model. This could mean two things: either the test data is very difficult, or the training data is not good. To rule out the latter, we trained only on ANLI ($\\sim $163k training examples): doing so with RoBERTa matches the performance of BERT on MNLI when it is trained on the much larger, fully in-domain SNLI+MNLI combined dataset (943k training examples), with both getting $\\sim $86, which is impressive. Hence, this shows that our new challenge test sets are so difficult that the current state-of-the-art model cannot do better than a hypothesis-only prior."]},{"idx":21,"heading":"Analysis","paragraphs":["We perform two types of model error analysis. First we evaluate two popular existing test sets that were created to expose model weaknesses, and show that our dataset discourages models from learning spurious statistical facts, relative to other large popular datasets (e.g., SNLI and MNLI). Secondly, we explore, by round, the types of inferences our writers successfully employed to stump models, by performing hand-annotation on 500 examples from each round's development set."]},{"idx":22,"heading":"Analysis ::: Performance on challenge datasets","paragraphs":["Recently, several hard test sets have been made available for revealing the biases NLI models learn from their training datasets BIBREF32, BIBREF17, BIBREF12, BIBREF33. We examined model performance on two of these: the SNLI-Hard BIBREF12 test set, which consists of examples that hypothesis-only models label incorrectly, and the NLI stress tests BIBREF33, in which sentences containing antonyms pairs, negations, high word overlap, i.a., are heuristically constructed. We test our models on these stress tests, after tuning on each test's respective development set to account for potential domain mismatches. For comparison, we also report accuracies from the original papers: for SNLI-Hard we present the results from BIBREF12's implementation of the hierarchical tensor-based Densely Interactive Inference Network BIBREF34 on MNLI, and for the NLI stress tests, we present the performance of BIBREF33's implementation of InferSent BIBREF3 trained on SNLI. Our results are in Table TABREF23.","We observe that all of our models far outperform the models presented in original papers for these common stress tests, with our two RoBERTa models performing best. Both perform well on SNLI-Hard and achieve accuracy levels in the high 80s on the `antonym' (AT), `numerical reasoning' (NR), `length' (LN), `spelling error'(SE) sub-datasets, and show marked improvement on both `negation' (NG), and `word overlap' (WO). Training a RoBERTa model also on ANLI appears to be particularly useful for the NR, WO, NG and AT NLI stress tests."]},{"idx":23,"heading":"Analysis ::: Reasoning types","paragraphs":["A dynamically evolving dataset offers the unique opportunity to track how model error rates change over time. Since each round's development set contains only verified examples, we can investigate two interesting questions: which types of inference do writers employ to fool the models, and are base models differentially sensitive to different types of reasoning? The results are summarized in Table TABREF27.","We employed an expert linguist annotator to devise an ontology of inference types that would be specific to NLI. While designing an appropriate ontology of types of inference is far from straightforward, we found that a unified ontology could be utilized to characterize examples from all three rounds, which suggests that it has at least some generalizeable applicability. The ontology was used to label 500 examples from each ANLI development set.","The inference ontology contains six types of inference: Numerical & Quantitative (i.e., reasoning about cardinal and ordinal numbers, inferring dates and ages from numbers, etc.), Reference & Names (coreferences between pronouns and forms of proper names, knowing facts about name gender, etc.), Basic Inferences (conjunctions, negations, cause-and-effect, comparatives and superlatives etc.), Lexical Inference (inferences made possible by lexical information about synonyms, antonyms, etc.), Tricky Inferences (wordplay, linguistic strategies such as syntactic transformations/reorderings, or inferring writer intentions from contexts), and reasoning from outside knowledge or additional facts (e.g., “You can't reach the sea directly from Djibouti”). The quality of annotations was also tracked; if a pair was ambiguous or had a label that seemed incorrect (from the expert annotator's perspective), it was flagged. Round 1–3 development sets contained few `Quality' tags; the incidence of quality issues was stable at between 3% and 4% per round. Any one example can have multiple types, and every example contained at least one tag.","As rounds 1 and 2 were both built with contexts from the same genre (Wikipedia), we might expect writers to arrive at similar strategies. However, since the model architectures used in the first two rounds differ, writers might be sufficiently creative in finding different exploits in each. For round 3, we expect some difference in reasoning types to be present, because we used source material from several domains as our contexts. In sum, any change between rounds could be due to any of the following factors: inherent differences between data collection, model architectures and model training data, random selection of contexts, or slight differences in writer pool or writer preferences.","We observe that both round 1 and 2 writers rely heavily on numerical and quantitative reasoning in over 30% of the development set—the percentage in A2 (32%) dropped roughly 6% from A1 (38%)—while round 3 writers use numerical or quantitative reasoning for only 17%. The majority of numerical reasoning types were references to cardinal numbers that referred to dates and ages. Inferences predicated on references and names were present in about 10% of rounds 1 & 3 development sets, and reached a high of 20% in round 2, with coreference featuring prominently. Basic inference types increased in prevalence as the rounds increased, ranging from 18%–30%, as did Lexical inferences (increasing from 13%–33%). The percentage of sentences relying on reasoning and outside facts remains roughly the same, in the mid-50s, perhaps slightly increasing after round 1. For round 3, we observe that the model used to collect it appears to be more susceptible to Basic, Lexical, and Tricky inference types. This finding is compatible with the idea that the models trained on adversarial data are more impressive and perform better, encouraging writers to devise more creative examples containing harder types of inference in order to stump them."]},{"idx":24,"heading":"Related work ::: Bias in datasets","paragraphs":["Machine learning methods are well-known to pick up on spurious statistical patterns. For instance, in image captioning, a simple baseline of utilizing the captions of nearest neighbors in the training set was shown to yield impressive BLEU scores BIBREF35. In the first visual question answering dataset BIBREF36, biases like “2” being the correct answer to 39% of the questions starting with “how many” allowed learning algorithms to perform well while ignoring the visual modality altogether BIBREF37, BIBREF38. The field has a tendency to overfit on static targets, even if that does not happen deliberately BIBREF39.","In NLI, Gururangan2018annotation, Poliak2018hypothesis and Tsuchiya2018performance showed that hypothesis-only baselines often perform far better than chance. It has been shown that NLI systems can often be broken merely by performing simple lexical substitutions BIBREF15, and that they struggle with quantifiers BIBREF40 and certain superficial syntactic properties BIBREF17. In reading comprehension and question answering, Kaushik2018howmuch showed that question- and passage-only models can perform surprisingly well, while Jia2017adversarial added adversarially constructed sentences to passages, leading to a drastic drop in performance. Many text classification datasets do not require sophisticated linguistic reasoning, as shown by the surprisingly good performance of random encoders BIBREF41. Similar observations were made in machine translation BIBREF42 and dialogue BIBREF43. In short, the field is rife with dataset bias and papers trying address this important problem. This work can be viewed as a natural extension: if such biases exist, they will allow humans to fool the models, adding useful examples to the training data until the bias is dynamically mitigated."]},{"idx":25,"heading":"Related work ::: Dynamic datasets.","paragraphs":["Concurrently with this work, Anonymous2020adversarialfilters proposed AFLite, an iterative approach for filtering adversarial data points to avoid spurious biases. Kaushik2019learningdifference offer a causal account of spurious patterns, and counterfactually augment NLI datasets by editing examples to break the model. The former is an example of a model-in-the-loop setting, where the model is iteratively probed and improved. The latter is human-in-the-loop training, where humans are used to find problems with one single model. In this work, we employ both strategies iteratively, in a form of human-and-model-in-the-loop training, to collect completely new examples, in a potentially never-ending loop BIBREF18. Relatedly, lan2017ppdb propose a method for continuously growing a dataset of paraphrases.","Human-and-model-in-the-loop training is not a new idea. Mechanical Turker Descent proposes a gamified environment for the collaborative training of grounded language learning agents over multiple rounds BIBREF19. The “Build it Break it Fix it” strategy in the security domain BIBREF44 has been adapted to NLP BIBREF20 as well as dialogue BIBREF45. The QApedia framework BIBREF46 continuously refines and updates its content repository using humans in the loop, while human feedback loops have been used to improve image captioning systems BIBREF47. wallace2018trick leverage trivia experts to create a model-driven adversarial question writing procedure and generate a small set of challenge questions that QA-models fail on.","There has been a flurry of work in constructing datasets with an adversarial component, such as Swag BIBREF48 and HellaSwag BIBREF49, CODAH BIBREF50, Adversarial SQuAD BIBREF51, Lambada BIBREF52 and others. Our dataset is not to be confused with abductive NLI BIBREF53, which calls itself $\\alpha $NLI, or ART."]},{"idx":26,"heading":"Discussion & Conclusion","paragraphs":["In this work, we used a human-and-model-in-the-loop entailment training method to collect a new benchmark for natural language understanding. The benchmark is designed to be challenging to current state of the art models. Annotators were employed to act as adversaries, and encouraged to find vulnerabilities that fool the model into predicting the wrong label, but that another person would correctly classify. We found that non-expert annotators, in this gamified setting and with appropriate incentives to fool the model, are remarkably creative at finding and exploiting weaknesses in models. We collected three rounds, and as the rounds progressed, the models became more robust and the test sets for each round became more difficult. Training on this new data yielded the state of the art on existing NLI benchmarks.","The ANLI benchmark presents a new challenge to the community. It was carefully constructed to mitigate issues with previous datasets, and was designed from first principles to last longer—if the test set saturates, the field can simply train up a new model, collect more data and find itself confronted yet again with a difficult challenge.","The dataset also presents many opportunities for further study. For instance, we collected annotator-provided explanations for each example that the model got wrong. We provided inference labels for the development set, opening up possibilities for interesting more fine-grained studies of NLI model performance. While we verified the development and test examples, we did not verify the correctness of each training example, which means there is probably some room for improvement there.","The benchmark is meant to be a challenge for measuring NLU progress, even for as yet undiscovered models and architectures. We plan for the benchmark itself to adapt to these new models by continuing to build new challenge rounds. As a first next step, it would be interesting to examine results when annotators are confronted with a wide variety of model architectures. We hope that the dataset will prove to be an interesting new challenge for the community. Luckily, if it does turn out to saturate quickly, we will always be able to collect a new round."]},{"idx":27,"heading":"Acknowledgments","paragraphs":["YN and MB were sponsored by DARPA MCS Grant #N66001-19-2-4031, ONR Grant #N00014-18-1-2871, and DARPA YFA17-D17AP00022."]},{"idx":28,"heading":"Figures and Tables","paragraphs":["Figure 1: Adversarial NLI data collection procedure, via human-and-model-in-the-loop entailment training (HAMLET). The four steps make up one round of data collection.","Table 1: Examples from development set. ‘An’ refers to round number, ‘orig.’ is the original annotator’s gold label, ‘pred.’ is the model prediction, ‘valid.’ is the validator labels, ‘reason’ was provided by the original annotator, ‘Annotations’ is the tags determined by linguist expert annotator.","Table 2: Dataset statistics: ‘Model error rate’ is the percentage of examples that model got wrong; ‘unverified’ is the simple percentage, while ‘verified’ is the percentage that were additionally verified by 2 human annotators.","Table 3: Model Performance. ‘Data’ refers to training dataset (‘S’ refers to SNLI, ‘M’ to MNLI dev (-m=matched, -mm=mismatched), and ‘F’ to FEVER); ‘A1–A3’ refer to the rounds respectively. ‘-E’ refers to test set examples written by annotators exclusive to the test set. Datasets marked ‘?n’ were used to train the base model for round n, and their performance on that round is underlined.","Table 4: Analysis of hypothesis-only performance for the different rounds. Hypothesis-only models are marked H . The rows subscripted with X are XLNet models, all other rows are RoBERTa. S=SNLI, M=MNLI. ALL=S,M,F,ANLI.","Table 5: Model Performance on NLI stress tests (tuned on their respective dev. sets). All=S+M+F+ANLI. AT=‘Antonym’; ‘NR’=Numerical Reasoning; ‘LN’=Length; ‘NG’=Negation; ‘WO’=Word Overlap; ‘SE’=Spell Error. Previous models refers to the Naik et al. (2018) implementation of Conneau and Kiela (2018, InferSent) for the Stress Tests, and to the Gururangan et al. (2018) implementation of Gong et al. (2018, DIIN) for SNLI-Hard.","Table 6: Analysis of 500 development set examples per round. ‘Average’ lists the average percentage of each top level category in ANLI."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1911.01940","title":"Deepening Hidden Representations from Pre-trained Language Models for Natural Language Understanding","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Transformer-based pre-trained language models have proven to be effective for learning contextualized language representation. However, current approaches only take advantage of the output of the encoder's final layer when fine-tuning the downstream tasks. We argue that only taking single layer's output restricts the power of pre-trained representation. Thus we deepen the representation learned by the model by fusing the hidden representation in terms of an explicit HIdden Representation Extractor (HIRE), which automatically absorbs the complementary representation with respect to the output from the final layer. Utilizing RoBERTa as the backbone encoder, our proposed improvement over the pre-trained models is shown effective on multiple natural language understanding tasks and help our model rival with the state-of-the-art models on the GLUE benchmark."]},{"idx":1,"heading":"Introduction","paragraphs":["Language representation is essential to the understanding of text. Recently, pre-training language models based on Transformer BIBREF0 such as GPT BIBREF1, BERT BIBREF2, XLNet BIBREF3, and RoBERTa BIBREF4 have been shown to be effective for learning contextualized language representation. These models have since continued to achieve new state-of-the-art results on a variety of natural processing tasks. They include question answering BIBREF5, BIBREF6, natural language inference BIBREF7, BIBREF8, named entity recognition BIBREF9, sentiment analysis BIBREF10 and semantic textual similarity BIBREF11, BIBREF12.","Normally, Transformer-based models are pre-trained on large-scale unlabed corpus in a unspervised manner, and then fine-turned on the downstream tasks through introducing task-specific output layer. When fine-tuning on the supervised downstream tasks, the models pass directly the output of Transformer encoder's final layer, which is consided as the contextualized representation of input text, to the task-specific layer.","However, due to the numerous layers (i.e., Transformer blocks) and considerable depth of these pre-training models, we argue that the output of the last layer may not always be the best representation of the input text during the fine-tuning for downstream task. BIBREF2 shows diverse combinations of different layers' outputs of the pre-trained BERT result in distinct performance on CoNNL-2003 Named Entity Recognition (NER) task BIBREF9. BIBREF13 points out for pre-trained language models, including Transformer, the most transferable contextualized representations of input text tend to occur in the middle layers, while the top layers specialize for language modeling. Therefore, the onefold use of last layer's output may restrict the power of the pre-trained representation.","In this paper, we introduce RTRHI: Refined Transformer Representation with Hidden Information based on the fine-tuning approach with Transformer-based model, which leverages the hidden information in the Transformer's hiddens layer to refine the language representation. Our approch consists of two main additional components:","HIdden Representation Extractor (HIRE) dynamically learns a complementary representation which contains the information that the final layer's output fails to capture. We put 2-layer bidirectional GRU beside the encoder to summarize the output of each layer into a single vector which will be used to compute the contribution score.","Fusion layer integrates the hidden information extracted by the HIRE with Transformer final layer's output through two steps of different functionalities, leading to a refined contextualized language representation.","Taking advantage of the robustness of RoBERTa by using it as the Transformer-based encoder of RTRHI, we conduct experiments on GLUE benchmark BIBREF14, which consists of nine Natural Language Understanding (NLU) tasks. RTRHI outperforms our baseline model RoBERTa on 5/9 of them and advances the state-of-the-art on SST-2 dataset. Even though we don't make any modification to the encoder's internal architecture or redefine the pre-training procedure with different objectives or datasets, we still get the comparable performance with other state-of-the-art models on the GLUE leaderboard. These results highlight RTRHI's excellent ability to refine Transformed-based model's language representation."]},{"idx":2,"heading":"Related Work","paragraphs":["Transformer-based language models take Transformer BIBREF0 as their model architecture, but pre-trained with different objectives or language corpus. OpenAI GPT BIBREF1 is the first model which introduced Transformer architecture into unsupervised pre-training. The model is pre-trained on 12-layer left-to-right Transformer with BooksCorpus BIBREF15 dataset. But instead of using a left-to-right architecture like GPT, BERT BIBREF2 adopts Masked LM objective when pre-training, which enables the representation to incorporate context from both direction. The next sentence prediction (NSP) objective is also used by BERT to better understand the relationship between two sentences. The training procedure is conducted on a combination of BooksCorpus plus English Wikipedia. XLNet BIBREF3, as a generalized autoregressive language model, uses a permutation language modeling objective during pre-training on the other hand. In addition to BooksCorpus and English Wikipedia, it also uses Giga5, ClueWeb 2012-B and Common Crawl for pre-training. Trained with dynamic masking, large mini-batches and a larger byte-level BPE, full-sentences without NSP, RoBERTa BIBREF4 improves BERT's performance on the downstream tasks. The pre-training corpora includes BooksCorpus, CC-News, Openwebtext and Stories. By fine-tuning on the downstream tasks in a supervised manner, these powerful Transformer-based models all push the state-of-the-art results on the various NLP tasks to a new level.","Recent works have proposed new methods for fine-tuning the downstream tasks, including multi-task learning BIBREF16, adversarial training BIBREF17 or incorporating semantic information into language representation BIBREF18."]},{"idx":3,"heading":"Model and Method ::: Transformer-based encoder layer","paragraphs":["Transformer-based encoder layer is responsible for encoding input text into a sequence of high-dimensional vectors, which is consided as the contextualized representation of the input sequence. Let {$w_1,\\dots , w_n$} represent a sequence of $n$ words of input text, we use a Transformer-based encoder to encode the input sequence, thereby to obtain its universal contextualized representation $R \\in \\mathbb {R}^{n\\times d}$:","where $d$ is the hidden size of the encoder. It should be noted that $R$ is the output of Transformer-based encoder's last layer which has the same length as the input text. We call it preliminary representation in this paper to distinguish it with the one that we introduce in section SECREF6. Here, we omit a rather extensive formulations of Transformer and refer readers to BIBREF1, BIBREF2 and BIBREF3 for more details."]},{"idx":4,"heading":"Model and Method ::: Hidden Representation Extractor","paragraphs":["Since Transformer-based encoder normally has many identical layers stacked together, for example, $\\text{BERT}_{\\text{LARGE}}$ and $\\text{XLNet}_{\\text{LARGE}}$ all contain 24 layers of the identical structure, the output of the final layer may not be the most perfect candidate to fully represent the information contained in the input text.","Trying to solve this problem, we introduce an HIdden Representation Extractor (HIRE) beside the encoder to draw from the hidden states the information that the output of the last layer fails to capture. Since each layer's hidden states don't carry the information of same importance to represent a certain input sequence, we adopt a mechanism which can compute the importance dynamically. We name the importance as contribution score.","The input to the HIRE is $\\lbrace H_0, \\dots , H_j, \\dots , H_l\\rbrace $ where $ 0 < j \\le l$ and $l$ represents the number of layers in the encoder. Here $H_0$ is the initial embedding of input text, which is the input of the encoder's first layer but is updated during training and $H_j \\in \\mathbb {R}^{n\\times d}$ is the hidden-state of the encoder at the output of layer $j$. For the sake of simplicity, we call them all hidden-state afterwards.","For each hidden-state of encoder, we use the same 2-layer Bidirectional Gated Recurrent Unit (GRU) BIBREF19 to summarize it. Instead of taking the whole output of GRU as the representation of the hidden state, we concatenate GRU's each layer and each direction's final state together. In this way, we manage to summarize the hidden-state into a fixed-sized vector. Hence, we obtain $U \\in \\mathbb {R}^{(l+1) \\times 4d}$ with $U_i$ the summarized vector of $H_i$:","where $0 \\le i \\le l$. Then the importance value $\\alpha _i$ for hidden-state $H_i$ is calculated by:","where $W^T \\in \\mathbb {R}^{4d} $ and $b\\in \\mathbb {R}$ are trainable parameters. Let $S$ represent the computation scores for all hidden-states. $S$ is computed as follows:","It should be noted that $\\sum _{i=0}^{l}S_i = 1$ where $S_i$ is the weight of hidden-state $i$ when computing the representation. Subsequently, we obtain the input sequence's new representation $A$ by:","With the same shape as the output of Transformer-based encoder's final layer, HIRE's output $A$ is expected to contain the additional useful information from the encoder's hidden-states which is helpful for a better understanding of the input text and we call it complementary representation."]},{"idx":5,"heading":"Model and Method ::: Fusion Layer","paragraphs":["This layer fuses the information contained in the output of Tansformed-based encoder and the one extracted from encoders' hidden states by HIRE.","Given the preliminary representation $R$, instead of letting it flow directly into task-specfic output layer, we combine it together with the complementary representation $A$ to yeild $M$, which we define by:","where $\\circ $ is elementwise multiplication (Hadamard Product) and $[;]$ is concatenation across the last dimension.","Later, two-layer bidirectional GRU, with the output size of $d$ for each direction, is used to fully fuse the information contained in the preliminary representation and the additional useful information included in the complementary representation. We concatenate the outputs of the GPUs in two dimensions together, and we hence obtain the final contextualized representation $F$ of input text:","The use of GRUs enables the complete interaction between the two different kinds of information mentioned before. Therefore, $F$ is expected to be a refined universal representation of input text."]},{"idx":6,"heading":"Model and Method ::: Output layer","paragraphs":["The output layer is task-specific, which means we can adopt HIRE and fusion layer to other downstream tasks by only changing the output layer, such as question answering.","GLUE benchmark contains two types of tasks: 1. classification; 2. regression. For classification tasks, given the input text's contextualized representation $F$, following BIBREF2, we take the first row $C \\in \\mathbb {R}^{2d}$ of $F$ corresponding to the first input token ([CLS]) as the aggregate representation. Let $m$ be the number of labels in the datasets, we pass $C$ through a feed-forward network(FFN):","with $W_1 \\in \\mathbb {R}^{2d\\times d}$, $W_2 \\in \\mathbb {R}^{d\\times m}$, $b_1\\in \\mathbb {R}^{d}$ and $b_2\\in \\mathbb {R}^{m}$ the only parameters that we introduce in output layer. Finally, the probability distribution of predicted label is computed as:","For regression task, we obtain $Q$ in the same manner with $m = 1$, and take $Q$ as the predicted value."]},{"idx":7,"heading":"Model and Method ::: Training","paragraphs":["For classification tasks, the training loss to be minimized is defined by the Cross-Entropy:","where $\\theta $ is the set of all parameters in the model, $N$ is the number of examples in the dataset, $p_{i,c}$ is the predicted probability of class $c$ for example $i$ and $y$ is the binary indicator defined as below:","For regression tasks, we define the training loss by mean squared error (MSE):","where $Q_i$ is the predicted value for example $i$ and $y_i$ is the ground truth value for example $i$ and $N$, $\\theta $ are same as the ones in equation DISPLAY_FORM19."]},{"idx":8,"heading":"Experiments ::: Dataset","paragraphs":["We conducted the experiments on the General Language Understanding Evaluation (GLUE) benchmark BIBREF14 to evaluate our method's performance. GLUE is a collection of 9 diverse datasets for training, evaluating, and analyzing natural language understanding models. Three different tasks are presented in GLUE benchmark according to the original paper:","Single-sentence tasks: The Corpus of Linguistic Acceptability (CoLA) BIBREF20 requires the model to determine whether a sentence is grammatically acceptable; the Stanford Sentiment Treebank (SST-2) BIBREF10 is to predict the sentiment of movie reviews with label of positive or negative.","Similarity and paraphrase tasks: Similarity and paraphrase tasks are to predict whether each pair of sentences captures a paraphrase/semantic equivalence relationship. The Microsoft Research Paraphrase Corpus (MRPC) BIBREF12, the Quora Question Pairs (QQP) BIBREF21 and the Semantic Textual Similarity Benchmark (STS-B) BIBREF11 are presented in this category.","Natural Language Inference (NLI) tasks: Natural language inference is the task of determining whether a “hypothesis” is true (entailment), false (contradiction), or undetermined (neutral) given a “premise”. GLUE benchmark contains the following tasks: the Multi-Genre Natural Language Inference Corpus (MNLI) BIBREF7, the Stanford Question Answering Dataset (QNLI) BIBREF22, the Recognizing Textual Entailment (RTE) BIBREF23 and the Winograd Schema Challenge (WNLI) BIBREF24.","Four official metrics are adopted to evaluate the model performance: Matthews correlation BIBREF25, accuracy, F1 score, Pearson and Spearman correlation coefficients. More details will be presented in section SECREF25."]},{"idx":9,"heading":"Experiments ::: Implementation","paragraphs":["Our implementation of RTRHI is based on the PyTorch implementation of Transformer .","Preprocessing: Following BIBREF4, we adopt GPT-2 BIBREF27 tokenizer with a Byte-Pair Encoding (BPE) vocabulary of subword units size 50K. We format the input sequence in the following way. Given single sequence $X$, we add <s> token at the begining and </s> token at the end: <s>X</s>. For a pair of sequences $(X, Y)$, we additionally use </s> to separate these two sequences: <s>X</s>Y</s>.","Model configurations: We use RoBERTa-Large as the Transformer-based encoder and load the pre-training weights of RoBERTa BIBREF4. Like BERT-Large, RoBERTa-Large model contains 24 Transformer-blocks, with the hidden size being 1024 and the number of self-attention heads being 16 BIBREF4, BIBREF2.","Optimization: We use Adam optimizer BIBREF28 with $\\beta _1=0.9$, $\\beta _2=0.98$ and $\\epsilon = 10^{-6}$ and the learning rate is selected amongst {5e-6, 1e-5, 2e-5, 3e-5} with a warmup rate ranging from 0.06 to 0.25 depending on the nature of the task. The number of training epochs ranges from 4 to 10 with the early stop and the batch size is selected amongst {16, 32, 48}. In addition to that, we clip the gradient norm within 1 to prevent exploding gradients problem occuring in the recurrent neural networks in our model.","Regularization: We employ two types of regularization methods during training. We apply dropout BIBREF29 of rate 0.1 to all layers in the Transformer-based encoder and GRUs in the HIRE and fusion layer. We additionally adopt L2 weight decay of 0.1 during training."]},{"idx":10,"heading":"Experiments ::: Main results","paragraphs":["Table TABREF22 compares our method RTRHI with a list of Transformer-based models on the development set. To obtain a direct and fair comparison with our baseline model RoBERTa, following the original paper BIBREF4, we fine-tune RTRHI separately for each of the GLUE tasks, using only task-specific training data. The single-model results for each task are reported. We run our model with five different random seeds but the same hyperparameters and take the median value. Due to the problematic nature of WNLI dataset, we exclude its results in this table. The results shows that RTRHI consistently outperforms RoBERTa on 4 of the GLUE task development sets, with an improvement of 1.6 points, 0.4 pionts, 0.4/0.2 points, 0.3 points on CoLA, SST-2, MNLI and QNLI respectively. And on the QQP and RTE task, our model get the same result as RoBERTa. It should be noted that the improvement is entirely attributed to the introduction of HIdden Representation Extractor and Fusion Layer in our model.","Table TABREF26 presents the results of RTRHI and other models on the test set that have been submitted to the GLUE leaderboard. Following BIBREF4, we fine-tune STS-B and MRPC starting from the MNLI single-task model. Given the simplicity between RTE, WNLI and MNLI, and the large-scale nature of MNLI dataset (393k), we also initialize RTRHI with the weights of MNLI single-task model before fine-tuning on RTE and WNLI. We submitted the ensemble-model results to the leaderboard. The results show that RTRHI still boosts the strong RoBERTa baseline model on the test set. To be specific, RTRHI outperforms RoBERTa over CoLA, SST-2, MRPC, SST-B, MNLI-mm with an improvement of 0.8 points, 0.4 points, 0.7/0.9 points, 0.2/0.1 points and 0.2 points respectively. In the meantime, RTRHI gets the same results as RoBERTa on QQP and WNLI. By category, RTRHI has better performance than RoBERTa on the single sentence tasks, similarity and paraphrase tasks. It's worth noting that our model obtains state-of-art results on SST-2 dataset, with a score of 97.1. The results is quite promising since HIRE does not make any modification with the encoder internal architecture BIBREF3 or redefine the pre-training procedure BIBREF4 and we still get the comparable results with them."]},{"idx":11,"heading":"Analysis","paragraphs":["We compare the contribution score's distribution of different NLU tasks. For each task, we run our best single model over the development set and the results are calculated by averaging the values across all the examples within each dataset. The results are showed in Figure FIGREF27. From the top to the bottom of the headmap, the results are placed in the following order: single-sentence tasks, similarity and paraphrase tasks and natural language inference tasks. From figure FIGREF27, we find that the distribution differs among the different tasks, which demonstrates RTRHI's dynamic ability to adapt for distinct task when computing the complementary representation. The most important contribution occurs below the final layer for all the tasks except MRPC and RTE. All layers have a close contribution for MRPC and RTE task.","Figure FIGREF28 presents the distribution of contribution scores over different layers for each example of SST-2 dataset. The number on the ordinate axis denotes the index of the example. We observe that even though there are subtle differences among these example, they follow certain same patterns when calculating the complementary representation, for example, layer 21 and 22 contribute the most for almost all the examples and also the layers around them. But the figure shows also that for some examples, all layers contribute almost equally."]},{"idx":12,"heading":"Conclusion","paragraphs":["In this paper, we have introduced RTRHI, a novel approach that refines language representation by leveraging the Transformer-based model's hidden layers. Specifically, an HIdden Representation Extractor is used to dynamically generate complementary imformation which will be incorporated with preliminary representation in the Fusion Layer. The experimental results demonstrate the effectiveness of refined language representation for natural language understanding. The analysis highlights the distinct contribution of each layer's output for diverse task and different example. We expect future work could be conducted in the following domains: (1) explore sparse version of Hidden Representation Extractor for more effective computation and less memory usage; (2) incorporating extra knowledge information BIBREF30 or structured semantic information BIBREF18 with current language representation in the fusion layer during fine-tuning; (3) integrate multi-tasks training BIBREF31 or knowledge distillation BIBREF32, BIBREF33 into our model."]},{"idx":13,"heading":"Figures and Tables","paragraphs":["Figure 1: Architecture of RTRHI.","Figure 2: Architecture of the Hidden Representation Extractor. The GRUs share the same parameters.","Table 1: GLUE Dev results. RTRHI results are based on single model trained with single task and a median over five runs with different random seed but the same hyperparameter is reported for each task. The results of MTDNN, XLNETLARGE, ALBERT(1.5M) and RoBERTa are from Liu et al. (2019a), Yang et al. (2019), Lan et al. (2019) and Liu et al. (2019b). See the lower-most row for the performance of our approach.","Figure 3: Distribution of contribution scores over different layers when computing the complementary representation for various NLU tasks. The contribution scores are normalized by SoftMax to sum up to 1 for each row. The numbers on the abscissa axis indicate the corresponding layer with 0 being the first layer and 23 being the last layer.","Table 2: GLUE Test results, scored by the official evaluation server. All the results are obtained from GLUE leaderboard (https://gluebenchmark.com/leaderboard) at the time of submitting RTRHI (3 November, 2019). The number below each task’s name indicates the size of training dataset. The state-of-the-art results are in bold. RTRHI takes RoBERTa as its Transformer-based encoder. Mcc, acc and pearson denote Matthews correlation, accuracy and Person correlation coefficient respectively.","Figure 4: Distribution of contribution scores over different layers for each example of SST-2 dataset. The number on the ordinate axis denotes the index of the example."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1911.02855","title":"Dice Loss for Data-imbalanced NLP Tasks","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Many NLP tasks such as tagging and machine reading comprehension are faced with the severe data imbalance issue: negative examples significantly outnumber positive examples, and the huge number of background examples (or easy-negative examples) overwhelms the training. The most commonly used cross entropy (CE) criteria is actually an accuracy-oriented objective, and thus creates a discrepancy between training and test: at training time, each training instance contributes equally to the objective function, while at test time F1 score concerns more about positive examples. In this paper, we propose to use dice loss in replacement of the standard cross-entropy objective for data-imbalanced NLP tasks. Dice loss is based on the Sorensen-Dice coefficient or Tversky index, which attaches similar importance to false positives and false negatives, and is more immune to the data-imbalance issue. To further alleviate the dominating influence from easy-negative examples in training, we propose to associate training examples with dynamically adjusted weights to deemphasize easy-negative examples.Theoretical analysis shows that this strategy narrows down the gap between the F1 score in evaluation and the dice loss in training. With the proposed training objective, we observe significant performance boost on a wide range of data imbalanced NLP tasks. Notably, we are able to achieve SOTA results on CTB5, CTB6 and UD1.4 for the part of speech tagging task; SOTA results on CoNLL03, OntoNotes5.0, MSRA and OntoNotes4.0 for the named entity recognition task; along with competitive results on the tasks of machine reading comprehension and paraphrase identification."]},{"idx":1,"heading":"Introduction","paragraphs":["Data imbalance is a common issue in a variety of NLP tasks such as tagging and machine reading comprehension. Table TABREF3 gives concrete examples: for the Named Entity Recognition (NER) task BIBREF2, BIBREF3, most tokens are backgrounds with tagging class $O$. Specifically, the number of tokens tagging class $O$ is 5 times as many as those with entity labels for the CoNLL03 dataset and 8 times for the OntoNotes5.0 dataset; Data-imbalanced issue is more severe for MRC tasks BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8 with the value of negative-positive ratio being 50-200.","Data imbalance results in the following two issues: (1) the training-test discrepancy: Without balancing the labels, the learning process tends to converge to a point that strongly biases towards class with the majority label. This actually creates a discrepancy between training and test: at training time, each training instance contributes equally to the objective function while at test time, F1 score concerns more about positive examples; (2) the overwhelming effect of easy-negative examples. As pointed out by meng2019dsreg, significantly large number of negative examples also means that the number of easy-negative example is large. The huge number of easy examples tends to overwhelm the training, making the model not sufficiently learned to distinguish between positive examples and hard-negative examples. The cross-entropy objective (CE for short) or maximum likelihood (MLE) objective, which is widely adopted as the training objective for data-imbalanced NLP tasks BIBREF9, BIBREF10, BIBREF11, BIBREF12, BIBREF13, BIBREF14, BIBREF15, handles neither of the issues.","To handle the first issue, we propose to replace CE or MLE with losses based on the Sørensen–Dice coefficient BIBREF0 or Tversky index BIBREF1. The Sørensen–Dice coefficient, dice loss for short, is the harmonic mean of precision and recall. It attaches equal importance to false positives (FPs) and false negatives (FNs) and is thus more immune to data-imbalanced datasets. Tversky index extends dice loss by using a weight that trades precision and recall, which can be thought as the approximation of the $F_{\\beta }$ score, and thus comes with more flexibility. Therefore, We use dice loss or Tversky index to replace CE loss to address the first issue.","Only using dice loss or Tversky index is not enough since they are unable to address the dominating influence of easy-negative examples. This is intrinsically because dice loss is actually a hard version of the F1 score. Taking the binary classification task as an example, at test time, an example will be classified as negative as long as its probability is smaller than 0.5, but training will push the value to 0 as much as possible. This gap isn't a big issue for balanced datasets, but is extremely detrimental if a big proportion of training examples are easy-negative ones: easy-negative examples can easily dominate training since their probabilities can be pushed to 0 fairly easily. Meanwhile, the model can hardly distinguish between hard-negative examples and positive ones. Inspired by the idea of focal loss BIBREF16 in computer vision, we propose a dynamic weight adjusting strategy, which associates each training example with a weight in proportion to $(1-p)$, and this weight dynamically changes as training proceeds. This strategy helps to deemphasize confident examples during training as their $p$ approaches the value of 1, makes the model attentive to hard-negative examples, and thus alleviates the dominating effect of easy-negative examples.","Combing both strategies, we observe significant performance boosts on a wide range of data imbalanced NLP tasks. Notably, we are able to achieve SOTA results on CTB5 (97.92, +1.86), CTB6 (96.57, +1.80) and UD1.4 (96.98, +2.19) for the POS task; SOTA results on CoNLL03 (93.33, +0.29), OntoNotes5.0 (92.07, +0.96)), MSRA 96.72(+0.97) and OntoNotes4.0 (84.47,+2.36) for the NER task; along with competitive results on the tasks of machine reading comprehension and paraphrase identification.","The rest of this paper is organized as follows: related work is presented in Section 2. We describe different training objectives in Section 3. Experimental results are presented in Section 4. We perform ablation studies in Section 5, followed by a brief conclusion in Section 6."]},{"idx":2,"heading":"Related Work ::: Data Resample","paragraphs":["The idea of weighting training examples has a long history. Importance sampling BIBREF17 assigns weights to different samples and changes the data distribution. Boosting algorithms such as AdaBoost BIBREF18 select harder examples to train subsequent classifiers. Similarly, hard example mining BIBREF19 downsamples the majority class and exploits the most difficult examples. Oversampling BIBREF20, BIBREF21 is used to balance the data distribution. Another line of data resampling is to dynamically control the weights of examples as training proceeds. For example, focal loss BIBREF16 used a soft weighting scheme that emphasizes harder examples during training. In self-paced learning BIBREF22, example weights are obtained through optimizing the weighted training loss which encourages learning easier examples first. At each training step, self-paced learning algorithm optimizes model parameters and example weights jointly. Other works BIBREF23, BIBREF24 adjusted the weights of different training examples based on training loss. Besides, recent work BIBREF25, BIBREF26 proposed to learn a separate network to predict sample weights."]},{"idx":3,"heading":"Related Work ::: Data Imbalance Issue in Object Detection","paragraphs":["The background-object label imbalance issue is severe and thus well studied in the field of object detection BIBREF27, BIBREF28, BIBREF29, BIBREF30, BIBREF31. The idea of hard negative mining (HNM) BIBREF30 has gained much attention recently. shrivastava2016ohem proposed the online hard example mining (OHEM) algorithm in an iterative manner that makes training progressively more difficult, and pushes the model to learn better. ssd2016liu sorted all of the negative samples based on the confidence loss and picking the training examples with the negative-positive ratio at 3:1. pang2019rcnn proposed a novel method called IoU-balanced sampling and aploss2019chen designed a ranking model to replace the conventional classification task with a average-precision loss to alleviate the class imbalance issue. The efforts made on object detection have greatly inspired us to solve the data imbalance issue in NLP."]},{"idx":4,"heading":"Losses ::: Notation","paragraphs":["For illustration purposes, we use the binary classification task to demonstrate how different losses work. The mechanism can be easily extended to multi-class classification.","Let $\\lbrace x_i\\rbrace $ denote a set of instances. Each $x_i$ is associated with a golden label vector $y_i = [y_{i0},y_{i1} ]$, where $y_{i1}\\in \\lbrace 0,1\\rbrace $ and $y_{i0}\\in \\lbrace 0,1\\rbrace $ respectively denote the positive and negative classes, and thus $y_i$ can be either $[0,1]$ or $[0,1]$. Let $p_i = [p_{i0},p_{i1} ]$ denote the probability vector, and $p_{i1}$ and $p_{i0}$ respectively denote the probability that a model assigns the positive and negative label to $x_i$."]},{"idx":5,"heading":"Losses ::: Cross Entropy Loss","paragraphs":["The vanilla cross entropy (CE) loss is given by:","As can be seen from Eq.DISPLAY_FORM8, each $x_i$ contributes equally to the final objective. Two strategies are normally used to address the the case where we wish that not all $x_i$ are treated equal: associating different classes with different weighting factor $\\alpha $ or resampling the datasets. For the former, Eq.DISPLAY_FORM8 is adjusted as follows:","where $\\alpha _i\\in [0,1]$ may be set by the inverse class frequency or treated as a hyperparameter to set by cross validation. In this work, we use $\\lg (\\frac{n-n_t}{n_t}+K)$ to calculate the coefficient $\\alpha $, where $n_t$ is the number of samples with class $t$ and $n$ is the total number of samples in the training set. $K$ is a hyperparameter to tune. The data resampling strategy constructs a new dataset by sampling training examples from the original dataset based on human-designed criteria, e.g., extract equal training samples from each class. Both strategies are equivalent to changing the data distribution and thus are of the same nature. Empirically, these two methods are not widely used due to the trickiness of selecting $\\alpha $ especially for multi-class classification tasks and that inappropriate selection can easily bias towards rare classes BIBREF32."]},{"idx":6,"heading":"Losses ::: Dice coefficient and Tversky index","paragraphs":["Sørensen–Dice coefficient BIBREF0, BIBREF33, dice coefficient (DSC) for short, is a F1-oriented statistic used to gauge the similarity of two sets. Given two sets $A$ and $B$, the dice coefficient between them is given as follows:","In our case, $A$ is the set that contains of all positive examples predicted by a specific model, and $B$ is the set of all golden positive examples in the dataset. When applied to boolean data with the definition of true positive (TP), false positive (FP), and false negative (FN), it can be then written as follows:","For an individual example $x_i$, its corresponding DSC loss is given as follows:","As can be seen, for a negative example with $y_{i1}=0$, it does not contribute to the objective. For smoothing purposes, it is common to add a $\\gamma $ factor to both the nominator and the denominator, making the form to be as follows:","As can be seen, negative examples, with $y_{i1}$ being 0 and DSC being $\\frac{\\gamma }{ p_{i1}+\\gamma }$, also contribute to the training. Additionally, milletari2016v proposed to change the denominator to the square form for faster convergence, which leads to the following dice loss (DL):","Another version of DL is to directly compute set-level dice coefficient instead of the sum of individual dice coefficient. We choose the latter due to ease of optimization.","Tversky index (TI), which can be thought as the approximation of the $F_{\\beta }$ score, extends dice coefficient to a more general case. Given two sets $A$ and $B$, tversky index is computed as follows:","Tversky index offers the flexibility in controlling the tradeoff between false-negatives and false-positives. It degenerates to DSC if $\\alpha =\\beta =0.5$. The Tversky loss (TL) for the training set $\\lbrace x_i,y_i\\rbrace $ is thus as follows:"]},{"idx":7,"heading":"Losses ::: Self-adusting Dice Loss","paragraphs":["Consider a simple case where the dataset consists of only one example $x_i$, which is classified as positive as long as $p_{i1}$ is larger than 0.5. The computation of $F1$ score is actually as follows:","Comparing Eq.DISPLAY_FORM14 with Eq.DISPLAY_FORM22, we can see that Eq.DISPLAY_FORM14 is actually a soft form of $F1$, using a continuous $p$ rather than the binary $\\mathbb {I}( p_{i1}>0.5)$. This gap isn't a big issue for balanced datasets, but is extremely detrimental if a big proportion of training examples are easy-negative ones: easy-negative examples can easily dominate training since their probabilities can be pushed to 0 fairly easily. Meanwhile, the model can hardly distinguish between hard-negative examples and positive ones, which has a huge negative effect on the final F1 performance.","To address this issue, we propose to multiply the soft probability $p$ with a decaying factor $(1-p)$, changing Eq.DISPLAY_FORM22 to the following form:","One can think $(1-p_{i1})$ as a weight associated with each example, which changes as training proceeds. The intuition of changing $p_{i1}$ to $(1-p_{i1}) p_{i1}$ is to push down the weight of easy examples. For easy examples whose probability are approaching 0 or 1, $(1-p_{i1}) p_{i1}$ makes the model attach significantly less focus to them. Figure FIGREF23 gives gives an explanation from the perspective in derivative: the derivative of $\\frac{(1-p)p}{1+(1-p)p}$ with respect to $p$ approaches 0 immediately after $p$ approaches 0, which means the model attends less to examples once they are correctly classified.","A close look at Eq.DISPLAY_FORM14 reveals that it actually mimics the idea of focal loss (FL for short) BIBREF16 for object detection in vision. Focal loss was proposed for one-stage object detector to handle foreground-background tradeoff encountered during training. It down-weights the loss assigned to well-classified examples by adding a $(1-p)^{\\beta }$ factor, leading the final loss to be $(1-p)^{\\beta }\\log p$.","In Table TABREF18, we show the losses used in our experiments, which is described in the next section."]},{"idx":8,"heading":"Experiments","paragraphs":["We evaluate the proposed method on four NLP tasks: part-of-speech tagging, named entity recognition, machine reading comprehension and paraphrase identification. Baselines in our experiments are optimized by using the standard cross-entropy training objective."]},{"idx":9,"heading":"Experiments ::: Part-of-Speech Tagging","paragraphs":["Part-of-speech tagging (POS) is the task of assigning a label (e.g., noun, verb, adjective) to each word in a given text. In this paper, we choose BERT as the backbone and conduct experiments on three Chinese POS datasets. We report the span-level micro-averaged precision, recall and F1 for evaluation. Hyperparameters are tuned on the corresponding development set of each dataset."]},{"idx":10,"heading":"Experiments ::: Part-of-Speech Tagging ::: Datasets","paragraphs":["We conduct experiments on the widely used Chinese Treebank 5.0, 6.0 as well as UD1.4.","CTB5 is a Chinese dataset for tagging and parsing, which contains 507,222 words, 824,983 characters and 18,782 sentences extracted from newswire sources.","CTB6 is an extension of CTB5, containing 781,351 words, 1,285,149 characters and 28,295 sentences.","UD is the abbreviation of Universal Dependencies, which is a framework for consistent annotation of grammar (parts of speech, morphological features, and syntactic dependencies) across different human languages. In this work, we use UD1.4 for Chinese POS tagging."]},{"idx":11,"heading":"Experiments ::: Part-of-Speech Tagging ::: Baselines","paragraphs":["We use the following baselines:","Joint-POS: shao2017character jointly learns Chinese word segmentation and POS.","Lattice-LSTM: lattice2018zhang constructs a word-character lattice.","Bert-Tagger: devlin2018bert treats part-of-speech as a tagging task."]},{"idx":12,"heading":"Experiments ::: Part-of-Speech Tagging ::: Results","paragraphs":["Table presents the experimental results on the POS task. As can be seen, the proposed DSC loss outperforms the best baseline results by a large margin, i.e., outperforming BERT-tagger by +1.86 in terms of F1 score on CTB5, +1.80 on CTB6 and +2.19 on UD1.4. As far as we are concerned, we are achieving SOTA performances on the three datasets. Weighted cross entropy and focal loss only gain a little performance improvement on CTB5 and CTB6, and the dice loss obtains huge gain on CTB5 but not on CTB6, which indicates the three losses are not consistently robust in resolving the data imbalance issue. The proposed DSC loss performs robustly on all the three datasets."]},{"idx":13,"heading":"Experiments ::: Named Entity Recognition","paragraphs":["Named entity recognition (NER) refers to the task of detecting the span and semantic category of entities from a chunk of text. Our implementation uses the current state-of-the-art BERT-MRC model proposed by xiaoya2019ner as a backbone. For English datasets, we use BERT$_\\text{Large}$ English checkpoints, while for Chinese we use the official Chinese checkpoints. We report span-level micro-averaged precision, recall and F1-score. Hyperparameters are tuned on the development set of each dataset."]},{"idx":14,"heading":"Experiments ::: Named Entity Recognition ::: Datasets","paragraphs":["For the NER task, we consider both Chinese datasets, i.e., OntoNotes4.0 BIBREF34 and MSRA BIBREF35, and English datasets, i.e., CoNLL2003 BIBREF36 and OntoNotes5.0 BIBREF37.","CoNLL2003 is an English dataset with 4 entity types: Location, Organization, Person and Miscellaneous. We followed data processing protocols in BIBREF14.","English OntoNotes5.0 consists of texts from a wide variety of sources and contains 18 entity types. We use the standard train/dev/test split of CoNLL2012 shared task.","Chinese MSRA performs as a Chinese benchmark dataset containing 3 entity types. Data in MSRA is collected from news domain. Since the development set is not provided in the original MSRA dataset, we randomly split the training set into training and development splits by 9:1. We use the official test set for evaluation.","Chinese OntoNotes4.0 is a Chinese dataset and consists of texts from news domain, which has 18 entity types. In this paper, we take the same data split as wu2019glyce did."]},{"idx":15,"heading":"Experiments ::: Named Entity Recognition ::: Baselines","paragraphs":["We use the following baselines:","ELMo: a tagging model from peters2018deep.","Lattice-LSTM: lattice2018zhang constructs a word-character lattice, only used in Chinese datasets.","CVT: from kevin2018cross, which uses Cross-View Training(CVT) to improve the representations of a Bi-LSTM encoder.","Bert-Tagger: devlin2018bert treats NER as a tagging task.","Glyce-BERT: wu2019glyce combines glyph information with BERT pretraining.","BERT-MRC: The current SOTA model for both Chinese and English NER datasets proposed by xiaoya2019ner, which formulate NER as machine reading comprehension task."]},{"idx":16,"heading":"Experiments ::: Named Entity Recognition ::: Results","paragraphs":["Table shows experimental results on NER datasets. For English datasets including CoNLL2003 and OntoNotes5.0, our proposed method outperforms BERT-MRCBIBREF38 by +0.29 and +0.96 respectively. We observe huge performance boosts on Chinese datasets, achieving F1 improvements by +0.97 and +2.36 on MSRA and OntoNotes4.0, respectively. As far as we are concerned, we are setting new SOTA performances on all of the four NER datasets."]},{"idx":17,"heading":"Experiments ::: Machine Reading Comprehension","paragraphs":["Machine reading comprehension (MRC) BIBREF39, BIBREF40, BIBREF41, BIBREF40, BIBREF42, BIBREF15 has become a central task in natural language understanding. MRC in the SQuAD-style is to predict the answer span in the passage given a question and the passage. In this paper, we choose the SQuAD-style MRC task and report Extract Match (EM) in addition to F1 score on validation set. All hyperparameters are tuned on the development set of each dataset."]},{"idx":18,"heading":"Experiments ::: Machine Reading Comprehension ::: Datasets","paragraphs":["The following five datasets are used for MRC task: SQuAD v1.1, SQuAD v2.0 BIBREF4, BIBREF6 and Quoref BIBREF8.","SQuAD v1.1 and SQuAD v2.0 are the most widely used QA benchmarks. SQuAD1.1 is a collection of 100K crowdsourced question-answer pairs, and SQuAD2.0 extends SQuAD1.1 allowing no short answer exists in the provided passage.","Quoref is a QA dataset which tests the coreferential reasoning capability of reading comprehension systems, containing 24K questions over 4.7K paragraphs from Wikipedia."]},{"idx":19,"heading":"Experiments ::: Machine Reading Comprehension ::: Baselines","paragraphs":["We use the following baselines:","QANet: qanet2018 builds a model based on convolutions and self-attention. Convolution to model local interactions and self-attention to model global interactions.","BERT: devlin2018bert treats NER as a tagging task.","XLNet: xlnet2019 proposes a generalized autoregressive pretraining method that enables learning bidirectional contexts."]},{"idx":20,"heading":"Experiments ::: Machine Reading Comprehension ::: Results","paragraphs":["Table shows the experimental results for MRC tasks. With either BERT or XLNet, our proposed DSC loss obtains significant performance boost on both EM and F1. For SQuADv1.1, our proposed method outperforms XLNet by +1.25 in terms of F1 score and +0.84 in terms of EM and achieves 87.65 on EM and 89.51 on F1 for SQuAD v2.0. Moreover, on QuoRef, the proposed method surpasses XLNet results by +1.46 on EM and +1.41 on F1. Another observation is that, XLNet outperforms BERT by a huge margin, and the proposed DSC loss can obtain further performance improvement by an average score above 1.0 in terms of both EM and F1, which indicates the DSC loss is complementary to the model structures."]},{"idx":21,"heading":"Experiments ::: Paraphrase Identification","paragraphs":["Paraphrases are textual expressions that have the same semantic meaning using different surface words. Paraphrase identification (PI) is the task of identifying whether two sentences have the same meaning or not. We use BERT BIBREF11 and XLNet BIBREF43 as backbones and report F1 score for comparison. Hyperparameters are tuned on the development set of each dataset."]},{"idx":22,"heading":"Experiments ::: Paraphrase Identification ::: Datasets","paragraphs":["We conduct experiments on two widely used datasets for PI task: MRPC BIBREF44 and QQP.","MRPC is a corpus of sentence pairs automatically extracted from online news sources, with human annotations of whether the sentence pairs are semantically equivalent. The MRPC dataset has imbalanced classes (68% positive, 32% for negative).","QQP is a collection of question pairs from the community question-answering website Quora. The class distribution in QQP is also unbalanced (37% positive, 63% negative)."]},{"idx":23,"heading":"Experiments ::: Paraphrase Identification ::: Results","paragraphs":["Table shows the results for PI task. We find that replacing the training objective with DSC introduces performance boost for both BERT and XLNet. Using DSC loss improves the F1 score by +0.58 for MRPC and +0.73 for QQP."]},{"idx":24,"heading":"Ablation Studies ::: The Effect of Dice Loss on Accuracy-oriented Tasks","paragraphs":["We argue that the most commonly used cross-entropy objective is actually accuracy-oriented, whereas the proposed dice loss (DL) performs as a hard version of F1-score. To explore the effect of the dice loss on accuracy-oriented tasks such as text classification, we conduct experiments on the Stanford Sentiment Treebank sentiment classification datasets including SST-2 and SST-5. We fine-tune BERT$_\\text{Large}$ with different training objectives. Experiment results for SST are shown in . For SST-5, BERT with CE achieves 55.57 in terms of accuracy, with DL and DSC losses slightly degrade the accuracy performance and achieve 54.63 and 55.19, respectively. For SST-2, BERT with CE achieves 94.9 in terms of accuracy. The same as SST-5, we observe a slight performance drop with DL and DSC, which means that the dice loss actually works well for F1 but not for accuracy."]},{"idx":25,"heading":"Ablation Studies ::: The Effect of Hyperparameters in Tversky index","paragraphs":["As mentioned in Section SECREF10, Tversky index (TI) offers the flexibility in controlling the tradeoff between false-negatives and false-positives. In this subsection, we explore the effect of hyperparameters (i.e., $\\alpha $ and $\\beta $) in TI to test how they manipulate the tradeoff. We conduct experiments on the Chinese OntoNotes4.0 NER dataset and English QuoRef MRC dataset to examine the influence of tradeoff between precision and recall. Experiment results are shown in Table . The highest F1 for Chinese OntoNotes4.0 is 84.67 when $\\alpha $ is set to 0.6 while for QuoRef, the highest F1 is 68.44 when $\\alpha $ is set to 0.4. In addition, we can observe that the performance varies a lot as $\\alpha $ changes in distinct datasets, which shows that the hyperparameters $\\alpha ,\\beta $ play an important role in the proposed method."]},{"idx":26,"heading":"Conclusion","paragraphs":["In this paper, we alleviate the severe data imbalance issue in NLP tasks. We propose to use dice loss in replacement of the standard cross-entropy loss, which performs as a soft version of F1 score. Using dice loss can help narrow the gap between training objectives and evaluation metrics. Empirically, we show that the proposed training objective leads to significant performance boost for part-of-speech, named entity recognition, machine reading comprehension and paraphrase identification tasks."]},{"idx":27,"heading":"Figures and Tables","paragraphs":["Table 1: Number of positive and negative examples and their ratios for different data-imbalanced NLP tasks.","Table 2: Different losses and their formulas.","Figure 1: An illustration of the derivative of f1 = p1+p and f2 = (1−p)p","Table 3: Experimental results for POS datasets. WeightCE denotes weighted cross-entropy, FL denotes focal loss, DL denotes dice loss and DSC denotes adjusted dice coefficient.","Table 4: Experimental results for NER task. WeightCE denotes weighted cross-entropy, FL denotes focal loss, DL denotes dice loss and DSC denotes adjusted dice coefficient.","Table 5: Experimental results for MRC task. FL denotes focal loss, DL denotes dice loss and DSC denotes adjusted dice coefficient.","Table 6: Experimental results for PI task. FL denotes focal loss, DL denotes dice loss and DSC denotes adjusted dice coefficient.","Table 7: The effect of dice loss on sentiment classification. BERT+CE refers to fine-tune BERT model and set cross-entropy as the training objective.","Table 8: The effect of α in Tversky Index."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1911.03090","title":"What Would Elsa Do? Freezing Layers During Transformer Fine-Tuning","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Pretrained transformer-based language models have achieved state of the art across countless tasks in natural language processing. These models are highly expressive, comprising at least a hundred million parameters and a dozen layers. Recent evidence suggests that only a few of the final layers need to be fine-tuned for high quality on downstream tasks. Naturally, a subsequent research question is, \"how many of the last layers do we need to fine-tune?\" In this paper, we precisely answer this question. We examine two recent pretrained language models, BERT and RoBERTa, across standard tasks in textual entailment, semantic similarity, sentiment analysis, and linguistic acceptability. We vary the number of final layers that are fine-tuned, then study the resulting change in task-specific effectiveness. We show that only a fourth of the final layers need to be fine-tuned to achieve 90% of the original quality. Surprisingly, we also find that fine-tuning all layers does not always help."]},{"idx":1,"heading":"Introduction","paragraphs":["Transformer-based pretrained language models are a battle-tested solution to a plethora of natural language processing tasks. In this paradigm, a transformer-based language model is first trained on copious amounts of text, then fine-tuned on task-specific data. BERT BIBREF0, XLNet BIBREF1, and RoBERTa BIBREF2 are some of the most well-known ones, representing the current state of the art in natural language inference, question answering, and sentiment classification, to list a few. These models are extremely expressive, consisting of at least a hundred million parameters, a hundred attention heads, and a dozen layers.","An emerging line of work questions the need for such a parameter-loaded model, especially on a single downstream task. BIBREF3, for example, note that only a few attention heads need to be retained in each layer for acceptable effectiveness. BIBREF4 find that, on many tasks, just the last few layers change the most after the fine-tuning process. We take these observations as evidence that only the last few layers necessarily need to be fine-tuned.","The central objective of our paper is, then, to determine how many of the last layers actually need fine-tuning. Why is this an important subject of study? Pragmatically, a reasonable cutoff point saves computational memory across fine-tuning multiple tasks, which bolsters the effectiveness of existing parameter-saving methods BIBREF5. Pedagogically, understanding the relationship between the number of fine-tuned layers and the resulting model quality may guide future works in modeling.","Our research contribution is a comprehensive evaluation, across multiple pretrained transformers and datasets, of the number of final layers needed for fine-tuning. We show that, on most tasks, we need to fine-tune only one fourth of the final layers to achieve within 10% parity with the full model. Surprisingly, on SST-2, a sentiment classification dataset, we find that not fine-tuning all of the layers leads to improved quality."]},{"idx":2,"heading":"Background and Related Work ::: Pretrained Language Models","paragraphs":["In the pretrained language modeling paradigm, a language model (LM) is trained on vast amounts of text, then fine-tuned on a specific downstream task. BIBREF6 are one of the first to successfully apply this idea, outperforming state of the art in question answering, textual entailment, and sentiment classification. Their model, dubbed ELMo, comprises a two-layer BiLSTM pretrained on the Billion Word Corpus BIBREF7.","Furthering this approach with more data and improved modeling, BIBREF0 pretrain deep 12- and 24-layer bidirectional transformers BIBREF8 on the entirety of Wikipedia and BooksCorpus BIBREF9. Their approach, called BERT, achieves state of the art across all tasks in the General Language Understanding Evaluation (GLUE) benchmark BIBREF10, as well as the Stanford Question Answering Dataset (BIBREF11).","As a result of this development, a flurry of recent papers has followed this more-data-plus-better-models principle. Two prominent examples include XLNet BIBREF1 and RoBERTa BIBREF2, both of which contest the present state of the art. XLNet proposes to pretrain two-stream attention-augmented transformers on an autoregressive LM objective, instead of the original cloze and next sentence prediction (NSP) tasks from BERT. RoBERTa primarily argues for pretraining longer, using more data, and removing the NSP task for BERT."]},{"idx":3,"heading":"Background and Related Work ::: Layerwise Interpretability","paragraphs":["The prevailing evidence in the neural network literature suggests that earlier layers extract universal features, while later ones perform task-specific modeling. BIBREF12 visualize the per-layer activations in image classification networks, finding that the first few layers function as corner and edge detectors, and the final layers as class-specific feature extractors. BIBREF13 demonstrate that the low- and high-level notions of content and style are separable in convolutional neural networks, with lower layers capturing content and higher layers style.","Pretrained transformers. In the NLP literature, similar observations have been made for pretrained language models. BIBREF14 analyze BERT's attention and observe that the bottom layers attend broadly, while the top layers capture linguistic syntax. BIBREF4 find that the last few layers of BERT change the most after task-specific fine-tuning. Similar to our work, BIBREF5 fine-tune the top layers of BERT, as part of their baseline comparison for their model compression approach. However, none of the studies comprehensively examine the number of necessary final layers across multiple pretrained transformers and datasets."]},{"idx":4,"heading":"Experimental Setup","paragraphs":["We conduct our experiments on NVIDIA Tesla V100 GPUs with CUDA v10.1. We run the models from the Transformers library (v2.1.1; BIBREF15) using PyTorch v1.2.0."]},{"idx":5,"heading":"Experimental Setup ::: Models and Datasets","paragraphs":["We choose BERT BIBREF0 and RoBERTa BIBREF2 as the subjects of our study, since they represent state of the art and the same architecture. XLNet BIBREF1 is another alternative; however, they use a slightly different attention structure, and our preliminary experiments encountered difficulties in reproducibility with the Transformers library. Each model has base and large variants that contain 12 and 24 layers, respectively. We denote them by appending the variant name as a subscript to the model name.","Within each variant, the two models display slight variability in parameter count—110 and 125 million in the base variant, and 335 and 355 in the large one. These differences are mostly attributed to RoBERTa using many more embedding parameters—exactly 63% more for both variants. For in-depth, layerwise statistics, see Table TABREF4.","For our datasets, we use the GLUE benchmark, which comprises the tasks in natural language inference, sentiment classification, linguistic acceptability, and semantic similarity. Specifically, for natural language inference (NLI), it provides the Multigenre NLI (MNLI; BIBREF16), Question NLI (QNLI; BIBREF10), Recognizing Textual Entailment (RTE; BIBREF17), and Winograd NLI BIBREF18 datasets. For semantic textual similarity and paraphrasing, it contains the Microsoft Research Paraphrase Corpus (MRPC; BIBREF19), the Semantic Textual Similarity Benchmark (STS-B; BIBREF20), and Quora Question Pairs (QQP; BIBREF21). Finally, its single-sentence tasks consist of the binary-polarity Stanford Sentiment Treebank (SST-2; BIBREF22) and the Corpus of Linguistic Acceptability (CoLA; BIBREF23)."]},{"idx":6,"heading":"Experimental Setup ::: Fine-Tuning Procedure","paragraphs":["Our fine-tuning procedure closely resembles those of BERT and RoBERTa. We choose the Adam optimizer BIBREF24 with a batch size of 16 and fine-tune BERT for 3 epochs and RoBERTa for 10, following the original papers. For hyperparameter tuning, the best learning rate is different for each task, and all of the original authors choose one between $1 \\times 10^{-5}$ and $5 \\times 10^{-5}$; thus, we perform line search over the interval with a step size of $1 \\times 10^{-5}$. We report the best results in Table TABREF5.","On each model, we freeze the embeddings and the weights of the first $N$ layers, then fine-tune the rest using the best hyperparameters of the full model. Specifically, if $L$ is the number of layers, we explore $N = \\frac{L}{2}, \\frac{L}{2} + 1, \\dots , L$. Due to computational limitations, we set half as the cutoff point. Additionally, we restrict our comprehensive all-datasets exploration to the base variant of BERT, since the large model variants and RoBERTa are much more computationally intensive. On the smaller CoLA, SST-2, MRPC, and STS-B datasets, we comprehensively evaluate both models. These choices do not substantially affect our analysis."]},{"idx":7,"heading":"Analysis ::: Operating Points","paragraphs":["We report three relevant operating points in Tables TABREF6–TABREF9: two extreme operating points and an intermediate one. The former is self-explanatory, indicating fine-tuning all or none of the nonoutput layers. The latter denotes the number of necessary layers for reaching at least 90% of the full model quality, excluding CoLA, which is an outlier.","From the reported results in Tables TABREF6–TABREF9, fine-tuning the last output layer and task-specific layers is insufficient for all tasks—see the rows corresponding to 0, 12, and 24 frozen layers. However, we find that the first half of the model is unnecessary; the base models, for example, need fine-tuning of only 3–5 layers out of the 12 to reach 90% of the original quality—see Table TABREF7, middle subrow of each row group. Similarly, fine-tuning only a fourth of the layers is sufficient for the large models (see Table TABREF9); only 6 layers out of 24 for BERT and 7 for RoBERTa."]},{"idx":8,"heading":"Analysis ::: Per-Layer Study","paragraphs":["In Figure FIGREF10, we examine how the relative quality changes with the number of frozen layers. To compute a relative score, we subtract each frozen model's results from its corresponding full model. The relative score aligns the two baselines at zero, allowing the fair comparison of the transformers. The graphs report the average of five trials to reduce the effects of outliers.","When every component except the output layer and the task-specific layer is frozen, the fine-tuned model achieves only 64% of the original quality, on average. As more layers are fine-tuned, the model effectiveness often improves drastically—see CoLA and STS-B, the first and fourth vertical pairs of subfigures from the left. This demonstrates that gains decompose nonadditively with respect to the number of frozen initial layers. Fine-tuning subsequent layers shows diminishing returns, with every model rapidly approaching the baseline quality at fine-tuning half of the network; hence, we believe that half is a reasonable cutoff point for characterizing the models.","Finally, for the large variants of BERT and RoBERTa on SST-2 (second subfigure from both the top and the left), we observe a surprisingly consistent increase in quality when freezing 12–16 layers. This finding suggests that these models may be overparameterized for SST-2."]},{"idx":9,"heading":"Conclusions and Future Work","paragraphs":["In this paper, we present a comprehensive evaluation of the number of final layers that need to be fine-tuned for pretrained transformer-based language models. We find that only a fourth of the layers necessarily need to be fine-tuned to obtain 90% of the original quality. One line of future work is to conduct a similar, more fine-grained analysis on the contributions of the attention heads."]},{"idx":10,"heading":"Acknowledgments","paragraphs":["This research was supported by the Natural Sciences and Engineering Research Council (NSERC) of Canada, and enabled by computational resources provided by Compute Ontario and Compute Canada."]},{"idx":11,"heading":"Figures and Tables","paragraphs":["Table 1: Parameter statistics for the base and large variants of BERT and RoBERTa. Note that “per-layer” indicates the number of parameters in one intermediate layer, which is more relevant to our study.","Table 2: Reproduced results of BERT and RoBERTa on the development sets.","Table 3: Development set results of BERT, with none, some, and all of the nonoutput layer weights fine-tuned. Results are averaged across five runs.","Table 5: Development set results of all large models, with none, some, and all of the nonoutput layer weights fine-tuned. Results are averaged across five runs.","Table 4: Development set results of all base models, with none, some, and all of the nonoutput layer weights fine-tuned. Results are averaged across five runs.","Figure 1: Relative change in quality compared to the full models, with respect to the number of frozen initial layers, represented by the x-axes."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1911.03977","title":"Multimodal Intelligence: Representation Learning, Information Fusion, and Applications","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Deep learning has revolutionized speech recognition, image recognition, and natural language processing since 2010, each involving a single modality in the input signal. However, many applications in artificial intelligence involve more than one modality. It is therefore of broad interest to study the more difficult and complex problem of modeling and learning across multiple modalities. In this paper, a technical review of the models and learning methods for multimodal intelligence is provided. The main focus is the combination of vision and natural language, which has become an important area in both computer vision and natural language processing research communities. This review provides a comprehensive analysis of recent work on multimodal deep learning from three angles --- learning multimodal representations, the fusion of multimodal signals at various levels, and multimodal applications. On multimodal representation learning, we review the key concept of embedding, which unifies the multimodal signals into the same vector space and thus enables cross-modality signal processing. We also review the properties of the many types of embedding constructed and learned for general downstream tasks. On multimodal fusion, this review focuses on special architectures for the integration of the representation of unimodal signals for a particular task. On applications, selected areas of a broad interest in current literature are covered, including image-to-text caption generation, text-to-image generation, and visual question answering. We believe this review can facilitate future studies in the emerging field of multimodal intelligence for the community."]},{"idx":1,"heading":"Introduction","paragraphs":["Significant progress has been made in the field of machine learning in the past years due to the rapid development of deep learning BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5. Dating back to the dramatic increase in the accuracy of large-scale automatic speech recognition (ASR) using fully connected deep neural networks (DNN) and deep auto-encoders around 2010 BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11, BIBREF12, BIBREF13, BIBREF14, BIBREF15, BIBREF16, and followed by a set of breakthroughs in computer vision (CV) using deep convolutional neural network (CNN) models BIBREF17 for large-scale image classification around 2012 BIBREF18, BIBREF19, BIBREF20, BIBREF21 and large-scale object detection BIBREF22, BIBREF23, BIBREF24 around 2014, a set of major milestones have been achieved in pattern recognition with single input modality. Subsequently, in natural language processing (NLP), recurrent neural network (RNN) based semantic slot filling methods BIBREF25 achieved new state-of-the-art in spoken language understanding, and RNN-encoder-decoder models with attention mechanism BIBREF26, also referred to as sequence to sequence models BIBREF27, produced superior performance in machine translation in an end-to-end fashion BIBREF28, BIBREF29. For other NLP tasks without much training data, such as question answering (QA) and machine reading comprehension, generative pre-training that transfers parameters from a language model (LM) pre-trained on a large out-of-domain data set using unsupervised or self learning then followed by fine-tuning on small in-domain data sets, achieved record-breaking results over a set of tasks BIBREF30, BIBREF31, BIBREF32.","Despite the advances in vision, speech, and language processing, many problems in artificial intelligence involve more than one modality, such as an intelligent personal assistant (IPA) that should understand human communicative intentions embedded not only in spoken language, but also in body and pictorial languages BIBREF33. Therefore, it is of broad interests to study the modeling and learning approaches across multiple modalities BIBREF34. Benefiting from the advances in image processing and language understanding BIBREF35, a set of tasks that combine both image and text have drawn much attention, which include visual grounding tasks like referring expression understanding and phrase localization BIBREF36, BIBREF37, BIBREF38, image captioning BIBREF39, BIBREF40, BIBREF41, visual QA (VQA) BIBREF42, BIBREF43, BIBREF44, text-to-image generation BIBREF45, BIBREF46, BIBREF47, and visual-language navigation BIBREF48 etc. In these tasks, natural language plays a key role in helping the machine to “understand” the content of the images, where “understand” means to capture the underlying correlations between the semantics embedded in language with the visual features obtained from the images. In addition to text, vision can be combined with speech as well. Such tasks include audio-visual speech recognition BIBREF49, BIBREF50, BIBREF51, speaker recognition BIBREF52, BIBREF53, BIBREF54, as well as speech diarisation BIBREF55, BIBREF56, separation BIBREF57, BIBREF58 and enhancement BIBREF59, which mostly focused on the use of visual features to improve the robustness of the audio-only methods.","In this paper, a technical review of the models and learning methods for multimodal intelligence is provided. The main focus is the combination of CV and NLP, which has become an important area for both research communities covering many different tasks and technologies. To provide a more structured perspective, we organize the methods selected in this technical review according to three key topics: representation, fusion, and applications.","Learning representations for the input data is a core problem for deep learning. For multimodal tasks, collecting paralleled data across all modalities can be quite difficult and leveraging pre-trained representations with desired properties, such as suitable for zero-shot or few-shot learning, is often an effective solution to the issue. Both supervised and unsupervised training based multimodal representation learning methods are reviewed.","The fusion of the features or representations of the single modalities is undoubtedly a centric problem of any multimodal task. Different from previous studies that often categorise the related work into early, middle and late stage methods based on the stage that fusion happens in the procedure, we classify them according to the actual operation used in the fusion, such as attention and bilinear pooling, since it becomes difficult to classify some recent complex approaches into stages.","Three types of applications are reviewed in this paper, namely image captioning, text-to-image synthesis and VQA. This is to give an idea how representation learning and fusion can be applied to specific tasks, and to provide a viewpoint of the situation of the current development of the multimodal applications, especially those integrating vision with natural languages. Visual reasoning methods for VQA are also discussed in the end.","This paper is organised as follows. Section SECREF2 reviews the recent progress on developing representations for single or multiple modalities. Section SECREF3 introduces the commonly used fusion methods, particularly attention and bilinear pooling. Applications including caption generation, text-to-image synthesis, VQA, and visual reasoning are introduced in Section SECREF4, followed by conclusions."]},{"idx":2,"heading":"Representations","paragraphs":["Deep learning, as a special area in representational learning, studies the use of artificial neural networks (ANNs) with many hidden layers to automatically discover the representations or features suitable for specific tasks from the raw data BIBREF60. In practice, it is often found that better representations can simplify the subsequent learning tasks and therefore has a great value. Over the past decade, it becomes feasible to learn effective and robust representations for single modalities, such as text BIBREF61, BIBREF62, BIBREF63, BIBREF30, BIBREF31, BIBREF32, BIBREF64, BIBREF65, BIBREF66, BIBREF67, BIBREF68 and image BIBREF18, BIBREF19, BIBREF20, BIBREF21, BIBREF22, BIBREF23, BIBREF24, due to the availability of large data and development of deep learning. For multimodal representations, though attracting more and more attentions, it still remains a challenging problem due to the complex cross-modal interactions and possible mismatch between the training and test data of each modal.","In this section, commonly used types of single modal representations, such as text and image, are first reviewed which often serve as cornerstones for learning multimodal representations. Afterwards, both supervised and unsupervised methods for learning a joint representation space for multiple modalities are introduced. To empower the model to handle data samples with some missing modality, zero-shot learning problem is studied to increase the similarity of the representational spaces across the involved modalities. At last, inspired by the great success of adapting pre-trained LMs for downstream tasks in NLP, methods that leverage large unimodal data sets to improve the learning of multimodal representations are also discussed."]},{"idx":3,"heading":"Representations ::: Unimodal Embeddings","paragraphs":["The representations obtained using ANN models are often distributed, which entails that elements composing the representations can be set separately to allow more concepts to be encoded efficiently in a relatively low-dimensional space BIBREF64. This can be compared with the symbolic representations, such as the one-hot encoding that uses an element with value one to indicate the presence of the associated symbol or category, and value zero for the rest elements. In deep learning, the term embedding often refers to a mapping from a one-hot encoding representing a word or an image category to a distributed representation as a vector of real-valued numbers."]},{"idx":4,"heading":"Representations ::: Unimodal Embeddings ::: Visual representation","paragraphs":["The image embeddings can be acquired as the output values from the final CNN layers from models that classify images into categories, such as AlexNet BIBREF18, VGG nets BIBREF19, and residual neural network (ResNet) BIBREF20. AlexNet is a model with five CNN layers with rectified linear unit (ReLU) activation functions whose kernel sizes are $11\\times 11$, $5\\times 5$, and $3\\times 3$. A VGG net often has 16 or 34 CNN layers, with all of them using very small $3\\times 3$ kernels. ResNet can have a depth of up to 152 layers, mostly with $3\\times 3$ kernels, due to the invention of residual connections. Comparing to the aforementioned models, GoogLeNet has a more different structure formed by stacking multiple Inception structures BIBREF21. The naïve Inception structure is a concatenation of CNN layers with $1\\times 1$, $3\\times 3$, and $5\\times 5$ sized kernels, and a max pooling layer with $3\\times 3$ kernels, and can be viewed as a sparsely connected convolutional architecture to reduce overfitting and computational cost. Later versions of the Inception models improve the structures further by factorizing the kernels and adding residual connections. AlexNet, GoogLeNet, and ResNet are the winners of the 2012, 2014, and 2015 ImageNet Large Scale Visual Recognition Competition for image classification respectively BIBREF69, BIBREF70. Alternatively, features with more direct relationships with the semantics can be used as visual embeddings, such as convolutional features and the associated class labels from selected regions found by object detection models, such as region with CNN features (R-CNN) BIBREF22, Fast R-CNN BIBREF23, and Faster R-CNN BIBREF24 etc."]},{"idx":5,"heading":"Representations ::: Unimodal Embeddings ::: Language representations","paragraphs":["Text embeddings can be derived from a neural network language model (NNLM) BIBREF65, which estimates the probability of a text sequence by factorizing it into word probabilities based on the chain rule of probability. A feedforward neural network with a linear projection layer and a non-linear hidden layer is often used as an implementation of an $n$-gram LM, which takes the previous $n-1$ words as the input to predict the probability of the current word. Each word is presented in one-hot encoding based on the vocabulary and converted into a real-valued vector, the word embedding, using the projection layer. An improved NNLM is to replace the feedforward model with an RNN, such as a long short-term memory (LSTM) BIBREF71 or gated recurrent unit (GRU) BIBREF72 model, which allows the use of information from all past words stored in a fixed-length recurrent vector when predicting the current word. Apart from NNLMs, continuous bag-of-words model (CBOW) and skip-gram model are two simple feedforward structures that learn word embedding either by predicting the current word based on the past and future context words or vice versa BIBREF66. The method of global vectors (GloVe) shows that effective text-embedding can be learnt using a global log-bilinear regression model based on the co-occurrence counts of words BIBREF73. Meanwhile, a serious of deep structured semantic models (DSSM) were proposed since 2013 for sentence level embedding learning through optimizing semantic-similarity driven objectives, with various neural network structures in a pseudo-simense network setting BIBREF61, BIBREF62, BIBREF63, BIBREF74, BIBREF75, BIBREF76, BIBREF77.","More recently, in order to transfer to use in downstream natural language understanding tasks without much training data, studies focus on learning general text embeddings by predicting word probabilities using NNLMs with complex structures on a large text corpus. Embeddings from language models (ELMo) uses a combined embedding from multiple layers of bidirectional LSTMs for forward and backward directions BIBREF30. Generative pre-training (GPT) and bidirectional encoder representations for Transformers (BERT) use the decoder and encoder part of the Transformer model to estimate the probability of the current subword unit BIBREF31, BIBREF32. Other technologies, such as masked language model and multi-task training are used in these methods nowadays BIBREF32. Besides word and subword levels, text embedding can be learnt at phrase, sentence, and even paragraph levels, such as the skip-though vectors that extends the skip-gram method to the sequence-to-sequence framework BIBREF78, BIBREF27. It uses two decoders to predict the previous and next sentences given the embedding of the current sentence generated by the encoder."]},{"idx":6,"heading":"Representations ::: Unimodal Embeddings ::: Vector arithmetic for word and image embeddings","paragraphs":["It is well-known that word embeddings can learn not only syntactic but also the semantic regularities. A famous example showed vector(“King”)$-$vector(“Man”)$+$vector(“Woman”) results in a vector closest to vector(“Queen”) where vector($\\cdot $) denotes the vector representation of a word learnt by a RNN LM BIBREF79. Similar phenomenon has been observed for vision embeddings. It was shown that when using a generative adversarial network (GAN) BIBREF80, there exists a similar vector arithmetic that the representation of an image with a man wearing glasses subtracted by that of a man without glasses and finally add the representation of a woman without glasses will lead to the representation of a woman wearing glasses BIBREF81. This reveals GAN can capture image representation that distangles the concept of gender from the concept of wearing glasses. Such encouraging progress in text and image representations encouraged further studies on the joint representations of these two modalities. More details about GAN based image generation can be found later in Section SECREF31."]},{"idx":7,"heading":"Representations ::: Multimodal Representations","paragraphs":["Although significant progress has been made in the learning of representations for vision or language, it is theoretically insufficient to model a complete set of human concepts using only unimodal data. For example, the concept of “beautiful music” is clearly grounded in the auditory perceptron and one can be struggled to describe this by natural language or other approaches. Therefore, it is important to learn a joint embedding to leverage the complementarity from multimodal data to represent the concepts better. Both supervised and unsupervised training approaches are of broad interest and can be applied to tasks with different data availability. Meanwhile, by assuming the corresponding representations to have similar neighbourhood structures across modalities, the representation of a concept with zero training sample in one modal can be found based on its representations grounded in other modalities which have training data. For instance, when using zero-shot training for image labelling, the closest word vectors can be retrieved as labels by projecting images of objects unseen in the training set onto the linguistic space. More recently, inspired by Transformer and BERT from NLP, it becomes increasingly popular to apply these models to develop better bimodal representations combining vision and language."]},{"idx":8,"heading":"Representations ::: Multimodal Representations ::: Unsupervised training methods","paragraphs":["Joint embeddings for multimodal data can be learnt by simply reconstructing the raw input using multiple streams of deep Boltzmann machines or autoencoders with a shared layer as the shared representation space BIBREF82, BIBREF83, BIBREF84. Alternatively, with the development of methods for single modal representations, the shared representation space can be constructed based on those of the involved single modalities. For example, in BIBREF85, Fang et al. propose a deep multimodal similarity model (DMSM), which extended the text-modal DSSM to learning embedding representations of text and image in an unified vector space. BIBREF84, BIBREF86 perform simple fusion of the word and image embeddings with addition or concatenation. BIBREF87 learns to increase the similarity between corresponding Skip-Gram word embedding and AlexNet derived image features. BIBREF88, BIBREF89 maximize the correlation and mutual information between embeddings of different modalities respectively. BIBREF90 modifies the distance between CBOW word embeddings according to the similarities between their visual instantiations, which are found by clustering abstract scenes in an unsupervised way.","Further studies found correlating image regions/fragments with sentence fragments or attribute words generates fine-grained multimodal embeddings BIBREF91, by finding the alignments of the image and sentence fragments automatically. BIBREF92 unifies the embeddings of concepts at different levels, including objects, attributes, relations and full scenes. BIBREF93 proposed a stacked cross attention network (SCAN) to learning fine-grained word and image-object aligned embedding for image-text matching. BIBREF47 employs a deep attentional multimodal similarity model (DAMSM) extending DMSM with attention models to measure the similarity between image sub-regions and words as an additional loss for text-to-image generation."]},{"idx":9,"heading":"Representations ::: Multimodal Representations ::: Supervised training methods","paragraphs":["Supervisions can be used to improve multimodal representation learning. BIBREF94 factorizes the representations into two sets of independent factors: multimodal discriminative factors for supervised training and intra-modality generative factors for unsupervised training. The discriminative factors are shared across all modalities and are useful for discriminative tasks, whereas the generative factors can be used to reconstruct missing modalities. With detailed text annotations, BIBREF95 proposed to learn word embeddings from their visual co-occurrences (ViCo) when applying to the same natural scene image or image region. ViCo is found to be complementary to the GloVe embedding by better representing similarities and differences between visual concepts that are difficult to obtain from text corpora alone. BIBREF96 applies multiple supervised training tasks to different layers of the vision-language encoder. The order of the training tasks is arranged following the idea of curriculum learning to increase the complexity of training objective step-by-step."]},{"idx":10,"heading":"Representations ::: Multimodal Representations ::: Methods for zero-shot learning","paragraphs":["Zero-shot learning often applies to vision related tasks due to the difficulty to acquire sufficient labelled images for training for all possible object categories. Not all types of multimodal representations are suitable for zero-shot learning since they may require pair-wised data from both modalities to present at the same time. Here we review methods that rely on extra language source to remedy this issue.","Deep learning based zero-shot learning started by training a linear mapping layer between different pre-trained embeddings BIBREF97, BIBREF98. The deep visual-semantic embedding (DeViSE) model is built upon Skip-Gram word embedding and AlexNet visual features and allows both pre-trained models to be jointly trained with the linear mapping layer BIBREF98. It achieved a large-scale test with 1000 seen classes and 2000 unseen classes. Better representations could be learnt when correlated autoencoders are used to reconstruct the representations for each modality, which improves one-shot and few-shot image retrieval performance comparing to DeViSE BIBREF99. Richer information source can be used for both modalities, including words selected from Wikipedia articles and features derived from multiple CNN layers BIBREF100. Rather than direct text attribute input, sentence embedding generated by recurrent models can be used as the text interface for zero-shot learning to achieve competitive results BIBREF101. Moving beyond empirical findings, recent study analyzed the properties of deep learning based cross-modal mapping using a similarity measure BIBREF102."]},{"idx":11,"heading":"Representations ::: Multimodal Representations ::: Transformer based methods","paragraphs":["Transformer is a prevalent sequence-based encoder-decoder model formed by stacking many blocks of feedforward layers with multi-head self-attention models BIBREF103. The parameters in all blocks shared across time similar to the time-delayed neural networks BIBREF104 and quasi-RNN BIBREF105 without an explicit temporal order. Compared with RNN based encoder-decoder models BIBREF26, it can have higher training efficiency due to the additional parallel degree across-time and superior performance on longer sequences benefited from the removing of first-order Markovian assumption imposed to the RNNs. BERT, the encoder part of Transformer pre-trained on a large text corpus as a masked LM, becomes a standard choice for word piece embeddings for downstream tasks, particularly since it utilizes both past and future information easily. It is natural to generalize the text-only BERT to cover images as well that can be used as the pre-trained multimodal embeddings.","A straightforward way to extend the unimodal BERT to bimodal, is to include new tokens to indicate the input of visual features, such as Unicoder-VL BIBREF106, VL-BERT BIBREF107, VisualBERT BIBREF108, VideoBERT BIBREF109, and B2T2 BIBREF110. LXMERT BIBREF111, ViLBERT BIBREF112, and OmniNet BIBREF113 modify the Transformer model by introducing an extra encoder or attention structures for visual features. More details about the modified structures can be found from Section SECREF18. Furthermore, recent NLP study found that multitask training can improve the generalization ability of the BERT representations BIBREF114. Most of the aforementioned bimodal BERT style models adopt multitask training to improve their performance on downstream tasks like VQA, image and video captioning etc. Although it would be useful to rigorously compare the performance of these models to understand the impact of different design choices, it is hard to do so since different amount of parameters and pre-training data are used across papers."]},{"idx":12,"heading":"Fusion","paragraphs":["Fusion is a key research problem in multimodal studies, which integrates information extracted from different unimodal data into one compact multimodal representation. There is a clear connection between fusion and multimodal representation. We classify an approach into the fusion category if its focus is the architectures for integrating unimodal representations for particular a task.","Traditionally, fusion methods are divided based on the stage it appears in the procedure. Early fusion, or feature-level fusion, directly combines the features extracted from each type of unimodal data to stress the intra-modality interactions and can cause the inter-modality interactions to be suppressed. Late fusion, on the other hand, refers to model-level fusion that builds a separate model for each modality and combines their output BIBREF115, BIBREF116, BIBREF117, BIBREF118, BIBREF119. The late fusion methods are strong in modelling intra-modality interactions with the modality-specific models but may suffer from the limited power of simple output value combination since the inter-modality interactions are rather complex. Recent studies focus on the intermediate or middle-level methods that allows fusion to happen at multiple layers of a deep model.","In this section, a review on intermediate fusion is focused – not only as it is more flexible, but also because the boundaries between stages are less clear due to the use of unimodal features derived from pre-trained backbone models. Three types of methods mostly used to fuse text with image features are included: simple operation-based, attention-based, as well as tensor-based methods."]},{"idx":13,"heading":"Fusion ::: Simple Operation-based Fusion","paragraphs":["In deep learning, vectorized features from different information sources can be integrated using a simple operation, such as concatenation or weighted sum, which often has only a few or even no parameter associated since the joint training of deep models can adapt the layers for high-level feature extractions to adjust for the required operation.","Concatenation can be used to combine either low-level input features BIBREF119, BIBREF120, BIBREF121 or high-level features extracted by the pre-trained models BIBREF121, BIBREF122, BIBREF123.","For weighted sum with scalar weights, an iterative method is proposed BIBREF124 that requires the pre-trained vector representations for each modality to have the same number of elements arranged in an order that is suitable for element-wise addition. This is often achieved by jointly training a fully connected layer for dimension control and reordering for each modality, together with the scalar weights for fusion.","A recent study BIBREF125 employs neural architecture search with progressive exploration BIBREF126, BIBREF127, BIBREF128 to find suitable settings for a number of fusion functions. Each fusion function is configured by which layers to fuse and whether to use concatenation or weighted sum as the fusion operation. Other weak functions can also be used to fuse multiple layers from each modality BIBREF129."]},{"idx":14,"heading":"Fusion ::: Attention-based Fusion","paragraphs":["Attention mechanism is widely used for fusion, which often refers to weighted sum a set of vectors using scalar weights dynamically generated by a small “attention” model at each time-step BIBREF129, BIBREF130. Multiple glimpses (output heads) are often used by the attention model to generate multiple sets of dynamic weights for the summation, whose resulted values can be concatenated to reserve more information. When applying attention mechanism to an image, image feature vectors relevant to different regions are weighted differently to produce an attended image vector."]},{"idx":15,"heading":"Fusion ::: Attention-based Fusion ::: Image attention","paragraphs":["BIBREF131 extends an LSTM model for text question processing with an image attention model conditioned on the previous LSTM hidden state, whose input is a concatenation of the current word embedding with the attended image feature. The final LSTM hidden state is regarded as the fused multimodal representation to predict the answer for pointing and grounded VQA. The attention model for sequence-based encoder-decoder model is used to attend to the image features for image captioning BIBREF132. Further for VQA, attention model conditioned on both image and query feature vectors is applied to pinpoint the image regions relevant to the answer BIBREF133. Similarly, stacked attention networks (SANs) are proposed to use multiple layers of attention models to query an image multiple times to infer the answer progressively by simulating a multi-step reasoning procedure BIBREF134. At each layer, a refined query vector is generated and send to the next layer by adding the previous query vector to the attended image vector produced using the current attention model. Spatial memory network (SMem) is a multi-hop method for VQA, which aligns words to image regions in the first hop and performs image attention based on the entire question in the second hop to derive the answer BIBREF135.","In BIBREF136, dynamic memory network (DMN) is augmented to use separate input modules to encode the question and image, which uses attention based GRUs to update episodic memory iteratively to retrieve the required information. Bottom-up and top-down attention method (Up-Down), as its name suggested, simulates human visual system using a combination of two visual attention mechanisms BIBREF137. The bottom-up attention mechanism proposes a set of salient image regions found by a Faster R-CNN, and the top-down attention mechanism uses a concatenation of visual and linguistic features to estimate the attention weights and produce the attended image feature vector for image captioning or VQA. The attended image feature vector can be fused with the linguistic feature again using an element-wise product. Complementary image features derived from different models, such as ResNet and Faster R-CNN, are used for multiple image attention mechanisms BIBREF138. Moreover, the reverse of image attention that generates attended text feature with image and text input is used for text-to-image synthesis in BIBREF47 and BIBREF139."]},{"idx":16,"heading":"Fusion ::: Attention-based Fusion ::: Image and text co-attention","paragraphs":["Different from the aforementioned image attention methods, co-attention mechanism has a symmetric attention structure to generate not only an attended image feature vector, but also an attended language vector BIBREF140. The parallel co-attention uses a joint representation to derive the image and language attention distributions simultaneously; alternating co-attention, on the other hand, has a cascade structure that first generates the attended image vector using the linguistic features, followed by the attended language vector generated using the attended image vector.","Similar to the parallel co-attention, dual attention network (DAN) estimates attention distributions for image and language simultaneously to derive their attended feature vectors BIBREF141. The attention models are conditioned on both feature and memory vectors of the relevant modality. This is a key difference to co-attention since the memory vectors can be iteratively updated at each reasoning step by repeating the DAN structure. The memory vectors can be either shared for VQA or modality-specific for image-text matching. Stacked latent attention (SLA) improves SAN by concatenating the original attended image vector with values from earlier layers of the attention model to retain the latent information from intermediate reasoning stages BIBREF142. A parallel co-attention like twin stream structure is also included to attend to both image and language features that also allows to reason iteratively using multiple SLA layers. Dual recurrent attention units (DRAU) implements the parallel co-attention structure with LSTM models for text and image to attend to each input location of the representations obtain by convolving image features with a stack of CNN layers BIBREF143. To model high-order interactions between modalities, high-order correlations between two data modalities can be computed as the inner product of two feature vectors and used to construct high-order attention models to derive the attended feature vectors for both modalities BIBREF144."]},{"idx":17,"heading":"Fusion ::: Attention-based Fusion ::: Attention in bimodal Transformer","paragraphs":["Recall Section SECREF14, the bimodal extensions to BERT rely on different tokens to indicate whether a vector is a word piece or an image, and the attention models fuse images with words in bimodal input sequences BIBREF106, BIBREF107, BIBREF108, BIBREF109, BIBREF110. OmniNet uses the gated multi-head attention model in each decoder block to fuse the vectors from the other modalities with that produced for the current modality by the previous layers in the the block BIBREF113. LXMERT uses independent encoders to learn the intra-modality features for each modality, and a cross-modality encoder sitting above them to learn the cross-modality features using extra cross-attention layers BIBREF111. ViLBERT extends BERT to include two encoder streams to process visual and textual inputs separately, which can interact through parallel co-attention layers BIBREF111."]},{"idx":18,"heading":"Fusion ::: Attention-based Fusion ::: Other attention like mechanisms","paragraphs":["Gated multimodal unit is a method that can be viewed as the attention of image and text based on gating BIBREF145. It performs weighted sum of visual and textual feature vectors based on dimension-specific scalar weights generated dynamically by the gating mechanism. Similarly, element-wise multiplication can be used to fuse visual and textual representations, which is used to create the building blocks of a multimodal residual network (MRN) based on deep residual learning BIBREF146. Dynamic parameter prediction network (DPPnet) uses a dynamic weight matrix to transform the visual feature vectors, whose parameters are dynamically generated by hashing the text feature vector BIBREF147."]},{"idx":19,"heading":"Fusion ::: Bilinear Pooling-based Fusion","paragraphs":["Bilinear pooling is a method often used to fuse a visual feature vector with a textual feature vector into a joint representation space by computing their outer product, which allows a multiplicative interaction between all elements in both vectors and is also termed as second order pooling BIBREF148. Comparing to simple vector combination operations (assuming each vector has $n$ elements), such as weighted sum, element-wise multiplication, or concatenation that result in $n$ or $2n$ dimensional representations, bilinear pooling leads to an $n^2$ dimensional representation by linearizing the outer product resulted matrix into a vector and is therefore more expressive. The bilinear representation is often linearly transformed into an output vector using a two-dimensional weight matrix, which is equivalent to use a three-dimensional tensor operator to fuse the two input feature vectors. Each feature vector can be extended with an extra value one to reserve input single modal features in the bilinear representation via outer product BIBREF149. However, given its high dimensionality, typically on the order of hundreds of thousands to a few million, bilinear pooling often requires decomposing the weight tensor to have the associated model to be trained properly and efficiently."]},{"idx":20,"heading":"Fusion ::: Bilinear Pooling-based Fusion ::: Factorization for bilinear pooling","paragraphs":["Since bilinear representations are found to be closely related to the polynomial kernels, different low-dimensional approximations can be used to acquire compact bilinear representations BIBREF150. Count Sketches and convolutions can be used to approximate the polynomial kernels BIBREF151, BIBREF152 that leads to the multimodal compact bilinear pooling (MCB) method BIBREF153. Alternatively, by enforcing a low rank to the weight tensor, multimodal low-rank bilinear pooling (MLB) factorizes the three-dimensional weight tensor for bilinear pooling into three two-dimensional weight matrices BIBREF154. More precisely, the visual and textual feature vectors are linearly projected to low-dimensional modality-specific factors by the two input factor matrices, which are then fused using element-wise multiplication followed by a linear projection with the third matrix, the output factor matrix. Multimodal factorized bilinear pooling (MFB) modifies MLB by using an extra operation to pool the element-wise multiplication results by summing the values within each non-overlapped one-dimensional window BIBREF155. Multiple MFB models can be cascaded to model high-order interactions between input features and is called multi-modal factorized high-order pooling (MFH) BIBREF156.","MUTAN, a multimodal tensor-based Tucker decomposition method, uses Tucker decomposition BIBREF157 to factorize the original three-dimensional weight tensor operator with a small-dimensional core tensor and the three two-dimensional weight matrices used by MLB BIBREF158. The core tensor models the interactions across modalities. Comparing to MUTAN, MCB can be seen as MUTAN with fixed diagonal input factor matrices and a sparse fixed core tensor, while MLB is MUTAN with the core tensor set to identity. Recently, BLOCK, a block superdiagonal fusion framework is proposed to use block-term decomposition BIBREF159 to compute bilinear pooling BIBREF160. BLOCK generalizes MUTAN as a summation of multiple MUTAN models to provide a richer modeling of interactions between modalities. The MUTAN core tensors can be arranged as a superdiagonal tensor, similar to the submatrices of a block diagonal matrix. Furthermore, bilinear pooling can be generalized to more than two modalities, such as BIBREF149 and BIBREF161 that use outer products to model the interactions among the representations for video, audio, and language."]},{"idx":21,"heading":"Fusion ::: Bilinear Pooling-based Fusion ::: Bilinear pooling and attention mechanism","paragraphs":["Bilinear pooling can be used along with attention mechanism. MCB/MLB fused bimodal representation can be used as the input feature of an attention model to derive the attended image feature vector, which is fused with the textual feature vector using MCB/MLB again to form the final joint representation BIBREF153, BIBREF154. MFB/MFH can be used for alternating co-attention to learn the join representation BIBREF155, BIBREF156. Bilinear attention network (BAN) uses MLB to fuse image and text to produce a bilinear attention map as the attention distributions, which is used as the weight tensor for bilinear pooling to fuse the image and text features again BIBREF162."]},{"idx":22,"heading":"Applications","paragraphs":["In this section, selected applications for multimodal intelligence that combine vision and language are discussed, which include image captioning, text-to-image generation, and VQA. It is worth noting that there are other applications, such as text-based image retrieval BIBREF93, BIBREF163, BIBREF164, and visual-and-language navigation (VLN) BIBREF165, BIBREF166, BIBREF167, BIBREF168, BIBREF169, BIBREF170, BIBREF171, BIBREF172, BIBREF173, that we have not included in this paper due to space limitation.","Caption generation is a task that aims to automatically generate a natural language description of an image. It requires a level of image understanding beyond normal image recognition and object detection.","A reverse of caption generation is text-to-image synthesis, which often generates image pixels according to a description sentence or some key words provided by human.","VQA is related to caption generation, which often takes an image as the input and a free-form, open-ended natural language question about the image, to output a classification result as the output of the answer. Natural language understanding is required as the questions are in free form. Other capabilities such as knowledge based reasoning and common-sense reasoning can be important since the questions are open-ended.","Visual reasoning can be included in all of the aforementioned tasks. Visual reasoning methods for VQA are reviewed in the end.","Detailed task specifications, data sets, and selected work for each task will be introduced in this section."]},{"idx":23,"heading":"Applications ::: Caption Generation","paragraphs":["Image captioning BIBREF174 requires to generate a description of an image and is one of the earliest task that studies multi-modal combination of image and text. We mainly review the deep learning based methods for caption generation. Image captioning, such as BIBREF85, BIBREF175, BIBREF39, divide the task into several sub-tasks and generate caption in a step-by-step manner. Authors in BIBREF85 first trained a deep CNN model to detect the words from images, then built a log-linear language model to compose the words into sentences. Similarly, BIBREF175 fed the image feature into a log-linear language model to generate sentences. In contrast, BIBREF39 tried to find the exact matching of objects in images and words in sentences to determine if an image and a sentence match with each other.","Similar to the RNN-based encoder-decoder methods for machine translation BIBREF26, BIBREF176, BIBREF177, BIBREF178 propose to generate captions from images in an end-to-end manner via the encoder-decoder architecture. In those models, a CNN, typically pre-trained on ImageNet BIBREF69, encoded the image into a continuous vector, which is then fed into a RNN/LSTM decoder to generate the caption directly. Those works all followed the same architecture, but varied slightly the choice of CNN architecture and how the image vector was fed into the decoder. Though powerful and convenient, the encoder-decoder architecture lacks to ability capture the fine grained relationship between objects in images and words in sentences. To overcome this, attention-based encoder-decoder model BIBREF179 was proposed and has become the standard benchmark for this task since then. In the attention encoder-decoder model, before generating the next word, the decoder first calculates the matching score (attention) with objects in the image, then conditions on the weighted image feature to generate the next token. There has been lots of work that tried to improve the attention model by incorporating more structures. For example, BIBREF180 added a gate at every decoding step to determine if the next word should be generated using image information; BIBREF181 combined detected words and image features as inputs to the decoder network. More recently, there has been a lot works that add more structure/knowledge from either image BIBREF137 or text side BIBREF182. Specifically, BIBREF137 used an object detector to localize the features for image object and then generates the caption based on the localized features. It improved the previous state of art model by a large margin in a variety of evaluation metrics.","Image captions with richer information could be generated when incorporated with external knowledge. For example, based on an model that can recognize celebrities BIBREF183, a CaptionBot app is developed which can not only describe the facts (such as activities) in a picture, but also describe who is doing that if the person in the picture is recognized BIBREF184. Further, beside simply generating a factual description of the image, other approaches were also proposed for explicitly controlling the style BIBREF185, semantic content BIBREF181, and diversity BIBREF186 of the generated caption."]},{"idx":24,"heading":"Applications ::: Text-to-Image Synthesis","paragraphs":["Text-to-image synthesis or generation that relies on natural language to control image generation, is a fundamental problem in computer vision. It is considered as a difficult problem since it least involves two tasks: high quality image generation and language understanding. The generated image is required to be both visually realistic and semantically consistent to the language description. Deep learning based text-to-image synthesis can perhaps be dated back to the use of LSTM for iterative hand-writting generation BIBREF187. This iterative image generation idea is later extended to form the deep recurrent attentive writer (DRAW) method that combines an LSTM based sequential variational auto-encoder (VAE) with a spatial attention mechanism BIBREF188. alignDRAW modifies DRAW to use natural language based descriptions to synthesis images with general content BIBREF189. An attention model is used to compute the alignment between the input words and the patches drawn iteratively."]},{"idx":25,"heading":"Applications ::: Text-to-Image Synthesis ::: GAN based methods","paragraphs":["Comparing to VAE, conditional-GAN (CGAN) is found to be able to synthesis highly compelling images of specific categories that a human might mistake for real BIBREF190, BIBREF191. A GAN model consists of a generator that synthesize candidates based on input noises and a discriminator that evaluates them. Adversarial training is employed to make the generator to capture the true data distribution so that the discriminator can no longer discriminate the synthesized data from the real ones BIBREF80. CGAN extends the standard GAN structure by conditioning on extra category labels for both generator and discriminator. GAN-INT-CLS allows to synthesize visually plausible 64$\\times $64 images using the embeddings of natural language descriptions to replace the category labels in CGAN BIBREF192. The automatic evaluation of the quality of text conditioned images can be less straightforward. To find the discriminability of GAN generated images, inception score (IS) BIBREF193 and Fréchet inception distance BIBREF194 (FID) are often used. Multi-scale structural similarity (MS-SSIM) BIBREF195 is commonly used to evaluate the diversity of images. To evaluate whether a generated image is semantically consistent with the input text description, R-precision BIBREF47 and visual-semantic similarity BIBREF196 are used as the metrics."]},{"idx":26,"heading":"Applications ::: Text-to-Image Synthesis ::: Generating high quality images","paragraphs":["Though basically reflecting the meaning of the descriptions, it is found the images produced by GAN-INT-CLS do not have necessary details and vivid object parts, and therefore leads to the StackGAN method BIBREF197. StackGAN decomposes image synthesize into more manageable sub-problems through a sketch-refinement process by stacking two CGANs trained separately. The first GAN produces 64$\\times $64 low-resolution images by sketching the primitive shape and colors of the object based on the text, and the second GAN is trained after to generate 256$\\times $256 images by rectifying defects and adding compelling details in the low-resolution image. StackGAN$++$ improves this idea by adding an extra GAN to generate 128$\\times $128 images in between and training all GANs jointlyBIBREF198. To ensure the generated image semantically match the text precisely, BIBREF47 proposed attentional GAN (AttnGAN), which also stacks three GANs for different image resolutions BIBREF47, and while the first GAN is conditioned on the sentence embedding, the next two GANs are conditioned on bimodal embeddings produced by attention models fusing word-level features with low-resolution images. It is shown attention mechanism can help GAN to focus on words that are most relevant to the sub-region drawn at each stage. Apart from stacking the generators, BIBREF199 shows that high resolution images can also be generated with a dynamic memory module."]},{"idx":27,"heading":"Applications ::: Text-to-Image Synthesis ::: Generating semantically consistent images","paragraphs":["To improve the semantic consistency between relevant image and text features, DAMSM is proposed for AttnGAN BIBREF47. BIBREF196 tackles the same problem by leveraging hierarchical representations with extra adversarial constraints to discriminate not only real/fake image pairs, but also real/fake image-text pairs at multiple image resolutions in the discriminator, and is named as hierarchically-nested discriminator GAN (HDGAN). Similarly, text conditioned auxiliary classifier GAN (TAC-GAN) introduces an extra image classification task to the discriminator BIBREF200, whereas text-conditioned semantic classifier GAN (Text-SeGAN) alternates the classifier with a regression task to estimate the semantic relevance between image and text BIBREF201. Analogous to cycle consistency BIBREF202, MirrorGAN is proposed to improve the semantic consistency between the two modalities using an extra image captioning module BIBREF203."]},{"idx":28,"heading":"Applications ::: Text-to-Image Synthesis ::: Semantic layout control for complex scenes","paragraphs":["With the success in the generation of realistic and semantically consistent images for single objects, such as birds BIBREF204 or flowers BIBREF205, state-of-the-art text-to-image synthesis methods still struggle to generate complex scenes with many objects and relationships, such as those in the Microsoft COCO data set BIBREF206. In the pioneering work BIBREF207, not only text descriptions but also locations of objects specified by keypoints or bounding boxes are used as the input. Later, detailed semantic layout, such as a scene graph, is used to replace the natural language sentence as a more direct description of objects and their relationships BIBREF208, BIBREF209, BIBREF210. Meanwhile, efforts are made to keep natural language input while incorporating the idea of semantic layout. BIBREF211 includes extra object pathway to both generator and discriminator to explicit control the object locations. BIBREF212 employs a two-stage procedure that first builds a semantic layout automatically from the input sentence with LSTM based box and shape generators, and then synthesizes the image using image generator and discriminators. Since fine-grained word/object level information is not explicitly used for generation, such synthesized images do not contain enough details to make them look realistic. The object-driven attentive GAN (Obj-GAN) improves the two-stage generation idea using a pair of object-driven attentive image generator and object-wise discriminator BIBREF139. At every generation step, the generator uses the text description as a semantic layout and synthesizes the image region within a bounding box by focusing on the words that are most relevant to the object in it. ObjGAN is found to be more robust and interpretable, and significantly improves the object generation quality for complex scenes."]},{"idx":29,"heading":"Applications ::: Text-to-Image Synthesis ::: Other topics","paragraphs":["In addition to the layout, other types of fine-grained control in image generation have also been studied in literature. Attribute2Image BIBREF213 studies the use of attributes in face generation, such as age and gender etc. BIBREF214 uses the same idea for face editing, such as to remove the beard or change the hair color. Text-adaptive GAN BIBREF215 allows semantic modification of input images for birds and flowers via natural language. BIBREF216 enforces to learn the representation content and style as two disentangled variables using a dual inference mechanism based on cycle-consistency for text-to-image synthesis. The success of these methods validate GAN is able to learn some semantic concepts as disentangled representations, as in Section SECREF9. Text2Scene is another interesting work that generates compositional scene representation from natural langauge step-by-step without using GANs BIBREF217. It is shown with minor modifications, Text2Scene can generate cartoon like, semantic layout, and real image like scenes. Dialogue based interaction is studied to control image synthesis, in order to improve complex scene generation progressively BIBREF218, BIBREF219, BIBREF220, BIBREF221, BIBREF222. Meanwhile, text-to-image synthesis is extended to multiple images or videos, where visual consistency is required among the generated images BIBREF223, BIBREF224, BIBREF225."]},{"idx":30,"heading":"Applications ::: Visual Question Answering ::: Task definition","paragraphs":["VQA extends text-based QA from NLP by asking questions related to the visual information presented in an image or a video clip. For image based VQA, it is often considered as a visual Turing test, in which the system is required to understand any form of natural language-based questions and to answer them in a natural way. However, it is often simplified as a classification task defined in different ways to focus on the core problem BIBREF131, BIBREF226, BIBREF227, BIBREF42, BIBREF43. Initial works generated the questions using templates or by converting from description sentences using syntax trees BIBREF226, BIBREF228. Later later studies focus on the use of free-form natural language questions authored either by human or powerful deep generative models, such as GAN and VAE BIBREF43, BIBREF228, BIBREF229, BIBREF230. Different from the open-ended questions presented in complete sentence form, possible answers are often presented as a large set of classes (e.g. 3000) related to yes/no, counts, object classes and instances etc. To focus on the core understanding and reasoning problems, VQA can be simplified as to classify visual and textual features into the answer related classes.","Alternatively, VQA can be defined as to select among multiple (e.g. 4) choices, and each choice is associated with each answer presented in the form of a natural language sentence BIBREF131. This setup can be implemented as a classification to the choices based on features of the image, question, and answer candidates BIBREF153. There exist other types of VQA task definitions, such as the Visual Madlibs dataset that requires to answer the questions by “fill-in-the-blanks” BIBREF44. Furthermore, visual dialogue can be viewed as to answer a sequence of questions grounded in an image BIBREF231, BIBREF232, which extends VQA by requiring to generate more human like responses and to infer the context from the dialogue history."]},{"idx":31,"heading":"Applications ::: Visual Question Answering ::: Common data sets and approaches","paragraphs":["The first VQA data set, DAQUAR, uses real-world images with both template-based and human annotated questions BIBREF226. COCO-QA has more QA pairs than DAQUAR by converting image descriptions from the MS COCO data set into questions BIBREF228. Such questions are in general easier since they allow the model to rely more on the rough image rather than logical reasoning. VQA v1 and v2 are the most popular data sets for VQA consisting of open-ended questions and both real and abstract scenes BIBREF43, BIBREF233. A VQA Challenge based on these data sets is held annually as a CVPR workshop since 2016. Visual7W is a part of the Visual Genome data set for VQA with multiple choices BIBREF131. It contains questions related to “what”, “who”, and “how” for spatial reasoning, and “where”, “when”, and “why” questions for high-level common-sense reasoning. The 7th type of the questions in Visual7W are the “which” questions, which are also termed as the pointing questions, whose answer choices are associated with bounding boxes of objects in the image. Approaches designed for these data sets often focus on fusing image and question vectors with the previously discussed attention- and bilinear pooling-based methods, such as SAN, co-attention, Up-Down, MCB, MLB, and BAN etc."]},{"idx":32,"heading":"Applications ::: Visual Question Answering ::: Integrating external knowledge source","paragraphs":["Since most of the VQA questions in these data sets are about simple counting, colors, and object detections that do not need any external knowledge to answer, a further development of the task is to include more difficult questions that require knowing more than what the questions entail or what information is contained in the images. Both knowledge-based reasoning for VQA (KB-VQA) and fact-based VQA (FVQA) data sets incorporate structured knowledge base, which often requires extra steps to query the knowledge base that makes the method no longer an end-to-end trainable approach BIBREF234, BIBREF235. Different from the structured knowledge bases, outside knowledge VQA (OK-VQA) uses external knowledge in the form of natural language sentences collected by retrieving Wikipedia articles with search queries extracted from the question, and an extra ArticleNet model is trained to find the answers from the retrieved articles BIBREF236."]},{"idx":33,"heading":"Applications ::: Visual Question Answering ::: Discounting language priors","paragraphs":["Though significant achievements have been made, recent studies point out that the common VQA benchmarks suffer from strong and prevalent priors – most bananas are yellow and mostly the sky is blue, which can often cause the VQA model to over-fit to these statistical biases and tendencies from the answer distributions, and largely circumvent the need to understand the visual scenes. Based on the objects, attributes, and relations provided through the scene graphs from Visual Genome, a new data set, GQA, was created to greatly reduce such biases by generating questions with a functional program that controls the reasoning steps BIBREF237. New splits for VQA v1 and VQA v2 are generated to have different answer distributions for every question of the training and test sets, which are referred to as VQA under challenging priors (VQA-CP v1 and VQA-CP v2) BIBREF238. Recent methods propose to handle the biased priors with adversarial training or additional train only structures BIBREF239, BIBREF240."]},{"idx":34,"heading":"Applications ::: Visual Question Answering ::: Other issues","paragraphs":["Another problem that current VQA methods suffers from is the low robustness against linguistic variations from the questions. A data set, VQA-Rephrasings, modified the VQA v2.0 validation set with human authored rephrasing of the questions BIBREF202. A cycle-consistency BIBREF241 based method that improves the linguistic robustness by enforcing consistencies between the original and rephrased questions, and between the true answer and the answers predicted based on the original and rephrased questions. BIBREF242 suggests that attention mechanism can cause VQA models to suffer from counting the object proposals, and an extra model component was proposed as a solution. Moreover, it is the fact that the current VQA methods cannot even read text from images. A method is proposed to address this problem by fusing not text extracted from the image using optical character recognition BIBREF243. VizWiz is a goal oriented VQA data set collected by blind people taking possibly low quality pictures and asking questions in spoken English, which also include many text related questions BIBREF244."]},{"idx":35,"heading":"Applications ::: Visual Reasoning","paragraphs":["This section focuses on the study of a very interesting problem – visual reasoning, which is about how to conduct accurate, explicit, and expressive understanding and reasoning. Visual reasoning can involved by many language and vision based bimodal tasks, such as caption generation and text-to-image synthesis. However, in this section we mostly focus on the related methods for VQA as visual reasoning is particularly important when answering complicated questions. SAN is often considered as a pioneering work related to implicit visual reasoning since to its stacked structure can be viewed as to perform multiple reasoning steps. Shortly afterwards, feature-wise linear modulation (FiLM) is proposed to refine visual features iteratively using feature-wise affine transforms based on the scaling factors and bias values generated dynamically from the textual features BIBREF245. Multimodal relational network (MuRel) also has a structure with multiple MuRel cells based on bilinear pooling, which can be used iteratively BIBREF246."]},{"idx":36,"heading":"Applications ::: Visual Reasoning ::: Neural module network based methods","paragraphs":["Neural module network (NMN) is a method which composes a collection of jointly trained neural “modules” into a deep model for answering the question BIBREF247. A dependency parser first helps to convert the natural language question into a fixed and rule-based network layout, and specify both the set of modules used to answer the question and the connections between them. Then a deep model is assembled based on the layout to produce the prediction of the answers. SHAPES, a synthetic dataset consists of complex questions about simple arranges of ordered shapes, was also proposed to focus on the compositional phenomena of questions BIBREF247. A later study learns the model layout predictor jointly with the parameters of the modules by re-ranking a list of layout candidates using reinforcement learning, which is termed as dynamic NMN (D-NMN) BIBREF248. Modules such as “find” or “relate” operation uses attention models to focus on one or two regions of the input image and makes the forwarding of the assembled deep model similar to running a functional program BIBREF248. An end-to-end version of NMN (N2NMN) used an RNN question encoder to convert the input question into a layout policy without requiring the aid of a parser BIBREF249. The work is based on a more recent data set called compositional language and elementary visual reasoning diagnostics (CLEVR). As its name suggests, CLEVR is a synthetic diagnostic data set testing a range of visual reasoning abilities of objections and relationships with minimal biases and detailed annotations describing the kind of reasoning each question requires BIBREF250. Other implementations of NMN include the program generator and execution engine method (PG+EE) that shares generic design among some operations BIBREF251, stack-NMN that improves the parser and incorporates question features into the modules BIBREF252, and transparency by design network (TbD-net) that redesigns some modules from PG+EE to maintain the transparency of the reasoning procedure BIBREF253."]},{"idx":37,"heading":"Applications ::: Visual Reasoning ::: Other types of end-to-end reasoning methods","paragraphs":["Another end-to-end approach is the memory, attention, and composition (MAC) network that decomposes the question into a series of attention-based reasoning steps and perform each of them using a recurrent MAC cell that maintains a separation between the control and memory hidden states. Each hidden state is generated by an ANN model constructed based on attention and gating mechanisms BIBREF254. More recently, both deterministic symbolic programs and probabilistic symbolic models have been used as the execution engine for the generated programs to improve the transparency and data efficiency, which result in the neural-symbolic VQA (NS-VQA) and probabilistic neural-symbolic models (prob-NMN) respectively BIBREF255, BIBREF256. As an extension of NS-VQA, the neuro-symbolic concept learner (NS-CL) uses a neuro-symbolic reasoning module to execute programs on the scene representation. NS-CL can have its program generator, reasoning module, and visual perception components jointly trained in an end-to-end fashion without requiring any component-level supervisions BIBREF257. Its perception module learns visual concepts based on the language descriptions of the objects and facilitates learning new words and parsing new sentences.","We finish this section by reviewing the relation networks (RN), which has a simple structure that uses an ANN as the function to model the relationship between any pair of visual and textual features, and the resulted output values are accumulated and transformed by another ANN BIBREF258. Though RN merely models the relationship without any form of induction reasoning, it achieves very high VQA accuracy on CLEVR. This inspires a re-thinking of the connection between correlation and induction."]},{"idx":38,"heading":"Conclusion","paragraphs":["This paper reviews the area of modeling and machine learning across multiple modalities based on deep learning, particularly the combination of vision and natural language. In particular, we propose to organize the many pieces of work in the language-vision multimodal intelligence field from three aspects, which include multimodal representations, the fusion of multimodal signals, and the applications of multimodal intelligence. In the section of representations, both single modal and multimodal representations are reviewed under the key concept of embedding. The multimodal representation unifies the involved signals of different modalities into the same vector space for general downstream tasks. On multimodal fusion, special architectures, such as attention mechanism and bilinear pooling, are discussed. In the application section, three selected areas of broad interest are presented, which include image caption generation, text-to-image synthesis, and visual question answering. A set of visual reasoning methods for VQA is also discussed. Our review covers task definition, data set specification, development of commonly used methods, as well as issues and trends, and therefore can facilitate future studies in this emerging field of multimodal intelligence for our community.","Chao Zhang is an advisor of JD.com speech team, and a research associate in speech and natural language processing at the University of Cambridge. He received his B.E. and M.S. degrees in 2009 and 2012 respectively, both from the Department of Computer Science & Technology, Tsinghua University, and a Ph.D. degree in 2017 from Cambridge University Engineering Department.","Zichao Yang is a quantitative researcher at Citadel. Prior to that, he received his Phd in computer science from Carnegie Mellon University. His research interests are in machine learning, deep learning and their applications in natural language processing and computer vision. He has published dozens of papers in NeurIPS, ICML, CVPR, ICCV, EMNLP, NAACL etc.","Xiaodong He (IEEE Member 2003, Senior member 2008, Fellow 2019) is the Deputy Managing Director of JD AI Research, and Head of the Deep learning, NLP and Speech Lab. He is also Affiliate Professor of ECE at the University of Washington (Seattle). His research interests are mainly in deep learning, natural language processing, speech recognition, computer vision, information retrieval, and multimodal intelligence. He has held editorial positions on multiple IEEE Journals and the Transactions of the ACL, and served in the organizing committee/program committee of major speech and language processing conferences. He is a member of the IEEE SLTC for the term of 2015-2017 and the Chair of the IEEE Seattle Section in 2016. He received the Bachelor degree from Tsinghua University in 1996, MS degree from Chinese Academy of Sciences in 1999, and the PhD degree from the University of Missouri – Columbia in 2003.","Li Deng has been the Chief Artificial Intelligence Officer of Citadel since May 2017. Prior to Citadel, he was the Chief Scientist of AI, the founder of the Deep Learning Technology Center, and Partner Research Manager at Microsoft and Microsoft Research , Redmond (2000-2017). Prior to Microsoft, he was an assistant professor (1989-1992), tenured associate (1992-1996), and full professor (1996-1999) at the University of Waterloo in Ontario, Canada. He also held faculty or research positions at Massachusetts Institute of Technology (Cambridge, 1992-1993), Advanced Telecommunications Research Institute (ATR, Kyoto, Japan, 1997-1998), and HK University of Science and Technology (Hong Kong, 1995). He is a Fellow of the Academy of Engineering of Canada, a Fellow of the Washington State Academy of Sciences, a Fellow of the IEEE, a Fellow of the Acoustical Society of America, and a Fellow of the International Speech Communication Association. He has also been an Affiliate Professor at the University of Washington, Seattle.","He was an elected member of the Board of Governors of the IEEE Signal Processing Society, and was Editors-in-Chief of IEEE Signal Processing Magazine and of IEEE/ACM Transactions on Audio, Speech, and Language Processing (2008-2014), for which he received the IEEE SPS Meritorious Service Award. In recognition of the pioneering work on disrupting speech recognition industry using large-scale deep learning, he received the 2015 IEEE SPS Technical Achievement Award for “Outstanding Contributions to Automatic Speech Recognition and to Deep Learning”. He also received dozens of best paper and patent awards for the contributions to artificial intelligence, machine learning, information retrieval, multimedia signal processing, speech processing and recognition, and human language technology. He is an author or co-author of six technical books on deep learning, speech processing, pattern recognition and machine learning, and, the latest, natural language processing (Springer, June 2018)."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1911.04474","title":"TENER: Adapting Transformer Encoder for Named Entity Recognition","sections":[{"idx":0,"heading":"Abstract","paragraphs":["The Bidirectional long short-term memory networks (BiLSTM) have been widely used as an encoder in models solving the named entity recognition (NER) task. Recently, the Transformer is broadly adopted in various Natural Language Processing (NLP) tasks owing to its parallelism and advantageous performance. Nevertheless, the performance of the Transformer in NER is not as good as it is in other NLP tasks. In this paper, we propose TENER, a NER architecture adopting adapted Transformer Encoder to model the character-level features and word-level features. By incorporating the direction and relative distance aware attention and the un-scaled attention, we prove the Transformer-like encoder is just as effective for NER as other NLP tasks."]},{"idx":1,"heading":"Introduction","paragraphs":["The named entity recognition (NER) is the task of finding the start and end of an entity in a sentence and assigning a class for this entity. NER has been widely studied in the field of natural language processing (NLP) because of its potential assistance in question generation BIBREF0, relation extraction BIBREF1, and coreference resolution BIBREF2. Since BIBREF3, various neural models have been introduced to avoid hand-crafted features BIBREF4, BIBREF5, BIBREF6.","NER is usually viewed as a sequence labeling task, the neural models usually contain three components: word embedding layer, context encoder layer, and decoder layer BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10. The difference between various NER models mainly lies in the variance in these components.","Recurrent Neural Networks (RNNs) are widely employed in NLP tasks due to its sequential characteristic, which is aligned well with language. Specifically, bidirectional long short-term memory networks (BiLSTM) BIBREF11 is one of the most widely used RNN structures. BIBREF4 was the first one to apply the BiLSTM and Conditional Random Fields (CRF) BIBREF12 to sequence labeling tasks. Owing to BiLSTM's high power to learn the contextual representation of words, it has been adopted by the majority of NER models as the encoder BIBREF5, BIBREF6, BIBREF9, BIBREF10.","Recently, Transformer BIBREF13 began to prevail in various NLP tasks, like machine translation BIBREF13, language modeling BIBREF14, and pretraining models BIBREF15. The Transformer encoder adopts a fully-connected self-attention structure to model the long-range context, which is the weakness of RNNs. Moreover, Transformer has better parallelism ability than RNNs. However, in the NER task, Transformer encoder has been reported to perform poorly BIBREF16, our experiments also confirm this result. Therefore, it is intriguing to explore the reason why Transformer does not work well in NER task.","In this paper, we analyze the properties of Transformer and propose two specific improvements for NER.","The first is that the sinusoidal position embedding used in the vanilla Transformer is aware of distance but unaware of the directionality. In addition, this property will lose when used in the vanilla Transformer. However, both the direction and distance information are important in the NER task. For example in Fig FIGREF3, words after “in\" are more likely to be a location or time than words before it, and words before “Inc.\" are mostly likely to be of the entity type “ORG\". Besides, an entity is a continuous span of words. Therefore, the awareness of distance might help the word better recognizes its neighbor. To endow the Transformer with the ability of direction- and distance-awareness, we adopt the relative positional encoding BIBREF17, BIBREF18, BIBREF19. instead of the absolute position encoding. We propose a revised relative positional encoding that uses fewer parameters and performs better.","The second is an empirical finding. The attention distribution of the vanilla Transformer is scaled and smooth. But for NER, a sparse attention is suitable since not all words are necessary to be attended. Given a current word, a few contextual words are enough to judge its label. The smooth attention could include some noisy information. Therefore, we abandon the scale factor of dot-production attention and use an un-scaled and sharp attention.","With the above improvements, we can greatly boost the performance of Transformer encoder for NER.","Other than only using Transformer to model the word-level context, we also tried to apply it as a character encoder to model word representation with character-level information. The previous work has proved that character encoder is necessary to capture the character-level features and alleviate the out-of-vocabulary (OOV) problem BIBREF6, BIBREF5, BIBREF7, BIBREF20. In NER, CNN is commonly used as the character encoder. However, we argue that CNN is also not perfect for representing character-level information, because the receptive field of CNN is limited, and the kernel size of the CNN character encoder is usually 3, which means it cannot correctly recognize 2-gram or 4-gram patterns. Although we can deliberately design different kernels, CNN still cannot solve patterns with discontinuous characters, such as “un..ily” in “unhappily\" and “unnecessarily\". Instead, the Transformer-based character encoder shall not only fully make use of the concurrence power of GPUs, but also have the potentiality to recognize different n-grams and even discontinuous patterns. Therefore, in this paper, we also try to use Transformer as the character encoder, and we compare four kinds of character encoders.","In summary, to improve the performance of the Transformer-based model in the NER task, we explicitly utilize the directional relative positional encoding, reduce the number of parameters and sharp the attention distribution. After the adaptation, the performance raises a lot, making our model even performs better than BiLSTM based models. Furthermore, in the six NER datasets, we achieve state-of-the-art performance among models without considering the pre-trained language models or designed features."]},{"idx":2,"heading":"Related Work ::: Neural Architecture for NER","paragraphs":["BIBREF3 utilized the Multi-Layer Perceptron (MLP) and CNN to avoid using task-specific features to tackle different sequence labeling tasks, such as Chunking, Part-of-Speech (POS) and NER. In BIBREF4, BiLSTM-CRF was introduced to solve sequence labeling questions. Since then, the BiLSTM has been extensively used in the field of NER BIBREF7, BIBREF21, BIBREF22, BIBREF5.","Despite BiLSTM's great success in the NER task, it has to compute token representations one by one, which massively hinders full exploitation of GPU's parallelism. Therefore, CNN has been proposed by BIBREF23, BIBREF24 to encode words concurrently. In order to enlarge the receptive field of CNNs, BIBREF23 used iterative dilated CNNs (ID-CNN).","Since the word shape information, such as the capitalization and n-gram, is important in recognizing named entities, CNN and BiLSTM have been used to extract character-level information BIBREF7, BIBREF6, BIBREF5, BIBREF23, BIBREF8.","Almost all neural-based NER models used pre-trained word embeddings, like Word2vec and Glove BIBREF25, BIBREF26. And when contextual word embeddings are combined, the performance of NER models will boost a lot BIBREF27, BIBREF28, BIBREF29. ELMo introduced by BIBREF28 used the CNN character encoder and BiLSTM language models to get contextualized word representations. Except for the BiLSTM based pre-trained models, BERT was based on Transformer BIBREF15."]},{"idx":3,"heading":"Related Work ::: Transformer","paragraphs":["Transformer was introduced by BIBREF13, which was mainly based on self-attention. It achieved great success in various NLP tasks. Since the self-attention mechanism used in the Transformer is unaware of positions, to avoid this shortage, position embeddings were used BIBREF13, BIBREF15. Instead of using the sinusoidal position embedding BIBREF13 and learned absolute position embedding, BIBREF17 argued that the distance between two tokens should be considered when calculating their attention score. BIBREF18 reduced the computation complexity of relative positional encoding from $O(l^2d)$ to $O(ld)$, where $l$ is the length of sequences and $d$ is the hidden size. BIBREF19 derived a new form of relative positional encodings, so that the relative relation could be better considered."]},{"idx":4,"heading":"Related Work ::: Transformer ::: Transformer Encoder Architecture","paragraphs":["We first introduce the Transformer encoder proposed in BIBREF13. The Transformer encoder takes in an matrix $H \\in \\mathbb {R}^{l \\times d}$, where $l$ is the sequence length, $d$ is the input dimension. Then three learnable matrix $W_q$, $W_k$, $W_v$ are used to project $H$ into different spaces. Usually, the matrix size of the three matrix are all $\\mathbb {R}^{d \\times d_k}$, where $d_k$ is a hyper-parameter. After that, the scaled dot-product attention can be calculated by the following equations,","where $Q_t$ is the query vector of the $t$th token, $j$ is the token the $t$th token attends. $K_j$ is the key vector representation of the $j$th token. The softmax is along the last dimension. Instead of using one group of $W_q$, $W_k$, $W_v$, using several groups will enhance the ability of self-attention. When several groups are used, it is called multi-head self-attention, the calculation can be formulated as follows,","where $n$ is the number of heads, the superscript $h$ represents the head index. $[head^{(1)}; ...; head^{(n)}]$ means concatenation in the last dimension. Usually $d_k \\times n = d$, which means the output of $[head^{(1)}; ...; head^{(n)}]$ will be of size $\\mathbb {R}^{l \\times d}$. $W_o$ is a learnable parameter, which is of size $\\mathbb {R}^{d \\times d}$.","The output of the multi-head attention will be further processed by the position-wise feed-forward networks, which can be represented as follows,","where $W_1$, $W_2$, $b_1$, $b_2$ are learnable parameters, and $W_1 \\in \\mathbb {R}^{d \\times d_{ff}}$, $W_2 \\in \\mathbb {R}^{d_{ff} \\times d}$, $b_1 \\in \\mathbb {R}^{d_{ff}}$, $b_2 \\in \\mathbb {R}^{d}$. $d_{ff}$ is a hyper-parameter. Other components of the Transformer encoder includes layer normalization and Residual connection, we use them the same as BIBREF13."]},{"idx":5,"heading":"Related Work ::: Transformer ::: Position Embedding","paragraphs":["The self-attention is not aware of the positions of different tokens, making it unable to capture the sequential characteristic of languages. In order to solve this problem, BIBREF13 suggested to use position embeddings generated by sinusoids of varying frequency. The $t$th token's position embedding can be represented by the following equations","where $i$ is in the range of $[0, \\frac{d}{2}]$, $d$ is the input dimension. This sinusoid based position embedding makes Transformer have an ability to model the position of a token and the distance of each two tokens. For any fixed offset $k$, $PE_{t+k}$ can be represented by a linear transformation of $PE_{t}$ BIBREF13."]},{"idx":6,"heading":"Proposed Model","paragraphs":["In this paper, we utilize the Transformer encoder to model the long-range and complicated interactions of sentence for NER. The structure of proposed model is shown in Fig FIGREF12. We detail each parts in the following sections."]},{"idx":7,"heading":"Proposed Model ::: Embedding Layer","paragraphs":["To alleviate the problems of data sparsity and out-of-vocabulary (OOV), most NER models adopted the CNN character encoder BIBREF5, BIBREF30, BIBREF8 to represent words. Compared to BiLSTM based character encoder BIBREF6, BIBREF31, CNN is more efficient. Since Transformer can also fully exploit the GPU's parallelism, it is interesting to use Transformer as the character encoder. A potential benefit of Transformer-based character encoder is to extract different n-grams and even uncontinuous character patterns, like “un..ily” in “unhappily” and “uneasily”. For the model's uniformity, we use the “adapted Transformer” to represent the Transformer introduced in next subsection.","The final word embedding is the concatenation of the character features extracted by the character encoder and the pre-trained word embeddings."]},{"idx":8,"heading":"Proposed Model ::: Encoding Layer with Adapted Transformer","paragraphs":["Although Transformer encoder has potential advantage in modeling long-range context, it is not working well for NER task. In this paper, we propose an adapted Transformer for NER task with two improvements."]},{"idx":9,"heading":"Proposed Model ::: Encoding Layer with Adapted Transformer ::: Direction- and Distance-Aware Attention","paragraphs":["Inspired by the success of BiLSTM in NER tasks, we consider what properties the Transformer lacks compared to BiLSTM-based models. One observation is that BiLSTM can discriminatively collect the context information of a token from its left and right sides. But it is not easy for the Transformer to distinguish which side the context information comes from.","Although the dot product between two sinusoidal position embeddings is able to reflect their distance, it lacks directionality and this property will be broken by the vanilla Transformer attention. To illustrate this, we first prove two properties of the sinusoidal position embeddings.","Property 1 For an offset $k$ and a position $t$, $PE_{t+k}^TPE_{t}$ only depends on $k$, which means the dot product of two sinusoidal position embeddings can reflect the distance between two tokens.","Based on the definitions of Eq.(DISPLAY_FORM11) and Eq.(), the position embedding of $t$-th token is PEt = [ c (c0t)","(c0t)","$\\vdots $","(cd2-1t)","(cd2-1t)","], where $d$ is the dimension of the position embedding, $c_i$ is a constant decided by $i$, and its value is $1/10000^{2i/d}$.","Therefore,","where Eq.(DISPLAY_FORM17) to Eq.() is based on the equation $\\cos (x-y) = \\sin (x)\\sin (y) + \\cos (x)\\cos (y)$.","Property 2 For an offset $k$ and a position $t$, $PE_{t}^TPE_{t-k}=PE_{t}^TPE_{t+k}$, which means the sinusoidal position embeddings is unware of directionality.","Let $j=t-k$, according to property 1, we have","The relation between $d$, $k$ and $PE_t^TPE_{t+k}$ is displayed in Fig FIGREF18. The sinusoidal position embeddings are distance-aware but lacks directionality.","However, the property of distance-awareness also disappears when $PE_t$ is projected into the query and key space of self-attention. Since in vanilla Transformer the calculation between $PE_t$ and $PE_{t+k}$ is actually $PE_t^TW_q^TW_kPE_{t+k}$, where $W_q, W_k$ are parameters in Eq.(DISPLAY_FORM7). Mathematically, it can be viewed as $PE_t^TWPE_{t+k}$ with only one parameter $W$. The relation between $PE_t^TPE_{t+k}$ and $PE_t^TWPE_{t+k}$ is depicted in Fig FIGREF19.","Therefore, to improve the Transformer with direction- and distance-aware characteristic, we calculate the attention scores using the equations below:","where $t$ is index of the target token, $j$ is the index of the context token, $Q_t, K_j$ is the query vector and key vector of token $t, j$ respectively, $W_q, W_v \\in \\mathbb {R}^{d \\times d_k}$. To get $H_{d_k}\\in \\mathbb {R}^{l \\times d_k}$, we first split $H$ into $d/d_k$ partitions in the second dimension, then for each head we use one partition. $\\mathbf {u} \\in \\mathbb {R}^{d_k}$, $\\mathbf {v} \\in \\mathbb {R}^{d_k}$ are learnable parameters, $R_{t-j}$ is the relative positional encoding, and $R_{t-j} \\in \\mathbb {R}^{d_k}$, $i$ in Eq.() is in the range $[0, \\frac{d_k}{2}]$. $Q_t^TK_j$ in Eq.() is the attention score between two tokens; $Q_t^TR_{t-j}$ is the $t$th token's bias on certain relative distance; $u^TK_j$ is the bias on the $j$th token; $v^TR_{t-j}$ is the bias term for certain distance and direction.","Based on Eq.(), we have","because $\\sin (-x)=-\\sin (x), \\cos (x)=\\cos (-x)$. This means for an offset $t$, the forward and backward relative positional encoding are the same with respect to the $\\cos (c_it)$ terms, but is the opposite with respect to the $\\sin (c_it)$ terms. Therefore, by using $R_{t-j}$, the attention score can distinguish different directions and distances.","The above improvement is based on the work BIBREF17, BIBREF19. Since the size of NER datasets is usually small, we avoid direct multiplication of two learnable parameters, because they can be represented by one learnable parameter. Therefore we do not use $W_k$ in Eq.(DISPLAY_FORM22). The multi-head version is the same as Eq.(DISPLAY_FORM8), but we discard $W_o$ since it is directly multiplied by $W_1$ in Eq.(DISPLAY_FORM9)."]},{"idx":10,"heading":"Proposed Model ::: Encoding Layer with Adapted Transformer ::: Un-scaled Dot-Product Attention","paragraphs":["The vanilla Transformer use the scaled dot-product attention to smooth the output of softmax function. In Eq.(), the dot product of key and value matrices is divided by the scaling factor $\\sqrt{d_k}$.","We empirically found that models perform better without the scaling factor $\\sqrt{d_k}$. We presume this is because without the scaling factor the attention will be sharper. And the sharper attention might be beneficial in the NER task since only few words in the sentence are named entities."]},{"idx":11,"heading":"Proposed Model ::: CRF Layer","paragraphs":["In order to take advantage of dependency between different tags, the Conditional Random Field (CRF) was used in all of our models. Given a sequence $\\mathbf {s}=[s_1, s_2, ..., s_T]$, the corresponding golden label sequence is $\\mathbf {y}=[y_1, y_2, ..., y_T]$, and $\\mathbf {Y}(\\mathbf {s})$ represents all valid label sequences. The probability of $\\mathbf {y}$ is calculated by the following equation","where $f(\\mathbf {y}_{t-1},\\mathbf {y}_t,\\mathbf {s})$ computes the transition score from $\\mathbf {y}_{t-1}$ to $\\mathbf {y}_t$ and the score for $\\mathbf {y}_t$. The optimization target is to maximize $P(\\mathbf {y}|\\mathbf {s})$. When decoding, the Viterbi Algorithm is used to find the path achieves the maximum probability."]},{"idx":12,"heading":"Experiment ::: Data","paragraphs":["We evaluate our model in two English NER datasets and four Chinese NER datasets.","(1) CoNLL2003 is one of the most evaluated English NER datasets, which contains four different named entities: PERSON, LOCATION, ORGANIZATION, and MISC BIBREF34.","(2) OntoNotes 5.0 is an English NER dataset whose corpus comes from different domains, such as telephone conversation, newswire. We exclude the New Testaments portion since there is no named entity in it BIBREF8, BIBREF7. This dataset has eleven entity names and seven value types, like CARDINAL, MONEY, LOC.","(3) BIBREF35 released OntoNotes 4.0. In this paper, we use the Chinese part. We adopted the same pre-process as BIBREF36.","(4) The corpus of the Chinese NER dataset MSRA came from news domain BIBREF37.","(5) Weibo NER was built based on text in Chinese social media Sina Weibo BIBREF38, and it contained 4 kinds of entities.","(6) Resume NER was annotated by BIBREF33.","Their statistics are listed in Table TABREF28. For all datasets, we replace all digits with “0”, and use the BIOES tag schema. For English, we use the Glove 100d pre-trained embedding BIBREF25. For the character encoder, we use 30d randomly initialized character embeddings. More details on models' hyper-parameters can be found in the supplementary material. For Chinese, we used the character embedding and bigram embedding released by BIBREF33. All pre-trained embeddings are finetuned during training. In order to reduce the impact of randomness, we ran all of our experiments at least three times, and its average F1 score and standard deviation are reported.","We used random-search to find the optimal hyper-parameters, hyper-parameters and their ranges are displayed in the supplemental material. We use SGD and 0.9 momentum to optimize the model. We run 100 epochs and each batch has 16 samples. During the optimization, we use the triangle learning rate BIBREF39 where the learning rate rises to the pre-set learning rate at the first 1% steps and decreases to 0 in the left 99% steps. The model achieves the highest development performance was used to evaluate the test set. The hyper-parameter search range and other settings can be found in the supplementary material. Codes are available at https://github.com/fastnlp/TENER."]},{"idx":13,"heading":"Experiment ::: Results on Chinese NER Datasets","paragraphs":["We first present our results in the four Chinese NER datasets. Since Chinese NER is directly based on the characters, it is more straightforward to show the abilities of different models without considering the influence of word representation.","As shown in Table TABREF29, the vanilla Transformer does not perform well and is worse than the BiLSTM and CNN based models. However, when relative positional encoding combined, the performance was enhanced greatly, resulting in better results than the BiLSTM and CNN in all datasets. The number of training examples of the Weibo dataset is tiny, therefore the performance of the Transformer is abysmal, which is as expected since the Transformer is data-hungry. Nevertheless, when enhanced with the relative positional encoding and unscaled attention, it can achieve even better performance than the BiLSTM-based model. The superior performance of the adapted Transformer in four datasets ranging from small datasets to big datasets depicts that the adapted Transformer is more robust to the number of training examples than the vanilla Transformer. As the last line of Table TABREF29 depicts, the scaled attention will deteriorate the performance."]},{"idx":14,"heading":"Experiment ::: Results on English NER datasets","paragraphs":["The comparison between different NER models on English NER datasets is shown in Table TABREF32. The poor performance of the Transformer in the NER datasets was also reported by BIBREF16. Although performance of the Transformer is higher than BIBREF16, it still lags behind the BiLSTM-based models BIBREF5. Nonetheless, the performance is massively enhanced by incorporating the relative positional encoding and unscaled attention into the Transformer. The adaptation not only makes the Transformer achieve superior performance than BiLSTM based models, but also unveil the new state-of-the-art performance in two NER datasets when only the Glove 100d embedding and CNN character embedding are used. The same deterioration of performance was observed when using the scaled attention. Besides, if ELMo was used BIBREF28, the performance of TENER can be further boosted as depicted in Table TABREF33."]},{"idx":15,"heading":"Experiment ::: Analysis of Different Character Encoders","paragraphs":["The character-level encoder has been widely used in the English NER task to alleviate the data sparsity and OOV problem in word representation. In this section, we cross different character-level encoders (BiLSTM, CNN, Transformer encoder and our adapted Transformer encoder (AdaTrans for short) ) and different word-level encoders (BiLSTM, ID-CNN and AdaTrans) to implement the NER task. Results on CoNLL2003 and OntoNotes 5.0 are presented in Table TABREF34 and Table TABREF34, respectively.","The ID-CNN encoder is from BIBREF23, and we re-implement their model in PyTorch. For different combinations, we use random search to find its best hyper-parameters. Hyper-parameters for character encoders were fixed. The details can be found in the supplementary material.","For the results on CoNLL2003 dataset which is depicted in Table TABREF34, the AdaTrans performs as good as the BiLSTM in different character encoder scenario averagely. In addition, from Table TABREF34, we can find the pattern that the AdaTrans character encoder outpaces the BiLSTM and CNN character encoders when different word-level encoders being used. Moreover, no matter what character encoder being used or none being used, the AdaTrans word-level encoder gets the best performance. This implies that when the number of training examples increases, the AdaTrans character-level and word-level encoder can better realize their ability."]},{"idx":16,"heading":"Experiment ::: Convergent Speed Comparison","paragraphs":["We compare the convergent speed of BiLSTM, ID-CNN, Transformer, and TENER in the development set of the OntoNotes 5.0. The curves are shown in Fig FIGREF37. TENER converges as fast as the BiLSTM model and outperforms the vanilla Transformer."]},{"idx":17,"heading":"Conclusion","paragraphs":["In this paper, we propose TENER, a model adopting Transformer Encoder with specific customizations for the NER task. Transformer Encoder has a powerful ability to capture the long-range context. In order to make the Transformer more suitable to the NER task, we introduce the direction-aware, distance-aware and un-scaled attention. Experiments in two English NER tasks and four Chinese NER tasks show that the performance can be massively increased. Under the same pre-trained embeddings and external knowledge, our proposed modification outperforms previous models in the six datasets. Meanwhile, we also found the adapted Transformer is suitable for being used as the English character encoder, because it has the potentiality to extract intricate patterns from characters. Experiments in two English NER datasets show that the adapted Transformer character encoder performs better than BiLSTM and CNN character encoders."]},{"idx":18,"heading":"Supplemental Material ::: Character Encoder","paragraphs":["We exploit four kinds of character encoders. For all character encoders, the randomly initialized character embeddings are 30d. The hidden size of BiLSTM used in the character encoder is 50d in each direction. The kernel size of CNN used in the character encoder is 3, and we used 30 kernels with stride 1. For Transformer and adapted Transformer, the number of heads is 3, and every head is 10d, the dropout rate is 0.15, the feed-forward dimension is 60. The Transformer used the sinusoid position embedding. The number of parameters for the character encoder (excluding character embedding) when using BiLSTM, CNN, Transformer and adapted Transformer are 35830, 3660, 8460 and 6600 respectively. For all experiments, the hyper-parameters of character encoders stay unchanged."]},{"idx":19,"heading":"Supplemental Material ::: Hyper-parameters","paragraphs":["The hyper-parameters and search ranges for different encoders are presented in Table TABREF40, Table TABREF41 and Table TABREF42."]},{"idx":20,"heading":"Figures and Tables","paragraphs":["Figure 1: An example for NER. The relative direction is important in the NER task, because words before “Inc.” are mostly to be an organization, words after “in” are more likely to be time or location. Besides, the distance between words is also important, since only continuous words can form an entity, the former “Louis Vuitton” can not form an entity with the “Inc.”.","Figure 2: Model structure of TENER for English NER tasks. In TENER, Transformer encoder is used not only to extract the word-level contextual information, but also to encode character-level information in a word.","Figure 3: Dot product between two sinusoidal position embeddings whose distance is k. It is clear that the product is symmetrical, and with the increment of |k|, it has a trend to decrease, but this decrease is not monotonous.","Figure 4: The upper line is the product between PETt PEt+k. The lower two lines are the products of PETt WPEt+k with two random W s. Although PETt PEt+k can reflect the distance, the PETt WPEt+k has no clear pattern.","Table 1: Details of Datasets.","Table 2: The F1 scores on Chinese NER datasets. ♣,♠ are results reported in (Zhang and Yang, 2018) and (Gui et al., 2019a), respectively. “w/ scale” means TENER using the scaled attention in Eq.(19). ∗ their results are not directly comparable with ours, since they used 100d pre-trained character and bigram embeddings. Other models use the same embeddings.","Table 4: The F1 scores on English NER datasets. We only list results based on non-contextualized embeddings, and methods utilized pre-trained language models, pre-trained features, or higher dimension word vectors are excluded. TENER (Ours) uses the Transformer encoder both in the character-level and wordlevel. “w/ scale” means TENER using the scaled attention in Eq.(19). “w/ CNN-char” means TENER using CNN as character encoder instead of AdaTrans.","Table 3: Performance of models with ELMo as their embeddings in English NER datasets. “BiLSTM” is our run. In the larger OntoNotes5.0, TENER achieves much better F1 score.","Table 5: F1 scores in the CoNLL2003 and OntoNotes 5.0. “Char” means character-level encoder, and “Word” means word-level encoder. “AdaTrans” means our adapted Transformer encoder.","Figure 5: Convergent speed in the development dataset of OntoNotes 5.0 for four kinds of models.","Table 6: The hyper-parameters and hyper-parameter search ranges for BiLSTM.","Table 7: The hyper-parameters and hyper-parameter search ranges for ID-CNN.","Table 8: The hyper-parameters and hyper-parameter search ranges for Transformer and adapted Transformer in Chinese and English NER datasets."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1911.09241","title":"Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Existing analysis work in machine reading comprehension (MRC) is largely concerned with evaluating the capabilities of systems. However, the capabilities of datasets are not assessed for benchmarking language understanding precisely. We propose a semi-automated, ablation-based methodology for this challenge; By checking whether questions can be solved even after removing features associated with a skill requisite for language understanding, we evaluate to what degree the questions do not require the skill. Experiments on 10 datasets (e.g., CoQA, SQuAD v2.0, and RACE) with a strong baseline model show that, for example, the relative scores of a baseline model provided with content words only and with shuffled sentence words in the context are on average 89.2% and 78.5% of the original score, respectively. These results suggest that most of the questions already answered correctly by the model do not necessarily require grammatical and complex reasoning. For precise benchmarking, MRC datasets will need to take extra care in their design to ensure that questions can correctly evaluate the intended skills."]},{"idx":1,"heading":"Introduction","paragraphs":["Machine reading comprehension (MRC) is a testbed for evaluating natural language understanding (NLU), by letting machines answer questions about given texts BIBREF1. Although MRC could be the most suitable task for evaluating NLU BIBREF2 and the performance of systems is comparable to humans on some existing datasets BIBREF3, it has been found that the quality of existing datasets might be insufficient for requiring precise understanding BIBREF4. Whereas these analyses are useful to investigate the performance of systems, however, it is still necessary to verify the fine-grained capabilities of datasets for benchmarking NLU.","In the design of MRC datasets, it is implicitly assumed that questions test a cognitive process of language understanding BIBREF5. As various aspects of such a process, we can use requisite skills for answering questions such as coreference resolution and commonsense reasoning BIBREF6. Considering skills as metrics would be useful for analyzing datasets. However, for most datasets, the skills required to answer existing questions are not identified, or significant human annotation is needed.","In this study, we propose a semi-automated, ablation-based methodology to analyze the capabilities of MRC datasets to benchmark NLU. Our motivation is to investigate to what extent a dataset allows unintended solutions that do not need requisite skills. This leads to the following intuition: if a question is correctly answered (or solvable) even after removing features associated with a given skill, the question does not require the skill. We show an example of our ablation method in Figure FIGREF1. Suppose we wish to analyze a dataset's capacity to evaluate understanding of texts beyond the information of part-of-speech (POS) tags. To this end, we replace context and question words with POS tags and ID numbers. If a model can still correctly answer this modified question, the question does not necessarily require deep understanding of texts but matching word patterns only. Questions of this kind might be insufficient for developing a model that understands texts deeply as they may reduce models to recognizing superficial word overlaps.","Our methodology uses a set of requisite skills and corresponding ablation methods. Inspired by the computational model of reading comprehension BIBREF7, we exemplify 12 skills on two classes: reading and reasoning (Section SECREF3). Then, we present a large-scale analysis over 10 existing datasets using a strong baseline model (Section SECREF4). In Section SECREF5, we perform a complementary inspection of questions with our ablation methods in terms of the solvability of questions and the reconstructability of ablated features. Finally we discuss, in Section SECREF6, two requirements for developing MRC to benchmark NLU: the control of question solvability and the comprehensiveness of requisite skills.","Our contributions are as follows:","We propose a semi-automated methodology to analyze the benchmarking capacity of MRC datasets in terms of requisite skills for answering questions.","With an example set of 12 skills and corresponding input-ablation methods, we use our methodology and examine 10 existing datasets with two answering styles.","Our analysis shows that the relative performance on questions with content words only, shuffled sentence words, and shuffled sentence order averaged 89.2%, 78.5%, and 95.4% of the original performance, indicating that the questions might be inadequate for evaluating grammatical and complex reasoning.","These results suggest that most of the questions currently solved in MRC may be insufficient for evaluating various skills. A limitation of our method is that it can not draw conclusions regarding questions that remain unsolved, and thus we need to assume a reasonable level of performance for existing models on the dataset to be analysed. Given our findings, we posit that MRC datasets should be carefully designed, e.g., by filtering questions using methods such as the ones we propose, so that their questions correctly benchmark the intended NLU skills."]},{"idx":2,"heading":"Related Work","paragraphs":["We briefly survey existing interpretation methods and skill-based analyses for NLU tasks.","Interpretation methods. A challenge with the MRC task is that we do not know the extent to which a successful model precisely understands natural language. To analyze a model's behavior, existing studies mainly proposed modification of the input. For example, BIBREF4 showed that the performance of existing models on SQuAD BIBREF0 significantly degrades when manually verified distracting sentences are added to the given context. In addition, BIBREF8 demonstrated that MRC models do not necessarily change their predictions even when most question tokens are dropped. Likewise, for the natural language inference task, BIBREF9 proposed to hide the premise and to evaluate a model using only the hypothesis. These kinds of analyses are helpful for detecting biases that are unintentionally included in datasets. Nonetheless, to assure that a dataset can evaluate various aspects of NLU, more fine-grained detail is needed than what is allowed by inspection using existing methods.","Skills as units of interpretation. In the topic of interpretable machine learning, BIBREF10 defined the concept of cognitive chunks as the basic units of explanation. In the MRC task, we consider that requisite skills to answer questions are appropriate as such units. A skill-based analysis was conducted by BIBREF11, who proposed classifications of knowledge and reasoning. Prior to this, BIBREF6 also defined a set of 13 requisite skills. However, there are two main issues with these approaches: (i) the human annotation does not necessarily reveal unintended biases that machines can make use of, and (ii) it requires costly annotation efforts. Therefore, we posit that a machine-based analysis is needed and that it should be performed in an automated manner."]},{"idx":3,"heading":"Dataset Diagnosis by Input Ablation ::: Formulation","paragraphs":["Our methodology uses a set of requisite skills and corresponding ablation methods. By checking the solvability of questions after applying the ablation methods, we can quantify to what degree the questions allow unintended solutions that do not require the requisite skills. Users can define an arbitrary set of skills to suit their purposes. We develop a method $\\sigma _i$ that ablates features necessary for the corresponding skill $s_i$ in a set of requisite skills $S$. For $(x, y) \\in X \\times Y$, whenever $f(x) = y$, if $f(\\sigma _i(x)) = y$, we recognize that $x$ is solvable without $s_i$.","Here, $X$ is the input, $Y$ is the gold labels, $(x, y)$ is a pair consisting of an input instance and its gold-standard answer, and $f$ is a model. When the performance gap between the original and the modified dataset is small, we can infer that most of the questions already solved are solvable without $s_i$. On the other hand, if the gap is large, a sizable proportion of the solved questions may require $s_i$.","We note that we cannot draw general conclusions for instances given by conditions other than the abovementioned one. Consider the case where $f(x) = y$ and $f(\\sigma _i(x)) \\ne y$, for example. This only means that $f$ cannot solve $x$ without the features ablated by $\\sigma _i$. We cannot conclude that $x$ requires $s_i$ in every model because there might exist a model that can solve $x$ without $s_i$. However, if there is at least one model $f$ that solves $x$ without $s_i$, this may indicate an unintended way to solve $x$ while ignoring $s_i$. Therefore our methodology only requires a single baseline model. Users can choose an arbitrary model for their purposes."]},{"idx":4,"heading":"Dataset Diagnosis by Input Ablation ::: Example Set of Requisite Skills","paragraphs":["In this section, we exemplify a skill set that consists of 12 skills along with two classes; reading and reasoning (Table TABREF5). In psychology, there is a tradition of theoretical research on human text comprehension. The construction–integration model BIBREF7 is one of the most acknowledged theories. This model assumes that human text comprehension consists of two processes: (i) construction, in which a reader elaborates concepts and propositions in the text and (ii) integration, in which the reader associates the propositions to understand them consistently. We associate this two-step process with our two classes. Reading skills. This class deals with six skills of observing and recognizing word appearances, which are performed before reasoning. In MRC, it has been shown that some existing questions can be solved by reading a limited number of words in the question and the context (e.g., by simply attending to context tokens that are similar to those of the questions BIBREF12). Our goal of this class is, therefore, to ensure that the questions require the reading of the whole question and context uniformly.","Reasoning skills. This class comprises six skills of relational reasoning among described entities and events such as pronoun coreference resolution and logical reasoning. Although these skills are essential for sophisticated NLU, it is difficult to precisely determine whether these types of reasoning are genuinely required in answering a question. Therefore, in this class, we define reasoning-related skills that are performed using the explicit information contained in the context (e.g., $s_9$ explicit logical reasoning and $s_{12}$ reasoning about explicit causality).","In the following, we highlight some of the defined skills. Skill $s_1$ is inspired by BIBREF8 and BIBREF12. Although their studies proposed dropping question tokens based on their model-based importance or the question length, we instead drop tokens other than interrogatives as interpretable features. Our vocabulary anonymization ($s_4$) is mainly inspired by BIBREF13 where they anonymized named entities to make their MRC task independent of prior knowledge. Our shuffle-based methods ($s_6$ to $s_8$) are inspired by existing analyses for other tasks BIBREF14, BIBREF15, BIBREF16. Among them, our purpose for $s_7$ is to analyze whether a question requires precise reasoning performed over syntactic and grammatical aspects in each sentence. The remaining skills are described in Appendix A.","Although our proposed definitions can be extended, they are sufficient for the purpose of demonstrating and evaluating our approach. In Section SECREF6, we discuss further directions to develop purpose-oriented skill sets."]},{"idx":5,"heading":"Experiments and Further Analyses ::: Experimental Settings","paragraphs":["Datasets. We use 10 datasets. For answer extraction datasets in which a reader chooses a text span in a given context, we use (1) CoQA BIBREF17, (2) DuoRC BIBREF18, (3) HotpotQA (distractor) BIBREF19, (4) SQuAD v1.1 BIBREF0, and (5) SQuAD v2.0 BIBREF20. For multiple choice datasets in which a reader chooses a correct option from multiple options, we use (6) ARC (Challenge) BIBREF21, (7) MCTest BIBREF22, (8) MultiRC BIBREF23, (9) RACE BIBREF24, and (10) SWAG BIBREF25. For the main analysis, we applied our ablation methods to development sets. We included SWAG because its formulation can be viewed as a multiple-choice MRC task and we would like to analyze the reasons for the high performance reported for the baseline model on this dataset BIBREF3. For preprocessing the datasets, we use CoreNLP BIBREF26. We specify further details in Appendix B.","Models. As the baseline model, we used BERT-large BIBREF3. We fine-tuned it on the original training set of each dataset and evaluated it on a modified development set. For $\\sigma _4$ vocabulary anonymization, we train the model after the anonymization. For ARC, MCTest, and MultiRC, we fine-tuned a model that had already been trained on RACE to see the performance gained by transfer learning BIBREF27. We report the hyperparameters of our models in Appendix C. Although we trained the baseline model on the original training set, it is assumed that the upper-bound performance can be achieved by a model trained on the modified training set. Therefore, in Section SECREF16, we also see the extent to which the performance improves when the model is trained on the modified training set.","Ablation methods. $\\sigma _2$ and $\\sigma _3$: we use a set of stopwords from NLTK BIBREF28 as function words. All other words are regarded as content words. We do not drop punctuation. When a token is dropped, it is replaced with an [UNK] token to preserve the correct answer span. $\\sigma _4$: we use the same ID for the same word in a single given context but different IDs for different contexts. For inflectional words, we anonymize them using their lemma. For example, are would be replaced with @verb2 (= is) if it appeared in Figure FIGREF1. In addition, to retain the information of the POS tags, we append its POS tag after each inflectional anonymized word (e.g., is is replaced with @verb{ID} [VBZ]). $\\sigma _6$: because it is necessary to maintain the correct answer span in the answer extraction datasets, we split the context into segments that have the same length as the gold answer span and shuffle them. $\\sigma _7$: as with $\\sigma _6$, we split each sentence into segments and shuffle them within each sentence. For $\\sigma _6$ to $\\sigma _8$, we averaged the scores over five runs with different seeds and report their variances in Appendix D."]},{"idx":6,"heading":"Experiments and Further Analyses ::: Results of Reading and Reasoning Skills","paragraphs":["We report the results for the skills in Table TABREF10. In the following, % indicates a relative change from the original F1/accuracy unless specified otherwise. In this section, we describe the notable findings for several skills. The observations for all other skills are explained in Appendix F.","$s_2$ and $s_3$: recognizing content words and function words. On all datasets, the relative changes for $s_2$ were greater than those for $s_3$. However, it is remarkable that even with function words alone, the model could achieve 53.0% and 17.4% F1 on CoQA and SQuAD v1.1, respectively. On ARC, RACE, and SWAG, the model showed more than 40% accuracy ($>$25% of random choice). As for content words only, on all answer extraction datasets, the performance was greater than 78.7% that of the original. On all multiple-choice datasets, it was more than 90.2%. These results imply that most of the questions already solved do not necessarily require grammatical and syntactic reasoning, in which function words are used.","$s_4$: recognizing vocabulary beyond POS tags. Surprisingly, for SQuAD v1.1, the baseline model achieved 61.2% F1. It only uses 248 tokens as the vocabulary with the anonymization tags and no other actual tokens. For the other answer extraction datasets, the largest drop (73.6% relative) is by HotpotQA; it has longer context documents than the other datasets, which seemingly makes its questions more difficult. To verify the effect of its longer documents, we also evaluated the baseline model on HotpotQA without distracting paragraphs. We found that the model's performance was 56.4% F1 (the original performance was 76.3% F1 and its relative drop was 26.1%) which is much higher than that on the context with distracting paragraphs (16.8% F1). This indicates that adding longer distracting documents contributes to encouraging machines to understand a given context beyond matching word patterns. On the other hand, the performance on the multiple choice datasets was significantly worse; if multiple choices do not have sufficient word overlap with the given context, there is no way to infer the correct answer option. Therefore, this result shows that multiple choice datasets might have a capacity for requiring more complex understanding beyond matching patterns between the question and the context than the answer extraction datasets.","$s_6$: recognizing the context word order (context words shuffle). We found that for the answer extraction datasets, the relative performance decreased by 55.6% on average. A moderate number of questions are solvable even with the context words shuffled. We also found that, surprisingly, the average decrease was 21.3% for the multiple choice datasets. The drop on MCTest is more prominent than that on the others. We posit that this is because its limited vocabulary makes questions more context dependent. ARC, in contrast, uses factoid texts, and appears less context dependent.","$s_7$: grasping sentence-level compositionality (sentence words shuffle). The performance with sentence words shuffled was greater than 60% and 80% those of the original dataset on the answer extraction and multiple-choice datasets, respectively. This result means that most of the solved questions are solvable even with the sentence words shuffled. However, we should not say that all questions must require this skill; a question can require the performance of some complex reasoning (e.g., logical and multi-hop reasoning) and merely need to identify the sentence that gives the correct answer without precisely understanding that sentence. Nevertheless, if the question is not intended to require such reasoning, we should care whether it can be solved with only a (sentence-level) bag of words. In order to ensure that a model can understand the precise meaning of a described event, we may need to include questions to evaluate the grammatical and syntactic understanding into a dataset. $s_8$: discourse relation understanding (sentence order shuffle). The smallest drop, excluding SWAG, which has one context sentence, was $-$1.3%, on SQuAD v1.1. Except for HotpotQA, the datasets show small drops (less than 10%), which indicates that most solved questions do not require understanding of adjacent discourse relations and are solvable even if the sentences appear in an unnatural order. For SQuAD v2.0, we observed that the model recall increases for the no-answer questions. Because F1 score is computed between the has- and no-answer question subsets, the scores tend to be higher than those for SQuAD v1.1."]},{"idx":7,"heading":"Experiments and Further Analyses ::: Further Analyses","paragraphs":["To complement the observations in Section SECREF11, we performed further experiments as follows.","The whole question and/or context ablation. To correctly interpret the result for $s_1$, we should know the performance on the empty questions. Likewise, for multiple-choice questions, the performance on the empty context should be investigated to reveal biases contained in the answer options. Therefore, we report the baseline results on the whole question and/or context ablations.","Our results are reported in Table TABREF17. Although the performance on SQuAD v2.0 was relatively high, we found that the model predicted no answer for all of the questions (in this dataset, almost half of the questions are no answer). The other answer extraction datasets showed a relative drop of 80–90%. This result is not surprising since this setting forces the model to choose an answer span arbitrarily. On the multiple-choice datasets, on the other hand, the accuracies were higher than those of random choice (50% for MultiRC and 25% for the others), which implies that some bias exists in the context and/or the options.","Training and evaluating on the modified context. A question that was raised during the main analysis is what would happen if the model was trained on the modified input. For example, given that the performance with the content words only is high, we would like to know the upper bound performance when the model is forced to ignore function words also during training. Hence we trained the model with the ablations for the following skills: $s_3$ content words only; $s_6$ context word shuffle; and $s_7$ sentence word shuffle. The results are reported in the bottom rows of Table TABREF17. On almost all datasets, the baseline model trained on the ablation training set ($s_3^{\\prime }$, $s_6^{\\prime }$, and $s_7^{\\prime }$) displayed higher scores than that on the original training set ($s_3$, $s_6$, and $s_7$). On CoQA, for instance, the relative change from the original score was only $-$8.3% when the model was trained on $s_3$ content words only. Although $s_3^{\\prime }$ and $s_7^{\\prime }$ with RACE were exceptions, their learning did not converge within the specified number of epochs. We observed that for all datasets the relative upper bounds of performance were on average 92.5%, 80.1%, and 91.8% for $s_3$, $s_6$, and $s_7$, respectively. These results support our observations in Section SECREF11, that is, the questions allow solutions that do not necessarily require these skills, and thus fall short of testing precise NLU. Even without tuning on the ablation training set, however, our methods can make an optimistic estimation of questions that are possibly dubious for evaluating intended skills.","Data leakage in BERT for SWAG. BERT's performance on SWAG is close to the performance by humans (88.0%). However, the questions and corresponding options for SWAG are generated by a language model trained on the BookCorpus BIBREF31, on which BERT's language model is also pretrained. We therefore suspect that there is severe data leakage in BERT's language model as reported in BIBREF32. To confirm this issue, we trained a model without the context (i.e., the first given sentence). The accuracy on the development set, which was also without the context, was 74.9% (a relative decrease of 12.2%). This result suggests that we need to pay more attention to the relations of corpora on which a model is trained and evaluated, but leave further analysis for future work."]},{"idx":8,"heading":"Qualitative Evaluation","paragraphs":["In this section, we qualitatively investigate our ablation methods in terms of the human solvability of questions and the reconstructability of ablated features.","We analyze questions of SQuAD v1.1 and RACE which cover both answering styles and are influential in the community. We randomly sampled 20 questions from each dataset that are correctly solved (100% F1 and accuracy) by the baseline model on the original datasets. Our analysis covers four ablation methods ($\\sigma _3$ content words only (involving $\\sigma _{10,11,12}$), $\\sigma _4$ vocabulary anonymization, $\\sigma _6$ context word shuffle, and $\\sigma _7$ sentence word shuffle) which provided specific insights in Section SECREF4."]},{"idx":9,"heading":"Qualitative Evaluation ::: Human Solvability after the Ablation","paragraphs":["Motivation. In Section SECREF4, we observed that the baseline model exhibits remarkably high performance on some ablation tests. To interpret this result, we investigate if a question is solvable by humans and the model. Concretely, the question after the ablation can be (A) solvable by both humans and the model, (B) solvable by humans but unsolvable by the model, (C) unsolvable by humans but solvable by the model, or (D) unsolvable by both humans and the model. For Case A, the question is easy and does not require complex language understanding. For Cases B and C, the model may use unintended solutions because (B) it does not use the same solution as humans or (C) it cleverly uses biases that humans cannot recognize. For Case D, the question may require the skill intended by the ablation method. Although Cases A to C are undesirable for evaluating the systems' skills, it seems to be useful to distinguish them for further improvement of the dataset creation. We therefore perform the annotation of questions with human solvability; We define that a question is solvable if a reasonable rationale for answering the question can be found in the context.","Results. Table TABREF20 shows the human solvability along with the baseline model's performance on the sampled questions. The model's performance is taken from the model trained on the original datasets except for the vocabulary anonymization method. For the content words only on both datasets, the human solvability is higher than the baseline performance. Although these gaps are not significant, we might be able to infer that the baseline model relies on content words more than humans (Case B). Given that the high performance of both humans and the baseline model, most of the questions fall into Case A, i.e., they are easy and do not necessarily require complex reasoning involving the understanding of function words.","For the other three methods, the human solvability is lower than the baseline performance. This result indicates that the questions correctly solved only by the baseline model may contain unintended biases (Case C). For example, the gap in the context word shuffle of RACE is significant (30.0% vs. 75.0%). Figure FIGREF21 shows a question that is unsolvable for humans but can be solved by the baseline model. We conjecture that while humans cannot detect biases easily, the model can exploit biases contained in the answer options and their relations to the given context."]},{"idx":10,"heading":"Qualitative Evaluation ::: Reconstructability of Ablated Features","paragraphs":["Motivation. We also seek to investigate the reconstructability of ablated features. Even if a question falls under Case A in the previous section, it might require the skill intended by the ablation; If a reader is able to guess the dropped information and uses it to solve the question, we cannot say that the question does not require the corresponding skill. For example, even after dropping function words ($\\sigma _3$), we might be able to guess which function word to fill in a cloze based on grammaticality and lexical knowledge. Such reconstructable features possibly exist for some ablation methods. However, they are not critical if they are unnecessary for answering questions. We can list the following cases: ablated features are ($\\alpha $) unreconstructable and unnecessary, ($\\beta $) unreconstructable and necessary, ($\\gamma $) reconstructable and unnecessary, and ($\\delta $) reconstructable and necessary. To verify that ablation methods work, we need to confirm that there are few questions of Case $\\delta $. The other cases are not critical to our observations in the main experiment. We therefore perform the annotation with the following queries: (i) are ablated features reconstructable? and (ii) are reconstructable features really necessary for answering? When the answers for both queries are yes, a question is in Case $\\delta $. In the annotation, we define that features in a question are reconstructable if the features existing around the rationale for answering the question are guessable. We also require that these features are necessary to decide the answer if the correct answer becomes undecidable without them.","Results. For both datasets, the annotation shows that, not surprisingly, almost all features are unreconstructable in the shuffled sentence/context words and the vocabulary anonymization (except for one example in RACE). When these questions are solvable / unsolvable by humans, we can say that features are unnecessary (Case $\\alpha $) / necessary (Case $\\beta $) for answering the questions. In contrast, the annotators could guess function words for some questions even if these words are dropped (SQuAD: 55.0% and RACE: 15.0%). The annotation of the necessity also shows that, however, reconstructable features (function words in this case) for all the questions are not necessary to answer them (i.e., Case $\\gamma $). Therefore, we could not find any question in Case $\\delta $. We report the annotation results in Appendix H. It is not easy for the annotator to completely ignore the information of reconstructed features. We leave designing a solid, scalable annotation scheme for future work.","In summary, we found that almost all ablated features are unreconstructable. Although for some questions ablated features are reconstructable for the content words only, these words are not necessarily required for answering the questions. Overall, this result supports our observations in Section SECREF4, i.e., questions already solved in existing datasets do not necessarily require complex language understanding."]},{"idx":11,"heading":"Discussion","paragraphs":["In this section, we discuss two requirements for developing the MRC task as an NLU benchmark. The control of question solvability. Not to allow the model to focus on unintended objectives, we need to ensure that each question is unsolvable without its intended requisite skill. Therefore, when benchmarking, we first need to identify necessary features whose presence determines the question's solvability. To identify them, we might need to perform ablation testing with humans. Further, we need to evaluate a model in both regular and ablation settings. This is because a model may detect some biases that enable it to solve the question; such biases can actually be false for humans and may be acquired by the model through overfitting to datasets. Nonetheless, there is a case in which, even if we can identify necessary features, the model can have prior, true knowledge (e.g., world knowledge) of the correct answer. In this case, the model can answer the question without the context. To avoid this circumvention, we may need to evaluate the model on fictional texts. Comprehensiveness of requisite skills. Another aspect of NLU benchmarking is the comprehensiveness of skills. Our proposed approach can be expanded in two further directions: (i) inner-sentence and (ii) multiple-sentence levels. For (i), we can focus on understanding of specific linguistic phenomena. This includes logical and semantic understanding such as in FraCaS BIBREF33 and SuperGLUE BIBREF34. To investigate particular syntactic phenomena, we might be able to use existing analysis methods BIBREF35. For (ii), our skills can include complex/implicit reasoning, e.g., spatial reasoning BIBREF36 and lexically dependent causal reasoning BIBREF37. Although we do not need to include all of these skills in a single dataset, we need to consider the generalization of models across them."]},{"idx":12,"heading":"Conclusion","paragraphs":["Existing analysis work in MRC is largely concerned with evaluating the capabilities of systems. By contrast, in this work, we proposed an analysis methodology for the benchmarking capacity of datasets. Our methodology consists of input-ablation tests, in which each ablation method is associated with a skill requisite for MRC. We exemplified 12 skills and analyzed 10 datasets. The experimental results suggest that for benchmarking sophisticated NLU, datasets should be more carefully designed to ensure that questions correctly evaluate the intended skills. In future work, we will develop a skill-oriented method for crowdsourcing questions."]},{"idx":13,"heading":"Acknowledgments","paragraphs":["We would like to thank Max Bartolo, Pasquale Minervini, and the anonymous reviewers for their insightful comments. This work was supported by JSPS KAKENHI Grant Numbers 18H03297 and 18J12960 and JST ACT-X Grant Number JPMJAX190G."]},{"idx":14,"heading":"Our Defined Requisite Skills","paragraphs":["Reading skills. As $s_2$ and $s_3$, we propose limiting the information available in the context by dropping content and function words respectively, which is intended to ascertain the extent to which a question depends on the given word type (e.g., a preposition in before a time-related expression for a when question). Skill $s_5$ provides a heuristic of the relative levels of attention between a question and the context. Skill $s_6$ is used to ensure that a model can extract the information conditioned on the word order.","Reasoning skills. Skill $s_8$ is for the understanding of discourse relations between adjacent sentences, which relies on information given by the sentence order in the context. When we shuffle the sentence order, various relations, such as causality and temporality, are expected to be broken. Skills $s_9$ to $s_{12}$ are defined more specifically; we drop tokens that explicitly emphasize important roles in specific skills such as if and not in logical reasoning."]},{"idx":15,"heading":"Experimental Details","paragraphs":["In this section, we provide details of the specifications used in our experiments.","Datasets. For CoQA, since this dataset allows for yes/no/unknown questions, we appended these words to the end of the context. These special words were not allowed to be dropped. Additionally, we appended the previous question-answer pair prior to the current question so that the model can consider the history of the QA conversation. To compute the performance on SQuAD v2.0, we used the best F1 value that was derived from the predictions with a no-answer threshold of $0.0$. For DuoRC, we used the ParaRC dataset (the official preprocessed version provided by the authors). When training a model on DuoRC and HotpotQA, we used the first answer span; i.e., the document spans that have no answer span were not used in training. For MCTest and RACE, we computed accuracy by combining MC160 with MC500 and Middle with High, respectively. For MultiRC, which is allowed to have multiple correct options for a question, we cast a pair consisting of a question and one option as a two-option multiple choice (i.e., whether its option is true or false) and computed the micro-averaged accuracy for the evaluation. The SWAG dataset is a multiple-choice task of predicting which event is most likely to occur next to a given sentence and the subject (noun phrase) of a subsequent event. We cast the first sentence as the context and the subject of the second sentence as the question. To compute F1 scores for the answer extraction datasets, we used the official evaluation scripts provided for the answer extraction datasets.","Ablation methods. For $\\sigma _4$ vocabulary anonymization, we used the tags as shown in Table TABREF23 and @other tags for the other POS tags. For $\\sigma _{10}$ logical words dropped, as logic-related terms, we used the following: all, any, each, every, few, if, more, most, no, nor, not, other, same, some, and than. For $\\sigma _{12}$ causal words dropped, as causality-related terms, we used the following: as, because, cause, since, therefore, and why. For $\\sigma _3^{\\prime }$ training with content words only, we dropped function words as well as punctuation marks so that the model would see only content words.","We show examples of questions for the ablation method $\\sigma _{4}$ in Figure FIGREF24."]},{"idx":16,"heading":"Hyperparameters of the Baseline Model","paragraphs":["Hyperparameters used in the baseline model are shown in Table TABREF25."]},{"idx":17,"heading":"Performance Variances in Shuffle Methods","paragraphs":["We report the variance for shuffling methods $s_6$ context words shuffle, $s_7$ sentence words shuffle, and $s_8$ sentence order shuffle in Table TABREF26."]},{"idx":18,"heading":"Statistics of the Examined MRC Datasets","paragraphs":["Table TABREF27 shows the statistics for the examined MRC datasets."]},{"idx":19,"heading":"Full Observations of the Main Results","paragraphs":["In this appendix, we describe the results for the reading and reasoning skills not mentioned in Section 4.2. $s_1$: recognizing question words. For the first four answer-extraction datasets, the performance decreased by more than 70%. For the multiple-choice datasets, the performance decreased by an average of 23.9%.","$s_5$: attending to the whole context other than similar sentences. Even with only the most similar sentences, the baseline models achieved a performance level greater than half their original performances in 8 out of 10 datasets. In contrast, HotpotQA showed the largest decrease in performance. This result reflects the fact that this dataset contains questions requiring multi-hop reasoning across multiple sentences.","$s_9$–$s_{12}$: various types of reasoning. For these skills, we can see that the performance drops were small; given that the drop for $s_3$ recognizing content words alone was under 20%, we can infer that specific types of reasoning might not be critical for answering the questions. Some types of reasoning, however, might play an essential role for some datasets: $s_9$ numerical reasoning in HotpotQA (whose questions sometimes require answers with numbers) and $s_{11}$ pronoun coreference resolution in DuoRC (consisting of movie scripts)."]},{"idx":20,"heading":"Detailed Results of SQuAD v2.0","paragraphs":["We report the ablation results for has-answer and no-answer questions in SQuAD v2.0 in Table TABREF28."]},{"idx":21,"heading":"The Annotation Results","paragraphs":["Table TABREF29 shows the frequency of questions for Cases $\\alpha $ to $\\delta $ for SQuAD v1.1 and RACE. See Section 5.2 for details."]},{"idx":22,"heading":"Figures and Tables","paragraphs":["Figure 1: Example of an ablation test that anonymizes context and question words, applied to a question from SQuAD v1.1 (Rajpurkar et al. 2016) with the correct answer in underscored. We found that the baseline model can achieve 61.2% F1 on SQuAD v1.1 even after the anonymization.","Table 1: Example set of requisite skills {si} and corresponding ablation methods {σi}. f is a model and (x, y) is a pair consisting of an input instance and its gold-standard answer. We interpret that for x s.t. f(x) = y, if f(σi(x)) = y, then x is solvable without si.","Table 2: The performances (%) of the baseline model with the ablation tests on the development set. Values in smaller font are changes (%) relative to the original baseline performance, and the rightmost column (“Rel. avg.”) shows their averages.","Table 3: Results of further analyses: the performance (%) after dropping all question (“Q”) and/or context (“C”) words, and that of the baseline model both trained and evaluated on the modified inputs.","Table 4: Comparison of the human solvability and the baseline model’s performance (%) on questions that are sampled from the ablation tests.","Figure 2: Example of questions with shuffled context words from RACE. Although the question appears unsolvable for humans, the baseline model predicts the correct answer.","Table 5: Examples of anonymization tags and corresponding POS tags (OntoNotes 5 version of Penn Treebank tag set). We use @noun, @verb, @adj, @adv, and @number for content words.","Table 6: Hyperparameters used in the experiments, where d is the size of the token sequence fed into the model, b is the training batch size, lr is the learning rate, and ep is the number of training epochs. We set the learning rate warmup in RACE to 0.05 and 0.1 for the other datasets. We used stride = 128 for documents longer than d tokens.","Table 7: Ablation results with variances in parentheses for shuffle-related skills (s6, s7, and s8) for five different runs.","Table 8: Statistics of the datasets examined and average numbers of tokens dropped by our ablation methods σi (i = 1, 2, 3, 5, 9, ..., 12). The tokens are counted after tokenization of the punctuation. Values in smaller font denote the proportion (%) of questions that contain dropped tokens.","Table 9: Results on the dev set of SQuAD v2.0 for subsets with normal (Has-ans) and no-answer (No-ans) questions.","Table 10: Frequency of questions for Cases α to δ for SQuAD v1.1 and RACE. Ablated features are (α) unreconstructable and unnecessary, (β) unreconstructable and necessary, (γ) reconstructable and unnecessary, and (δ) reconstructable and necessary. Questions for Case δ are problematic for interpreting our main observations.","Figure 3: Examples of questions after the application of our ablation method σ3 (content words only). Blacked-out text denotes words that were dropped by the ablation function."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1911.09247","title":"How to Ask Better Questions? A Large-Scale Multi-Domain Dataset for Rewriting Ill-Formed Questions","sections":[{"idx":0,"heading":"Abstract","paragraphs":["We present a large-scale dataset for the task of rewriting an ill-formed natural language question to a well-formed one. Our multi-domain question rewriting MQR dataset is constructed from human contributed Stack Exchange question edit histories. The dataset contains 427,719 question pairs which come from 303 domains. We provide human annotations for a subset of the dataset as a quality estimate. When moving from ill-formed to well-formed questions, the question quality improves by an average of 45 points across three aspects. We train sequence-to-sequence neural models on the constructed dataset and obtain an improvement of 13.2% in BLEU-4 over baseline methods built from other data resources. We release the MQR dataset to encourage research on the problem of question rewriting."]},{"idx":1,"heading":"Introduction","paragraphs":["Understanding text and voice questions from users is a difficult task as it involves dealing with “word salad” and ill-formed text. Ill-formed questions may arise from imperfect speech recognition systems, search engines, dialogue histories, inputs from low bandwidth devices such as mobile phones, or second language learners, among other sources. However, most downstream applications involving questions, such as question answering and semantic parsing, are trained on well-formed natural language. In this work, we focus on rewriting textual ill-formed questions, which could improve the performance of such downstream applications.","BIBREF0 introduced the task of identifying well-formed natural language questions. In this paper, we take a step further to investigate methods to rewrite ill-formed questions into well-formed ones without changing their semantics. We create a multi-domain question rewriting dataset (MQR) from human contributed Stack Exchange question edit histories. This dataset provides pairs of questions: the original ill-formed question and a well-formed question rewritten by the author or community contributors. The dataset contains 427,719 question pairs which come from 303 domains. The MQR dataset is further split into TRAIN and DEV/TEST, where question pairs in DEV/TEST have less $n$-gram overlap but better semantic preservation after rewriting. Table TABREF2 shows some example question pairs from the MQR DEV split.","Our dataset enables us to train models directly for the task of question rewriting. We train neural generation models on our dataset, including Long-Short Term Memory networks (LSTM; BIBREF1) with attention BIBREF2 and transformers BIBREF3. We show that these models consistently improve the well-formedness of questions although sometimes at the expense of semantic drift. We compare to approaches that do not use our training dataset, including general-purpose sentence paraphrasing, grammatical error correction (GEC) systems, and round trip neural machine translation. Methods trained on our dataset greatly outperform those developed from other resources. Augmenting our training set with additional question pairs such as Quora or Paralex question pairs BIBREF4 has mixed impact on this task. Our findings from the benchmarked methods suggest potential research directions to improve question quality.","To summarize our contributions:","We propose the task of question rewriting: converting textual ill-formed questions to well-formed ones while preserving their semantics.","We construct a large-scale multi-domain question rewriting dataset MQR from human generated Stack Exchange question edit histories. The development and test sets are of high quality according to human annotation. The training set is of large-scale. We release the MQR dataset to encourage research on the question rewriting task.","We benchmark a variety of neural models trained on the MQR dataset, neural models trained with other question rewriting datasets, and other paraphrasing techniques. We find that models trained on the MQR and Quora datasets combined followed by grammatical error correction perform the best in the MQR question rewriting task."]},{"idx":2,"heading":"Related Work ::: Query and Question Rewriting","paragraphs":["Methods have been developed to reformulate or expand search queries BIBREF5. Sometimes query rewriting is performed for sponsored search BIBREF6, BIBREF7. This work differs from our goal as we rewrite ill-formed questions to be well-formed.","Some work rewrites queries by searching through a database of query logs to find a semantically similar query to replace the original query. BIBREF8 compute query similarities for query ranking based on user click information. BIBREF9 learn paraphrases of questions to improve question answering systems. BIBREF10 translate queries from search engines into natural language questions. They used Bing's search logs and their corresponding clicked question page as a query-to-question dataset. We work on question rewriting without any database of question logs.","Actively rewriting questions with reinforcement learning has been shown to improve QA systems BIBREF11. This work proposes to rewrite questions to fulfill more general quality criteria."]},{"idx":3,"heading":"Related Work ::: Paraphrase Generation","paragraphs":["A variety of paraphrase generation techniques have been proposed and studied BIBREF12, BIBREF13, BIBREF14, BIBREF15, BIBREF16, BIBREF17. Recently, BIBREF18 use a variational autoencoder to generate paraphrases from sentences and BIBREF19 use deep reinforcement learning to generate paraphrases. Several have generated paraphrases by separately modeling syntax and semantics BIBREF20, BIBREF21.","Paraphrase generation has been used in several applications. BIBREF22 use paraphrase generation as a data augmentation technique for natural language understanding. BIBREF20 and BIBREF23 generate adversarial paraphrases with surface form variations to measure and improve model robustness. BIBREF24 generate paraphrases using machine translation on parallel text and use the resulting sentential paraphrase pairs to learn sentence embeddings for semantic textual similarity.","Our work focuses on question rewriting to improve question qualities, which is different from general sentence paraphrasing."]},{"idx":4,"heading":"Related Work ::: Text Normalization","paragraphs":["Text normalization BIBREF25 is the task of converting non-canonical language to “standard” writing. Non-canonical language frequently appears in informal domains such as social media postings or other conversational text, user-generated content, such as search queries or product reviews, speech transcriptions, and low-bandwidth input settings such as those found with mobile devices. Text normalization is difficult to define precisely and therefore difficult to provide gold standard annotations and evaluate systems for BIBREF26. In our setting, rewriting questions is defined implicitly through the choices made by the Stack Exchange community with the goals of helpfulness, clarity, and utility."]},{"idx":5,"heading":"Task Definition: Question Rewriting","paragraphs":["Given a question $q_i$, potentially ill-formed, the question rewriting task is to convert it to a well-formed natural language question $q_w$ while preserving its semantics and intention. Following BIBREF0, we define a well-formed question as one satisfying the following constraints:","The question is grammatically correct. Common grammatical errors include misuse of third person singular or verb tense.","The question does not contain spelling errors. Spelling errors refer specifically to typos and other misspellings, but not to grammatical errors such as third person singular or tense misuse in verbs.","The question is explicit. A well-formed question must be explicit and end with a question mark. A command or search query-like fragment is not well-formed."]},{"idx":6,"heading":"MQR Dataset Construction and Analysis","paragraphs":["We construct our Multi-Domain Question Rewriting (MQR) dataset from human contributed Stack Exchange question edit histories. Stack Exchange is a question answering platform where users post and answer questions as a community. Stack Exchange has its own standard of good questions, and their standard aligns well with our definition of well-formed questions. If questions on Stack Exchange do not meet their quality standards, members of the community often volunteer to edit the questions. Such edits typically correct spelling and grammatical errors while making the question more explicit and easier to understand.","We use 303 sub areas from Stack Exchange data dumps. The full list of area names is in the appendix. We do not include Stack Overflow because it is too specific to programming related questions. We also exclude all questions under the following language sub areas: Chinese, German, Spanish, Russian, Japanese, Korean, Latin, Ukrainian. This ensures that the questions in MQR are mostly English sentences. Having questions from 303 Stack Exchange sites makes the MQR dataset cover a broad range of domains.","We use “PostHistory.xml” and “Posts.xml” tables of each Stack Exchange site data dump. If a question appears in both “PostHistory.xml” and “Posts.xml”, it means the question was modified. We treat the most up-to-date Stack Exchange questions as a well formed-question and treat its version from “PostHistory.xml” as ill-formed. “PostHistory.xml” only keeps one edit for each question, so the MQR dataset does not contain duplicated questions.","The questions in the Stack Exchange raw data dumps do not always fulfill our data quality requirements. For example, some questions after rewriting are still not explicit. Sometimes rewriting introduces or deletes new information and cannot be done correctly without more context or the question description. We thus perform the following steps to filter the question pairs:","All well-formed questions in the pairs must start with “how”, “why”, “when”, “what”, “which”, “who”, “whose”, “do”, “where”, “does”, “is”, “are”, “must”, “may”, “need”, “did”, “was”, “were”, “can”, “has”, “have”, “are”. This step is performed to make sure the questions are explicit questions but not statements or commands.","To ensure there are no sentences written in non-English languages, we keep questions that contain 80% or more of valid English characters, including punctuation.","This yields the MQR dataset. We use the following heuristic criteria to split MQR into TRAIN, DEV, and TEST sets:","The BLEU scores between well-formed and ill-formed questions (excluding punctuation) are lower than 0.3 in DEV and TEST to ensure large variations after rewriting.","The lists of verbs and nouns between well-formed and ill-formed questions have a Jaccard similarity greater than 0.8 in DEV and TEST. We split DEV and TEST randomly and equally. This yields 2,112 instances in DEV and 2,113 instances in TEST.","The rest of the question edit pairs (423,495 instances) are placed in the TRAIN set.","Examples are shown in Table TABREF2. We release our TRAIN/DEV/TEST splits of the MQR dataset to encourage research in question rewriting."]},{"idx":7,"heading":"MQR Dataset Construction and Analysis ::: Dataset Quality","paragraphs":["To understand the quality of the question rewriting examples in the MQR dataset, we ask human annotators to judge the quality of the questions in the DEV and TEST splits (abbreviated as DEVTEST onward). Specifically, we take both ill-formed and well-formed questions in DEVTEST and ask human annotators to annotate the following three aspects regarding each question BIBREF0:","Is the question grammatically correct?","Is the spelling correct? Misuse of third person singular or past tense in verbs are considered grammatical errors instead of spelling errors. Missing question mark in the end of a question is also considered as spelling errors.","Is the question an explicit question, rather than a search query, a command, or a statement?","The annotators were asked to annotate each aspect with a binary (0/1) answer. Examples of questions provided to the annotators are in Table TABREF13. We consider all “How to” questions (“How to unlock GT90 in Gran Turismo 2?”) as grammatical. Although it is not a complete sentence, this kind of question is quite common in our dataset and therefore we choose to treat it as grammatically correct.","The ill-formed and well-formed questions are shuffled so the annotators do not have any prior knowledge or bias regarding these questions during annotation. We randomly sample 300 questions from the shuffled DEVTEST questions, among which 145 examples are well-formed and 155 are ill-formed. Two annotators produce a judgment for each of the three aspects for all 300 questions.","The above annotation task considers a single question at a time. We also consider an annotation task related to the quality of a question pair, specifically whether the two questions in the pair are semantically equivalent. If rewriting introduces additional information, then the question rewriting task may require additional context to be performed, even for a human writer. This may happen when a user changes the question content or the question title is modified based on the additional description about the question. In the MQR dataset, we focus on question rewriting tasks that can be performed without extra information.","We randomly sample 100 question pairs from DEVTEST for annotation of semantic equivalence. Two annotators produced binary judgments for all 100 pairs. Example pairs are shown in Table TABREF14.","Table TABREF15 summarizes the human annotations of the quality of the DEVTEST portion of the MQR dataset. We summed up the binary scores from two annotators. There are clear differences between ill-formed and well-formed questions. Ill-formed question are indeed ill-formed and well-formed questions are generally of high quality. The average score over three aspects improves by 45 points from ill-formed to well-formed questions. Over 90% of the question pairs possess semantic equivalence, i.e., they do not introduce or delete information. Therefore, the vast majority of rewrites can be performed without extra information.","The Cohen's Kappa inter-rater reliability scores BIBREF27 are 0.83, 0.77, and 0.89 respectively for the question quality annotations, and 0.86 for question semantic equivalence. These values show good inter-rater agreement on the annotations of the qualities and semantic equivalences of the MQR question pairs."]},{"idx":8,"heading":"MQR Dataset Construction and Analysis ::: Dataset Domains","paragraphs":["As the MQR dataset is constructed from 303 sub areas of the Stack Exchange networks, it covers a wide range of question domains. Table TABREF16 summarizes the number of categories in the TRAIN and DEVTEST portions of the MQR dataset, as well as the mean, standard deviation, minimum, and maximum number of instances per categories.","The number of questions from each sub area is not evenly distributed due to the fact that some sub areas are more popular and have more questions than the others, but the DEV/TEST splits still cover a reasonably large range of domains.","The most common categories in DEV and TEST are “diy”(295), “askubuntu”(288), “math”(250), “gaming”(189), and “physics”(140). The least common categories are mostly “Meta Stack Exchange” websites where people ask questions regarding the policies of posting questions on Stack Exchange sites. The most common categories in TRAIN are “askubuntu”(6237), “math”(5933), “gaming”(3938), “diy”(2791), and “2604”(scifi)."]},{"idx":9,"heading":"Models and Experiments","paragraphs":["In this section, we describe the models and methods we benchmarked to perform the task of question rewriting.","To evaluate model performance, we apply our trained models to rewrite the ill-formed questions in TEST and treat the well-formed question in each pair as the reference sentence. We then compute BLEU-4 BIBREF28, ROUGE-1, ROUGE-2, ROUGE-L BIBREF29, and METEOR BIBREF30 scores. As a baseline, we also evaluate the original ill-formed question using the automatic metrics."]},{"idx":10,"heading":"Models and Experiments ::: Models Trained on MQR ::: Transformer.","paragraphs":["We use the Tensor2Tensor BIBREF31 implementation of the transformer model BIBREF3. We use their “transformer_base” hyperparameter setting. The details are as follows: batch size 4096, hidden size 512, 8 attention heads, 6 transformer encoder and decoder layers, learning rate 0.1 and 4000 warm-up steps. We train the model for 250,000 steps and perform early stopping using the loss values on the DEV set.","In following sections, when a transformer model is used, we follow the same setting as described above."]},{"idx":11,"heading":"Models and Experiments ::: Models Trained on MQR ::: LSTM Sequence to Sequence Model with Attention.","paragraphs":["We use the attention mechanism proposed by BIBREF2. We use the Tensor2Tensor implementation BIBREF31 with their provided Luong Attention hyperparameter settings. We set batch size to 4096. The hidden size is 1000 and we use 4 LSTM hidden layers following BIBREF2."]},{"idx":12,"heading":"Models and Experiments ::: Methods Built from Other Resources","paragraphs":["We also benchmark other methods involving different training datasets and models. All the methods in this subsection use transformer models."]},{"idx":13,"heading":"Models and Experiments ::: Methods Built from Other Resources ::: Round Trip Neural Machine Translation.","paragraphs":["Round trip neural machine translation is an effective approach for question or sentence paraphrasing BIBREF32, BIBREF9, BIBREF20. It first translates a sentence to another pivot language, then translates it back to the original language. We consider the use of both German (De) and French (Fr) as the pivot language, so we require translation systems for En$\\leftrightarrow $De and En$\\leftrightarrow $Fr.","The English-German translation models are trained on WMT datasets, including News Commentary 13, Europarl v7, and Common Crawl, and evaluated on newstest2013 for early stopping. On the newstest2013 dev set, the En$\\rightarrow $De model reaches a BLEU-4 score of 19.6, and the De$\\rightarrow $En model reaches a BLEU-4 score of 24.6.","The English-French models are trained on Common Crawl 13, Europarl v7, News Commentary v9, Giga release 2, and UN doc 2000. On the newstest2013 dev set, the En$\\rightarrow $Fr model reaches a BLEU-4 score of 25.6, and the Fr$\\rightarrow $En model reaches a BLEU-4 score of 26.1."]},{"idx":14,"heading":"Models and Experiments ::: Methods Built from Other Resources ::: Grammatical Error Correction (GEC).","paragraphs":["As some ill-formed questions are not grammatical, we benchmark a state-of-the-art grammatical error correction system on this task. We use the system of BIBREF33, a GEC ensemble model trained from Wikipedia edit histories and round trip translations."]},{"idx":15,"heading":"Models and Experiments ::: Methods Built from Other Resources ::: Paraphrase Generator Trained on ParaNMT.","paragraphs":["We also train a paraphrase generation model on a subset of the ParaNMT dataset BIBREF24, which was created automatically by using neural machine translation to translate the Czech side of a large Czech-English parallel corpus. We use the filtered subset of 5M pairs provided by the authors. For each pair of paraphrases (S1 and S2) in the dataset, we train the model to rewrite from S1 to S2 and also rewrite from S2 to S1. We use the MQR DEV set for early stopping during training."]},{"idx":16,"heading":"Models and Experiments ::: Results","paragraphs":["Table TABREF30 shows the performance of the models and methods described above. Among these methods models trained on MQR work best. GEC corrects grammatical errors and spelling errors, so it also improves the question quality in rewriting. Round trip neural machine translation is a faithful rewrite of the questions, and it naturally corrects some spelling and grammatical errors during both rounds of translation due to the strong language models present in the NMT models. However, it fails in converting commands and statements into questions.","The paraphrase generator trained on ParaNMT does not perform well, likely because of domain difference (there are not many questions in ParaNMT). It also is unlikely to convert non-question sentences into explicit questions."]},{"idx":17,"heading":"Models and Experiments ::: Additional Training Data","paragraphs":["We consider two additional data resources to improve question rewriting models.","The first resource is the Quora Question Pairs dataset. This dataset contains question pairs from Quora, an online question answering community. Some question pairs are marked as duplicate by human annotators and other are not. We consider all Quora Question Pairs (Q1 and Q2) marked as duplicate as additional training data. We train the model to rewrite from Q1 to Q2 and also from Q2 to Q1. This gives us 298,364 more question pairs for training.","The second resource is the Paralex dataset BIBREF4. The questions in Paralex are scraped from WikiAnswers, where questions with similar content are clustered. As questions in the Paralex dataset may be noisy, we use the annotation from BIBREF0. Following their standard, we treat all questions with scores higher than 0.8 as well-formed questions. For each well-formed question, we take all questions in the same Paralex question cluster and construct pairs to rewrite from other questions in the cluster to the single well-formed question. This gives us 169,682 extra question pairs for training.","We also tried adding “identity” training examples in which the well-formed questions from the MQR TRAIN set are repeated to form a question pair.","The results of adding training data are summarized in Table TABREF31. Adding the identity pairs improves the ROUGE and METEOR scores, which are focused more on recall, while harming BLEU, which is focused on precision. We hypothesize that adding auto-encoding data improves semantic preservation, which is expected to help the recall-oriented metrics. Adding Quora Question Pairs improves performance on TEST but adding Paralex pairs does not. The reason may stem from domain differences: WikiAnswers (used in Paralex) is focused on factoid questions answered by encyclopedic knowledge while Quora and Stack Exchange questions are mainly answered by community contributors. Semantic drift occurs more often in Paralex question pairs as Paralex is constructed from question clusters, and a cluster often contains more than 5 questions with significant variation."]},{"idx":18,"heading":"Models and Experiments ::: Combining Methods","paragraphs":["In addition to the aforementioned methods, we also try combining multiple approaches. Table TABREF32 shows results when combining GEC and the Quora-augmented transformer model. We find that combining GEC and a transformer question rewriting model achieves better results than each alone. In particular, it is best to first rewrite the question using the transformer trained on MQR + Quora, then run GEC on the output.","We also tried applying the transformer (trained on MQR) twice, but it hurts the performance compared to applying it only once (see Table TABREF32)."]},{"idx":19,"heading":"Models and Experiments ::: Human Evaluation","paragraphs":["To better evaluate model performance, we conduct a human evaluation on the model rewritten questions following the same guidelines from the “Dataset Quality” subsection. Among the 300 questions annotated earlier, we chose the ill-formed questions from the TEST split, which yields 75 questions. We evaluate questions rewritten by three methods (Transformer (MQR + Quora), GEC, and Transformer (MQR + Quora) $\\rightarrow $ GEC), and ask annotators to determine the qualities of the rewritten questions. To understand if question meanings change after rewriting, we also annotate whether a model rewritten question is semantically equivalent to the ill-formed question or equivalent to the well-formed one.","Table TABREF33 shows the annotations from two annotators. When the two annotators disagree, a judge makes a final decision. Note that the examples annotated here are a subset of those annotated in Table TABREF15, so the first row is different from the ill-formed questions in Table TABREF15. According to the annotations, the GEC method slightly improves the question quality scores. Although Table TABREF30 shows that GEC improves the question quality by some automatic metrics, it simply corrects a few grammatical errors and the rewritten questions still do not meet the standards of human annotators. However, the GEC model is good at preserving question semantics.","The Transformer (MQR + Quora) model and Transformer (MQR + Quora) $\\rightarrow $ GEC excel at improving question quality in all three aspects, but they suffer from semantic drift. This suggests that future work should focus on solving the problem of semantic drift when building question rewriting models.","Table TABREF34 shows two example questions rewritten by different methods. The questions rewritten by GEC remain unchanged but are still of low quality, whereas ParaNMT and round trip NMT make a variety of changes, resulting in large variations in question quality and semantics. Methods trained on MQR excel at converting ill-formed questions into explicit ones (e.g., adding “What is” in the first example and “How to” in the second example), but sometimes make grammatical errors (e.g., Trans. (MQR + Quora) misses “a” in the second example). According to Table TABREF32, combining neural models trained on MQR and GEC achieves the best results in automatic metrics. However, they still suffer from semantic drift. In the first example of Table TABREF34, the last two rewrites show significant semantic mistakes, generating non-existent words “widebutcherblock” and “widebitcherblock”."]},{"idx":20,"heading":"Conclusion and Future Work","paragraphs":["We proposed the task of question rewriting and produced a novel dataset MQR to target it. Our evaluation shows consistent gains in metric scores when using our dataset compared to systems derived from previous resources. A key challenge for future work is to design better models to rewrite ill-formed questions without changing their semantics. Alternatively, we could attempt to model the process whereby question content changes. Sometimes community members do change the content of questions in online forums. Such rewrites typically require extra context information, such as the question description. Additional work will be needed to address this context-sensitive question rewriting task."]},{"idx":21,"heading":"Acknowledgments","paragraphs":["We thank Shankar Kumar, Zi Yang, Yiran Zhang, Rahul Gupta, Dekang Lin, Yuchen Lin, Guan-lin Chao, Llion Jones, and Amarnag Subramanya for their helpful discussions and suggestions."]},{"idx":22,"heading":"Figures and Tables","paragraphs":["Table 1: Examples of pairs of ill-formed and well-formed questions from the MQR dataset.","Table 2: Examples given to annotators for binary question quality scores.","Table 3: Example question pairs given to annotators to judge semantic equivalence.","Table 4: Summary of manual annotations for instances sampled from the DEV and TEST portions of the MQR dataset. “Quality” are the average quality scores, broken down into three aspects. “Semantic Equivalence” is the percentage of question pairs in which the ill-formed and well-formed questions are semantically equivalent. The scores are averages of binary scores across both annotators.","Table 5: Statistics of question pairs (“instances”) from Stack Exchange categories in the MQR dataset.","Table 6: Results on MQR TEST set. The “Ill-formed” shows metric scores for the questions in TEST without rewriting. The next portion shows results for models trained on the TRAIN portion of MQR. The lower portion shows results for methods using other models and/or datasets.","Table 7: Results showing how additional training data affects performance for the transformer model.","Table 8: Methods combining transformer trained on MQR + Quora with GEC. “A→ B” means running method A followed by method B on method A’s output.","Table 9: Results of human evaluation of three models on 75 test examples.","Table 10: Examples of ill-formed question rewritten by models with human annotations. (S = Spelling, G = Grammar, and E = Explicit) The last column shows semantic equivalence with the well-formed questions."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1912.11637","title":"Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Self-attention based Transformer has demonstrated the state-of-the-art performances in a number of natural language processing tasks. Self-attention is able to model long-term dependencies, but it may suffer from the extraction of irrelevant information in the context. To tackle the problem, we propose a novel model called \\textbf{Explicit Sparse Transformer}. Explicit Sparse Transformer is able to improve the concentration of attention on the global context through an explicit selection of the most relevant segments. Extensive experimental results on a series of natural language processing and computer vision tasks, including neural machine translation, image captioning, and language modeling, all demonstrate the advantages of Explicit Sparse Transformer in model performance. We also show that our proposed sparse attention method achieves comparable or better results than the previous sparse attention method, but significantly reduces training and testing time. For example, the inference speed is twice that of sparsemax in Transformer model. Code will be available at \\url{this https URL}"]},{"idx":1,"heading":"Introduction","paragraphs":["Understanding natural language requires the ability to pay attention to the most relevant information. For example, people tend to focus on the most relevant segments to search for the answers to their questions in mind during reading. However, retrieving problems may occur if irrelevant segments impose negative impacts on reading comprehension. Such distraction hinders the understanding process, which calls for an effective attention.","This principle is also applicable to the computation systems for natural language. Attention has been a vital component of the models for natural language understanding and natural language generation. Recently, BIBREF0 proposed Transformer, a model based on the attention mechanism for Neural Machine Translation(NMT). Transformer has shown outstanding performance in natural language generation tasks. More recently, the success of BERT BIBREF1 in natural language processing shows the great usefulness of both the attention mechanism and the framework of Transformer.","However, the attention in vanilla Transformer has a obvious drawback, as the Transformer assigns credits to all components of the context. This causes a lack of focus. As illustrated in Figure FIGREF1, the attention in vanilla Transformer assigns high credits to many irrelevant words, while in Explicit Sparse Transformer, it concentrates on the most relevant $k$ words. For the word “tim”, the most related words should be \"heart\" and the immediate words. Yet the attention in vanilla Transformer does not focus on them but gives credits to some irrelevant words such as “him”.","Recent works have studied applying sparse attention in Transformer model. However, they either add local attention constraints BIBREF2 which break long term dependency or hurt the time efficiency BIBREF3. Inspired by BIBREF4 which introduce sparse credit assignment to the LSTM model, we propose a novel model called Explicit Sparse Transformer which is equipped with our sparse attention mechanism. We implement an explicit selection method based on top-$k$ selection. Unlike vanilla Transformer, Explicit Sparse Transformer only pays attention to the $k$ most contributive states. Thus Explicit Sparse Transformer can perform more concentrated attention than vanilla Transformer.","We first validate our methods on three tasks. For further investigation, we compare our methods with previous sparse attention methods and experimentally answer how to choose k in a series of qualitative analyses. We are surprised to find that the proposed sparse attention method can also help with training as a regularization method. Visual analysis shows that Explicit Sparse Transformer exhibits a higher potential in performing a high-quality alignment. The contributions of this paper are presented below:","We propose a novel model called Explicit Sparse Transformer, which enhances the concentration of the Transformer's attention through explicit selection.","We conducted extensive experiments on three natural language processing tasks, including Neural Machine Translation, Image Captioning and Language Modeling. Compared with vanilla Transformer, Explicit Sparse Transformer demonstrates better performances in the above three tasks.","Compared to previous sparse attention methods for transformers, our methods are much faster in training and testing, and achieves comparable results."]},{"idx":2,"heading":"Explicit Sparse Transformer","paragraphs":["The review to the attention mechanism and the attention-based framework of Transformer can be found in Appendix SECREF35.","Lack of concentration in the attention can lead to the failure of relevant information extraction. To this end, we propose a novel model, Explicit Sparse Transformer, which enables the focus on only a few elements through explicit selection. Compared with the conventional attention, no credit will be assigned to the value that is not highly correlated to the query. We provide a comparison between the attention of vanilla Transformer and that of Explicit Sparse Transformer in Figure FIGREF5.","Explicit Sparse Transformer is still based on the Transformer framework. The difference is in the implementation of self-attention. The attention is degenerated to the sparse attention through top-$k$ selection. In this way, the most contributive components for attention are reserved and the other irrelevant information are removed. This selective method is effective in preserving important information and removing noise. The attention can be much more concentrated on the most contributive elements of value. In the following, we first introduce the sparsification in self-attention and then extend it to context attention.","In the unihead self-attention, the key components, the query $Q[l_{Q}, d]$, key $K[l_{K}, d]$ and value $V[l_{V}, d]$, are the linear transformation of the source context, namely the input of each layer, where $Q = W_{Q}x$, $K = W_{K}x$ and $V = W_{V}x$. Explicit Sparse Transformer first generates the attention scores $P$ as demonstrated below:","Then the model evaluates the values of the scores $P$ based on the hypothesis that scores with larger values demonstrate higher relevance. The sparse attention masking operation $\\mathcal {M}(\\cdot )$ is implemented upon $P$ in order to select the top-$k$ contributive elements. Specifically, we select the $k$ largest element of each row in $P$ and record their positions in the position matrix $(i, j)$, where $k$ is a hyperparameter. To be specific, say the $k$-th largest value of row $i$ is $t_{i}$, if the value of the $j$-th component is larger than $t_i$, the position $(i, j)$ is recorded. We concatenate the threshold value of each row to form a vector $t = [t_1, t_2, \\cdots , t_{l_{Q}}]$. The masking functions $\\mathcal {M}(\\cdot , \\cdot )$ is illustrated as follows:","With the top-$k$ selection, the high attention scores are selected through an explicit way. This is different from dropout which randomly abandons the scores. Such explicit selection can not only guarantee the preservation of important components, but also simplify the model since $k$ is usually a small number such as 8, detailed analysis can be found in SECREF28. The next step after top-$k$ selection is normalization:","where $A$ refers to the normalized scores. As the scores that are smaller than the top k largest scores are assigned with negative infinity by the masking function $\\mathcal {M}(\\cdot , \\cdot )$, their normalized scores, namely the probabilities, approximate 0. We show the back-propagation process of Top-k selection in SECREF50. The output representation of self-attention $C$ can be computed as below:","The output is the expectation of the value following the sparsified distribution $A$. Following the distribution of the selected components, the attention in the Explicit Sparse Transformer model can obtain more focused attention. Also, such sparse attention can extend to context attention. Resembling but different from the self-attention mechanism, the $Q$ is no longer the linear transformation of the source context but the decoding states $s$. In the implementation, we replace $Q$ with $W_{Q}s$, where $W_{Q}$ is still learnable matrix.","In brief, the attention in our proposed Explicit Sparse Transformer sparsifies the attention weights. The attention can then become focused on the most contributive elements, and it is compatible to both self-attention and context attention. The simple implementation of this method is in the Appendix SECREF55."]},{"idx":3,"heading":"Results","paragraphs":["We conducted a series of experiments on three natural language processing tasks, including neural machine translation, image captioning and language modeling. Detailed experimental settings are in Appendix SECREF42."]},{"idx":4,"heading":"Results ::: Neural Machine Translation ::: Dataset","paragraphs":["To evaluate the performance of Explicit Sparse Transformer in NMT, we conducted experiments on three NMT tasks, English-to-German translation (En-De) with a large dataset, English-to-Vietnamese (En-Vi) translation and German-to-English translation (De-En) with two datasets of medium size. For En-De, we trained Explicit Sparse Transformer on the standard dataset for WMT 2014 En-De translation. The dataset consists of around 4.5 million sentence pairs. The source and target languages share a vocabulary of 32K sub-word units. We used the newstest 2013 for validation and the newstest 2014 as our test set. We report the results on the test set.","For En-Vi, we trained our model on the dataset in IWSLT 2015 BIBREF20. The dataset consists of around 133K sentence pairs from translated TED talks. The vocabulary size for source language is around 17,200 and that for target language is around 7,800. We used tst2012 for validation, and tst2013 for testing and report the testing results. For De-En, we used the dataset in IWSLT 2014. The training set contains 160K sentence pairs and the validation set contains 7K sentences. Following BIBREF21, we used the same test set with around 7K sentences. The data were preprocessed with byte-pair encoding BIBREF22. The vocabulary size is 14,000."]},{"idx":5,"heading":"Results ::: Neural Machine Translation ::: Result","paragraphs":["Table TABREF10 presents the results of the baselines and our Explicit Sparse Transformer on the three datasets. For En-De, Transformer-based models outperform the previous methods. Compared with the result of Transformer BIBREF0, Explicit Sparse Transformer reaches 29.4 in BLEU score evaluation, outperforming vanilla Transformer by 0.3 BLEU score. For En-Vi, vanilla Transformer reaches 30.2, outperforming previous best method BIBREF7. Our model, Explicit Sparse Transformer, achieves a much better performance, 31.1, by a margin of 0.5 over vanilla Transformer. For De-En, we demonstrate that Transformer-based models outperform the other baselines. Compared with Transformer, our Explicit Sparse Transformer reaches a better performance, 35.6. Its advantage is +0.3. To the best of our knowledge, Explicit Sparse Transformer reaches a top line performance on the dataset."]},{"idx":6,"heading":"Results ::: Image Captioning ::: Dataset","paragraphs":["We evaluated our approach on the image captioning task. Image captioning is a task that combines image understanding and language generation. We conducted experiments on the Microsoft COCO 2014 dataset BIBREF23. It contains 123,287 images, each of which is paired 5 with descriptive sentences. We report the results and evaluate the image captioning model on the MSCOCO 2014 test set for image captioning. Following previous works BIBREF24, BIBREF25, we used the publicly-available splits provided by BIBREF26. The validation set and test set both contain 5,000 images."]},{"idx":7,"heading":"Results ::: Image Captioning ::: Result","paragraphs":["Table TABREF17 shows the results of the baseline models and Explicit Sparse Transformer on the COCO Karpathy test split. Transformer outperforms the mentioned baseline models. Explicit Sparse Transformer outperforms the implemented Transformer by +0.4 in terms of BLEU-4, +0.3 in terms of METEOR, +0.7 in terms of CIDEr. , which consistently proves its effectiveness in Image Captioning."]},{"idx":8,"heading":"Results ::: Language Modeling ::: Dataset","paragraphs":["Enwiki8 is large-scale dataset for character-level language modeling. It contains 100M bytes of unprocessed Wikipedia texts. The inputs include Latin alphabets, non-Latin alphabets, XML markups and special characters. The vocabulary size 205 tokens, including one for unknown characters. We used the same preprocessing method following BIBREF33. The training set contains 90M bytes of data, and the validation set and the test set contains 5M respectively."]},{"idx":9,"heading":"Results ::: Language Modeling ::: Result","paragraphs":["Table TABREF23 shows the results of the baseline models and Explicit Sparse Transformer-XL on the test set of enwiki8. Compared with the other strong baselines, Transformer-XL can reach a better performance, and Explicit Sparse Transformer outperforms Transformer-XL with an advantage."]},{"idx":10,"heading":"Discussion","paragraphs":["In this section, we performed several analyses for further discussion of Explicit Sparse Transformer. First, we compare the proposed method of topk selection before softmax with previous sparse attention method including various variants of sparsemax BIBREF3, BIBREF42, BIBREF43. Second, we discuss about the selection of the value of $k$. Third, we demonstrate that the top-k sparse attention method helps training. In the end, we conducted a series of qualitative analyses to visualize proposed sparse attention in Transformer."]},{"idx":11,"heading":"Discussion ::: Comparison with other Sparse Attention Methods","paragraphs":["We compare the performance and speed of our method with the previous sparse attention methods on the basis of strong implemented transformer baseline. The training and inference speed are reported on the platform of Pytorch and IWSLT 2014 De-En translation dataset, the batch size for inference is set to 128 in terms of sentence and half precision training(FP-16) is applied.","As we can see from Table TABREF25, the proposed sparse attention method achieve the comparable results as previous sparse attention methods, but the training and testing speed is 2x faster than sparsemax and 10x faster than Entmax-alpha during the inference. This is due to the fact that our method does not introduce too much computation for calculating sparse attention scores.","The other group of sparse attention methods of adding local attention constraints into attention BIBREF2, BIBREF41, do not show performance on neural machine translation, so we do not compare them in Table TABREF25."]},{"idx":12,"heading":"Discussion ::: How to Select a Proper k?","paragraphs":["The natural question of how to choose the optimal $k$ comes with the proposed method. We compare the effect of the value of $k$ at exponential scales. We perform experiments on En-Vi and De-En from 3 different initializations for each value of $K$, and report the mean BLEU scores on the valid set. The figure FIGREF27 shows that regardless of the value of 16 on the En-Vi dataset, the model performance generally rises first and then falls as $k$ increases. For $k\\in \\lbrace 4,8,16,32\\rbrace $, setting the value of $k$ to 8 achieves consistent improvements over the transformer baseline."]},{"idx":13,"heading":"Discussion ::: Do the proposed sparse attention method helps training?","paragraphs":["We are surprised to find that only adding the sparsification in the training phase can also bring an improvement in the performance. We experiment this idea on IWSLT En-Vi and report the results on the valid set in Table TABREF30, . The improvement of 0.3 BLEU scores shows that vanilla Transformer may be overparameterized and the sparsification encourages the simplification of the model."]},{"idx":14,"heading":"Discussion ::: Do the Explicit Sparse Transformer Attend better?","paragraphs":["To perform a thorough evaluation of our Explicit Sparse Transformer, we conducted a case study and visualize the attention distributions of our model and the baseline for further comparison. Specifically, we conducted the analysis on the test set of En-Vi, and randomly selected a sample pair of attention visualization of both models.","The visualization of the context attention of the decoder's bottom layer in Figure FIGREF33. The attention distribution of the left figure is fairly disperse. On the contrary, the right figure shows that the sparse attention can choose to focus only on several positions so that the model can be forced to stay focused. For example, when generating the phrase “for thinking about my heart”(Word-to-word translation from Vietnamese), the generated word cannot be aligned to the corresponding words. As to Explicit Sparse Transformer, when generating the phrase \"with all my heart\", the attention can focus on the corresponding positions with strong confidence.","The visualization of the decoder's top layer is shown in Figure FIGREF34. From the figure, the context attention at the top layer of the vanilla Transformer decoder suffers from focusing on the last source token. This is a common behavior of the attention in vanilla Transformer. Such attention with wrong alignment cannot sufficiently extract enough relevant source-side information for the generation. In contrast, Explicit Sparse Transformer, with simple modification on the vanilla version, does not suffer from this problem, but instead focuses on the relevant sections of the source context. The figure on the right demonstrating the attention distribution of Explicit Sparse Transformer shows that our proposed attention in the model is able to perform accurate alignment."]},{"idx":15,"heading":"Related Work","paragraphs":["Attention mechanism has demonstrated outstanding performances in a number of neural-network-based methods, and it has been a focus in the NLP studies BIBREF44. A number of studies are proposed to enhance the effects of attention mechanism BIBREF45, BIBREF0, BIBREF4, BIBREF46. BIBREF45 propose local attention and BIBREF47 propose local attention for self-attention. BIBREF48 propose hard attention that pays discrete attention in image captioning. BIBREF49 propose a combination soft attention with hard attention to construct hierarchical memory network. BIBREF8 propose a temperature mechanism to change the softness of attention distribution. BIBREF50 propose an attention which can select a small proportion for focusing. It is trained by reinforcement learning algorithms BIBREF51. In terms of memory networks, BIBREF52 propose to sparse access memory.","BIBREF2 recently propose to use local attention and block attention to sparsify the transformer. Our approach differs from them in that our method does not need to block sentences and still capture long distance dependencies. Besides, we demonstrate the importance of Explicit Sparse Transformer in sequence to sequence learning. Although the variants of sparsemax BIBREF3, BIBREF42, BIBREF43 improve in machine translation tasks, we empirically demonstrate in SECREF24 that our method introduces less computation in the standard transformer and is much faster than those sparse attention methods on GPUs."]},{"idx":16,"heading":"Conclusion","paragraphs":["In this paper, we propose a novel model called Explicit Sparse Transformer. Explicit Sparse Transformer is able to make the attention in vanilla Transformer more concentrated on the most contributive components. Extensive experiments show that Explicit Sparse Transformer outperforms vanilla Transformer in three different NLP tasks. We conducted a series of qualitative analyses to investigate the reasons why Explicit Sparse Transformer outperforms the vanilla Transformer. Furthermore, we find an obvious problem of the attention at the top layer of the vanilla Transformer, and Explicit Sparse Transformer can alleviate this problem effectively with improved alignment effects."]},{"idx":17,"heading":"Appendix ::: Background ::: Attention Mechanism","paragraphs":["BIBREF44 first introduced the attention mechanism to learn the alignment between the target-side context and the source-side context, and BIBREF45 formulated several versions for local and global attention. In general, the attention mechanism maps a query and a key-value pair to an output. The attention score function and softmax normalization can turn the query $Q$ and the key $K$ into a distribution $\\alpha $. Following the distribution $\\alpha $, the attention mechanism computes the expectation of the value $V$ and finally generates the output $C$.","Take the original attention mechanism in NMT as an example. Both key $K \\in \\mathbb {R}^{n \\times d}$ and value $V \\in \\mathbb {R}^{n \\times d} $ are the sequence of output states from the encoder. Query $Q \\in \\mathbb {R}^{m \\times d}$ is the sequence of output states from the decoder, where $m$ is the length of $Q$, $n$ is the length of $K$ and $V$, and $d$ is the dimension of the states. Thus, the attention mechanism is formulated as:","where $f$ refers to the attention score computation."]},{"idx":18,"heading":"Appendix ::: Background ::: Transformer","paragraphs":["Transformer BIBREF0, which is fully based on the attention mechanism, demonstrates the state-of-the-art performances in a series of natural language generation tasks. Specifically, we focus on self-attention and multi-head attention.","The ideology of self-attention is, as the name implies, the attention over the context itself. In the implementation, the query $Q$, key $K$ and value $V$ are the linear transformation of the input $x$, so that $Q = W_{Q}x$, $K = W_{K}x$ and $V = W_{V}x$ where $W_{Q}$, $W_{K}$ and $W_{V}$ are learnable parameters. Therefore, the computation can be formulated as below:","where $d$ refers to the dimension of the states.","The aforementioned mechanism can be regarded as the unihead attention. As to the multi-head attention, the attention computation is separated into $g$ heads (namely 8 for basic model and 16 for large model in the common practice). Thus multiple parts of the inputs can be computed individually. For the $i$-th head, the output can be computed as in the following formula:","where $C^{(i)}$ refers to the output of the head, $Q^{(i)}$, $K^{(i)}$ and $V^{(i)}$ are the query, key and value of the head, and $d_k$ refers to the size of each head ($d_k = d/g$). Finally, the output of each head are concatenated for the output:","In common practice, $C$ is sent through a linear transformation with weight matrix $W_c$ for the final output of multi-head attention.","However, soft attention can assign weights to a lot more words that are less relevent to the query. Therefore, in order to improve concentration in attention for effective information extraction, we study the problem of sparse attention in Transformer and propose our model Explicit Sparse Transformer."]},{"idx":19,"heading":"Appendix ::: Experimental Details","paragraphs":["We use the default setting in BIBREF0 for the implementation of our proposed Explicit Sparse Transformer. The hyper parameters including beam size and training steps are tuned on the valid set."]},{"idx":20,"heading":"Appendix ::: Experimental Details ::: Neural Machine Translation","paragraphs":["Training： For En-Vi translation, we use default scripts and hyper-parameter setting of tensor2tensor v1.11.0 to preprocess, train and evaluate our model. We use the default scripts of fairseq v0.6.1 to preprocess the De-En and En-De dataset. We train the model on the En-Vi dataset for $35K$ steps with batch size of $4K$. For IWSLT 2015 De-En dataset, batch size is also set to $4K$, we update the model every 4 steps and train the model for 90epochs. For WMT 2014 En-De dataset, we train the model for 72 epochs on 4 GPUs with update frequency of 32 and batch size of 3584. We train all models on a single RTX2080TI for two small IWSLT datasets and on a single machine of 4 RTX TITAN for WMT14 En-De. In order to reduce the impact of random initialization, we perform experiments with three different initializations for all models and report the highest for small datasets.","Evaluation： We use case-sensitive tokenized BLEU score BIBREF55 for the evaluation of WMT14 En-De, and we use case-insensitive BLEU for that of IWSLT 2015 En-Vi and IWSLT 2014 De-En following BIBREF8. Same as BIBREF0, compound splitting is used for WMT 14 En-De. For WMT 14 En-De and IWSLT 2014 De-En, we save checkpoints every epoch and average last 10 checkpoints every 5 epochs, We select the averaged checkpoint with best valid BLEU and report its BLEU score on the test set. For IWSLT 2015 En-Vi, we save checkpoints every 600 seconds and average last 20 checkpoints."]},{"idx":21,"heading":"Appendix ::: Experimental Details ::: Image Captioning","paragraphs":["We still use the default setting of Transformer for training our proposed Explicit Sparse Transformer. We report the standard automatic evaluation metrics with the help of the COCO captioning evaluation toolkit BIBREF53, which includes the commonly-used evaluation metrics, BLEU-4 BIBREF55, METEOR BIBREF54, and CIDEr BIBREF56."]},{"idx":22,"heading":"Appendix ::: Experimental Details ::: Language Models","paragraphs":["We follow BIBREF40 and use their implementation for our Explicit Sparse Transformer. Following the previous work BIBREF33, BIBREF40, we use BPC ($E[− log_2 P(xt+1|ht)]$), standing for the average number of Bits-Per-Character, for evaluation. Lower BPC refers to better performance. As to the model implementation, we implement Explicit Sparse Transformer-XL, which is based on the base version of Transformer-XL. Transformer-XL is a model based on Transformer but has better capability of representing long sequences."]},{"idx":23,"heading":"Appendix ::: The Back-propagation Process of Top-k Selection","paragraphs":["The masking function $\\mathcal {M}(\\cdot , \\cdot )$ is illustrated as follow:","Denote $M=\\mathcal {M}(P, k)$. We regard $t_i$ as constants. When back-propagating,","The next step after top-$k$ selection is normalization:","where $A$ refers to the normalized scores. When backpropagating,","The softmax function is evidently differentiable, therefore, we have calculated the gradient involved in top-k selection."]},{"idx":24,"heading":"Appendix ::: Implementation","paragraphs":["Figure FIGREF56 shows the code for the idea in case of single head self-attention, the proposed method is easy to implement and plug in the successful Transformer model."]},{"idx":25,"heading":"Figures and Tables","paragraphs":["Figure 1: Illustration of self-attention in the models. The orange bar denotes the attention score of our proposed model while the blue bar denotes the attention scores of the vanilla Transformer. The orange line denotes the attention between the target word “tim” and the selected top-k positions in the sequence. In the attention of vanilla Transformer, ”tim” assigns too many non-zero attention scores to the irrelevant words. But for the proposal, the top-k largest attention scores removes the distraction from irrelevant words and the attention becomes concentrated.","Figure 2: The comparison between the attentions of vanilla Transformer and Explicit Sparse Transformer and the illustration of the attention module of Explicit Sparse Transformer. With the mask based on top-k selection and softmax function, only the most contributive elements are assigned with probabilities.","Table 1: Results on the En-De, En-Vi and De-En test sets. Compared with the baseline models, Explicit Sparse Transformer reaches improved performances, and it achieves the state-of-the-art performances in En-Vi and De-En.","Table 2: Results on the MSCOCO Karpathy test split.","Table 3: Comparison with state-of-the-art results on enwiki8. Explicit Sparse Transformer-XL refers to the Transformer with our sparsification method.","Table 4: In the Transformer model, the proposed method, top-k selection before softmax is faster than previous sparse attention methods and is comparable in terms of BLEU scores.","Figure 3: Analyse the value of K on IWSLT En-Vi and De-En datasets. ”inf” denotes the special case of the Explicit Sparse Transformer where all positions may be attended, same as the origin Transformer.","Table 5: Results of the ablation study of the sparsification at different phases on the En-Vi test set. “Base” denotes vanilla Transformer. “T” denotes only adding the sparsification in the training phase, and “T&P” denotes adding it at both phases as the implementation of Explicit Sparse Transformer does.","Figure 4: Figure 4(a) is the attention visualization of Transformer and Figure 4(b) is that of the Explicit Sparse Transformer. The red box shows that the attentions in vanilla Transformer at most steps are concentrated on the last token of the context.","Figure 5: Code for the main idea in Pytorch"]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"1912.11980","title":"Explicit Sentence Compression for Neural Machine Translation","sections":[{"idx":0,"heading":"Abstract","paragraphs":["State-of-the-art Transformer-based neural machine translation (NMT) systems still follow a standard encoder-decoder framework, in which source sentence representation can be well done by an encoder with self-attention mechanism. Though Transformer-based encoder may effectively capture general information in its resulting source sentence representation, the backbone information, which stands for the gist of a sentence, is not specifically focused on. In this paper, we propose an explicit sentence compression method to enhance the source sentence representation for NMT. In practice, an explicit sentence compression goal used to learn the backbone information in a sentence. We propose three ways, including backbone source-side fusion, target-side fusion, and both-side fusion, to integrate the compressed sentence into NMT. Our empirical tests on the WMT English-to-French and English-to-German translation tasks show that the proposed sentence compression method significantly improves the translation performances over strong baselines."]},{"idx":1,"heading":"Introduction","paragraphs":["Neural machine translation (NMT) is popularly implemented as an encoder-decoder framework BIBREF0, in which the encoder is right in charge of source sentence representation. Typically, the input sentence is implicitly represented as a contextualized source representation through deep learning networks. By further feeding the decoder, the source representation is used to learn dependent time-step context vectors for predicting target translation BIBREF1.","In state-of-the-art Transformer-based encoder, self-attention mechanisms are good at capturing the general information in a sentence BIBREF2, BIBREF3, BIBREF4. However, it is difficult to distinguish which kind of information lying deeply under the language is really salient for learning source representation. Intuitively, when a person reads a source sentence, he/she often selectively focuses on the basic sentence meaning, and re-reads the entire sentence to understand its meaning completely. Take the English sentence in Table TABREF2 as an example. We manually annotate its basic meaning as a shorter sequence of words than in the original sentence, called backbone information. Obviously, these words with the basic meaning contain more important information for human understanding than the remaining words in the sentence. We argue that such backbone information is also helpful for learning source representation, and is not explicitly considered by the existing NMT system to enrich the source sentence representation.","In this paper, we propose a novel explicit sentence compression approach to enhance the source representation for NMT. To this end, we first design three sentence compression models to accommodate the needs of various languages and scenarios, including supervised, unsupervised, and semi-supervised ways, to learn a backbone information words sequence (as shown in Table TABREF2) from the source sentence. We then propose three translation models, including backbone source-side fusion based NMT (BSFNMT), backbone target-side fusion (BTFNMT), and both-side fusion based NMT (BBFNMT), to introduce this backbone knowledge into the existing Transformer NMT system for improving translation predictions. Empirical results on the WMT14 English-to-German and English-to-French translation tasks show that the proposed approach significantly improves the translation performance over the strong even state-of-the-art NMT baselines."]},{"idx":2,"heading":"Explicit Sentence Compression","paragraphs":["Generally, sentence compression is a typical sequence generation task which aims to maximize the absorption and long-term retention of large amounts of data over a relatively short sequence for text understanding BIBREF5, BIBREF6. To distinguish the importance of words in the sentence and, more importantly, to dig out the most salient part in the sentence representation, we utilize the sentence compression method to explicitly distill the key knowledge that can retain the key meaning of the sentence, termed explicit sentence compression (ESC) in this paper. Depending on whether or not the sentence compression is trained using human annotated data, the proposed method can be implemented in three ways: supervised ESC, unsupervised ESC, and semi-supervised ESC."]},{"idx":3,"heading":"Explicit Sentence Compression ::: Supervised ESC","paragraphs":["Sentence compression usually relies on large-scale raw data together with their human-labeled data, which can be viewed as supervision, to train a sentence compression model BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11, BIBREF12. For example, BIBREF11 proposed an attentive encoder-decoder recurrent neural network (RNN) to model abstractive text summarization. BIBREF13 furture proposed MAsked Sequence to Sequence pre-training (MASS) for the encoder-decoder sentence compression framework which reported state-of-the-art performance on both the Gigaword Corpus and DUC Corpus.","Sentence compression can be conducted by a typical sequence-to-sequence model. The encoder represents the input sentence $S$ as a sequence of annotation vectors, and the decoder depends on the attention mechanism to learn the context vector for generating a compressed version $S^{^{\\prime }}$ with the key meaning of the input sentence. Recently, the new Transformer architecture proposed by BIBREF0, which fully relies on self-attention networks, has exhibited state-of-the-art translation performance for several language pairs. We follow this practice and attempt to apply the Transformer architecture to such a compression task."]},{"idx":4,"heading":"Explicit Sentence Compression ::: Unsupervised ESC","paragraphs":["A major challenge in supervised sentence compression is the scarce high quality human annotated parallel data. In practice, due to the lack of parallel annotated data, the supervised sentence compression model cannot be trained or the annotated data domain is different, resulting in the sentence compression model trained on the in-domain performing poorly on the out-of-domain.","Supervised sentence compression models have achieved impressive performances based on large corpora containing pairs of verbose and compressed sentences with human annotation BIBREF11, BIBREF13. However, the effectiveness relies heavily on the availability of large amounts of parallel original and human-annotated compressed sentences. This hinders the sentence compression approach from further improvements for many low-resource scenarios. Recently, motivated by recent progress in unsupervised cross-lingual embeddings, the unsupervised NMT BIBREF14, BIBREF15, BIBREF16 opened the door to solving the problem of sequence-to-sequence learning without any parallel sentence pairs. It takes advantage of the lossless (ideal situation) nature of machine translation between languages; i.e., it can translate language $L_1$ to language $L_2$ and back translate $L_2$ to language $L_1$. However, sentence compression does not have this feature. It is lossy from sentence $S$ to sentence $S^{^{\\prime }}$, which makes it difficult to restore from the compressed sentence $S^{^{\\prime }}$ to the original sentence $S$.","BIBREF17 added noises to extend the original sentences and trained a denoising auto-encoder to recover the original, constructing an end-to-end training network without any examples of compressed sentences in sequence to sequence framework. In doing so, the model has to exclude and reorder the noisy sentence input, and hence learns to output more semantic important, shorter but grammatically correct sentences. There are two types of noise used in the model: Additive Sampling Noise and Shuffle Noise.","Additive Sampling Noise: To extend the original sentence, we sample additional sentence from the training dataset randomly, and then sub-sample a subset of words from each without replacement. The newly sampled words are appended to the original sentence.","Shuffle Noise: In order for the model to learn to rephrase the input sentence to make the output shorter, we shuffle the resultant additive noisy sentence.","To gain a better quality for the compressed sentences, we transfer the method of BIBREF17 into the Transformer architecture instead of their suggested RNN architecture, which makes it conducive to deeper network training and a larger corpus."]},{"idx":5,"heading":"Explicit Sentence Compression ::: Semi-supervised ESC","paragraphs":["As pointed out in BIBREF13, sequence to sequence framework has attracted much attention recently due to the advances of deep learning by using large-scale data. Many language generation tasks have only a small scale of pair data which can't support to train a deep model with good generalization ability. In comparison, there is a lot of unpaired data which is earier to obtain.","We observe a performance degradation caused by different domains in the supervised ESC. According to the experimental results of BIBREF17, the accuracy of the unsupervised ESC is currently lower than the supervised one. Therefore, we have further adopted the semi-supervised explicit sentence compression model to alleviate this problem. Specifically, the unsupervised training (often referred to as pre-training) is performed on the unpaired data first and fine-tuning with the small scale paired data (supervised training) to obtain the ESC model with good performance and generalization ability."]},{"idx":6,"heading":"Explicit Sentence Compression ::: Compression Rate Control","paragraphs":["Explicit compression rate (length) control is a common method which has been used in previous sentence compression works. BIBREF18 examined several methods of introducing target output length information, and found that they were effective without negatively impacting summarization quality. BIBREF19 introduced a length marker token that induces the model to target an output of a desired length, coarsely divided into discrete bins. BIBREF17 augmented the decoder with an additional length countdown input which is a single scalar that ticks down to 0 when the generation reached the desired length.","Different with the length marker or length countdown input, to induce our model to output the compression sequence with desired length, we use beam search during generation to find the sequence $S^{^{\\prime }}$ that maximizes a score function $s(S^{^{\\prime }}, S)$ given a trained ESC model. The length normalization is introduced to account for the fact that we have to compare hypotheses of different length. Without some form of length-normalization regular $ln$, beam search will favor shorter sequences over longer ones on average since a negative log-probability is added at each step, yielding lower (more negative) scores for longer sentences. Moreover, a coverage penalty $cp$ is also added to favor the sequence that cover the source sentence meaning as much as possible according to the attention weights BIBREF20.","where $p_{i,j}$ is the attention probability of the $j$-th target word on the $i$-th source word. Parameters $\\alpha $ and $\\beta $ control the strength of the length normalization and the coverage penalty. Although $\\alpha $ can be used to control the compression ratio softly, we use the compression ratio $\\gamma $ to control the maximum length of decoding generation by hard requirements. When the decoding length $|S^{^{\\prime }}|$ is greater than $\\gamma |S|$, the decoding stops."]},{"idx":7,"heading":"NMT with ESC","paragraphs":["In this section, we first introduce the Transformer networks for machine translation. Then based on the fusion position of the backbone knowledge sequence, we propose three novel translation models: the backbone source-side fusion based NMT model (as shown in Figure FIGREF11), the backbone target-side based NMT model (as shown in Figure FIGREF12), and the backbone both-side based NMT. All of these models can make use of the source backbone knowledge generated by our sentence compression models."]},{"idx":8,"heading":"NMT with ESC ::: Transformer Networks","paragraphs":["A Transformer NMT model consists of an encoder and a decoder, which fully rely on self-attention networks (SANs), to translate a sentence in one language into another language with equivalent meaning. Formally, one input sentence $x$=$\\lbrace x_1, \\cdots , x_J\\rbrace $ of length $J$ is first mapped into a sequence of word vectors. Then the sequence and its position embeddings add up to form the input representation $v_x=\\lbrace v^x_1, \\cdots , v^x_J\\rbrace $. The sequence $\\lbrace v^x_1, \\cdots , v^x_J\\rbrace $ is then packed into a query matrix $\\textbf {Q}_x$, a key matrix $\\textbf {K}_x$, and a value matrix $\\textbf {V}_x$. For the SAN-based encoder, the self-attention sub-layer is first performed over $\\textbf {Q}$, $\\textbf {K}$, and $\\textbf {V}$ to the matrix of outputs as:","where $d_{model}$ represents the dimensions of the model. Similarly, the translated target words are used to generate the decoder hidden state $\\textbf {s}_i$ at the current time-step $i$. Generally, the self-attention function is further refined as multi-head self-attention to jointly consider information from different representation subspaces at different positions:","where the projections are parameter matrices $\\textbf {W}_{h}^{Q}$$\\in $$\\mathbb {R}^{d_{model}\\times d_k}$, $\\textbf {W}_{h}^{K}$$\\in $$\\mathbb {R}^{d_{model}\\times d_k}$, $\\textbf {W}_{h}^{V}$$\\in $$ \\mathbb {R}^{d_{model}\\times d_v}$, and $\\textbf {W}^{O}$$\\in $$\\mathbb {R}^{hd_{v}\\times d_{model}}$. For example, there are $H$=8 heads, $d_{model}$ is 512, and $d_k$=$d_v$=512/8=64. A position-wise feed-forward network (FFN) layer is applied over the output of multi-head self-attention, and then is added with the matrix $\\textbf {V}$ to generate the final source representation $H_{x}$=$\\lbrace H^{x}_1, \\cdots , H^{x}_J\\rbrace $:","The SAN of decoder then uses both $H_x$ and target context hidden state $H_{tgt}$ to learn the context vector $o_i$ by “encoder-decoder attention\":","Finally, the context vector $o_{i}$ is used to compute translation probabilities of the next target word $\\textit {y}_i$ by a linear, potentially multi-layered function:","where $\\textbf {L}_{o}$ and $\\textbf {L}_{w}$ are projection matrices."]},{"idx":9,"heading":"NMT with ESC ::: Backbone Source-side Fusion based NMT","paragraphs":["In the backbone source-side fusion based NMT (BSFNMT) model, given an input sentence $x$=$\\lbrace x_1, \\cdots , x_J\\rbrace $, there is an additional compressed sequence $x_c$=$\\lbrace x^c_1, \\cdots , x^c_K\\rbrace $ of length $K$ generated by the proposed sentence compression model. This compressed sequence is also input to the SAN shared with the original encoder with word vectors $v_c = \\lbrace v^c_i, \\cdots , v^c_K\\rbrace $ in shared vocabulary to learn its final representation $H_{c}$=$\\lbrace H^{c}_1, \\cdots , H^{c}_K\\rbrace $. In the proposed SFNMT model, we introduce an additional multi-head attention layer to fuse the compressed sentence and the original input sentence for learning a more effective source representation.","Specifically, for the multi-head attention-fusion layer, a compressed sentence-specific context representation $H_x^c$ is computed by the multi-head attention on the original sentence representation $H_x$ and the compressed sentence representation $H_c$:","$H_x^c$ and $H_x$ are added to form a fusion source representation $H_{x}^{^{\\prime }}$:","Finally, the $H_{x}^{^{\\prime }}$ instead of $H_{x}$ is input to the Eq. (DISPLAY_FORM17) in turn for predicting the target translations word by word."]},{"idx":10,"heading":"NMT with ESC ::: Backbone Target-side Fusion based NMT","paragraphs":["In the backbone target-side fusion based NMT (BTFNMT) model, both the original sentence and its compressed version are also represented as $H_x$ and $H_c$ respectively by the shared SANs. We then use a tuple ($H_x,H_c$) instead of the source-side fusion representation $H_x^{^{\\prime }}$ as the input to the decoder. Specifically, we introduce an additional “encoder-decoder attention\" module into the decoder to learn the compressed sequence context $b_i$ at the current time-step $i$:","Since we are here to treat the original sentence and the compressed sentence as two independent source contexts when encoding at the source side, we use a context gate $g_c$ for integrating two independent contexts of the source: original context $c_i$ and compressed context $b_i$. The gate $g_i$ is calculated by:","Therefore, the final target fusion context $c_i^{\\prime }$ is:","where $\\sigma $ is the logistic sigmoid function, $\\otimes $ is the point-wise multiplication, and $[\\cdot ]$ represent the concatenation operation.","The context $c_i^{\\prime }$ is input to replace the $c_i$ the Eq. (DISPLAY_FORM18) to compute the probabilities of next target word."]},{"idx":11,"heading":"NMT with ESC ::: Backbone Both-side Fusion based NMT","paragraphs":["In the backbone both-side fusion based NMT (BBFNMT) model, we combine BSFNMT and BTFNMT. Both the original representation $H_x$ and its compressed enhanced representation $H_x^{^{\\prime }}$ are as the input to the decoder. Similarly, we introduce an additional “encoder-decoder attention\" module into the decoder to learn the compressed sequence enhanced context $b_i^{^{\\prime }}$ at the current time-step $i$:","Then, the context gate $g_i$ consistent with BTFNMT is applied to combine the two context information $c_i$ and $b_i^{^{\\prime }}$."]},{"idx":12,"heading":"Experiments ::: Setup ::: Sentence Compression","paragraphs":["To evaluate the quality of our sentence compression model, we used the Annotated Gigaword corpus BIBREF21 as the benchmark BIBREF22. The data includes approximately 3.8 M training samples, 400 K validation samples, and 2 K test samples. The byte pair encoding (BPE) algorithm BIBREF23 was adopted for subword segmentation, and the vocabulary size was set at 40 K for our supervised, unsupervised and semi-supervised settings BIBREF24.","Baseline systems include AllText and F8W BIBREF22, BIBREF25. F8W is simply the first 8 words of the input, and AllText uses the whole text as the compression output. The $F_1$ score of ROUGE-1 (R-1), ROUGE-2 (R-2), and ROUGE-L (R-L) was used to evaluate this task BIBREF26. We use beam search with a beam size of 5, the length length normalization of 0.5, and the coverage penalty of 0.2.","For the semi-supervised setting, in order to make the results comparable to BIBREF13, we used the same 190M English monolingual unpaired data from WMT News Crawl datasets for pre-training (unsupervised training). We included the other pretraining methods: masked language modeling (MLM, BERT) BIBREF27, denoising auto-encoder (DAE) BIBREF28, and masked sequence to sequence (MASS) BIBREF13 to compare with our unsupervised pretraining method in the semi-supervised setting."]},{"idx":13,"heading":"Experiments ::: Setup ::: Machine Translation","paragraphs":["The proposed NMT model was evaluated on the WMT14 English-to-German (EN-DE) and English-to-French (EN-FR) tasks, which are both standard large-scale corpora for NMT evaluation. For the EN-DE translation task, 4.43 M bilingual sentence pairs from the WMT14 dataset were used as training data, including Common Crawl, News Commentary, and Europarl v7. The newstest2013 and newstest2014 datasets were used as the dev set and test set, respectively. For the EN-FR translation task, 36 M bilingual sentence pairs from the WMT14 dataset were were used as training data. Newstest12 and newstest13 were combined for validation and the newstest14 was the test set, following the setting of BIBREF29. The BPE algorithm BIBREF23 was also adopted, and the joint vocabulary size was set at 40 K. For the hyper-parameters of our Transformer (base/large) models, we followed the settings used in BIBREF0's work.","In addition, we also reported the state-of-the-art results in recent literatures, including modelling local dependencies (Localness) BIBREF30, fusing multiple-layer representations in SANs (Context-Aware) BIBREF31, and fusing all global context representations in SANs (global-deep context) BIBREF32. MultiBLEU was used to evaluate the translation task."]},{"idx":14,"heading":"Experiments ::: Main Results ::: Sentence Compression","paragraphs":["To evaluate the quality of our sentence compression model, we conducted a horizontal comparison between the proposed sentence compression model and other sentence compression models in different settings. Table TABREF34 shows the comparison results. We observed that the proposed unsupervised ESC model performed substantially better than Fevry and BIBREF17's unsupervised method. The proposed supervised ESC model also substantially outperformed the RNN-based Seq2seq and BIBREF11's baseline method. That is, our supervised model gave +2.0 improvements on R-1, R-2, and R-L scores over the RNN-based Seq2seq. This means that the proposed Transformer-based approaches can generate compressed sentences of high quality.","We further compared our semi-supervised model with the semi-supervised pretraining methods of MLM BIBREF27, DAE BIBREF28, and MASS BIBREF13. Our unsupervised pretrainining method outperformed the other unsupervised pretrainining ones on the sentence compression task consistently."]},{"idx":15,"heading":"Experiments ::: Main Results ::: Machine Translation","paragraphs":["According to the results in Table TABREF34, we chose the semi-supervised ESC model (which performed the best) to generate compressed sentences for the machine translation task. The main results on the WMT14 EN-DE and EN-FR translation tasks are shown in Table TABREF35. In the EN-DE task, we made the following observations:","1) The baseline Transformer (base) in this work achieved a performance comparable to the original Transformer (base) BIBREF0. This indicates that it is a strong baseline NMT system.","2) All BSFNMT, BTFNMT, and BBFNMT significantly outperformed the baseline Transformer (base/big) and only introduces a very small amount of extra parameters. This indicates that the learned compressed backbone information was beneficial for the Transformer translation system.","3) Among the proposed three methods, BTFNMT performed better than BSFNMT. This indicates that the backbone fusion at the target-side is better than at the source-side. In addition, BBFNMT (base/big) outperformed the comparison systems +Localness and +Context-Aware SANs. This indicates that the compression knowledge as an additional context can enhance NMT better.","4) BBFNMT (based) is comparable to the +global-deep context, the best comparison system, while BBFNMT (big) slightly outperformed +global-deep context by $0.16$ BLEU scores. In particular, the parameters of BBFNMT (base/big) model, which just increased $12.1/7.9$M over the Transformer (base/big), were only 70% of the +global-deep context model. This denotes that the BBFNMT model is more efficient than the +global-deep context model. In addition, the training speed of the proposed models slightly decreased ($8\\%$), compared to the corresponding baselines.","5) The proposed BBFNMT (base) slightly outperformed the Transformer (big) which contains much more parameters than BBFNMT (base). This indicates that our improvement is not likely to be due to the increased number of parameters.","For the EN-FR translation task, the proposed models gave similar improvements over the baseline systems and comparing methods (except that the Transformer (big) performed much more better than Transformer (base)). These results show that our method is robust for improving the translation of other language pairs."]},{"idx":16,"heading":"Experiments ::: Ablation Study ::: Evaluating Sentence Compression","paragraphs":["To demonstrate the effectiveness of sentence compression, we compared the compressed sentences ($\\gamma = 0.6$) generated in the Transformer translation system (BBFNMT) under different settings: AllText, F8W, RandSample (random sampling), supervised ESC, Unsupervised ESC and semi-supervised ESC. Table TABREF39 shows the results on newstest2014 for the EN-DE translation task.","We made the following observations: 1) Simply introducing AllText and F8W achieved few improvement, and RandSample is lower than the baseline. In comparison, all the +supervised ESC, +unsupervised ESC, and +semi-supervised ESC models substantially improved the performance over the baseline Transformer (base). This means that our ESC method provides a richer source information for machine translation tasks.","2) +Unsupervised ESC can gain better improvements over the +supervised ESC although supervised ESC model can achieve higher quality than the unsupervised ESC model in the benchmark test dataset. This may be due to that the annotated sentence compression training data is in different domain with the WMT EN-DE traing data. Meanwhile, +Semi-supervised ESC with annotated data fine-tuning outperformed both +Unsupervised and +supervised ESC."]},{"idx":17,"heading":"Experiments ::: Ablation Study ::: Effect of Encoder Parameters","paragraphs":["In our model, representations of the original sentence and its compressed version were learned by a shared encoder. To explore the effect of the encoder parameters, we also designed a BBFNMT with two independent encoders to learn representations of the original sentence and its compressed version, respectively. Table TABREF41 shows results on the newstest2014 test set for the WMT14 EN-DE translation task.","The BBFNMT (w/ independent params) slightly outperformed the proposed shared encoder model by a BLEU score of 0.15, but its parameters increased by approximately 30%. In contrast, the parameters in our model are comparable to the baseline Transformer (base). Considering the parameter scale, we took a shared encoder to learn source representation, which makes it easy to verify the effectiveness of the additional translation knowledge, such as our backbone knowledge."]},{"idx":18,"heading":"Experiments ::: Ablation Study ::: Evaluating Compression Ratio","paragraphs":["In order to verify the impact of different compression ratios on translation quality, we conducted experiments on EN-DE translation task with semi-supervised sentence compression in BBFNMT model.","We controled the compression ratio $\\gamma $ from 0 to 1.0. Consider two boundary conditions, when the compression ratio $\\gamma = 0$, it means no compression sequence generated, which is the same as the vanilla Transformer. When the compression ratio $\\gamma = 1.0$, it is equivalent to re-paraphrasing the source sentence using the sentence compression model (maintaining the same length) as the additional input for BBFNMT.","The experimental results are shown in Fig. FIGREF43. As can be seen from the results, in our experiments, sentence compression (re-paraphrasing) can bring performance improvement, even when the compression ratio $\\gamma =1.0$ and the sentence length is not shortened, re-paraphrasing can still bring slight improvement of translation quality. On the wmt14 EN-DE translation task, the compression ratio $\\gamma $ was set to 0.6 to get the best results."]},{"idx":19,"heading":"Related Work","paragraphs":["To let the translation have more focus over the source sentence information, efforts have been initiated on exploiting sentence segmentation, sentence simplification, and sentence compression for machine translation. BIBREF33 presented a approach to integrating the sentence skeleton information into a phrase-based statistic machine translation system. BIBREF34 proposed an approach to modeling syntactically-motivated skeletal structure of source sentence for statistic machine translation. BIBREF35 describe an early approach to skeleton-based translation, which decomposes input sentences into syntactically meaningful chunks. The central part of the sentence is identified and remains unaltered while other parts of the sentence are simplified. This process produces a set of partial, potentially overlapping translations which are recombined to form the final translation. BIBREF36 describe a “divide and translate” approach to dealing with complex input sentences. They parse the input sentences, replace subclauses with placeholders and later substitute them with separately translated clauses. Their method requires training translation models on clause-level aligned parallel data with placeholders in order for the translation model to deal with the placeholders correctly. BIBREF37 experimented with automatically segmenting the source sentence to overcome problems with overly long sentences. BIBREF38 showed that the spaces of original and simplified translations can be effectively combined using translation lattices and compare two decoding approaches to process both inputs at different levels of integration.","Different from these work, our proposed sentence compression model does not rely on any known linguistics motivated (such as syntax) skeleton simplification, but directly trains a computation motivated sentence compression model to learn to compress sentences and re-paraphrase them directly in seq2seq model. Though with a pure computation source, our sentence compression model can surprisingly generate more grammatically correct and refined sentences, and the words in the compressed sentence do not have to be the same as the original sentence. In the meantime, our sentence compression model can stably give source backbone representation exempt from unstable performance of a syntactic parser which is essential for syntactic skeleton simplification. Our sentence compression model can perform unsupervised training on large-scale data sets, and then use the supervised data for finetune, which is more promising from the results."]},{"idx":20,"heading":"Conclusion and Future work","paragraphs":["To give a more focused source representation, this paper makes the first attempt to propose an explicit sentence compression method to enhance state-of-the-art Transformer-based NMT. To demonstrate that the proposed sentence compression enhancement is indeed helpful for the neural machine translation, We evaluate the impact of the proposed model on the large-scale WMT14 English-to-German and English-to-French translation tasks. The experimental results on WMT14 EN-DE and EN-FR translation tasks show that our proposed NMT model can yield significantly improved results over strong baseline translation systems. In the future work, we will release a pre-trained language model that uses unsupervised sentence compression as the pre-training objective to demonstrate the performance of unsupervised sentence compression in representation learning."]},{"idx":21,"heading":"Acknowledgment","paragraphs":["The corresponding authors are Rui wang and Hai Zhao. Zuchao Li and Zhuosheng Zhang were internship research fellows at NICT when conducting this work. Hai Zhao was partially supported by National Key Research and Development Program of China (No. 2017YFB0304100) and Key Projects of National Natural Science Foundation of China (No. U1836222 and No. 61733011). Rui Wang was partially supported by JSPS grantin-aid for early-career scientists (19K20354): “Unsupervised Neural Machine Translation in Universal Scenarios” and NICT tenure-track researcher startup fund “Toward Intelligent Machine Translation”."]},{"idx":22,"heading":"Figures and Tables","paragraphs":["TABLE 1 An example of sentence compression.","Fig. 2. The architecture of proposed BSFNMT model.","Fig. 3. The architecture of proposed BTFNMT model.","TABLE 2 Performance on the sentence compression task","TABLE 3 Comparison with existing NMT systems on WMT14 EN-DE and EN-FR Translation Tasks. “++/+” after the BLEU score indicate that the proposed method was significantly better than the corresponding baseline Transformer (base or big) at significance level p<0.01/0.05. “#Speed” denotes the decoding speed measured in target tokens per second.","TABLE 5 The effect of encoder parameters.","TABLE 4 The effect of our ESC methods.","Fig. 3. Performances on EN-DE newstest2014 with different sentence compression ratios."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"2001.07820","title":"Elephant in the Room: An Evaluation Framework for Assessing Adversarial Examples in NLP","sections":[{"idx":0,"heading":"Abstract","paragraphs":["An adversarial example is an input transformed by small perturbations that machine learning models consistently misclassify. While there are a number of methods proposed to generate adversarial examples for text data, it is not trivial to assess the quality of these adversarial examples, as minor perturbations (such as changing a word in a sentence) can lead to a significant shift in their meaning, readability and classification label. In this paper, we propose an evaluation framework to assess the quality of adversarial examples based on the aforementioned properties. We experiment with five benchmark attacking methods and an alternative approach based on an auto-encoder, and found that these methods generate adversarial examples with poor readability and content preservation. We also learned that there are multiple factors that can influence the attacking performance, such as the the length of text examples and the input domain."]},{"idx":1,"heading":"Introduction","paragraphs":["Adversarial examples, a term introduced in BIBREF0, are inputs transformed by small perturbations that machine learning models consistently misclassify. The experiments are conducted in the context of computer vision (CV), and the core idea is encapsulated by an illustrative example: after imperceptible noises are added to a panda image, an image classifier predicts, with high confidence, that it is a gibbon. Interestingly, these adversarial examples can also be used to improve the classifier — either as additional training data BIBREF0 or as a regularisation objective BIBREF1 — thus providing motivation for generating effective adversarial examples.","The germ of this paper comes from our investigation of adversarial attack methods for natural language processing (NLP) tasks, e.g. sentiment classification, which drives us to quantify what is an “effective” or “good” adversarial example. In the context of images, a good adversarial example is typically defined according two criteria:","it has successfully fooled the classifier;","it is visually similar to the original example.","In NLP, defining a good adversarial example is a little more involving, because while criterion (b) can be measured with a comparable text similarity metric (e.g. BLEU or edit distance), an adversarial example should also:","be fluent or natural;","preserve its original label.","These two additional criteria are generally irrelevant for images, as adding minor perturbations to an image is unlikely to: (1) create an uninterpretable image (where else changing one word in a sentence can render a sentence incoherent), or (2) change how we perceive the image, say from seeing a panda to a gibbon (but a sentence's sentiment can be reversed by simply adding a negative adverb such as not). Without considering criterion (d), generating adversarial examples in NLP would be trivial, as the model can learn to simply replace a positive adjective (amazing) with a negative one (awful) to attack a sentiment classifier.","To the best of our knowledge, most studies on adversarial example generation in NLP have largely ignored these additional criteria BIBREF2, BIBREF3, BIBREF4, BIBREF5. We believe the lack of a rigorous evaluation framework partially explains why adversarial training for NLP models has not seen the same extent of improvement compared to CV models. As our experiments reveal, examples generated from most attacking methods are successful in fooling the classifier, but their language is often unnatural and the original label is not properly preserved.","The core contribution of our paper is to introduce a systematic, rigorous evaluation framework to assess the quality of adversarial examples for NLP. We focus on sentiment classification as the target task, as it is a popular application that highlights the importance of criteria discussed above. We test a number of attacking methods and also propose an alternative approach (based on an auto-encoder) for generating adversarial examples. We learn that a number of factors can influence the performance of adversarial attacks, including architecture of the classifier, sentence length and input domain."]},{"idx":2,"heading":"Related Work","paragraphs":["Most existing adversarial attack methods for text inputs are derived from those for image inputs. These methods can be categorised into three types including gradient-based attacks, optimisation-based attacks and model-based attacks.","Gradient-based attacks are mainly white-box attacks that rely on calculating the gradients of the target classifier with respect to the input representation. This class of attacking methods BIBREF6, BIBREF7, BIBREF6 are mainly derived from the fast gradient sign method (FGSM) BIBREF1, and it has been shown to be effective in attacking CV classifiers. However, these gradient-based methods could not be applied to text directly because perturbed word embeddings do not necessarily map to valid words. Other methods such as DeepFool BIBREF8 that rely on perturbing the word embedding space face similar roadblocks. BIBREF5 propose to use nearest neighbour search to find the closest word to the perturbed embedding.","Both optimisation-based and model-based attacks treat adversarial attack as an optimisation problem where the constraints are to maximise the loss of target classifiers and to minimise the difference between original and adversarial examples. Between these two, the former uses optimisation algorithms directly; while the latter trains a seperate model to generate adversarial examples and therefore involves a training process. Some of the most effective attacks for images are achieved by optimisation-based methods, such as the L-BFGS attack BIBREF1 and the C&W attack BIBREF9 in white-box attacks and the ZOO method BIBREF10 in black-box attacks. For texts, the white-box attack HotFlip BIBREF3 and black-box attack DeepWordBug BIBREF11 and TextBugger BIBREF12 are proposed in this category.","In a similar vein, a few model-based attacks have been proposed for images, e.g. BIBREF13 design a generative adversarial network (GAN) to generate the image perturbation from a noise map. The attacking method and target classifier typically form a single large network and the attacking method is trained using the loss from the target classifier. For this reason, it is not very straightforward to use these model-based techniques for text because there is a discontinuity in the network (since words in the adversarial examples are discrete) and so it is not fully differentiable."]},{"idx":3,"heading":"Methodology ::: Sentiment Classifiers","paragraphs":["There are a number of off-the-shelf neural models for sentiment classification BIBREF14, BIBREF15, most of which are based on long-short term memory networks (LSTM) BIBREF16 or convolutional neural networks (CNN) BIBREF14. In this paper, we pre-train three sentiment classifiers: BiLSTM, BiLSTM$+$A, and CNN. These classifiers are targeted by white-box attacking methods to generate adversarial examples (detailed in Section SECREF9). BiLSTM is composed of an embedding layer that maps individual words to pre-trained word embeddings; a number of bi-directional LSTMs that capture sequential contexts; and an output layer that maps the averaged LSTM hidden states to a binary output. BiLSTM$+$A is similar to BiLSTM except it has an extra self-attention layer which learns to attend to salient words for sentiment classification, and we compute a weighted mean of the LSTM hidden states prior to the output layer. Manual inspection of the attention weights show that polarity words such as awesome and disappointed are assigned with higher weights. Finally, CNN has a number of convolutional filters of varying sizes, and their outputs are concatenated, pooled and fed to a fully-connected layer followed by a binary output layer.","Recent development in transformer-based pre-trained models have produced state-of-the-art performance on a range of NLP tasks BIBREF17, BIBREF18. To validate the transferability of the attacking methods, we also fine-tune a BERT classifier for black-box tests. That is, we use the adversarial examples generated for attacking the three previous classifiers (BiLSTM, BiLSTM$+$A and CNN) as test data for BERT to measure its classification performance to understand whether these adversarial examples can fool BERT."]},{"idx":4,"heading":"Methodology ::: Benchmark Attacking Methods","paragraphs":["We experiment with five benchmark attacking methods for texts: FGM, FGVM, DeepFool BIBREF5, HotFlip BIBREF3) and TYC BIBREF4.","To perturb the discrete inputs, both FGM and FGVM introduce noises in the word embedding space via the fast gradient method BIBREF1 and reconstruct the input by mapping perturbed word embeddings to valid words via nearest neighbour search. Between FGM and FGVM, the former introduce noises that is proportional to the sign of the gradients while the latter introduce perturbations proportional to the gradients directly. The proportion is known as the overshoot value and denoted by $\\epsilon $. DeepFool uses the same trick to deal with discrete inputs except that, instead of using the fast gradient method, it uses the DeepFool method introduced in BIBREF8 for image to search for an optimal direction to perturb the word embeddings.","Unlike the previous methods, HotFlip and TYC rely on performing one or more atomic flip operations to replace words while monitoring the label change given by the target classifier. In HotFlip, the directional derivatives w.r.t. flip operations are calculated and the flip operation that results in the largest increase in loss is selected. TYC is similar to FGM, FGVM and DeepFool in that it also uses nearest neighbour search to map the perturbed embeddings to valid words, but instead of using the perturbed tokens directly, it uses greedy search or beam search to flip original tokens to perturbed ones one at a time in order of their vulnerability."]},{"idx":5,"heading":"Methodology ::: Model-based Attacking Method","paragraphs":["The benchmark methods we test (Section SECREF9) are gradient-based and optimisation-based attacks (Section SECREF2). We propose an alternative model-based method to train a seperate generative model to generate text adversarial examples. Denoting the generative model as $\\mathcal {G}$, we train it to generate an adversarial example $X^{\\prime }$ given an input example $X$ such that $X^{\\prime }$ is similar to $X$ but that it changes the prediction of the target classifier $\\mathcal {D}$, i.e. $X^{\\prime } \\sim X$ and $\\mathcal {D}(X^{\\prime }) \\ne \\mathcal {D}(X)$.","For images, it is straightforward to combine $\\mathcal {G}$ and $\\mathcal {D}$ in the same network and use the loss of $\\mathcal {D}$ to update $\\mathcal {G}$ because $X^{\\prime }$ is continuous and can be fed to $\\mathcal {D}$ while keeping the whole network differentiable. For texts, $X^{\\prime }$ are discrete symbols (typically decoded with argmax or beam-search) and so the network is not fully differentiable. To create a fully differentiable network with $\\mathcal {G}+\\mathcal {D}$, we propose to use Gumbel-Softmax to imitate the categorical distribution BIBREF19. As the temperature ($\\tau $) of Gumbel-Softmax sampling approaches 0, the distribution of the sampled output $X^*$ is identical to $X^{\\prime }$. We illustrate this in Figure FIGREF12, where $X^*$ is fed to $\\mathcal {D}$ during training while $X^{\\prime }$ is generated as the output (i.e. the adversarial example) at test time.","$\\mathcal {G}$ is designed as an auto-encoder to reconstruct the input (denoted as AutoEncoder), with the following objectives to capture the evaluation criteria described in Section SECREF1:","$L_{adv}$, the adversarial loss that maximises the cross-entropy of $\\mathcal {D}$;","$L_{seq}$, the auto-encoder loss to reconstruct $X^{\\prime }$ from $X^{\\prime }$;","$L_{sem}$, the cosine distance between the mean embeddings of $X$ and $X^*$.","$L_{adv}$ ensures that the adversarial examples are fooling the classifier (criterion (a)); $L_{seq}$ regulates the adversarial example so that it isn't too different from the original input and has a reasonable language (criteria (b) and (c)); and $L_{sem}$ constrains the underlying semantic content of the original and adversarial sentences to be similar (criterion (b)); so reduces the likelihood of a sentiment flip (indirectly for criteria (d)). Note that criteria (d) is the perhaps the most difficult aspect to handle, as its interpretation is ultimately a human judgement.","We use two scaling hyper-parameters $\\lambda _{ae}$ and $\\lambda _{seq}$ to weigh the three objectives: $\\lambda _{ae}(\\lambda _{seq} * L_{seq} + (1-\\lambda _{seq})*L_{sem}) + (1-\\lambda _{ae})*L_{adv}$. We introduce attention layers to AutoEncoder and test both greedy and beam search decoding to improve the quality of generated adversarial examples. During training we alternate between positive-to-negative and negative-to-positive attack for different mini-batches."]},{"idx":6,"heading":"Experiments ::: Datasets","paragraphs":["We construct three datasets based on IMDB reviews and Yelp reviews. The IMDB dataset is binarised and split into a training and test set, each with 25K reviews (2K reviews from the training set are reserved for development). We filter out any review that has more than 400 tokens, producing the final dataset (imdb400). For Yelp, we binarise the ratings, and create 2 datasets, where we keep only reviews with $\\le $ 50 tokens (yelp50) and $\\le $200 tokens (yelp200). We randomly partition both datasets into train/dev/test sets (90/5/5 for yelp50; 99/0.5/0.5 for yelp200). For all datasets, we use spaCy for tokenisation. We train and tune target classifiers (see Section SECREF8) using the training and development sets; and evaluate their performance on the original examples in the test sets as well as the adversarial examples generated by attacking methods for the test sets. Note that AutoEncoder also involves a training process, for which we train and tune AutoEncoder using the training and development sets in yelp50, yelp200 and imdb400. Statistics of the three datasets are presented in Table TABREF22. These datasets present a variation in the text lengths (e.g. the average number of words for yelp50, yelp200 and imdb400 is 34, 82 and 195 words respectively), training data size (e.g. the number of training examples for target classifiers for imdb400, yelp50 and yelp200 are 18K, 407K and 2M, respectively) and input domain (e.g. restaurant vs. movie reviews)."]},{"idx":7,"heading":"Experiments ::: Implementation Details","paragraphs":["We use the pre-trained glove.840B.300d embeddings BIBREF20 for all 6 attacking methods. For FGM, FGVM and DeepFool, we tune $\\epsilon $, the overshoot hyper-parameter (Section SECREF9) and keep the iterative step $n$ static (5). For TYC, besides $\\epsilon $ we also tune the upper limit of flipped words, ranging from 10%–100% of the maximum length. For HotFlip, we tune only the upper limit of flipped words, in the range of $[1, 7]$.","We pre-train AutoEncoder to reconstruct sentences in different datasets as we found that this improves the quality of the generated adversarial examples. During pre-training, we tune batch size, number of layers and number of units, and stop the training after the performance on the development sets stops improving for 20K steps. The model is then initialised with the pre-trained weights and trained based on objectives defined in Section SECREF11. During the training process we tune $\\lambda _{ae}$ and $\\lambda _{seq}$ while keeping the batch size (32) and learning rate ($1e^{-4}$) fixed. As part of our preliminary study, we also tested different values for the Gumbel-softmax temperature $\\tau $ and find that $\\tau =0.1$ performs the best. Embeddings are fixed throughout all training processes.","For target classifiers, we tune batch size, learning rate, number of layers, number of units, attention size (BiLSTM$+$A), filter sizes and dropout probability (CNN). For BERT, we use the default fine-tuning hyper-parameter values except for batch size, where we adjust based on memory consumption. Note that after the target classifiers are trained their weights are not updated when training or testing the attacking methods."]},{"idx":8,"heading":"Evaluation","paragraphs":["We propose both automatic metrics and manual evaluation strategies to assess the quality of adversarial examples, based on four criteria defined in Section SECREF1: (a) attacking performance (i.e. how well they fool the classifier); (b) textual similarity between the original input and the adversarial input; (c) fluency of the adversarial example; and (d) label preservation. Note that the automatic metrics only address the first 3 criteria (a, b and c); we contend that criterion (d) requires manual evaluation, as the judgement of whether the original label is preserved is inherently a human decision."]},{"idx":9,"heading":"Evaluation ::: Automatic Evaluation: Metrics","paragraphs":["As sentiment classification is our target task, we use the standard classification accuracy (ACC) to evaluate the attacking performance of adversarial examples (criterion (a)).","To assess the similarity between the original and (transformed) adversarial examples (criteria (b)), we compute BLEU scores BIBREF21.","To measure fluency, we first explore a supervised BERT model fine-tuned to predict linguistic acceptability BIBREF17. However, in preliminary experiments we found that BERT performs very poorly at predicting the acceptability of adversarial examples (e.g. it predicts word-salad-like sentences generated by FGVM as very acceptable), revealing the brittleness of these supervised models. We next explore an unsupervised approach BIBREF22, BIBREF23, using normalised sentence probabilities estimated by pre-trained language models for measuring acceptability. In the original papers, the authors tested simple recurrent language models; here we use modern pre-trained language models such as GPT-2 BIBREF24 and XLNet BIBREF18. Our final acceptability metric (ACPT) is based on normalised XLNet sentence probabilities: ${\\log P(s)} / ({((5+|s|)/(5+1))^\\alpha })$, where $s$ is the sentence, and $\\alpha $ is a hyper-parameter (set to 0.8) to dampen the impact of large values BIBREF25.","We only computed BLEU and ACPT scores for adversarial examples that have successfully fooled the classifier. Our rationale is that unsuccessful examples can artificially boost these scores by not making any modifications, and so the better approach is to only consider successful examples."]},{"idx":10,"heading":"Evaluation ::: Automatic Evaluation: Results","paragraphs":["We present the performance of the attacking methods against 3 target classifiers (Table TABREF23A; top) and on 3 datasets (Table TABREF23B; bottom). We choose 3 ACC thresholds for the attacking performance: T0, T1 and T2, which correspond approximately to accuracy scores of 90%, 80% and 70% for the Yelp datasets (yelp50, yelp200); and 80%, 70% and 50% for the IMDB datasets (imdb400). Each method is tuned accordingly to achieve a particular accuracy. Missing numbers (dashed lines) indicate the method is unable to produce the desired accuracy, e.g. HotFlip with only 1 word flip produces 81.5% accuracy (T1) when attacking CNN on yelp50, and so T0 accuracy is unachievable.","Looking at BLEU and ACPT, HotFlip is the most consistent method over multiple datasets and classifiers. AutoEncoder is also fairly competitive, producing largely comparable performance (except for the yelp200 dataset). Gradient-based methods FGM and FGVM perform very poorly. In general, they tend to produce word salad adversarial examples, as indicated by their poor BLEU scores. DeepFool similarly generates incoherent sentences with low BLEU scores, but occasionally produces good ACPT (BiLSTM at T1 and T2), suggesting potential brittleness of the unsupervised approach for evaluating acceptability.","Comparing the performance across different ACC thresholds, we observe a consistent pattern of decreasing performance over all metrics as the attacking performance increases from T0 to T2. These observations suggest that all methods are trading off fluency and content preservation as they attempt to generate stronger adversarial examples.","We now focus on Table TABREF23A, to understand the impact of model architecture for the target classifier. With 1 word flip as the upper limit for HotFlip, the accuracy of BiLSTM$+$A and BiLSTM drops to T0 (approximately 4% accuracy decrease) while the accuracy of CNN drops to T1 (approximately 13% accuracy decrease), suggesting that convolutional networks are more vulnerable to attacks (noting that it is the predominant architecture for CV). Looking at Table TABREF23B, we also find that the attacking performance is influenced by the input text length and the number of training examples for target classifiers. For HotFlip, we see improvements over BLEU and ACPT as text length increases from yelp50 to yelp200 to imdb400, indicating the performance of HotFlip is more affected by input lengths. We think this is because with more words it is more likely for HotFlip to find a vulnerable spot to target. While for TYC and AutoEncoder, we see improvements over BLEU and ACPT as the number of training examples for target classifier decreases from yelp200 (2M) to yelp50 (407K) to imdb400 (22K), indicating these two methods are less effective for attacking target classifiers that are trained with more data. Therefore, increasing the training data for target classifier may improve their robustness against adversarial examples that are generated by certain methods.","As a black-box test to check how well these adversarial examples generalise to fooling other classifiers, also known as transferability, we feed the adversarial examples from the 3 best methods, i.e. TYC, HotFlip and AutoEncoder, to a pre-trained BERT trained for sentiment classification (Figure FIGREF31) and measure its accuracy. Unsurprisingly, we observe that the attacking performance (i.e. the drop in ACC) is not as good as those reported for the white-box tests. Interestingly, we find that HotFlip, the best method, produces the least effective adversarial examples for BERT. Both TYC and AutoEncoder perform better here, as their generated adversarial examples do well in fooling BERT.","To summarise, our results demonstrate that the best white-box methods (e.g. HotFlip) may not produce adversarial examples that generalise to fooling other classifiers. We also saw that convolutional networks are more vulnerable than recurrent networks, and that dataset features such as text lengths and training data size (for target classifiers) can influence how difficult it is to perform adversarial attack."]},{"idx":11,"heading":"Evaluation ::: Human Evaluation: Design","paragraphs":["Automatic metrics provide a proxy to quantify the quality of the adversarial examples. To validate that these metrics work, we conduct a crowdsourcing experiment on Figure-Eight. Recall that the automatic metrics do not assess sentiment preservation (criterion (d)); we evaluate that aspect here.","We experiment with the 3 best methods (TYC, AutoEncoder and HotFlip) on 2 accuracy thresholds (T0 and T2), using BiLSTM$+$A as the classifier. For each method and threshold, we randomly sample 25 positive-to-negative and 25 negative-to-positive examples. To control for quality, we reserve and annotate 10% of the samples ourselves as control questions. Workers are first presented with 10 control questions as a quiz, and only those who pass the quiz with at least 80% accuracy can continue to work on the task. We display 10 questions per page, where one control question is embedded to continue monitor worker performance. The task is designed such that each control question can only be seen once per worker. We restrict our jobs to workers in United States, United Kingdoms, Australia, and Canada.","To evaluate the criteria discussed in Section SECREF1: (b) textual similarity, (c) fluency, and (d) sentiment preservation, we ask the annotators three questions:","Is snippet B a good paraphrase of snippet A?","$\\circledcirc $ Yes $\\circledcirc $ Somewhat yes $\\circledcirc $ No","How natural does the text read?","$\\circledcirc $ Very unnatural $\\circledcirc $ Somewhat natural $\\circledcirc $ Natural","What is the sentiment of the text?","$\\circledcirc $ Positive $\\circledcirc $ Negative $\\circledcirc $ Cannot tell","For question 1, we display both the adversarial input and the original input, while for question 2 and 3 we present only the adversarial example. As an upper-bound, we also run a survey on question 2 and 3 for 50 random original samples."]},{"idx":12,"heading":"Evaluation ::: Human Evaluation: Results","paragraphs":["We present the percentage of answers to each question in Figure FIGREF38. The green bars illustrate how well the adversarial examples paraphrase the original ones; blue how natural the adversarial examples read; and red whether the sentiment of the adversarial examples is consistent compared to the original.","Looking at the performance of the original sentences (“(a) Original samples”), we see that their language is largely fluent and their sentiment is generally consistent to the original examples', although it's worth noting that the review sentiment of imdb400 can be somewhat ambiguous (63% agreement). We think this is due to movie reviews being more descriptive and therefore creating potential ambiguity in their expression of sentiment.","On content preservation (criterion (b); green bars), all methods produce poor paraphrases on yelp50. For imdb400, however, the results are more promising; adversarial examples generated by HotFlip, in particular, are good, even at T2.","Next we look at fluency (criterion (c); blue bars). We see a similar trend: performance in imdb400 is substantially better than yelp50. In fact we see almost no decrease in fluency in the adversarial examples compared to the original in imdb400. In yelp50, HotFlip and AutoEncoder are fairly competitive, producing adversarial examples that are only marginally less fluent compared to the original at T0. At T2, however, these methods begin to trade off fluency. All in all, the paraphrasability and fluency surveys suggest that imdb400 is an easier dataset for adversarial experiments, and it is the predominant dataset used by most studies.","Lastly, we consider sentiment preservation (criterion (d); red bars). All methods perform poorly at preserving the original sentiment on both yelp50 and imdb400 datasets. The artifact is arguably more profound on shorter inputs, as the original examples in imdb400 have lower agreement in the first place (yelp50 vs. imdb400: 86% vs. 63%). Again both HotFlip and AutoEncoder are the better methods here (interestingly, we observe an increase in agreement as their attacking performance increases from T0 and T2).","Summarising our findings, HotFlip is generally the best method across all criteria, noting that its adversarial examples, however, have poor transferability. TYC generates good black-box adversarial examples but do not do well in terms of content preservation and fluency. AutoEncoder produces comparable results with HotFlip for meeting the four criteria and generates examples that generalise reasonably, but it is very sensitive to the increase of training examples for the target classifier. The ACPT metric appears to be effective in evaluating fluency, as we see good agreement with human evaluation. All said, we found that all methods tend to produce adversarial examples that do not preserve their original sentiments, revealing that these methods in a way “cheat” by simply flipping the sentiments of the original sentences to fool the classifier, and therefore the adversarial examples might be ineffective for adversarial training, as they are not examples that reveal potential vulnerabilities in the classifier."]},{"idx":13,"heading":"Conclusion","paragraphs":["We propose an evaluation framework for assessing the quality of adversarial examples in NLP, based on four criteria: (a) attacking performance, (b) textual similarity; (c) fluency; (d) label preservation. Our framework involves both automatic and human evaluation, and we test 5 benchmark methods and a novel auto-encoder approach. We found that the architecture of the target classifier is an important factor when it comes to attacking performance, e.g. CNNs are more vulnerable than LSTMs; dataset features such as length of text, training data size (for target classifiers) and input domains are also influencing factors that affect how difficulty it is to perform adversarial attack; and the predominant dataset (IMDB) used by most studies is comparatively easy for adversarial attack. Lastly, we also observe in our human evaluation that on shorter texts (Yelp) these methods produce adversarial examples that tend not to preserve their semantic content and have low readability. More importantly, these methods also “cheat” by simply flipping the sentiment in the adversarial examples, and this behaviour is evident on both datasets, suggesting they could be ineffective for adversarial training."]},{"idx":14,"heading":"Figures and Tables","paragraphs":["Figure 1: Framework to generate adversarial examples.","Table 1: Dataset statistics.","Table 2: Results based on automatic metrics. Top half (A) presents 3 different target classifiers evaluated on one dataset (yelp50); bottom half (B) tests 3 datasets using one classifier (BiLSTM+A). For ACPT, less negative values are better. Boldface indicates optimal performance in each column.","Figure 2: Accuracy of BERT on adversarial examples generated from TYC, HOTFLIP and AUTOENCODER for different target classifiers (top row) and for different datasets (bottom row). BERT’s accuracy on original examples are denoted as red line in each plot.","Figure 3: Human evaluation results."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"2002.01984","title":"UNCC Biomedical Semantic Question Answering Systems. BioASQ: Task-7B, Phase-B","sections":[{"idx":0,"heading":"Abstract","paragraphs":["In this paper, we detail our submission to the 2019, 7th year, BioASQ competition. We present our approach for Task-7b, Phase B, Exact Answering Task. These Question Answering (QA) tasks include Factoid, Yes/No, List Type Question answering. Our system is based on a contextual word embedding model. We have used a Bidirectional Encoder Representations from Transformers(BERT) based system, fined tuned for biomedical question answering task using BioBERT. In the third test batch set, our system achieved the highest MRR score for Factoid Question Answering task. Also, for List type question answering task our system achieved the highest recall score in the fourth test batch set. Along with our detailed approach, we present the results for our submissions, and also highlight identified downsides for our current approach and ways to improve them in our future experiments."]},{"idx":1,"heading":"Introduction","paragraphs":["BioASQ is a biomedical document classification, document retrieval, and question answering competition, currently in its seventh year. We provide an overview of our submissions to semantic question answering task (7b, Phase B) of BioASQ 7 (except for 'ideal answer' test, in which we did not participate this year). In this task systems are provided with biomedical questions and are required to submit ideal and exact answers to those questions. We have used BioBERT BIBREF0 based system , see also Bidirectional Encoder Representations from Transformers(BERT) BIBREF1, and we fine tuned it for the biomedical question answering task. Our system scored near the top for factoid questions for all the batches of the challenge. More specifially, in the third test batch set, our system achieved highest ‘MRR’ score for Factoid Question Answering task. Also, for List-type question answering task our system achieved highest recall score in the fourth test batch set. Along with our detailed approach, we present the results for our submissions and also highlight identified downsides for our current approach and ways to improve them in our future experiments. In last test batch results we placed 4th for List-type questions and 3rd for Factoid-type questions.)","The QA task is organized in two phases. Phase A deals with retrieval of the relevant document, snippets, concepts, and RDF triples, and phase B deals with exact and ideal answer generations (which is a paragraph size summary of snippets). Exact answer generation is required for factoid, list, and yes/no type question.","BioASQ organizers provide the training and testing data. The training data consists of questions, gold standard documents, snippets, concepts, and ideal answers (which we did not use in this paper, but we used last year BIBREF2). The test data is split between phases A and B. The Phase A dataset consists of the questions, unique ids, question types. The Phase B dataset consists of the questions, golden standard documents, snippets, unique ids and question types. Exact answers for factoid type questions are evaluated using strict accuracy (the top answer), lenient accuracy (the top 5 answers), and MRR (Mean Reciprocal Rank) which takes into account the ranks of returned answers. Answers for the list type question are evaluated using precision, recall, and F-measure."]},{"idx":2,"heading":"Related Work ::: BioAsq","paragraphs":["Sharma et al. BIBREF3 describe a system with two stage process for factoid and list type question answering. Their system extracts relevant entities and then runs supervised classifier to rank the entities. Wiese et al. BIBREF4 propose neural network based model for Factoid and List-type question answering task. The model is based on Fast QA and predicts the answer span in the passage for a given question. The model is trained on SQuAD data set and fine tuned on the BioASQ data. Dimitriadis et al. BIBREF5 proposed two stage process for Factoid question answering task. Their system uses general purpose tools such as Metamap, BeCas to identify candidate sentences. These candidate sentences are represented in the form of features, and are then ranked by the binary classifier. Classifier is trained on candidate sentences extracted from relevant questions, snippets and correct answers from BioASQ challenge. For factoid question answering task highest ‘MRR’ achieved in the 6th edition of BioASQ competition is ‘0.4325’. Our system is a neural network model based on contextual word embeddings BIBREF1 and achieved a ‘MRR’ score ‘0.6103’ in one of the test batches for Factoid Question Answering task."]},{"idx":3,"heading":"Related Work ::: A minimum background on BERT","paragraphs":["BERT stands for \"Bidirectional Encoder Representations from Transformers\" BIBREF1 is a contextual word embedding model. Given a sentence as an input, contextual embedding for the words are returned. The BERT model was designed so it can be fine tuned for 11 different tasks BIBREF1, including question answering tasks. For a question answering task, question and paragraph (context) are given as an input. A BERT standard is that question text and paragraph text are separated by a separator [Sep]. BERT question-answering fine tuning involves adding softmax layer. Softmax layer takes contextual word embeddings from BERT as input and learns to identity answer span present in the paragraph (context). This process is represented in Figure FIGREF4.","BERT was originally trained to perform tasks such as language model creation using masked words and next-sentence-prediction. In other words BERT weights are learned such that context is used in building the representation of the word, not just as a loss function to help learn a context-independent representation. For detailed understanding of BERT Architecture, please refer to the original BERT paper BIBREF1."]},{"idx":4,"heading":"Related Work ::: A minimum background on BERT ::: Comparison of Word Embeddings and Contextual Word Embeddings","paragraphs":["A ‘word embedding’ is a learned representation. It is represented in the form of vector where words that have the same meaning have a similar vector representation. Consider a word embedding model 'word2vec' BIBREF6 trained on a corpus. Word embeddings generated from the model are context independent that is, word embeddings are returned regardless of where the words appear in a sentence and regardless of e.g. the sentiment of the sentence. However, contextual word embedding models like BERT also takes context of the word into consideration."]},{"idx":5,"heading":"Related Work ::: Comparison of BERT and Bio-BERT","paragraphs":["‘BERT’ and BioBERT are very similar in terms of architecture. Difference is that ‘BERT’ is pretrained on Wikipedia articles, whereas BioBERT version used in our experiments is pretrained on Wikipedia, PMC and PubMed articles. Therefore BioBERT model is expected to perform well with biomedical text, in terms of generating contextual word embeddings.","BioBERT model used in our experiments is based on BERT-Base Architecture; BERT-Base has 12 transformer Layers where as BERT-Large has 24 transformer layers. Moreover contextual word embedding vector size is 768 for BERT-Base and more for BERT-large. According to BIBREF1 Bert-Large, fine-tuned on SQuAD 1.1 question answering data BIBREF7 can achieve F1 Score of 90.9 for Question Answering task where as BERT-Base Fine-tuned on the same SQuAD question answering BIBREF7 data could achieve F1 score of 88.5. One downside of the current version BioBERT is that word-piece vocabulary is the same as that of original BERT Model, as a result word-piece vocabulary does not include biomedical jargon. Lee et al. BIBREF0 created BioBERT, using the same pre-trained BERT released by Google, and hence in the word-piece vocabulary (vocab.txt), as a result biomedical jargon is not included in word-piece vocabulary. Modifying word-piece vocabulary (vocab.txt) at this stage would loose original compatibility with ‘BERT’, hence it is left unmodified.","In our future work we would like to build pre-trained ‘BERT’ model from scratch. We would pretrain the model with biomedical corpus (PubMed, ‘PMC’) and Wikipedia. Doing so would give us scope to create word piece vocabulary to include biomedical jargon and there are chances of model performing better with biomedical jargon being included in the word piece vocabulary. We will consider this scenario in the future, or wait for the next version of BioBERT."]},{"idx":6,"heading":"Experiments: Factoid Question Answering Task","paragraphs":["For Factoid Question Answering task, we fine tuned BioBERT BIBREF0 with question answering data and added new features. Fig. FIGREF4 shows the architecture of BioBERT fine tuned for question answering tasks: Input to BioBERT is word tokenized embeddings for question and the paragraph (Context). As per the ‘BERT’ BIBREF1 standards, tokens ‘[CLS]’ and ‘[SEP]’ are appended to the tokenized input as illustrated in the figure. The resulting model has a softmax layer formed for predicting answer span indices in the given paragraph (Context). On test data, the fine tuned model generates $n$-best predictions for each question. For a question, $n$-best corresponds that $n$ answers are returned as possible answers in the decreasing order of confidence. Variable $n$ is configurable. In our paper, any further mentions of ‘answer returned by the model’ correspond to the top answer returned by the model."]},{"idx":7,"heading":"Experiments: Factoid Question Answering Task ::: Setup","paragraphs":["BioASQ provides the training data. This data is based on previous BioASQ competitions. Train data we have considered is aggregate of all train data sets till the 5th version of BioASQ competition. We cleaned the data, that is, question-answering data without answers are removed and left with a total count of ‘530’ question answers. The data is split into train and test data in the ratio of 94 to 6; that is, count of '495' for training and '35' for testing.","The original data format is converted to the BERT/BioBERT format, where BioBERT expects ‘start_index’ of the actual answer. The ‘start_index corresponds to the index of the answer text present in the paragraph/ Context. For finding ‘start_index’ we used built-in python function find(). The function returns the lowest index of the actual answer present in the context(paragraph). If the answer is not found ‘-1’ is returned as the index. The efficient way of finding start_index is, if the paragraph (Context) has multiple instances of answer text, then ‘start_index’ of the answer should be that instance of answer text whose context actually matches with what’s been asked in the question.","Example (Question, Answer and Paragraph from BIBREF8):","Question: Which drug should be used as an antidote in benzodiazepine overdose?","Answer: 'Flumazenil'","Paragraph(context):","\"Flumazenil use in benzodiazepine overdose in the UK: a retrospective survey of NPIS data. OBJECTIVE: Benzodiazepine (BZD) overdose (OD) continues to cause significant morbidity and mortality in the UK. Flumazenil is an effective antidote but there is a risk of seizures, particularly in those who have co-ingested tricyclic antidepressants. A study was undertaken to examine the frequency of use, safety and efficacy of flumazenil in the management of BZD OD in the UK. METHODS: A 2-year retrospective cohort study was performed of all enquiries to the UK National Poisons Information Service involving BZD OD. RESULTS: Flumazenil was administered to 80 patients in 4504 BZD-related enquiries, 68 of whom did not have ventilatory failure or had recognised contraindications to flumazenil. Factors associated with flumazenil use were increased age, severe poisoning and ventilatory failure. Co-ingestion of tricyclic antidepressants and chronic obstructive pulmonary disease did not influence flumazenil administration. Seizure frequency in patients not treated with flumazenil was 0.3%\".","Actual answer is 'Flumazenil', but there are multiple instances of word 'Flu-mazenil'. Efficient way to identify the start-index for 'Flumazenil'(answer) is to find that particular instance of the word 'Flumazenil' which matches the context for the question. In the above example 'Flumazenil' highlighted in bold is the actual instance that matches question's context. Unfortunately, we could not identify readily available tools that can achieve this goal. In our future work, we look forward to handling these scenarios effectively.","Note: The creators of 'SQuAD' BIBREF7 have handled the task of identifying answer's start_index effectively. But 'SQuAD' data set is much more general and does not include biomedical question answering data."]},{"idx":8,"heading":"Experiments: Factoid Question Answering Task ::: Training and error analysis","paragraphs":["During our training with the BioASQ data, learning rate is set to 3e-5, as mentioned in the BioBERT paper BIBREF0. We started training the model with 495 available train data and 35 test data by setting number of epochs to 50. After training with these hyper-parameters training accuracy(exact match) was 99.3%(overfitting) and testing accuracy is only 4%. In the next iteration we reduced the number of epochs to 25 then training accuracy is reduced to 98.5% and test accuracy moved to 5%. We further reduced number of epochs to 15, and the resulting training accuracy was 70% and test accuracy 15%. In the next iteration set number of epochs to 12 and achieved train accuracy of 57.7% and test accuracy 23.3%. Repeated the experiment with 11 epochs and found training accuracy to be 57.7% and test accuracy to be same 22%. In the next iteration we set number of epochs to '9' and found training accuracy of 48% and test accuracy of 15%. Hence optimum number of epochs is taken as 12 epochs.","During our error analysis we found that on test data, model tends to return text in the beginning of the context(paragraph) as the answer. On analysing train data, we found that there are '120'(out of '495') question answering data instances having start_index:0, meaning 120( 25%) question answering data has first word(s) in the context(paragraph) as the answer. We removed 70% of those instances in order to make train data more balanced. In the new train data set we are left with '411' question answering data instances. This time we got the highest test accuracy of 26% at 11 epochs. We have submitted our results for BioASQ test batch-2, got strict accuracy of 32% and our system stood in 2nd place. Initially, hyper-parameter- 'batch size' is set to ‘400’. Later it is tuned to '32'. Although accuracy(exact answer match) remained at 26%, model generated concise and better answers at batch size ‘32’, that is wrong answers are close to the expected answer in good number of cases.","Example.(from BIBREF8)","Question: Which mutated gene causes Chediak Higashi Syndrome?","Exact Answer: ‘lysosomal trafficking regulator gene’.","The answer returned by a model trained at ‘400’ batch size is ‘Autosomal-recessive complicated spastic paraplegia with a novel lysosomal trafficking regulator’, and from the one trained at ‘32’ batch size is ‘lysosomal trafficking regulator’.","In further experiments, we have fine tuned the BioBERT model with both ‘SQuAD’ dataset (version 2.0) and BioAsq train data. For training on ‘SQuAD’, hyper parameters- Learning rate and number of epochs are set to ‘3e-3’ and ‘3’ respectively as mentioned in the paper BIBREF1. Test accuracy of the model boosted to 44%. In one more experiment we trained model only on ‘SQuAD’ dataset, this time test accuracy of the model moved to 47%. The reason model did not perform up to the mark when trained with ‘SQuAD’ alongside BioASQ data could be that in formatted BioASQ data, start_index for the answer is not accurate, and affected the overall accuracy."]},{"idx":9,"heading":"Our Systems and Their Performance on Factoid Questions","paragraphs":["We have experimented with several systems and their variations, e.g. created by training with specific additional features (see next subsection). Here is their list and short descriptions. Unfortunately we did not pay attention to naming, and the systems evolved between test batches, so the overall picture can only be understood by looking at the details.","When we started the experiments our objective was to see whether BioBERT and entailment-based techniques can provide value for in the context of biomedical question answering. The answer to both questions was a yes, qualified by many examples clearly showing the limitations of both methods. Therefore we tried to address some of these limitations using feature engineering with mixed results: some clear errors got corrected and new errors got introduced, without overall improvement but convincing us that in future experiments it might be worth trying feature engineering again especially if more training data were available.","Overall we experimented with several approaches with the following aspects of the systems changing between batches, that is being absent or present:","training on BioAsq data vs. training on SQuAD","using the BioAsq snippets for context vs. using the documents from the provided URLs for context","adding or not the LAT, i.e. lexical answer type, feature (see BIBREF9, BIBREF10 and an explanation in the subsection just below).","For Yes/No questions (only) we experimented with the entailment methods.","We will discuss the performance of these models below and in Section 6. But before we do that, let us discuss a feature engineering experiment which eventually produced mixed results, but where we feel it is potentially useful in future experiments."]},{"idx":10,"heading":"Our Systems and Their Performance on Factoid Questions ::: LAT Feature considered and its impact (slightly negative)","paragraphs":["During error analysis we found that for some cases, answer being returned by the model is far away from what it is being asked in the Question.","Example: (from BIBREF8)","Question: Hy's law measures failure of which organ?","Actual Answer: ‘Liver’.","The answer returned by one of our models was ‘alanine aminotransferase’, which is an enzyme. The model returns an enzyme, when the question asked for the organ name. To address this type of errors, we decided to try the concepts of ‘Lexical Answer Type’ (LAT) and Focus Word, which was used in IBM Watson, see BIBREF11 for overview; BIBREF10 for technical details, and BIBREF9 for details on question analysis. In an example given in the last source we read:","POETS & POETRY: He was a bank clerk in the Yukon before he published \"Songs of a Sourdough\" in 1907.","The focus is the part of the question that is a reference to the answer. In the example above, the focus is \"he\".","LATs are terms in the question that indicate what type of entity is being asked for.","The headword of the focus is generally a LAT, but questions often contain additional LATs, and in the Jeopardy! domain, categories are an additional source of LATs.","(...) In the example, LATs are \"he\", \"clerk\", and \"poet\".","For example in the question \"Which plant does oleuropein originate from?\" (BIBREF8). The LAT here is ‘plant’. For the BioAsq task we did not need to explicitly distinguish between the focus and the LAT concepts. In this example, the expectation is that answer returned by the model is a plant. Thus it is conceivable that the cosine distance between contextual embedding of word 'plant' in the question and contextual embedding for the answer present in the paragraph(context) is comparatively low. As a result model learns to adjust its weights during training phase and returns answers with low cosine distance with the LAT.","We used Stanford CoreNLP BIBREF12 library to write rules for extracting lexical answer type present in the question, both 'parts of speech'(POS) and dependency parsing functionality was used. We incorporated the Lexical Answer Type into one of our systems, UNCC_QA1 in Batch 4. This system underperformed our system FACTOIDS by about 3% in the MRR measure, but corrected errors such as in the example above."]},{"idx":11,"heading":"Our Systems and Their Performance on Factoid Questions ::: LAT Feature considered and its impact (slightly negative) ::: Assumptions and rules for deriving lexical answer type.","paragraphs":["There are different question types: ‘Which’, ‘What’, ‘When’, ‘How’ etc. Each type of question is being handled differently and there are commonalities among the rules written for different question types. Question words are identified through parts of speech tags: 'WDT', 'WRB' ,'WP'. We assumed that LAT is a ‘Noun’ and follows the question word. Often it was also a subject (nsubj). This process is illustrated in Fig.FIGREF15.","LAT computation was governed by a few simple rules, e.g. when a question has multiple words that are 'Subjects’ (and ‘Noun’), a word that is in proximity to the question word is considered as ‘LAT’. These rules are different for each \"Wh\" word.","Namely, when the word immediately following the question word is a Noun, window size is set to ‘3’. The window size ‘3’ means we iterate through the next ‘3’ words to check if any of the word is both Noun and Subject, If so, such word is considered the ‘LAT’; else the word that is present very next to the question word is considered as the ‘LAT’.","For questions with words ‘Which’ , ‘What’, ‘When’; a Noun immediately following the question word is very often the LAT, e.g. 'enzyme' in Which enzyme is targeted by Evolocumab?. When the word immediately following the question word is not a Noun, e.g. in What is the function of the protein Magt1? the window size is set to ‘5’, and we iterate through the next ‘5’ words (if present) and search for the word that is both Noun and Subject. If present, the word is considered as the ‘LAT’; else, the Noun in close proximity to the question word and following it is returned as the ‘LAT’.","For questions with question words: ‘When’, ‘Who’, ‘Why’, the ’LAT’ is a question word itself. For the word ‘How', e.g. in How many selenoproteins are encoded in the human genome?, we look at the adjective and if we find one, we take it to be the LAT, otherwise the word 'How' is considered as the ‘LAT’.","Perhaps because of using only very simple rules, the accuracy for ‘LAT’ derivation is 75%; that is, in the remaining 25% of the cases the LAT word is identified incorrectly. Worth noting is that the overall performance the system that used LATs was slightly inferior to the system without LATs, but the types of errors changed. The training used BioBERT with the LAT feature as part of the input string. The errors it introduces usually involve finding the wrong element of the correct type e.g. wrong enzyme when two similar enzymes are described in the text, or 'neuron' when asked about a type of cell with a certain function, when the answer calls for a different cell category, adipocytes, and both are mentioned in the text. We feel with more data and additional tuning or perhaps using an ensemble model, we might be able to keep the correct answers, and improve the results on the confusing examples like the one mentioned above. Therefore if we improve our ‘LAT’ derivation logic, or have larger datasets, then perhaps the neural network techniques they will yield better results."]},{"idx":12,"heading":"Our Systems and Their Performance on Factoid Questions ::: Impact of Training using BioAsq data (slightly negative)","paragraphs":["Training on BioAsq data in our entry in Batch 1 and Batch 2 under the name QA1 showed it might lead to overfitting. This happened both with (Batch 2) and without (Batch 1) hyperparameters tuning: abysmal 18% MRR in Batch 1, and slighly better one, 40% in Batch 2 (although in Batch 2 it was overall the second best result in MRR but 16% lower than the highest score).","In Batch 3 (only), our UNCC_QA3 system was fine tuned on BioAsq and SQuAD 2.0 BIBREF7, and for data preprocessing Context paragraph is generated from relevant snippets provided in the test data. This system underperformed, by about 2% in MRR, our other entry UNCC_QA1, which was also an overall category winner for this batch. The latter was also trained on SQuAD, but not on BioAsq. We suspect that the reason could be the simplistic nature of the find() function described in Section 3.1. So, this could be an area where a better algorithm for finding the best occurrence of an entity could improve performance."]},{"idx":13,"heading":"Our Systems and Their Performance on Factoid Questions ::: Impact of Using Context from URLs (negative)","paragraphs":["In some experiments, for context in testing, we used documents for which URL pointers are provided in BioAsq. However, our system UNCC_QA3 underperformed our other system tested only on the provided snippets.","In Batch 5 the underperformance was about 6% of MRR, compared to our best system UNCC_QA1, and by 9% to the top performer."]},{"idx":14,"heading":"Performance on Yes/No and List questions","paragraphs":["Our work focused on Factoid questions. But we also have done experiments on List-type and Yes/No questions."]},{"idx":15,"heading":"Performance on Yes/No and List questions ::: Entailment improves Yes/No accuracy","paragraphs":["We started by answering always YES (in batch 2 and 3) to get the baseline performance. For batch 4 we used entailment. Our algorithm was very simple: Given a question we iterate through the candidate sentences and try to find any candidate sentence is contradicting the question (with confidence over 50%), if so 'No' is returned as answer, else 'Yes' is returned. In batch 4 this strategy produced better than the BioAsq baseline performance, and compared to our other systems, the use of entailment increased the performance by about 13% (macro F1 score). We used 'AllenNlp' BIBREF13 entailment library to find entailment of the candidate sentences with question."]},{"idx":16,"heading":"Performance on Yes/No and List questions ::: For List-type the URLs have negative impact","paragraphs":["Overall, we followed the similar strategy that's been followed for Factoid Question Answering task. We started our experiment with batch 2, where we submitted 20 best answers (with context from snippets). Starting with batch 3, we performed post processing: once models generate answer predictions (n-best predictions), we do post-processing on the predicted answers. In test batch 4, our system (called FACTOIDS) achieved highest recall score of ‘0.7033’ but low precision of 0.1119, leaving open the question of how could we have better balanced the two measures.","In the post-processing phase, we take the top ‘20’ (batch 3) and top 5 (batch 4 and 5), predicted answers, tokenize them using common separators: 'comma' , 'and', 'also', 'as well as'. Tokens with characters count more than ‘100’ are eliminated and rest of the tokens are added to the list of possible answers. BioASQ evaluation mechanism does not consider snippets with more than ‘100’ characters as a valid answer. Considering lengthy snippets in to the list of answers would reduce the mean precision score. As a final step, duplicate snippets in the answer pool are removed. For example, consider these top 3 answers predicted by the system (before post-processing):","{","\"text\": \"dendritic cells\",","\"probability\": 0.7554540733426441,","\"start_logit\": 8.466046333312988,","\"end_logit\": 9.536355018615723","},","{","\"text\": \"neutrophils, macrophages and","distinct subtypes of dendritic cells\",","\"probability\": 0.13806867348304214,","\"start_logit\": 6.766478538513184,","\"end_logit\": 9.536355018615723","},","{","\"text\": \"macrophages and distinct subtypes of dendritic\",","\"probability\": 0.013973475271178242,","\"start_logit\": 6.766478538513184,","\"end_logit\": 7.24576473236084","},","After execution of post-processing heuristics, the list of answers returned is as follows:","[\"dendritic cells\"],","[\"neutrophils\"],","[\"macrophages\"],","[\"distinct subtypes of dendritic cells\"]"]},{"idx":17,"heading":"Summary of our results","paragraphs":["The tables below summarize all our results. They show that the performance of our systems was mixed. The simple architectures and algorithm we used worked very well only in Batch 3. However, we feel we can built a better system based on this experience. In particular we observed both the value of contextual embeddings and of feature engineering (LAT), however we failed to combine them properly."]},{"idx":18,"heading":"Summary of our results ::: Factoid questions ::: Systems used in Batch 5 experiments","paragraphs":["System description for ‘UNCC_QA1’: The system was finetuned on the SQuAD 2.0. For data preprocessing Context / paragraph was generated from relevant snippets provided in the test data.","System description for ‘QA1’ : ‘LAT’ feature was added and finetuned with SQuAD 2.0. For data preprocessing Context / paragraph was generated from relevant snippets provided in the test data.","System description for ‘UNCC_QA3’ : Fine tuning process is same as it is done for the system ‘UNCC_QA1’ in test batch-5. Difference is during data preprocessing, Context/paragraph is generated from the relevant documents for which URLS are included in the test data."]},{"idx":19,"heading":"Summary of our results ::: List Questions","paragraphs":["For List-type questions, although post processing helped in the later batches, we never managed to obtain competitive precision, although our recall was good."]},{"idx":20,"heading":"Summary of our results ::: Yes/No questions","paragraphs":["The only thing worth remembering from our performance is that using entailment can have a measurable impact (at least with respect to a weak baseline). The results (weak) are in Table 3."]},{"idx":21,"heading":"Discussion, Future Experiments, and Conclusions ::: Summary:","paragraphs":["In contrast to 2018, when we submitted BIBREF2 to BioASQ a system based on extractive summarization (and scored very high in the ideal answer category), this year we mainly targeted factoid question answering task and focused on experimenting with BioBERT. After these experiments we see the promise of BioBERT in QA tasks, but we also see its limitations. The latter we tried to address with mixed results using feature engineering. Overall these experiments allowed us to secure a best and a second best score in different test batches. Along with Factoid-type question, we also tried ‘Yes/No’ and ‘List’-type questions, and did reasonably well with our very simple approach.","For Yes/No the moral worth remembering is that reasoning has a potential to influence results, as evidenced by our adding the AllenNLP entailment BIBREF13 system increased its performance.","All our data and software is available at Github, in the previously referenced URL (end of Section 2)."]},{"idx":22,"heading":"Discussion, Future Experiments, and Conclusions ::: Future experiments","paragraphs":["In the current model, we have a shallow neural network with a softmax layer for predicting answer span. Shallow networks however are not good at generalizations. In our future experiments we would like to create dense question answering neural network with a softmax layer for predicting answer span. The main idea is to get contextual word embedding for the words present in the question and paragraph (Context) and feed the contextual word embeddings retrieved from the last layer of BioBERT to the dense question answering network. The mentioned dense layered question answering neural network need to be tuned for finding right hyper parameters. An example of such architecture is shown in Fig.FIGREF30.","In one more experiment, we would like to add a better version of ‘LAT’ contextual word embedding as a feature, along with the actual contextual word embeddings for question text, and Context and feed them as input to the dense question answering neural network. By this experiment, we would like to find if ‘LAT’ feature is improving overall answer prediction accuracy. Adding ‘LAT’ feature this way instead of feeding this word piece embedding directly to the BioBERT (as we did in our above experiments) would not downgrade the quality of contextual word embeddings generated form ‘BioBERT'. Quality contextual word embeddings would lead to efficient transfer learning and chances are that it would improve the model's answer prediction accuracy.","We also see potential for incorporating domain specific inference into the task e.g. using the MedNLI dataset BIBREF14. For all types of experiments it might be worth exploring clinical BERT embeddings BIBREF15, explicitly incorporating domain knowledge (e.g. BIBREF16) and possibly deeper discourse representations (e.g. BIBREF17)."]},{"idx":23,"heading":"APPENDIX","paragraphs":["In this appendix we provide additional details about the implementations."]},{"idx":24,"heading":"APPENDIX ::: Systems and their descriptions:","paragraphs":["We used several variants of our systems when experimenting with the BioASQ problems. In retrospect, it would be much easier to understand the changes if we adopted some mnemonic conventions in naming the systems. So, we apologize for the names that do not reflect the modifications, and necessitate this list."]},{"idx":25,"heading":"APPENDIX ::: Systems and their descriptions: ::: Factoid Type Question Answering:","paragraphs":["We preprocessed the test data to convert test data to BioBERT format, We generated Context/paragraph by either aggregating relevant snippets provided or by aggregating documents for which URLS are provided in the BioASQ test data."]},{"idx":26,"heading":"APPENDIX ::: Systems and their descriptions: ::: System description for QA1:","paragraphs":["We generated Context/paragraph by aggregating relevant snippets available in the test data and mapped it against the question text and question id. We ignored the content present in the documents (document URLS were provided in the original test data). The model is finetuned with BioASQ data.","data preprocessing is done in the same way as it is done for test batch-1. Model fine tuned on BioASQ data.","‘LAT’/ Focus word feature added and fine tuned with SQuAD 2.0 [reference]. For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data."]},{"idx":27,"heading":"APPENDIX ::: Systems and their descriptions: ::: System description for UNCC_QA_1:","paragraphs":["System is finetuned on the SQuAD 2.0 [reference]. For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data.","‘LAT’/ Focus word feature added and fine tuned with SQuAD 2.0 [reference]. For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data.","The System is finetuned on the SQuAD 2.0. For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data."]},{"idx":28,"heading":"APPENDIX ::: Systems and their descriptions: ::: System description for UNCC_QA3:","paragraphs":["System is finetuned on the SQuAD 2.0 [reference] and BioASQ dataset[].For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data.","Fine tuning process is same as it is done for the system ‘UNCC_QA_1’ in test batch-5. Difference is during data preprocessing, Context/paragraph is generated form from the relevant documents for which URLS are included in the test data."]},{"idx":29,"heading":"APPENDIX ::: Systems and their descriptions: ::: System description for UNCC_QA2:","paragraphs":["Fine tuning process is same as for ‘UNCC_QA_1 ’. Difference is Context/paragraph is generated form from the relevant documents for which URLS are included in the test data. System ‘UNCC_QA_1’ got the highest ‘MRR’ score in the 3rd test batch set."]},{"idx":30,"heading":"APPENDIX ::: Systems and their descriptions: ::: System description for FACTOIDS:","paragraphs":["The System is finetuned on the SQuAD 2.0. For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data."]},{"idx":31,"heading":"APPENDIX ::: Systems and their descriptions: ::: List Type Questions:","paragraphs":["We attempted List type questions starting from test batch ‘2’. Used similar approach that's been followed for Factoid Question answering task. For all the test batch sets, in the data pre processing phase Context/ paragraph is generated either by aggregating relevant snippets or by aggregating documents(URLS) provided in the BioASQ test data.","For test batch-2, model (System: QA1) is finetuned on BioASQ data and submitted top ‘20’ answers predicted by the model as the list of answers. system ‘QA1’ achieved low F-Measure score:‘0.0786’ in the second test batch. In the further test batches for List type questions, we finetuned the model on Squad data set [reference], implemented post processing techniques (refer section 5.2) and achieved a better F-measure score: ‘0.2862’ in the final test batch set.","In test batch-3 (Systems : ‘QA1’/’’UNCC_QA_1’/’UNCC_QA3’/’UNCC_QA2’) top 20 answers returned by the model is sent for post processing and in test batch 4 and 5 only top 5 answers are sent for post processing. System UNCC_QA2(in batch 3) for List type question answering, Context is generated from documents for which URLS are provided in the BioASQ test data. for the rest of the systems (in test batch-3) for List Type question answering task snippets present in the BioaSQ test data are used to generate context.","In test batch-4 (System : ‘FACTOIDS’/’UNCC_QA_1’/’UNCC_QA3’’) top 5 answers returned by the model is sent for post processing. In case of system ‘FACTOIDS’ snippets in the test data were used to generate context. for systems ’UNCC_QA_1’ and ’UNCC_QA3’ context is generated from the documents for which URLS are provided in the BioASQ test data.","In test batch-5 ( Systems: ‘QA1’/’UNCC_QA_1’/’UNCC_QA3’/’UNCC_QA2’ ) our approach is the same as that of test batch-4 where top 5 answers returned by the model is sent for post processing. for all the systems (in test batch-5) context is generated from the snippets provided in the BioASQ test data."]},{"idx":32,"heading":"APPENDIX ::: Systems and their descriptions: ::: Yes/No Type Questions:","paragraphs":["For the first 3 test batches, We have submitted answer ‘Yes’ to all the questions. Later, we employed ‘Sentence Entailment’ techniques(refer section 6.0) for the fourth and fifth test batch sets. Our Systems with ‘Sentence Entailment’ approach (for ‘Yes’/ ‘No’ question answering): ‘UNCC_QA_1’(test batch-4), UNCC_QA3(test batch-5)."]},{"idx":33,"heading":"APPENDIX ::: Additional details for Yes/No Type Questions","paragraphs":["We used Textual Entailment in Batch 4 and 5 for ‘Yes’/‘No’ question type. The algorithm was very simple: Given a question we iterate through the candidate sentences, and look for any candidate sentences contradicting the question. If we find one 'No' is returned as answer, else 'Yes' is returned. (The confidence for contradiction was set at 50%) We used AllenNLP BIBREF13 entailment library to find entailment of the candidate sentences with question.","Flow Chart for Yes/No Question answer processing is shown in Fig.FIGREF51"]},{"idx":34,"heading":"APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions","paragraphs":["There are different question types, and we distinguished them based on the question words: ‘Which’, ‘What’, ‘When’, ‘How’ etc. Each type of question is being handled differently and there are commonalities among the rules written for different question types. How are question words identified? question words have parts of speech(POS): 'WDT', 'WRB', 'WP'.","Assumptions:","1) Lexical answer type (‘LAT’) or focus word is of type Noun and follows the question word.","2) The LAT word is a Subject. (This clearly not always true, but we used a very simple method). Note: ‘StanfordNLP’ dependency parsing tag for identifying subject is 'nsubj' or 'nsubjpass'.","3) When a question has multiple words that are of type Subject (and Noun), a word that is in proximity to the question word is considered as ‘LAT’.","4) For questions with question words: ‘When’, ‘Who’, ‘Why’, the ’LAT’ is a question word itself that is, ‘When’, ‘Who’, ‘Why’ respectively.","Rules and logic flow to traverse a question: The three cases below describe the logic flow of finding LATs. The figures show the grammatical structures used for this purpose."]},{"idx":35,"heading":"APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions ::: Case-1:","paragraphs":["Question with question word ‘How’.","For questions with question word 'How', the adjective that follows the question word is considered as ‘LAT’ (need not follow immediately). If an adjective is absent, word 'How' is considered as ‘LAT’. When there are multiple words that are adjectives, a word in close proximity to the question word and follows it is returned as ‘LAT’. Note: The part of speech tag to identify adjectives is 'JJ'. For Other possible question words like ‘whose’. ‘LAT’/Focus word is question words itself.","Example Question: How many selenoproteins are encoded in the human genome?"]},{"idx":36,"heading":"APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions ::: Case-2:","paragraphs":["Questions with question words ‘Which’ , ‘What’ and all other possible question words; a 'Noun' immediately following the question word.","Example Question: Which enzyme is targeted by Evolocumab?","Here, Focus word/LAT is ‘enzyme’ which is both Noun and Subject and immediately follows the question word.","When the word immediately following the question word is a noun, the window size is set to ‘3’. This size ‘3’ means that we iterate through the next ‘3’ words (if present) to check if any of the word is both 'Noun' and 'Subject', If so, the word is considered as ‘LAT’/Focus Word. Else the word that is present very next to the question word is considered as ‘LAT’."]},{"idx":37,"heading":"APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions ::: Case-3:","paragraphs":["Questions with question words ‘Which’ , ‘What’ and all other possible question words; word immediately following the question word is not a 'Noun'.","Example Question: What is the function of the protein Magt1?","Here, Focus word/LAT is ‘function ’ which is both Noun and Subject and does not immediately follow the question word.","When the very next word following the question word is not a Noun, window size is set to ‘5’. Window size ‘5’ corresponds that we iterate through the next ‘5’ words (if present) and search for the word that is both Noun and Subject. If present, the word is considered as ‘LAT’. Else, the 'Noun' close proximity to the question word and follows it is returned as ‘LAT’.","Ad we mentioned earlier, the accuracy for ‘LAT’ derivation is 75 percent. But clearly the simple logic described above can be improved, as shown in BIBREF9, BIBREF10. Whether this in turn produces improvements in this particular task is an open question."]},{"idx":38,"heading":"APPENDIX ::: Proposing Future Experiments","paragraphs":["In the current model, we have a shallow neural network with a softmax layer for predicting answer span. Shallow networks however are not good at generalizations. In our future experiments we would like to create dense question answering neural network with a softmax layer for predicting answer span. The main idea is to get contextual word embedding for the words present in the question and paragraph (Context) and feed the contextual word embeddings retrieved from the last layer of BioBERT to the dense question answering network. The mentioned dense layered question answering Neural network need to be tuned for finding right hyper parameters. An example of such architecture is shown in Fig.FIGREF30.","In another experiment we would like to only feed contextual word embeddings for Focus word/ ‘LAT’, paragraph/ Context as input to the question answering neural network. In this experiment we would neglect all embeddings for the question text except that of Focus word/ ‘LAT’. Our assumption and idea for considering focus word and neglecting remaining words in the question is that during training phase it would make more precise for the model to identify the focus of the question and map answers against the question’s focus. To validate our assumption, we would like to take sample question answering data and find the cosine distance between contextual embedding of Focus word and that of the actual answer and verify if the cosine distance is comparatively low in most of the cases.","In one more experiment, we would like to add a better version of ‘LAT’ contextual word embedding as a feature, along with the actual contextual word embeddings for question text, and Context and feed them as input to the dense question answering neural network. By this experiment, we would like to find if ‘LAT’ feature is improving overall answer prediction accuracy. Adding ‘LAT’ feature this way instead of feeding Focus word’s word piece embedding directly (as we did in our above experiments) to the BioBERT would not downgrade the quality of contextual word embeddings generated form ‘BioBERT'. Quality contextual word embeddings would lead to efficient transfer learning and chances are that it would improve the model's answer prediction accuracy."]},{"idx":39,"heading":"Figures and Tables","paragraphs":["Figure 1: BioBERT fine tuned for question answering task","Figure 2: A simple way of finding the lexical answer types, LATs, of factoid questions: using POS tags to find the question word (e.g. ’which’), and a dependency parse to find the LAT within the window of 3 words. If a noun is not found near the ”Wh-” word, we iterate looking for it, as in the second panel.","Figure 3: An example of a using BioBERT with additional features: Contextual word embedding for Lexical Answer Type (LAT) given as feature along with the actual contextual embeddings for the words in question and the paragraph. This change produced mixed results and no overall improvement.","Table 1: Factoid Questions. In Batch 3 we obtained the highest score. Also the relative distance between our best system and the top performing system shrunk between Batch 4 and 5.","Table 2: List Questions","Table 3: Yes/No Questions","Figure 4: Proposed extension. In addition to using BioBERT we propose to add a network that would train on embeddings of questions (Emb Quesi) and paragraphs (Emb Pari) with additional LAT features, or more broadly other semantic features, like relations that can be derived from text","Figure 5: Flow Chart for Yes/No Question answer processing","Figure 6: Dependency parse visualization for the example ’how’ question.","Figure 7: Dependency parse visualization for the example ’which’ question.","Figure 8: Dependency parse visualization for the question about the function of the protein Magt1."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"2002.04326","title":"ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Recent powerful pre-trained language models have achieved remarkable performance on most of the popular datasets for reading comprehension. It is time to introduce more challenging datasets to push the development of this field towards more comprehensive reasoning of text. In this paper, we introduce a new Reading Comprehension dataset requiring logical reasoning (ReClor) extracted from standardized graduate admission examinations. As earlier studies suggest, human-annotated datasets usually contain biases, which are often exploited by models to achieve high accuracy without truly understanding the text. In order to comprehensively evaluate the logical reasoning ability of models on ReClor, we propose to identify biased data points and separate them into EASY set while the rest as HARD set. Empirical results show that the state-of-the-art models have an outstanding ability to capture biases contained in the dataset with high accuracy on EASY set. However, they struggle on HARD set with poor performance near that of random guess, indicating more research is needed to essentially enhance the logical reasoning ability of current models."]},{"idx":1,"heading":"Introduction","paragraphs":["Machine reading comprehension (MRC) is a fundamental task in Natural Language Processing, which requires models to understand a body of text and answer a particular question related to the context. With success of unsupervised representation learning in NLP, language pre-training based models such as GPT-2 BIBREF0, BERT BIBREF1, XLNet BIBREF2 and RoBERTa BIBREF3 have achieved nearly saturated performance on most of the popular MRC datasets BIBREF4, BIBREF5, BIBREF6, BIBREF7. It is time to challenge state-of-the-art models with more difficult reading comprehension tasks and move a step forward to more comprehensive analysis and reasoning over text BIBREF8.","In natural language understanding, logical reasoning is an important ability to examine, analyze and critically evaluate arguments as they occur in ordinary language according to the definition from Law School Admission BIBREF9. It is a significant component of human intelligence and is essential in negotiation, debate and writing etc. However, existing reading comprehension datasets have none or merely a small amount of data requiring logical reasoning, e.g., 0% in MCTest dataset BIBREF10 and 1.2% in SQuAD BIBREF4 according to BIBREF11. One related task is natural language inference, which requires models to label the logical relationships of sentence pairs. However, this task only considers three types of simple logical relationships and only needs reasoning at sentence-level. To push the development of models in logical reasoning from simple logical relationship classification to multiple complicated logical reasoning and from sentence-level to passage-level, it is necessary to introduce a reading comprehension dataset targeting logical reasoning.","A typical example of logical reasoning questions is shown in Table TABREF5. Similar to the format of multiple-choice reading comprehension datasets BIBREF10, BIBREF5, it contains a context, a question and four options with only one right answer. To answer the question in this example, readers need to identify the logical connections between the lines to pinpoint the conflict, then understand each of the options and select an option that solves the conflict. Human minds need extensive training and practice to get used to complex reasoning, and it will take immense efforts for crowdsourcing workers to design such logical reasoning questions. Inspired by the datasets extracted from standardized examinations BIBREF5, BIBREF12, we build a dataset by selecting such logical reasoning questions from standardized exams such as GMAT and LSAT . We finally collect 6,138 pieces of logical reasoning questions, which constitute a Reading Comprehension dataset requiring logical reasoning (ReClor).","Human-annotated datasets usually contain biases BIBREF13, BIBREF14, BIBREF15, BIBREF16, BIBREF17, BIBREF18, which are often exploited by neural network models as shortcut solutions to achieve high testing accuracy. For data points whose options can be selected correctly without knowing the contexts and questions, we classify them as biased ones. In order to fully assess the logical reasoning ability of the models, we propose to identify the biased data points and group them as EASY set, and put the rest into HARD set. Based on our experiments on these separate sets, we find that even the state-of-the-art models can only perform well on EASY set and struggle on HARD set as shown in Figure FIGREF4. This phenomenon shows that current models can well capture the biases in the dataset but lack the ability to understand the text and reason based on connections between the lines. On the other hand, human beings perform similarly on both the EASY and HARD set. It is thus observed that there is still a long way to go to equip models with true logical reasoning ability.","The contributions of our paper are two-fold. First, we introduce ReClor, a new reading comprehension dataset requiring logical reasoning. We use option-only-input baselines trained with different random seeds to identify the data points with biases in the testing set, and group them as EASY set, with the rest as HARD set to facilitate comprehensive evaluation. Second, we evaluate several state-of-the-art models on ReClor and find these pre-trained language models can perform well on EASY set but struggle on the HARD set. This indicates although current models are good at exploiting biases in the dataset, they are far from capable of performing real logical reasoning yet."]},{"idx":2,"heading":"Related Work","paragraphs":["Reading Comprehension Datasets. A variety of reading comprehension datasets have been introduced to promote the development of this field. MCTest BIBREF10 is a dataset with 2,000 multiple-choice reading comprehension questions about fictional stories in the format similar to ReClor. BIBREF4 proposed SQuAD dataset, which contains 107,785 question-answer pairs on 536 Wikipedia articles. The authors manually labeled 192 examples of the dataset and found that the examples mainly require reasoning of lexical or syntactic variation. In an analysis of the above-mentioned datasets, BIBREF11 found that none of questions requiring logical reasoning in MCTest dataset BIBREF10 and only 1.2% in SQuAD dataset BIBREF4. BIBREF5 introduced RACE dataset by collecting the English exams for middle and high school Chinese students in the age range between 12 to 18. They hired crowd workers on Amazon Mechanical Turk to label the reasoning type of 500 samples in the dataset and show that around 70 % of the samples are in the category of word matching, paraphrasing or single-sentence reasoning. To encourage progress on deeper comprehension of language, more reading comprehension datasets requiring more complicated reasoning types are introduced, such as iterative reasoning about the narrative of a story BIBREF20, multi-hop reasoning across multiple sentences BIBREF21 and multiple documents BIBREF22, commonsense knowledge reasoning BIBREF23, BIBREF24, BIBREF25 and numerical discrete reasoning over paragraphs BIBREF8. However, to the best of our knowledge, although there are some datasets targeting logical reasoning in other NLP tasks mentioned in the next section, there is no dataset targeting evaluating logical reasoning in reading comprehension task. This work introduces a new dataset to fill this gap.","Logical Reasoning in NLP. There are several tasks and datasets introduced to investigate logical reasoning in NLP. The task of natural language inference, also known as recognizing textual entailment BIBREF26, BIBREF27, BIBREF28, BIBREF29, BIBREF30 requires models to take a pair of sentence as input and classify their relationship types, i.e., Entailment, Neutral, or Contradiction. SNLI BIBREF31 and MultiNLI BIBREF32 datasets are proposed for this task. However, this task only focuses on sentence-level logical relationship reasoning and the relationships are limited to only a few types. Another task related to logical reasoning in NLP is argument reasoning comprehension task introduced by BIBREF33 with a dataset of this task. Given an argument with a claim and a premise, this task aims to select the correct implicit warrant from two options. Although the task is on passage-level logical reasoning, it is limited to only one logical reasoning type, i.e., identifying warrants. ReClor and the proposed task integrate various logical reasoning types into reading comprehension, with the aim to promote the development of models in logical reasoning not only from sentence-level to passage-level, but also from simple logical reasoning types to the complicated diverse ones.","Datasets from Examinations. There have been several datasets extracted from human standardized examinations in NLP, such as RACE dataset BIBREF5 mentioned above. Besides, NTCIR QA Lab BIBREF34 offers comparative evaluation for solving real-world university entrance exam questions; The dataset of CLEF QA Entrance Exams Task BIBREF35 is extracted from standardized English examinations for university admission in Japan; ARC dataset BIBREF12 consists of 7,787 science questions targeting student grade level, ranging from 3rd grade to 9th; The dialogue-based multiple-choice reading comprehension dataset DREAM BIBREF36 contains 10,197 questions for 6,444 multi-turn multi-party dialogues from English language exams that are designed by human experts to assess the comprehension level of Chinese learners of English. Compared with these datasets, ReClor distinguishes itself by targeting logical reasoning."]},{"idx":3,"heading":"ReClor Data Collection and Analysis ::: Data collection","paragraphs":["The format of data in ReClor is similar to other multiple-choice reading comprehension datasets BIBREF10, BIBREF5, where a data point contains a context, a question and four answer options, among which only one option is right/most suitable. We collect reading comprehension problems that require complicated logical reasoning. However, producing such data requires the ability to perform complex logical reasoning, which makes it hard for crowdsourcing workers to generate such logical questions. Fortunately, we find the reading comprehension problems in some standardized tests, such as GMAT and LSAT, are highly in line with our expectation.","We construct a dataset containing 6,138 logical reasoning questions sourced from open websites and books. In the original problems, there are five answer options in which only one is right. To comply with fair use of law, we shuffle the order of answer options and randomly delete one of the wrong options for each data point, which results in four options with one right option and three wrong options. Furthermore, similar to ImageNet dataset, ReClor is available for non-commercial research purpose only. We are also hosting a public evaluation server on EvalAI BIBREF37 to benchmark progress on Reclor."]},{"idx":4,"heading":"ReClor Data Collection and Analysis ::: Data analysis","paragraphs":["As mentioned above, we collect 6,138 data points, in which 91.22% are from actual exams of GMAT and LSAT while others are from high-quality practice exams. They are divided into training set, validation set and testing set with 4,638, 500 and 1,000 data points respectively. The overall statistics of ReClor and comparison with other similar multiple-choice MRC datasets are summarized in Table TABREF9. As shown, ReClor is of comparable size and relatively large vocabulary size. Compared with RACE, the length of the context of ReCor is much shorter. In RACE, there are many redundant sentences in context to answer a question. However, in ReClor, every sentence in the context passages is important, which makes this dataset focus on evaluating the logical reasoning ability of models rather than the ability to extract relevant information from a long context. The length of answer options of ReClor is largest among these datasets. We analyze and manually annotate the types of questions on the testing set and group them into 17 categories, whose percentages and descriptions are shown in Table TABREF11. The percentages of different types of questions reflect those in the logical reasoning module of GMAT and LSAT. Some examples of different types of logical reasoning are listed in Figure FIGREF12, and more examples are listed in the Appendix . Taking two examples, we further express how humans would solve such questions in Table TABREF13, showing the challenge of ReClor."]},{"idx":5,"heading":"ReClor Data Collection and Analysis ::: Data Biases in the Dataset","paragraphs":["The dataset is collected from exams devised by experts in logical reasoning, which means it is annotated by humans and may introduce biases in the dataset. Recent studies have shown that models can utilize the biases in a dataset of natural language understanding to perform well on the task without truly understanding the text BIBREF13, BIBREF14, BIBREF15, BIBREF16, BIBREF17, BIBREF18. It is necessary to analyze such data biases to help evaluate models. In the ReClor dataset, the common context and question are shared across the four options for each data point, so we focus on the analysis of the difference in lexical choice and sentence length of the right and wrong options without contexts and questions. We first investigate the biases of lexical choice. We lowercase the options and then use WordPiece tokenization BIBREF39 of BERT$_{\\small \\textsc {BASE}}$ BIBREF1 to get the tokens. Similar to BIBREF16, for the tokens in options, we analyze their conditional probability of label $l \\in \\lbrace \\mathrm {right, wrong}\\rbrace $ given by the token $t$ by $p(l|t) =count(t, l) / count(t)$. The larger the correlation score is for a particular token, the more likely it contributes to the prediction of related option. Table SECREF14 reports tokens in training set which occur at least twenty times with the highest scores since many of the tokens with the highest scores are of low frequency. We further analyze the lengths of right and wrong options BIBREF17 in training set. We notice a slight difference in the distribution of sentence length for right and wrong options. The average length for wrong options is around 21.82 whereas that for right options is generally longer with an average length of 23.06.","tableTop 10 tokens that correlate to right options with more than 20 occurrences. figureThe distribution of the option length in ReClor with respect to right and wrong labels."]},{"idx":6,"heading":"Experiments ::: Baseline Models","paragraphs":["Many neural network based models such as FastText BIBREF40, Bi-LSTM, GPT BIBREF41, GPT-2 BIBREF0, BERT BIBREF1, XLNet BIBREF2, RoBERTa BIBREF3 have achieved impressive results in various NLP tasks. We challenge these neural models with ReClor to investigate how well they can perform. Details of the baseline models and implementation are shown in the Appendix and ."]},{"idx":7,"heading":"Experiments ::: Experiments to Find Biased Data","paragraphs":["As mentioned earlier, biases prevalently exist in human-annotated datasets BIBREF16, BIBREF17, BIBREF18, BIBREF42, which are often exploited by models to perform well without truly understanding the text. Therefore, it is necessary to find out the biased data points in ReClor in order to evaluate models in a more comprehensive manner BIBREF43. To this end, we feed the five strong baseline models (GPT, GPT-2, BERT$_{\\small \\textsc {BASE}}$, XLNet$_{\\small \\textsc {BASE}}$ and RoBERTa$_{\\small \\textsc {BASE}}$) with ONLY THE ANSWER OPTIONS for each problem. In other words, we purposely remove the context and question in the inputs. In this way, we are able to identify those problems that can be answered correctly by merely exploiting the biases in answer options without knowing the relevant context and question. However, the setting of this task is a multiple-choice question with 4 probable options, and even a chance baseline could have 25% probability to get it right. To eliminate the effect of random guess, we set four different random seeds for each model and pick the data points that are predicted correctly in all four cases to form the EASY set. Then, the data points which are predicted correctly by the models at random could be nearly eliminated, since any data point only has a probability of $(25\\%)^{4}=0.39\\%$ to be guessed right consecutively for four times. Then we unite the sets of data points that are consistently predicted right by each model, because intuitively different models may learn different biases of the dataset. The above process is formulated as the following expression,","where $_{\\mathrm {BERT}}^{\\mathrm {seed_1}}$ denotes the set of data points which are predicted correctly by BERT$_{\\small \\textsc {BASE}}$ with seed 1, and similarly for the rest. Table TABREF18 shows the average performance for each model trained with four different random seeds and the number of data points predicted correctly by all of them. Finally, we get 440 data points from the testing set $_{\\mathrm {TEST}}$ and we denote this subset as EASY set $_{\\mathrm {EASY}}$ and the other as HARD set $_{\\mathrm {HARD}}$."]},{"idx":8,"heading":"Experiments ::: Transfer learning Through Fine-tuning","paragraphs":["Among multiple-choice reading comprehension or QA datasets from exams, although the size of ReClor is comparable to those of ARC BIBREF12 and DREAM BIBREF36, it is much smaller than RACE BIBREF5. Recent studies BIBREF44, BIBREF45, BIBREF25, BIBREF46 have shown the effectiveness of pre-training on similar tasks or datasets then fine-tuning on the target dataset for transfer learning. BIBREF46 find that by first training on RACE BIBREF5 and then further fine-tuning on the target dataset, the performances of BERT$_{\\small \\textsc {BASE}}$ on multiple-choice dataset MC500 BIBREF10 and DREAM BIBREF36 can significantly boost from 69.5% to 81.2%, and from 63.2% to 70.2%, respectively. However, they also find that the model cannot obtain significant improvement even performs worse if it is first fine-tuned on span-based dataset like SQuAD BIBREF4. ReClor is a multiple-choice dataset, so we choose RACE for fine-tuning study."]},{"idx":9,"heading":"Experiments ::: Results and Analysis","paragraphs":["The performance of all tested models on the ReClor is presented in Table TABREF21. This dataset is built on questions designed for students who apply for admission to graduate schools, thus we randomly choose 100 samples from the testing set and divide them into ten tests, which are distributed to ten different graduate students in a university. We take the average of their scores and present it as the baseline of graduate students. The data of ReClor are carefully chosen and modified from only high-quality questions from standardized graduate entrance exams. We set the ceiling performance to 100% since ambiguous questions are not included in the dataset.","The performance of fastText is better than random guess, showing that word correlation could be used to help improve performance to some extent. It is difficult for Bi-LSTM to converge on this dataset. Transformer-based pre-training models have relatively good performance, close to the performance of graduate students. However, we find that these models only perform well on EASY set with around 75% accuracy, showing these models have an outstanding ability to capture the biases of the dataset, but they perform poorly on HARD set with only around 30% accuracy. In contrast, humans can still keep good performance on HARD set. We notice the difference in testing accuracy performed by graduate students on EASY and HARD set, but this could be due to the small number of students participated in the experiments. Therefore, we say humans perform relatively consistent on both biased and non-biased dataset.","It is noticed that if the models are first trained on RACE and then fine-tuned on ReClor, they could obtain significant improvement, especially on HARD set. The overall performance of RoBERTa$_{\\small \\textsc {LARGE}}$ is even better than that of graduate students. This similar phenomenon can also be observed on DREAM dataset BIBREF36 by BIBREF46, which shows the potential of transfer learning for reasoning tasks. However, even after fine-tuning on RACE, the best performance of these strong baselines on HARD set is around 50%, still lower than that of graduate students and far away from ceiling performance.","Experiments in different input settings are also done. Compared with the input setting of answer options only (A), the setting of questions and answer options (Q, A) can not bring significant improvement. This may be because some questions e.g., Which one of the following is an assumption required by the argument?, Which one of the following, if true, most strengthens the argument? can be used in the same reasoning types of question, which could not offer much information. Further adding context causes significant boost, showing the high informativeness of the context.","We further analyze the model performance with respect to different question types of logical reasoning. Some results are shown in Figure FIGREF22 and the full results are shown in Figure , and in the Appendix . Three models of BERT$_{\\small \\textsc {LARGE}}$, XLNet$_{\\small \\textsc {LARGE}}$ and RoBERTa$_{\\small \\textsc {LARGE}}$ perform well on most of types. On HARD set, the three models perform poorly on certain types such as Strengthen, Weaken and Role which require extensive logical reasoning. However, they perform relatively better on other certain types, such as Conclusion/Main Point and Match Structures that are more straight-forward. For the result of transfer learning, we analyze XLNet$_{\\small \\textsc {LARGE}}$ in detail. Though the overall performance is significantly boosted after fine-tuning on RACE first, the histograms in the bottom of Figure FIGREF22 show that on EASY set, accuracy of the model with fine-tuning on RACE is similar to that without it among most question types, while on HARD set, significant improvement on some question types is observed, such as Conclusion/Main Point and Most Strongly Supported. This may be because these types require less logical reasoning to some extent compared with other types, and similar question types may also be found in RACE dataset. Thus, the pre-training on RACE helps enhance the ability of logical reasoning especially of relatively simple reasoning types, but more methods are still needed to further enhance the ability especially that of relatively complex reasoning types."]},{"idx":10,"heading":"Conclusion","paragraphs":["In this paper, we introduce ReClor, a reading comprehension dataset requiring logical reasoning, with the aim to push research progress on logical reasoning in NLP forward from sentence-level to passage-level and from simple logical reasoning to multiple complicated one. We propose to identify biased data points and split the testing set into EASY and HARD group for biased and non-biased data separately. We further empirically study the different behaviors of state-of-the-art models on these two testing sets, and find recent powerful transformer-based pre-trained language models have an excellent ability to exploit the biases in the dataset but have difficulty in understanding and reasoning given the non-biased data with low performance close to or slightly better than random guess. These results show there is a long way to equip deep learning models with real logical reasoning abilities. We hope this work would inspire more research in future to adopt similar split technique and evaluation scheme when reporting their model performance. We also show by first fine-tuning on a large-scale dataset RACE then fine-tuning on ReClor, the models could obtain significant improvement, showing the potential of transfer learning to solve reasoning tasks."]},{"idx":11,"heading":"Conclusion ::: Acknowledgments","paragraphs":["We would like to thank three anonymous reviews for their insightful comments and suggestions; thank Rishabh Jain from Georgia Tech for helping build up the leaderboard of ReClor on EvalAI. Jiashi Feng was partially supported by NUS IDS R-263-000-C67-646, ECRA R-263-000-C87-133, MOE Tier-II R-263-000-D17-112 and AI.SG R-263-000-D97-490."]},{"idx":12,"heading":"Figures and Tables","paragraphs":["Figure 1: Performance comparison of state-of-the-art models and humans (graduate students) on EASY and HARD set of ReClor testing set.","Table 1: An example in the ReClor dataset which is modified from the Law School Admission Council (2019b).","Table 2: Statistics of several multiple-choice MRC datasets.","Table 3: The percentage and description of each logical reasoning type. The descriptions are adapted from those specified by Khan Academy (2019).","Figure 3: The distribution of the option length in ReClor with respect to right and wrong labels.","Figure 2: Examples of some question types. The correct options are marked by X. More examples are shown in the Appendix C.","Table 6: Average accuracy of each model using four different random seeds with only answer options as input, and the number of their common correct predictions.","Table 7: Accuracy (%) of models and human performance. The column Input means whether to input context (C), question (Q) and answer options (A). The RACE column represents whether to first use RACE to fine-tune before training on ReClor.","Figure 4: Performance of models on EASY (left) and HARD (right) testing sets and that of models. XLNetLARGE +Fine-Tune means the model is first fine-tuned on RACE before training on ReClor.","Table 8: Input formats of different models. Context, Question and Option represent the token sequences of the context, question and option respectively, and || denotes concatenation.","Table 9: Hyperparameters for finetuning pre-training language models on ReClor","Table 27: The definition and an example of the logical reasoning type - Others","Table 28: Overlap of each pair of models after intersection among 4 random seeds.","Figure 5: Accuracy of all baseline models on overall testing set","Figure 6: Accuracy of all baseline models on EASY set of testing set","Figure 7: Accuracy of all baseline models on HARD set of testing set","Figure 8: Performance of BERTLARGE (top) and RoBERTaLARGE (bottom) on EASY (left) and HARD (right) testing sets."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"2002.04745","title":"On Layer Normalization in the Transformer Architecture","sections":[{"idx":0,"heading":"Abstract","paragraphs":["The Transformer is widely used in natural language processing tasks. To train a Transformer however, one usually needs a carefully designed learning rate warm-up stage, which is shown to be crucial to the final performance but will slow down the optimization and bring more hyper-parameter tunings. In this paper, we first study theoretically why the learning rate warm-up stage is essential and show that the location of layer normalization matters. Specifically, we prove with mean field theory that at initialization, for the original-designed Post-LN Transformer, which places the layer normalization between the residual blocks, the expected gradients of the parameters near the output layer are large. Therefore, using a large learning rate on those gradients makes the training unstable. The warm-up stage is practically helpful for avoiding this problem. On the other hand, our theory also shows that if the layer normalization is put inside the residual blocks (recently proposed as Pre-LN Transformer), the gradients are well-behaved at initialization. This motivates us to remove the warm-up stage for the training of Pre-LN Transformers. We show in our experiments that Pre-LN Transformers without the warm-up stage can reach comparable results with baselines while requiring significantly less training time and hyper-parameter tuning on a wide range of applications."]},{"idx":1,"heading":"Introduction","paragraphs":["The Transformer BIBREF0 is one of the most commonly used neural network architectures in natural language processing. Layer normalization BIBREF1 plays a key role in Transformer's success. The originally designed Transformer places the layer normalization between the residual blocks, which is usually referred to as the Transformer with Post-Layer Normalization (Post-LN) BIBREF2. This architecture has achieved state-of-the-art performance in many tasks including language modeling BIBREF3, BIBREF4 and machine translation BIBREF5, BIBREF6. Unsupervised pre-trained models based on the Post-LN Transformer architecture also show impressive performance in many downstream tasks BIBREF7, BIBREF8, BIBREF9.","Despite its great success, people usually need to deal with the optimization of the Post-LN Transformer more carefully than convolutional networks or other sequence-to-sequence models BIBREF10. In particular, to train the model from scratch, any gradient-based optimization approach requires a learning rate warm-up stage BIBREF0, BIBREF11: the optimization starts with an extremely small learning rate, and then gradually increases it to a pre-defined maximum value in a pre-defined number of iterations. Such a warm-up stage not only slows down the optimization process but also brings more hyperparameter tunings. BIBREF10 has shown that the final model performance is quite sensitive to the value of the maximum learning rate and the number of warm-up iterations. Tuning such sensitive hyper-parameters is costly in training large-scale models, e.g., BERT BIBREF8 or XLNet BIBREF9.","In this paper, we try to alleviate this problem by finding ways to safely remove the learning rate warm-up stage. As the warm-up stage happens in the first several iterations, we investigate the optimization behavior at initialization using mean field theory BIBREF12, BIBREF13, BIBREF14, BIBREF15, BIBREF16, BIBREF17. According to our theoretical analysis, when putting the layer normalization between the residual blocks, the expected gradients of the parameters near the output layer are large. Therefore, without the warm-up stage, directly using a large learning rate to those parameters can make the optimization process unstable. Using a warm-up stage and training the model with small learning rates practically avoid this problem. Extensive experiments are provided to support our theoretical findings.","Our theory also shows that the layer normalization plays a crucial role in controlling the gradient scales. This motivates us to investigate whether there are some other ways of positioning the layer normalization that lead to well-behaved gradients. In particular, we study another variant, the Transformer with Pre-Layer Normalization (Pre-LN) BIBREF18, BIBREF19, BIBREF2. The Pre-LN Transformer puts the layer normalization inside the residual connection and equips with an additional final-layer normalization before prediction (Please see Figure SECREF1 for the differences between the two variants of the Transformer architectures). We show that at initialization, the gradients are well-behaved without any exploding or vanishing for the Pre-LN Transformer both theoretically and empirically.","Given the gradients are well-behaved in the Pre-LN Transformer, it is natural to consider removing the learning rate warm-up stage during training. We conduct a variety of experiments, including IWSLT14 German-English translation, WMT14 English-German translation, and BERT pre-training tasks. We show that, in all tasks, the learning rate warm-up stage can be safely removed, and thus, the number of hyper-parameter is reduced. Furthermore, we observe that the loss decays faster for the Pre-LN Transformer model. It can achieve comparable final performances but use much less training time. This is particularly important for training large-scale models on large-scale datasets.","Our contributions are summarized as follows:","$\\bullet $ We investigate two Transformer variants, the Post-LN Transformer and the Pre-LN Transformer, using mean field theory. By studying the gradients at initialization, we provide evidence to show why the learning rate warm-up stage is essential in training the Post-LN Transformer.","$\\bullet $ We are the first to show that the learning-rate warm-up stage can be removed for the Pre-LN Transformer, which eases the hyperparameter tuning. We further show that by using proper learning rate schedulers, the training time can be largely reduced on a wide range of applications."]},{"idx":2,"heading":"Related work","paragraphs":["Gradient descent-based methods BIBREF20, BIBREF21, BIBREF22, BIBREF23 are popularly used in optimizing deep neural networks. For convolutional neural networks and recurrent neural networks, a relatively large learning rate is usually set in the beginning, and then decreased along with the optimization process BIBREF24, BIBREF25, BIBREF26, BIBREF27, BIBREF28. The learning rate warm-up stage has only been shown essential in dealing with some very specific problems, e.g., the large-batch training. BIBREF29, BIBREF28, BIBREF30 showed that a learning rate warm-up stage is preferred when training neural networks with extremely large batch sizes.","However, the learning rate warm-up stage is essential and critical when optimizing the Transformer models in a majority of scenarios BIBREF0, BIBREF8, BIBREF3, BIBREF7, BIBREF31. BIBREF10 investigated the influence of different warm-up strategies for the optimization of the Post-LN Transformer model and found that without or with relatively less warm-up iterations, the optimization diverges. The Pre-LN Transformer has been proposed in several recent works BIBREF18, BIBREF19, BIBREF2 to alleviate some optimization issues when training deeper models, but the troublesome warm-up stage still remains in their training pipelines.","In a concurrent and independent work BIBREF11, the authors claimed that the benefit of the warm-up stage comes from reducing the variance for the adaptive learning rate in the Adam optimizer BIBREF20. They proposed to rectify the variance of adaptive learning rate by a new variant of Adam called RAdam. However, we find that not only for Adam, the learning rate warm-up stage also helps quite a lot for other optimizers. This may indicate that Adam is not the prerequisite for the necessity of the warm-up stage. Therefore, we re-identify the problem and find that it highly relates to the architecture, in particular, the position of layer normalization."]},{"idx":3,"heading":"Optimization for the Transformer ::: Transformer with Post-Layer Normalization","paragraphs":["The Transformer architecture usually consists of stacked Transformer layers BIBREF0, BIBREF8, each of which takes a sequence of vectors as input and outputs a new sequence of vectors with the same shape. A Transformer layer has two sub-layers: the (multi-head) self-attention sub-layer and the position-wise feed-forward network sub-layer. Residual connection BIBREF24 and layer normalization BIBREF1 are applied for both sub-layers individually. We first introduce each component of the Transformer layer and then present the entire architecture."]},{"idx":4,"heading":"Optimization for the Transformer ::: Transformer with Post-Layer Normalization ::: Self-attention sub-layer","paragraphs":["An attention function can be formulated as querying an entry with key-value pairs BIBREF0. The self-attention sub-layer uses scaled dot-product attention, which is defined as: $\\text{Attention}(Q,K,V)=\\text{softmax}(\\frac{QK^T}{\\sqrt{d}})V$, where $d$ is the dimensionality of the hidden representations, and $Q$ (Query), $K$ (Key), $V$ (Value) are specified as the hidden representations of the previous layer. The multi-head variant of the self-attention sub-layer is popularly used which allows the model to jointly attend to information from different representation sub-spaces, and is defined as","where $W_k^Q\\in \\mathbb {R}^{d \\times d_K}, W_k^K\\in \\mathbb {R}^{d \\times d_K}, W_k^V\\in \\mathbb {R}^{d\\times d_V},$ and $W^O\\in \\mathbb {R}^{Hd_V \\times d}$ are project parameter matrices, $H$ is the number of heads. $d_K$ and $d_V$ are the dimensionalities of Key and Value. Without any confusion, given a sequence of vectors $(x_1,...,x_n)$, we use $\\text{MultiHeadAtt}(x_{i},[ x_{ 1}, x_{2},\\cdots , x_{n}])$ as the multi-head self-attention mechanism on position $i$ which considers the attention from $x_i$ to the entire sequence, i.e., $\\text{MultiHeadAtt}(x_{i},[ x_{ 1}, x_{2},\\cdots , x_{n}])=\\text{Multi-head}(x_i, [x_1, \\dots , x_n], [x_1, \\dots , x_n])$."]},{"idx":5,"heading":"Optimization for the Transformer ::: Transformer with Post-Layer Normalization ::: Position-wise FFN sub-layer","paragraphs":["In addition to the self-attention sub-layer, each Transformer layer contains a fully connected network, which is applied to each position separately and identically. This sub-layer is a two-layer feed-forward network with a ReLU activation function. Given a sequence of vectors $h_1, ..., h_n$, the computation of a position-wise FFN sub-layer on any $h_i$ is defined as:","where $W^1$, $W^2$, $b^1$ and $b^2$ are parameters."]},{"idx":6,"heading":"Optimization for the Transformer ::: Transformer with Post-Layer Normalization ::: Residual connection and layer normalization","paragraphs":["Besides the two sub-layers described above, the residual connection and layer normalization are also key components to the Transformer. For any vector $v$, the layer normalization is computed as $\\text{LayerNorm}(v) = \\gamma \\frac{v - \\mu }{\\sigma } + \\beta $, in which $\\mu , \\sigma $ are the mean and standard deviation of the elements in $v$, i.e., $\\mu = \\frac{1}{d} \\sum _{k=1}^d v_{k}$ and $\\sigma ^2 = \\frac{1}{d}\\sum _{k=1}^d(v_{k} -\\mu )^2$. Scale $\\gamma $ and bias vector $\\beta $ are parameters.","Different orders of the sub-layers, residual connection and layer normalization in a Transformer layer lead to variants of Transformer architectures. One of the original and most popularly used architecture for the Transformer and BERT BIBREF0, BIBREF8 follows “self-attention (FFN) sub-layer $\\rightarrow $ residual connection $\\rightarrow $ layer normalization”, which we call the Transformer with Post-Layer normalization (Post-LN Transformer), as illustrated in Figure SECREF1."]},{"idx":7,"heading":"Optimization for the Transformer ::: Transformer with Post-Layer Normalization ::: Post-LN Transformer","paragraphs":["Denote $x_{l, i}$ as the input of the $l$-th Transformer layer at position $i$, where $x_{l,i}$ is a real-valued vector of dimension $d$, $i=1,2,...,n$, $l=1,2,...,L$. $n$ is the length of the sequence and $L$ is the number of layers. For completeness, we define $x_{0,i}$ as the input embedding at position $i$ which is usually a combination of word embedding and positional embedding. The computations inside the $l$-th layer are composed of several steps, and we use super-scripts on $x$ to present the input(output) of different steps as in Table 1 (left), where $W^{1,l}$, $W^{2,l}$, $b^{1,l}$ and $b^{2,l}$ are parameters of the FFN sub-layer in the $l$-th layer."]},{"idx":8,"heading":"Optimization for the Transformer ::: The learning rate warm-up stage","paragraphs":["We are interested in the learning rate warm-up stage in the optimization of the Post-LN Transformer. Different from the optimization of many other architectures in which the learning rate starts from a relatively large value and then decays BIBREF32, BIBREF33, a learning rate warm-up stage for the Post-LN Transformer seems critical BIBREF10. We denote the learning rate of the $t$-th iteration as $\\text{lr}(t)$ and the maximum learning rate during training as $\\text{lr}_{max}$. Given a predefined time frame $T_{\\text{warmup}}$, the learning rate scheduler for the first $T_{\\text{warmup}}$ iterations BIBREF34 is defined as","After this warm-up stage, the learning rate will be set by classical learning rate schedulers, such as the linear decay, the inverse square-root decay, or forced decay at particular iterations. We conduct experiments to show that this learning rate warm-up stage is essential for training Post-LN Transformer models."]},{"idx":9,"heading":"Optimization for the Transformer ::: The learning rate warm-up stage ::: Experimental setting","paragraphs":["We conduct experiments on the IWSLT14 German-to-English (De-En) machine translation task. We mainly investigate two aspects: whether the learning rate warm-up stage is essential and whether the final model performance is sensitive to the value of $T_{\\text{warmup}}$. To study the first aspect, we train the model with the Adam optimizer BIBREF20 and the vanilla SGD optimizer BIBREF35 respectively. For both optimziers, we check whether the warm-up stage can be removed. We follow BIBREF0 to set hyper-parameter $\\beta $ to be $(0.9,0.98)$ in Adam. We also test different $\\text{lr}_{max}$ for both optimizers. For Adam, we set $\\text{lr}_{max}=5e^{-4}$ or $1e^{-3}$, and for SGD, we set $\\text{lr}_{max}=5e^{-3}$ or $1e^{-3}$. When the warm-up stage is used, we set $T_{\\text{warmup}}=4000$ as suggested by the original paper BIBREF0. To study the second aspect, we set $T_{\\text{warmup}}$ to be 1/500/4000 (“1” refers to the no warm-up setting) and use $\\text{lr}_{max}=5e^{-4}$ or $1e^{-3}$ with Adam. For all experiments, a same inverse square root learning rate scheduler is used after the warm-up stage. We use both validation loss and BLEU BIBREF36 as the evaluation measure of the model performance. All other details can be found in the supplementary material."]},{"idx":10,"heading":"Optimization for the Transformer ::: The learning rate warm-up stage ::: Results and discussions","paragraphs":["We record the model checkpoints for every epoch during training and calculate the validation loss and BLEU score. The performance of the models are plotted in Figure FIGREF13 and Figure FIGREF14. The x-axis is the epoch number and the y-axis is the BLEU score/validation loss. \"w/o warm-up\" indicates “without the warm-up stage” while \"w/ warm-up\" indicates “with the warm-up stage”.","First, we can see that for both optimizers, the learning rate warm-up stage is essential. Without the warm-up stage, the BLEU score of the model trained with Adam optimizer can only achieve 8.45. As a comparison, the model trained using the warm-up stage can achieve around 34 in terms of BLEU score. The same trend can also be observed on the validation loss curves. Although the performance of the model trained with SGD is significantly worse than Adam, we can still see similar phenomena as Adam. The BLEU score is just above zero in 15 epochs without using the warm-up stage.","Second, we can see that the optimization process is sensitive to the value of $T_{\\text{warmup}}$, which means $T_{\\text{warmup}}$ is an important hyper-parameter in training the Post-LN Transformer. For example, when setting $T_{\\text{warmup}}=500$, the learned models with Adam achieve only 31.16 and 2.77 in term of BLEU score for $lr_{max}=5e^{-4}$ and $1e^{-3}$ respectively.","Such a warm-up stage has several disadvantages. First, its configuration significantly affects the final performance. The practitioners need a careful hyper-parameter tuning, which is computationally expensive for large-scale NLP tasks. Second, the warm-up stage could slow down the optimization. Standard optimization algorithms usually start with a large learning rate for fast convergence. However, when using the warm-up stage, the learning rate has to gradually increase from zero, which may make the training inefficient. BIBREF11 suggests that the warm-up stage plays a role in reducing the undesirably significant variance in Adam in the early stage of model training. However, according to our results, the warm-up stage also helps the training of SGD. This suggests that the benefit of the warm-up stage may be not for a particular optimizer."]},{"idx":11,"heading":"Optimization for the Transformer ::: Understanding the Transformer at initialization","paragraphs":["We can see that the Post-LN Transformer cannot be trained with a large learning rate from scratch. This motivates us to investigate what happens at the model initialization. We first introduce the parameter initialization setting for our theoretical analysis and then present our theoretical findings."]},{"idx":12,"heading":"Optimization for the Transformer ::: Understanding the Transformer at initialization ::: Notations","paragraphs":["We denote $\\mathcal {L}(\\cdot )$ as the loss function of one position, $\\tilde{\\mathcal {L}}(\\cdot )$ as the loss function of the whole sequence, $\\Vert \\cdot \\Vert _2$ and $\\Vert \\cdot \\Vert _F$ as the $l_2$ norm (spectral norm) and the Frobenius norm, $\\text{LN}(x)$ as the standard layer normalization with scale $\\gamma =1$ and bias $\\beta =0$, and $\\mathbf {J}_{LN}(x)=\\frac{\\partial \\text{LN}(x)}{\\partial x}$ as the Jacobian matrix of $\\text{LN}(x)$. Let $\\mathcal {O}(\\cdot )$ denote standard Big-O notation that suppress multiplicative constants."]},{"idx":13,"heading":"Optimization for the Transformer ::: Understanding the Transformer at initialization ::: Parameter Initialization","paragraphs":["The parameter matrices in each Transformer layer are usually initialized by the Xavier initialization BIBREF37. Given a matrix of size $n_{in}\\times n_{out}$, the Xavier initialization sets the value of each element by independently sampling from Gaussian distribution $N(0, \\frac{2}{n_{in}+n_{out}})$. The bias vectors are usually initialized as zero vectors. The scale $\\gamma $ in the layer normalization is set to one.","For theoretical analysis, we study a simpler setting. First, we focus on single-head attention instead of the multi-head variant and for all layers, we set the shape of $W^{Q,l}$, $W^{K,l}$, $W^{V,l}$, $W^{1,l}$,$W^{2,l}$ to be $d\\times d$. Second, we initialize the parameter matrices in the self-attention sub-layer $W^{Q,l}$ and $W^{K,l}$ to be zero matrices. In this setting, the attention is a uniform distribution at initialization and $\\text{MultiHeadAtt}(x_{l, i}^1,[ x_{l, 1}^1, x_{l, 2}^1,\\cdots , x_{l, n}^1])$ can be simplified as $\\frac{1}{n}\\sum _{j=1}^{n}x_{l, j} W^{V, l}$. Third, we assume the input vectors are also sampled from the same Gaussian distribution. This is reasonable since the inputs are linear combinations of word embeddings and learnable positional embeddings, both of which are initialized by Gaussian distributions."]},{"idx":14,"heading":"Optimization for the Transformer ::: Understanding the Transformer at initialization ::: Post-LN Transformer v.s. Pre-LN Transformer","paragraphs":["We compare the Post-LN Transformer with another variant of the Transformer architecture, the Transformer with Pre-Layer Normalization (Pre-LN). The Pre-LN Transformer was implemented in several systems BIBREF34, BIBREF38, BIBREF39. BIBREF2 suggested that the Pre-LN Transformer outperforms the Post-LN Transformer when the number of layers increases. Different from the Post-LN Transformer that puts the layer normalization between the residual blocks, the Pre-LN Transformer puts the layer normalization inside the residual connection and places it before all other non-linear transformations. Additionally, the Pre-LN Transformer uses a final layer normalization right before the prediction. We provide the mathematical formulations and visualizations of the Post-LN/Pre-LN Transformer in Table TABREF9 and Figure SECREF1.","For both architectures, each $x_{L,i}$ passes through a softmax layer to produce a distribution over the dictionary $V$. The loss function is defined on the softmax distribution. For example, in sequence prediction, the loss function is defined as $\\mathcal {L}(x_{L+1,i}^{post})=-\\log (\\text{softmax}_{y_i}(W^{emb}x_{L+1,i}^{post}))$ for the Post-LN Transformer and $\\mathcal {L}(x_{Final,i}^{pre})=-\\log (\\text{softmax}_{y_i}(W^{emb}x_{Final,i}^{pre}))$ for the Pre-LN Transformer, where $\\text{softmax}_{y_i}$ is the probability of ground truth token $y_i$ outputted by the softmax distribution and $W^{emb}$ is the word embedding matrix. The loss of the whole sequence is an average of the loss on each position. Without loss of generality, we assume that all the derivatives are bounded. We introduce the following concentration property of random variables which will be further used in the theorem.","Definition 1 A random variable $Z\\ge 0$ is called $(\\epsilon ,\\delta )$-bounded if with probability at least $1-\\delta $, $\\frac{Z-\\mathbb {E}Z}{\\mathbb {E}Z}\\le \\epsilon $, where $\\epsilon >0$ and $0<\\delta <1$.","Intuitively, if the random variable $Z$ is $(\\epsilon ,\\delta )$-bounded, then with a high probability its realization will not get too far away from its expectation. For example, if $Y$ is a $d$-dimensional standard Gaussian random vector, then $Z=\\Vert Y\\Vert _2^2$ is $(\\epsilon ,\\delta )$-bounded with $\\delta =\\exp (-d\\epsilon ^2/8)$, $0<\\epsilon <1$ (see supplementary material for details). As parameter matrices in self-attention sub-layers and FFN sub-layers are initialized by Gaussian distributions, if the norm of the hidden states in the Transformer satisfies the concentrated condition above, we have the following theorem to characterize the scale of the gradients.","Theorem 1 (Gradients of the last layer in the Transformer) Assume that $\\Vert x^{post,5}_{L,i}\\Vert _2^2$ and $\\Vert x^{pre}_{L+1,i}\\Vert _2^2$ are $(\\epsilon ,\\delta )$-bounded for all $i$, where $\\epsilon $ and $\\delta =\\delta (\\epsilon )$ are small numbers. Then with probability at least $0.99-\\delta -\\frac{\\epsilon }{0.9+\\epsilon }$, for the Post-LN Transformer with $L$ layers, the gradient of the parameters of the last layer satisfies","and for the Pre-LN Transformer with $L$ layers,","From Theorem 1, we can see that for the Post-LN Transformer, the scale of the gradients to the last FFN layer is of order $\\mathcal {O}(d\\sqrt{\\ln {d}})$ which is independent of $L$. For the Pre-LN Transformer, the scale of the gradients is much smaller. We first study the forward propagation of the Post-LN Transformer and the Pre-LN Transformer. Lemma 1 will be served as a basic tool to prove the main theorem and other lemmas.","Lemma 1 If $X\\in \\mathbb {R}^d$ is a Gaussian vector, $X\\sim N(0,\\sigma ^2 \\mathbf {I}_d)$, then $\\mathbb {E}(\\Vert \\text{ReLU}(X)\\Vert _2^2)=\\frac{1}{2}\\sigma ^2 d$.","Based on Lemma 1, we have the following lemma to estimate the scale of the hidden states in different layers for the Post-LN Transformer and the Pre-LN Transformer.","Lemma 2 At initialization, for the Post-LN Transformer, $\\mathbb {E}(\\Vert x_{l,i}^{post,5}\\Vert _2^2)=\\frac{3}{2}d$ for all $l>0$ and $i$. For the Pre-LN Transformer, $(1+\\frac{l}{2})d\\le \\mathbb {E}(\\Vert x^{pre}_{l,i}\\Vert _2^2)\\le (1+\\frac{3l}{2})d$ for all $l>0$ and $i$. Expectations are taken over the input and the randomness of initialization.","Lemma 2 studies the expected norm of the hidden states in both Post-LN/Pre-LN Transformer. It is obviously that in the Post-LN Transformer, the norm of $x_{l,i}^{post}$ is $\\sqrt{d}$ and thus we study the norm of $x^{post,5}_{l,i}$ instead. As we can see from Lemma 2, the scale of the hidden states in the Post-LN Transformer keeps to be the same in expectation while the scale of the hidden states in the Pre-LN Transformer grows linearly along with the depth. The next lemma shows that the scale of the hidden states highly relates to the scale of the gradient in the architectures using layer normalization.","Lemma 3 For $x\\in \\mathbb {R}^d$, we have $\\Vert \\mathbf {J}_{LN}(x)\\Vert _2=\\mathcal {O}(\\frac{\\sqrt{d}}{\\Vert x\\Vert _2})$ in which $\\mathbf {J}_{LN}(x)=\\frac{\\partial \\text{LN}(x)}{\\partial x}$.","The proof of Lemma 1, Lemma 2, Lemma 3, and Theorem 1 can be found in the supplementary material. The main idea is that the layer normalization will normalize the gradients. In the Post-LN Transformer, the scale of the inputs to the layer normalization is independent of $L$, and thus the gradients of parameters in the last layer are independent of $L$. While in the Pre-LN Transformer, the scale of the input to the final layer normalization is linear in $L$, and thus the gradients of all parameters will be normalized by $\\sqrt{L}$."]},{"idx":15,"heading":"Optimization for the Transformer ::: Understanding the Transformer at initialization ::: Extended theory to other layers/parameters","paragraphs":["We have provided a formal proof on the gradients of the last FFN sub-layer as above. In order to fully understand the optimization, we also make some preliminary analysis for other layers and other parameters. Our main result is that the gradient norm in the Post-LN Transformer is large for the parameters near the output and will be likely to decay as the layer index $l$ decreases. On the contrary, the gradient norm in the Pre- Transformer will be likely to stay the same for any layer $l$. All the preliminary theoretical results are provided in the supplementary material."]},{"idx":16,"heading":"Optimization for the Transformer ::: Empirical verification of the theory and discussion","paragraphs":["As our theory is derived based on several simplifications of the problem, we conduct experiments to study whether our theoretical insights are consistent with what we observe in real scenarios. The general model and training configuration exactly follow Section 3.2. The experiments are repeated ten times using different random seeds."]},{"idx":17,"heading":"Optimization for the Transformer ::: Empirical verification of the theory and discussion ::: On the concentration property","paragraphs":["Given an initialized model, we record the hidden states in the Post-LN/Pre-LN Transformer across batches and find that the norm of the hidden states satisfies the property ((0.1,0.125)-bounded)."]},{"idx":18,"heading":"Optimization for the Transformer ::: Empirical verification of the theory and discussion ::: On Theorem 1","paragraphs":["Theorem 1 suggests that for any sizes of the Post-LN Transformer, the scale of the gradient norm in the last FFN sub-layer remains the same. On the contrary, that of the Pre-LN Transformer decreases as the size of the model grows. We calculate and record the gradient norm in the last FFN sub-layer in 6-6/8-8/10-10/12-12/14-14 Post-LN/Pre-LN Transformer models at initialization. The results are plotted in Figure FIGREF27 and FIGREF28. The x-axis is the size of the model, and the y-axis is the value of the gradient norm of $W^2$ in the final FFN sub-layer. The figures show when the number of layers grows, the gradient norm remains in the Post-LN Transformer (around 1.6) and decreases in the Pre-LN Transformer. This observation is consistent with our theory."]},{"idx":19,"heading":"Optimization for the Transformer ::: Empirical verification of the theory and discussion ::: On the extended theory","paragraphs":["We calculate the gradient norm of each paramter matrix in 6-6 Post-LN/Pre-LN Transformer. We record the gradient for each parameter for different mini-batches. For elements in a parameter matrix, we calculate their expected gradients and use the Frobenius norm of those values as the scale of the expected gradient of the matrix. Figure FIGREF25 and FIGREF26 shows those statistics for FFN sub-layers. The x-axis indexes different Transformer layers. It can be seen from the figure, the scale of the expected gradients grows along with the layer index for the Post-LN Transformer. On the contrary, the scale almost keeps the same for different layers in the Pre-LN Transformer. These observations are consistent with our theoretical findings."]},{"idx":20,"heading":"Optimization for the Transformer ::: Empirical verification of the theory and discussion ::: The critical warm-up stage for Post-LN Transformer","paragraphs":["Given the analysis above, we hypothesize that the gradient scale is one of the reasons that the Post-LN Transformer needs a careful learning rate scheduling. Since the gradients are large for some layers, using a large learning rate without warm-up may make the training unstable.","To verify this argument, first, we study the gradient statistics for the Post-LN Transformer after the warm-up stage with Adam. It can be seen from Figure FIGREF25 and FIGREF26 that the scale of the gradients are very small, and the model can be trained with large learning rates. Second, we conduct an experiment to train the Post-LN Transformer from scratch using a fixed small learning rate, i.e., $1e^{-4}$, to verify whether using small-step updates mitigates the issue. The details are provided in the supplementary material. In general, using a very small and fixed learning rate can mitigate the problem and optimize the Post-LN Transformer to a certain extent but the convergence is significantly slower. Both experiments above are supportive to our claim."]},{"idx":21,"heading":"Experiments","paragraphs":["We find in the previous section that the gradients at initialization for Pre-LN Transformer are well-behaved. Given this observation, we deduce that the learning rate warm-up stage can be safely removed when training Pre-LN Transformer. In this section, we empirically verify it on two main tasks in NLP, machine translation and unsupervised pre-training."]},{"idx":22,"heading":"Experiments ::: Experiment Settings ::: Machine Translation","paragraphs":["We conduct our experiments on two widely used tasks: the IWSLT14 German-to-English (De-En) task and the WMT14 English-to-German (En-De) task. For the IWSLT14 De-En task, we use the same model configuration as in Section 3. For the WMT14 En-De task, we use the Transformer base setting. More details can be found in the supplementary material.","For training the Pre-LN Transformer, we remove the learning rate warm-up stage. On the IWSLT14 De-En task, we set the initial learning rate to be $5e^{-4}$ and decay the learning rate at the 8-th epoch by 0.1. On the WMT14 En-De task, we run two experiments in which the initial learning rates are set to be $7e^{-4}/1.5e^{-3}$ respectively. Both learning rates are decayed at the 6-th epoch followed by the inverse square root learning rate scheduler.","We train the Post-LN Transformer using the learning rate warm-up stage as the baseline. In both IWSLT14 De-En task and WMT14 En-De task, we set the number of the warm-up stage to be 4000 following BIBREF0 and then use the inverse square root learning rate scheduler. For all experiments above, we use the Adam optimizer and set the hyper-parameter $\\beta $ to be $(0.9,0.98)$. We set $lr_{max}$ as same as the initial learning rates of the Pre-LN Transformer in each corresponding experiment. Since BIBREF11 suggests that the learning rate warm-up stage can be removed using RAdam, we try this optimizer on the IWSLT14 De-En task. We use linear learning rate decay suggested by BIBREF11 and keep all other hyper-parameters to be the same as in other experiments."]},{"idx":23,"heading":"Experiments ::: Experiment Settings ::: Unsupervised Pre-training (BERT)","paragraphs":["We follow BIBREF8 to use English Wikipedia corpus and BookCorpus for pre-training. As the dataset BookCorpus BIBREF40 is no longer freely distributed. We follow the suggestions from BIBREF8 to crawl and collect BookCorpus on our own. The concatenation of two datasets contains roughly 3.4B words in total, which is comparable with the data corpus used in BIBREF8. We randomly split documents into one training set and one validation set. The training-validation ratio for pre-training is 199:1.","We use base model configuration in our experiments. Similar to the translation task, we train the Pre-LN BERT without the warm-up stage and compare it with the Post-LN BERT. We follow the same hyper-parameter configuration in BIBREF8 to train the Post-LN BERT using 10k warm-up steps with $\\text{lr}_{max}=1e^{-4}$. For the Pre-LN BERT, we use linear learning rate decay starting from $3e^{-4}$ without the warm-up stage. We have tried to use a larger learning rate (such as $3e^{-4}$) for the Post-LN BERT but found the optimization diverged."]},{"idx":24,"heading":"Experiments ::: Experiment Results ::: Machine Translation","paragraphs":["We record the model checkpoints for every epoch during training and calculate the validation loss and BLEU score. The performance of the models at different checkpoints are plotted in Figure FIGREF41 - FIGREF44.","First, as we can see from the figure, the learning rate warm-up stage is not critical anymore for training the Pre-LN Transformer and the performance of the learned model is competitive. For example, on the IWSLT14 De-En task, the BLEU score and validation loss of the Pre-LN Transformer can achieve around 34 and 4, which are comparable with the performance of the Post-LN Transformer.","Second, the Pre-LN Transformer converges faster than the Post-LN Transformer using the same $\\text{lr}_{max}$. On the IWSLT14 De-En task, the 9-th checkpoint of the Pre-LN Transformer achieves nearly the same performance (validation loss/BLEU score) as 15-th checkpoint of the Post-LN Transformer. Similar observations can be found in the WMT14 En-De task. Third, compared with RAdam, we find that the change of the position of layer normalization “dominates” the change of the optimizer. According to our experiments on the IWSLT14 De-En task, we can see that although RAdam trains the Post-LN Transformer well without the warm-up stage, it has little difference with Adam when training the Pre-LN Transformer."]},{"idx":25,"heading":"Experiments ::: Experiment Results ::: Unsupervised Pre-training (BERT)","paragraphs":["We record validation loss of the model checkpoints and plot them in Figure FIGREF47. Similar to the machine translation tasks, the learning rate warm-up stage can be removed for the Pre-LN model. The Pre-LN model can be trained faster. For example, the Post-LN model achieves 1.69 validation loss at 500k updates while the Pre-LN model achieves similar validation loss at 700k updates, which suggests there is a 40% speed-up rate. Note that $T_{warmup}$ (10k) is far less than the acceleration (200k) which suggests the Pre-LN Transformer is easier to optimize using larger learning rates. We also evaluate different model checkpoints on the downstream task MRPC and RTE (more details can be found in the supplementary material). The experiments results are plotted in Figure FIGREF48 and FIGREF49. We can see that the Pre-LN model also converges faster on the downstream tasks.","As a summary, all the experiments on different tasks show that training the Pre-LN Transformer does not rely on the learning rate warm-up stage and can be trained much faster than the Post-LN Transformer."]},{"idx":26,"heading":"Conclusion and Future Work","paragraphs":["In this paper, we study why the learning rate warm-up stage is important in training the Transformer and show that the location of layer normalization matters. We show that in the original Transformer, which locates the layer normalization outside the residual blocks, the expected gradients of the parameters near the output layer are large at initialization. This leads to an unstable training when using a large learning rate. We further show that the Transformer which locates the layer normalization inside the residual blocks, can be trained without the warm-up stage and converges much faster. In the future, we will investigate other strategies of positioning the layer normalization and understand the optimization of Transformer from a theoretical perspective."]},{"idx":27,"heading":"Experimental Settings ::: Machine Translation ::: Experiment on Section 3","paragraphs":["The training/validation/test sets of the IWSLT14 German-to-English (De-En) task contain about 153K/7K/7K sentence pairs, respectively. We use a vocabulary of 10K tokens based on a joint source and target byte pair encoding (BPE) BIBREF44. All of our experiments use a Transformer architecture with a 6-layer encoder and 6-layer decoder. The size of embedding is set to 512, the size of hidden nodes in attention sub-layer and position-wise feed-forward network sub-layer are set to 512 and 1024, and the number of heads is set to 4. Label smoothed cross entropy is used as the objective function by setting $\\epsilon = 0.1$ BIBREF46, and we apply dropout with a ratio 0.1. The batch size is set to be 4096 tokens. When we decode translation results from the model during inference, we set beam size as 5 and the length penalty as 1.2."]},{"idx":28,"heading":"Experimental Settings ::: Machine Translation ::: Experiment on Section 4","paragraphs":["The configuration of IWLST14 De-En task is the same as in Section 3. For the WMT14 En-De task, we replicate the setup of BIBREF0, which consists of about 4.5M training parallel sentence pairs, and uses a 37K vocabulary based on a joint source and target BPE. Newstest2013 is used as the validation set, and Newstest2014 is used as the test set. One of the basic configurations of the Transformer architecture is the base setting, which consists of a 6-layer encoder and 6-layer decoder. The size of the hidden nodes and embeddings are set to 512. The number of heads is 8. Label smoothed cross entropy is used as the objective function by setting $\\epsilon = 0.1$. The batch size is set to be 8192 tokens per GPU on 16 NVIDIA Tesla P40 GPUs."]},{"idx":29,"heading":"Experimental Settings ::: Unsupervised Pretraining","paragraphs":["We follow BIBREF8 to use English Wikipedia corpus and BookCorpus for the pre-training. As the dataset BookCorpus BIBREF40 is no longer freely distributed. We follow the suggestions from BIBREF8 to crawl and collect BookCorpus on our own. The concatenation of two datasets includes roughly 3.4B words in total, which is comparable with the data corpus used in BIBREF8. We first segment documents into sentences with Spacy; Then, we normalize, lower-case, and tokenize texts using Moses BIBREF43 and apply BPEBIBREF45. We randomly split documents into one training set and one validation set. The training-validation ratio for pre-training is 199:1. All experiments are conducted on 32 NVIDIA Tesla P40 GPUs.","The base model in BIBREF8 consists of 12 Transformer layers. The size of hidden nodes and embeddings are set to 768, and the number of heads is set to 12."]},{"idx":30,"heading":"Experimental Settings ::: GLUE Dataset ::: MRPC","paragraphs":["The Microsoft Research Paraphrase Corpus BIBREF42 is a corpus of sentence pairs automatically extracted from online news sources, with human annotations for whether the sentences in the pair are semantically equivalent, and the task is to predict the equivalence. The performance is evaluated by the accuracy."]},{"idx":31,"heading":"Experimental Settings ::: GLUE Dataset ::: RTE","paragraphs":["The Recognizing Textual Entailment (RTE) datasets come from a series of annual textual entailment challenges BIBREF41. The task is to predict whether sentences in a sentence pair are entailment. The performance is evaluated by the accuracy."]},{"idx":32,"heading":"Experimental Settings ::: GLUE Dataset ::: Fine-tuning on GLUE tasks","paragraphs":["We use the validation set for evaluation. To fine-tune the models, following BIBREF8, BIBREF39, we search the optimization hyper-parameters in a search space including different batch sizes (16/32), learning rates ($1e^{-5}$ - $1e^{-4}$) and number of epochs (3-8). We find that the validation accuracy are sensitive to random seeds, so we repeat fine-tuning on each task for 6 times using different random seeds and compute the 95% confidence interval of validation accuracy."]},{"idx":33,"heading":"Proof of Lemma 1","paragraphs":["Denote $X=(X_1,X_2,...,X_d)$ in which $X_i$ are i.i.d. Gaussian random variables with distribution $N(0,\\sigma ^2)$. Denote $\\rho _X(x)$ as the probability density function of $X_1$. Then $\\mathbb {E}(\\Vert \\text{ReLU}(X)\\Vert _2^2)=\\sum _{i=1}^d \\mathbb {E}[\\text{ReLU}(X_i)^2] =\\sum _{i=1}^d \\mathbb {E}[\\text{ReLU}(X_i)^2|X_i\\ge 0]\\mathbb {P}(X_i\\ge 0) =\\frac{d}{2}\\mathbb {E}[\\text{ReLU}(X_1)^2|X_1\\ge 0]=\\frac{d}{2}\\mathbb {E}[X_1^2|X_1\\ge 0] =\\frac{d}{2}\\int _{-\\infty }^{+\\infty } x^2 \\rho _{X|X>0}(x)dx =\\frac{d}{2}\\int _{0}^{+\\infty } x^2 2\\rho _{X}(x)dx =\\frac{1}{2}\\sigma ^2 d$."]},{"idx":34,"heading":"Proof of Lemma 2","paragraphs":["At initialization, the layer normalization is computed as $\\text{LN}(v) = \\frac{v - \\mu }{\\sigma }$. It is easy to see that layer normalization at initialization projects any vector $v$ onto the $d-1$-sphere of radius $\\sqrt{d}$ since $\\Vert \\text{LN}(v)\\Vert _2^2=\\Vert \\frac{v - \\mu }{\\sigma }\\Vert _2^2=\\frac{\\sum _{k=1}^d(v_{k} -\\mu )^2}{\\sigma ^2}=d$.","We first estimate the expected $l_2$ norm of each intermediate output $x^{post,1}_{l,i},\\cdots ,x^{post,5}_{l,i}$ for $l>0$. Using Xavier initialization, the elements in $W^{V,l}$ are i.i.d. Gaussian random variables sampled from $N(0,1/d)$. Since $\\Vert x_{l,i}^{post}\\Vert _2^2=d$ by the definition of Layer Normalization when $l>0$, we have","and $\\mathbb {E}(\\Vert x_{l,i}^{post,2}\\Vert _2^2)=\\mathbb {E}(\\Vert x_{l,i}^{post}\\Vert _2^2)+\\mathbb {E}(\\Vert x_{l,i}^{post,1}\\Vert _2^2)=\\mathbb {E}(\\Vert x_{l,i}^{post}\\Vert _2^2)+\\mathbb {E}(\\Vert \\frac{1}{n}\\sum _{i=1}^n x_{l,i}^{post}\\Vert _2^2)\\ge \\mathbb {E}(\\Vert x_{l,i}^{post}\\Vert _2^2)=d$.","Similarly, we have $\\Vert x_{l,i}^{post,3}\\Vert _2^2=d$ by the definition of Layer Normalization. Again, for the ReLU activation function, the elements in $W^{1,l}$ and $W^{2,l}$ are i.i.d. Gaussian random variables sampled from $N(0,1/d)$. According to Lemma 1, we have","Based on this, we can estimate the scale of $\\mathbb {E}(\\Vert x_{l,i}^{post,5}\\Vert _2^2)$ as follows.","Using similar technique we can bound $\\mathbb {E}(\\Vert x_{l,i}^{pre}\\Vert _2^2)$ for the Pre-LN Transformer.","It is easy to see that we have $\\mathbb {E}(\\Vert x_{l,i}^{pre}\\Vert _2^2) \\le \\mathbb {E}(\\Vert x_{l,i}^{pre,3}\\Vert _2^2) \\le \\mathbb {E}(\\Vert x_{l,i}^{pre}\\Vert _2^2)+d$. And similar to (10)-(12),","Combining both, we have $\\mathbb {E}(\\Vert x_{l,i}^{pre}\\Vert _2^2) +\\frac{1}{2}d\\le \\mathbb {E}(\\Vert x_{l+1,i}^{pre}\\Vert _2^2)\\le \\mathbb {E}(\\Vert x_{l,i}^{pre}\\Vert _2^2)+\\frac{3}{2}d$. Then we have $(1+\\frac{l}{2})d\\le \\mathbb {E}(\\Vert x_{l,i}^{pre}\\Vert _2^2)\\le (1+\\frac{3l}{2})d$ by induction."]},{"idx":35,"heading":"Proof of Lemma 3","paragraphs":["The proof of Lemma 3 is based on Lemma 4.1:","Lemma 4 Let $\\alpha \\in \\mathbb {R}^d$ be a vector such that $\\Vert \\alpha \\Vert _2=1$, then the eigenvalue of $I-\\alpha ^{\\top }\\alpha $ is either 1 or 0.","Let $\\lbrace e_1,...,e_d\\rbrace $ be unit vectors such that $e_1=\\alpha $ and $e_i\\bot e_j$ for all $(i,j)$. Then we have $e_1(I-\\alpha ^{\\top }\\alpha )=e_1-e_1\\alpha ^{\\top }\\alpha =e_1-\\alpha =0$ and $e_i(I-\\alpha ^{\\top }\\alpha )=e_i-e_i\\alpha ^{\\top }\\alpha =e_i$ for $i\\ne 1$. So $e_i$ are all the eigenvectors of $I-\\alpha ^{\\top }\\alpha $, and their corresponding eigenvalues are $(0,1,1,...,1)$. Hence we complete our proof.","[Proof of Lemma 3] Denote $y=x(I-\\frac{1}{d}\\textbf {1}^{\\top }\\textbf {1})$, where $\\textbf {1}=(1,1,...,1)\\in \\mathbb {R}^d$, then the layer normalization can be rewritten as","We explicitly calculate the Jacobian of layer normalization as","where $\\delta _{ij}=1$ when $i=j$ and $\\delta _{ij}=0$ when $i\\ne j$. In the matrix form,","and","Since the eigenvalue of the matrix $(I-\\frac{y^{\\top }y}{\\Vert y\\Vert _2^2})$ and $(I-\\frac{1}{d}\\textbf {1}^{\\top }\\textbf {1})$ are either 1 or 0 (by Lemma 4.1), we have $\\Vert (I-\\frac{y^{\\top }y}{\\Vert y\\Vert _2^2})\\Vert _2=\\mathcal {O}(1)$ and $\\Vert (I-\\frac{1}{d}\\textbf {1}^{\\top }\\textbf {1})\\Vert _2=\\mathcal {O}(1)$. So the spectral norm of $\\mathbf {J}_{LN}(x)$ is"]},{"idx":36,"heading":"Proof of Theorem 1","paragraphs":["The proof of Theorem 1 is based on Lemma 4.2:","Lemma 5 Let $Y$ be a random variable that is never larger than B. Then for all $a<B$,","Let $X=B-Y$, then $X\\ge 0$ and Markov's inequality tells us that","Hence","[Proof of Theorem 1] We prove Theorem 1 by estimating each element of the gradient matrix. Namely, we will analyze $\\frac{\\partial \\tilde{\\mathcal {L}}}{\\partial W^{2,L}_{pq}}$ for $p,q \\in \\lbrace 1,...,d\\rbrace $. The loss of the post-LN Transformer can be written as","Through back propagation, for each $i\\in \\lbrace 1,2,...,n\\rbrace $ the gradient of $\\mathcal {L}(x_{L+1,i})$ with respect to the last layer's parameter $W^{2,L}$ in the post-LN setting can be written as:","Here $[\\text{ReLU}(x_{L,i}^{post,3}W^{1,L})]_p$ means the $p$-th element of $\\text{ReLU}(x_{L,i}^{post,3}W^{1,L})$. So the absolute value of $\\frac{\\partial \\mathcal {L}(x_{L+1,i}^{post})}{\\partial W^{2,L}_{pq}}$ can be bounded by","which implies","Since all the derivatives are bounded, we have $\\Vert \\frac{\\partial \\mathcal {L}(x_{L+1,i}^{post})}{\\partial x_{L+1,i}^{post}}\\Vert ^2_2=\\mathcal {O}(1)$. So","Since $\\Vert x_{L,i}^{post,3}\\Vert _2^2=d$, $[x_{L,i}^{post,3}W^{1,L}]_p$ has distribution $N(0,1)$, using Chernoff bound we have","So","Thus with probability at least $0.99$, for all $p=1,2,...,d$ we have $\\text{ReLU}([x_{L,i}^{post,3}W^{1,L}]_p)^2\\le 2\\ln {100d}$.","Since with probability $1-\\delta (\\epsilon )$, $\\frac{|\\Vert x^{post,5}_{L,i}\\Vert _2^2-\\mathbb {E}\\Vert x^{post,5}_{L,i}\\Vert _2^2|}{\\mathbb {E}\\Vert x^{post,5}_{L,i}\\Vert _2^2}\\le \\epsilon $, we have $\\Vert x^{post,5}_{L,i}\\Vert _2^2\\le (1+\\epsilon )\\mathbb {E}\\Vert x^{post,5}_{L,i}\\Vert _2^2$. Using Lemma 4.2, we have","for an arbitrary constant $\\alpha _0>0$, which equals","So according to union bound, with probability at least $0.99-\\delta (\\epsilon )-\\frac{\\epsilon }{1+\\epsilon -\\alpha _0}$ we have $|\\frac{\\partial \\mathcal {L}(x_{L+1,i}^{post})}{\\partial W^{2,L}_{pq}}|^2=\\mathcal {O}(\\left[\\Vert \\mathbf {J}_{LN}(x_{L,i}^{post,5})\\Vert ^2_2|[\\text{ReLU}(x_{L,i}^{post,3}W^{1,L})]_p|^2\\right])\\le \\mathcal {O}(\\frac{2d\\ln {100d}}{\\Vert x_{L,i}^{post,5}\\Vert _2^2})\\le \\mathcal {O}(\\frac{d\\ln {d}}{\\alpha _0\\mathbb {E}\\Vert x_{L,i}^{post,5}\\Vert _2^2})=\\mathcal {O}(\\frac{\\ln {d}}{\\alpha _0})$. So we have","and",".","The loss of the pre-LN Transformer can be written as","Using the same technique, in the pre-LN setting the gradient of $\\mathcal {L}(x_{Final,i}^{pre})$ with respect to the last layer's parameter $W^{2,L}$ can be written as","So the absolute value of each component of the gradient is bounded by","Since $\\Vert x_{L,i}^{pre,4}\\Vert _2^2=d$ and $[ x_{L,i}^{pre,4}W^{1,L}]_p$ obeys distribution $N(0,1)$, using Chernoff bound we have","So","So with probability at least $0.99$, for all $p=1,2,...,d$ we have $\\text{ReLU}([ x_{L,i}^{pre,4}W^{1,L}]_p)^2\\le 2\\ln {100d}$.","Since with probability $1-\\delta (\\epsilon )$, $\\frac{|\\Vert x^{pre}_{L+1,i}\\Vert _2^2-\\mathbb {E}\\Vert x^{pre}_{L+1,i}\\Vert _2^2|}{\\mathbb {E}\\Vert x^{pre}_{L+1,i}\\Vert _2^2}\\le \\epsilon $, we have $\\Vert x^{pre}_{L+1,i}\\Vert _2^2\\le (1+\\epsilon )\\mathbb {E}\\Vert x^{pre}_{L+1,i}\\Vert _2^2$. Using Lemma 5, we have","which equals","According to union bound, with probability $0.99-\\delta (\\epsilon )-\\frac{\\epsilon }{1+\\epsilon -\\alpha _0}$ we have $|\\frac{\\partial \\mathcal {L}(x_{Final,i}^{pre})}{\\partial W^{2,L}_{pq}}|^2=\\mathcal {O}(\\left[\\Vert \\mathbf {J}_{LN}(x_{L+1,i}^{pre})\\Vert ^2_2|[\\text{ReLU}(x_{L,i}^{pre,4}W^{1,L})]_p|^2\\right])\\le \\mathcal {O}(\\frac{2d\\ln {100d}}{\\Vert x_{L+1,i}^{pre}\\Vert _2^2})\\le \\mathcal {O}(\\frac{d\\ln {d}}{\\alpha _0\\mathbb {E}\\Vert x_{L+1,i}^{pre}\\Vert _2^2})=\\mathcal {O}(\\frac{\\ln {d}}{\\alpha _0 L})$. So we have","Thus $\\Vert \\frac{\\partial \\tilde{\\mathcal {L}}}{\\partial W^{2,L}}\\Vert _F=\\sqrt{\\sum _{p,q=1}^d|\\frac{\\partial \\tilde{\\mathcal {L}}}{\\partial W^{2,L}_{pq}}|^2}\\le \\mathcal {O}(\\sqrt{\\frac{d^2\\ln {d}}{\\alpha _0L}})$.","Take $\\alpha _0=\\frac{1}{10}$, we have that with probability at least $0.99-\\delta (\\epsilon )-\\frac{\\epsilon }{0.9+\\epsilon }$, for the Post-LN Transformer we have $\\Vert \\frac{\\partial \\tilde{\\mathcal {L}}}{\\partial W^{2,L}}\\Vert _F\\le \\mathcal {O}(d\\sqrt{\\ln {d}})$ and for the Pre-LN Transformer we have $\\Vert \\frac{\\partial \\tilde{\\mathcal {L}}}{\\partial W^{2,L}}\\Vert _F\\le \\mathcal {O}(d\\sqrt{\\frac{\\ln {d}}{L}})$"]},{"idx":37,"heading":"Extension to other layers","paragraphs":["For simplicity, we denote $x_l=\\text{Concat}(x_{l,1},...,x_{l,n})\\in \\mathbb {R}^{nd}$ and $x_l^k=\\text{Concat}(x_{l,1}^k,...,x_{l,n}^k)\\in \\mathbb {R}^{nd}$ for $k=\\lbrace 1,2,3,4,5\\rbrace $. Then in the Post-LN Transformer, the gradient of the parameters in the $l$-th layer (take $W^{2,l}$ as an example) can be written as","where","The Jacobian matrices of the Post-LN Transformer layers are:","where Jji = diag(( xj,ipost,3(w11,j)), ...,( xj,ipost,3(wd1,j))) Rd d","Using Hölder's inequality, we have","Since $\\frac{\\partial x_{j+1}}{\\partial x_j^{post,5}}=diag(\\mathbf {J}_{LN}(x_{j,1}^{post,5}),...,\\mathbf {J}_{LN}(x_{j,n}^{post,5}))$, we have $\\sqrt{\\mathbb {E}\\left[\\Vert \\frac{\\partial x_{j+1}^{post}}{\\partial x_j^{post,5}}\\Vert _2^2\\right]} \\approx \\sqrt{\\mathbb {E}\\frac{d}{\\Vert x_{j,1}^{post,5}\\Vert ^2_2}} \\approx \\sqrt{\\frac{2}{3}}$ when $\\Vert x_{j,1}^{post,5}\\Vert ^2_2$ concentrates around its expectation $\\mathbb {E}\\Vert x_{j,1}^{post,5}\\Vert ^2_2$ which equals $\\frac{3}{2}d$ according to Lemma 2. Therefore, when we estimate the norm of $\\frac{\\partial \\tilde{\\mathcal {L}}}{\\partial W^{2,l}}$ for post-LN transformer, there exists a term $\\mathcal {O}(\\frac{2}{3}^{(L-l)/2})$, which exponentially decreases as $l$ goes smaller. Similarly, in the pre-LN Transformer, the gradient can be written as","where","The Jacobian matrices of the Pre-LN Transformer layers are:","If $l$ is sufficiently large, the norm of $\\mathbf {J}_{LN}(x_{j,i}^{pre})$ and $\\mathbf {J}_{LN}(x^{pre,3}_{j,i})$ are very small (of order $\\mathcal {O}(\\frac{1}{\\sqrt{j}})$) as $j$ is between $l+1$ and $L$, which means the eigenvalues of matrix $\\frac{\\partial x_{j+1}^{pre}}{\\partial x_j^{pre,3}}$ and $\\frac{\\partial x_{j}^{pre,3}}{\\partial x_j^{pre}}$ are close to 1. Then we can see that $\\mathbb {E}\\Vert \\frac{\\partial x^{pre}_{j+1}}{\\partial x_j^{pre,3}}\\Vert _2$ and $\\mathbb {E}\\Vert \\frac{\\partial x_{j}^{pre,3}}{\\partial x_j^{pre}}\\Vert _2$ are nearly 1, and the norm of $\\frac{\\partial \\tilde{\\mathcal {L}}}{\\partial W^{2,l}}$ for pre-LN transformer is independent of $l$ when $l$ is large."]},{"idx":38,"heading":"Examples of @!START@$(\\epsilon ,\\delta )$@!END@-bounded random variables","paragraphs":["In this section we give an example of $(\\epsilon ,\\delta )$-bounded random variable. This example comes from Example 2.5 in BIBREF47 and we give a short description below.","If $Z=(Z_1,...,Z_n)$ is a Gaussian vector with distribution $N(0,I_n)$, then $Y=\\Vert Z\\Vert ^2_2=\\sum _{k=1}^n Z_k^2$ has distribution $\\chi ^2_n$. And $\\mathbb {E}Y=\\sum _{k=1}^n \\mathbb {E}Z_k^2=n$","A random variable $X$ with mean $\\mu =\\mathbb {E}[X]$ is called sub-exponential if there are non-negative parameters $(\\nu ,\\alpha )$ such that $\\mathbb {E}[\\exp (\\lambda (X-\\mu ))]\\le \\exp (\\frac{\\nu ^2\\lambda ^2}{2})$ for all $|\\lambda |<\\frac{1}{\\alpha }$. The next proposition comes from Proposition 2.2 in BIBREF47.","Proposition 1 (Sub-exponential tail bound) Suppose that $X$ is sub-exponential with parameters $(\\nu ,\\alpha )$. Then","and from Example 2.5 in BIBREF47, the $\\chi ^2$ variable $Y$ is sub-exponential with parameters $(\\nu , \\alpha )=(2 \\sqrt{n}, 4)$. So we can derive the one-sided bound","So $Y$ is $(\\epsilon ,\\delta )$-bounded with $\\epsilon \\in (0,1)$ and $\\delta =\\exp (-n\\epsilon ^2/8)$."]},{"idx":39,"heading":"Small learning rate experiment","paragraphs":["Theoretically, we find that the gradients of the parameters near the output layers are very large for the Post-LN Transformer and suggest using large learning rates to those parameters makes the training unstable. To verify whether using small-step updates mitigates the issue, we use a very small but fixed learning rate and check whether it can optimize the Post-LN Transformer (without the learning rate warm-up step) to a certain extent. In detail, we use a fixed learning rate of $1e^{-4}$ at the beginning of the optimization, which is much smaller than the $\\text{lr}_{max}= 1e^{-3}$ in the paper. Please note that as the learning rates during training are small, the training converges slowly, and this setting is not very practical in real large-scale tasks. We plot the validation curve together with other baseline approaches in Figure 6. We can see from the figure, the validation loss (pink curve) is around 4.3 in 27 epochs. This loss is much lower than that of the Post-LN Transformer trained using a large learning rate (blue curve). But it is still worse than the SOTA performance (green curve)."]},{"idx":40,"heading":"Figures and Tables","paragraphs":["Figure 1. (a) Post-LN Transformer layer; (b) Pre-LN Transformer layer.","Table 1. Post-LN Transformer v.s. Pre-LN Transformer","Figure 2. Performances of the models optimized by Adam and SGD on the IWSLT14 De-En task.","Figure 3. The norm of gradients of 1. different layers in the 6-6 Transformer (a,b). 2. W 2,L in different size of the Transformer (c,d).","Figure 4. Performances of the models on the IWSLT14 De-En task and WMT14 En-De task","Figure 5. Performances of the models on unsupervised pre-training (BERT) and downstream tasks","Figure 6. Performances of the models on the IWSLT14 De-En task."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"2002.07306","title":"From English To Foreign Languages: Transferring Pre-trained Language Models","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Pre-trained models have demonstrated their effectiveness in many downstream natural language processing (NLP) tasks. The availability of multilingual pre-trained models enables zero-shot transfer of NLP tasks from high resource languages to low resource ones. However, recent research in improving pre-trained models focuses heavily on English. While it is possible to train the latest neural architectures for other languages from scratch, it is undesirable due to the required amount of compute. In this work, we tackle the problem of transferring an existing pre-trained model from English to other languages under a limited computational budget. With a single GPU, our approach can obtain a foreign BERT base model within a day and a foreign BERT large within two days. Furthermore, evaluating our models on six languages, we demonstrate that our models are better than multilingual BERT on two zero-shot tasks: natural language inference and dependency parsing."]},{"idx":1,"heading":"Introduction","paragraphs":["Pre-trained models BIBREF0, BIBREF1 have received much of attention recently thanks to their impressive results in many down stream NLP tasks. Additionally, multilingual pre-trained models enable many NLP applications for other languages via zero-short cross-lingual transfer. Zero-shot cross-lingual transfer has shown promising results for rapidly building applications for low resource languages. BIBREF2 show the potential of multilingual-BERT BIBREF0 in zero-shot transfer for a large number of languages from different language families on five NLP tasks, namely, natural language inference, document classification, named entity recognition, part-of-speech tagging, and dependency parsing.","Although multilingual models are an important ingredient for enhancing language technology in many languages, recent research on improving pre-trained models puts much emphasis on English BIBREF3, BIBREF4, BIBREF5. The current state of affairs makes it difficult to translate advancements in pre-training from English to non-English languages. To our best knowledge, there are only three available multilingual pre-trained models to date: (1) the multilingual-BERT (mBERT) that supports 104 languages, (2) cross-lingual language model BIBREF6 that supports 100 languages, and (3) Language Agnostic SEntence Representations BIBREF7 that supports 93 languages. Among the three models, LASER is based on neural machine translation approach and strictly requires parallel data to train.","Do multilingual models always need to be trained from scratch? Can we transfer linguistic knowledge learned by English pre-trained models to other languages? In this work, we develop a technique to rapidly transfer an existing pre-trained model from English to other languages in an energy efficient way BIBREF8. As the first step, we focus on building a bilingual language model (LM) of English and a target language. Starting from a pre-trained English LM, we learn the target language specific parameters (i.e., word embeddings), while keeping the encoder layers of the pre-trained English LM fixed. We then fine-tune both English and target model to obtain the bilingual LM. We apply our approach to autoencoding language models with masked language model objective and show the advantage of the proposed approach in zero-shot transfer. Our main contributions in this work are:","We propose a fast adaptation method for obtaining a bilingual BERT$_{\\textsc {base}}$ of English and a target language within a day using one Tesla V100 16GB GPU.","We evaluate our bilingual LMs for six languages on two zero-shot cross-lingual transfer tasks, namely natural language inference BIBREF9 and universal dependency parsing. We show that our models offer competitive performance or even better that mBERT.","We illustrate that our bilingual LMs can serve as an excellent feature extractor in supervised dependency parsing task."]},{"idx":2,"heading":"Bilingual Pre-trained LMs","paragraphs":["We first provide some background of pre-trained language models. Let $_e$ be English word-embeddings and $\\Psi ()$ be the Transformer BIBREF10 encoder with parameters $$. Let $_{w_i}$ denote the embedding of word $w_i$ (i.e., $_{w_i} = _e[w_1]$). We omit positional embeddings and bias for clarity. A pre-trained LM typically performs the following computations: (i) transform a sequence of input tokens to contextualized representations $[_{w_1},\\dots ,_{w_n}] = \\Psi (_{w_1}, \\dots , _{w_n}; )$, and (ii) predict an output word $y_i$ at $i^{\\text{th}}$ position $p(y_i | _{w_i}) \\propto \\exp (_{w_i}^\\top _{y_i})$.","Autoencoding LM BIBREF0 corrupts some input tokens $w_i$ by replacing them with a special token [MASK]. It then predicts the original tokens $y_i = w_i$ from the corrupted tokens. Autoregressive LM BIBREF3 predicts the next token ($y_i = w_{i+1}$) given all the previous tokens. The recently proposed XLNet model BIBREF5 is an autoregressive LM that factorizes output with all possible permutations, which shows empirical performance improvement over GPT-2 due to the ability to capture bidirectional context. Here we assume that the encoder performs necessary masking with respect to each training objective.","Given an English pre-trained LM, we wish to learn a bilingual LM for English and a given target language $f$ under a limited computational resource budget. To quickly build a bilingual LM, we directly adapt the English pre-traind model to the target model. Our approach consists of three steps. First, we initialize target language word-embeddings $_f$ in the English embedding space such that embeddings of a target word and its English equivalents are close together (§SECREF8). Next, we create a target LM from the target embeddings and the English encoder $\\Psi ()$. We then fine-tune target embeddings while keeping $\\Psi ()$ fixed (§SECREF14). Finally, we construct a bilingual LM of $_e$, $_f$, and $\\Psi ()$ and fine-tune all the parameters (§SECREF15). Figure FIGREF7 illustrates the last two steps in our approach."]},{"idx":3,"heading":"Bilingual Pre-trained LMs ::: Initializing Target Embeddings","paragraphs":["Our approach to learn the initial foreign word embeddings $_f \\in ^{|V_f| \\times d}$ is based on the idea of mapping the trained English word embeddings $_e \\in ^{|V_e| \\times d}$ to $_f$ such that if a foreign word and an English word are similar in meaning then their embeddings are similar. Borrowing the idea of universal lexical sharing from BIBREF11, we represent each foreign word embedding $_f[i] \\in ^d$ as a linear combination of English word embeddings $_e[j] \\in ^d$","where $_i\\in ^{|V_e|}$ is a sparse vector and $\\sum _j^{|V_e|} \\alpha _{ij} = 1$.","In this step of initializing foreign embeddings, having a good estimation of $$ could speed of the convergence when tuning the foreign model and enable zero-shot transfer (§SECREF5). In the following, we discuss how to estimate $_i\\;\\forall i\\in \\lbrace 1,2, \\dots , |V_f|\\rbrace $ under two scenarios: (i) we have parallel data of English-foreign, and (ii) we only rely on English and foreign monolingual data."]},{"idx":4,"heading":"Bilingual Pre-trained LMs ::: Initializing Target Embeddings ::: Learning from Parallel Corpus","paragraphs":["Given an English-foreign parallel corpus, we can estimate word translation probability $p(e\\,|\\,f)$ for any (English-foreign) pair $(e, f)$ using popular word-alignment BIBREF12 toolkits such as fast-align BIBREF13. We then assign:","Since $_i$ is estimated from word alignment, it is a sparse vector."]},{"idx":5,"heading":"Bilingual Pre-trained LMs ::: Initializing Target Embeddings ::: Learning from Monolingual Corpus","paragraphs":["For low resource languages, parallel data may not be available. In this case, we rely only on monolingual data (e.g., Wikipedias). We estimate word translation probabilities from word embeddings of the two languages. Word vectors of these languages can be learned using fastText BIBREF14 and then are aligned into a shared space with English BIBREF15, BIBREF16. Unlike learning contextualized representations, learning word vectors is fast and computationally cheap. Given the aligned vectors $\\bar{}_f$ of foreign and $\\bar{}_e$ of English, we calculate the word translation matrix $\\in ^{|V_f|\\times |V_e|}$ as","Here, we use $\\mathrm {sparsemax}$ BIBREF17 instead of softmax. Sparsemax is a sparse version of softmax and it puts zero probabilities on most of the word in the English vocabulary except few English words that are similar to a given foreign word. This property is desirable in our approach since it leads to a better initialization of the foreign embeddings."]},{"idx":6,"heading":"Bilingual Pre-trained LMs ::: Fine-tuning Target Embeddings","paragraphs":["After initializing foreign word-embeddings, we replace English word-embeddings in the English pre-trained LM with foreign word-embeddings to obtain the foreign LM. We then fine-tune only foreign word-embeddings on monolingual data. The training objective is the same as the training objective of the English pre-trained LM (i.e., masked LM for BERT). Since the trained encoder $\\Psi ()$ is good at capturing association, the purpose of this step is to further optimize target embeddings such that the target LM can utilized the trained encoder for association task. For example, if the words Albert Camus presented in a French input sequence, the self-attention in the encoder more likely attends to words absurde and existentialisme once their embeddings are tuned."]},{"idx":7,"heading":"Bilingual Pre-trained LMs ::: Fine-tuning Bilingual LM","paragraphs":["We create a bilingual LM by plugging foreign language specific parameters to the pre-trained English LM (Figure FIGREF7). The new model has two separate embedding layers and output layers, one for English and one for foreign language. The encoder layer in between is shared. We then fine-tune this model using English and foreign monolingual data. Here, we keep tuning the model on English to ensure that it does not forget what it has learned in English and that we can use the resulting model for zero-shot transfer (§SECREF3). In this step, the encoder parameters are also updated so that in can learn syntactic aspects (i.e., word order, morphological agreement) of the target languages."]},{"idx":8,"heading":"Zero-shot Experiments","paragraphs":["We build our bilingual LMs, named RAMEN, starting from BERT$_{\\textsc {base}}$, BERT$_{\\textsc {large}}$, RoBERTa$_{\\textsc {base}}$, and RoBERTa$_{\\textsc {large}}$ pre-trained models. Using BERT$_{\\textsc {base}}$ allows us to compare the results with mBERT model. Using BERT$_{\\textsc {large}}$ and RoBERTa allows us to investigate whether the performance of the target LM correlates with the performance of the source LM. We evaluate our models on two cross-lingual zero-shot tasks: (1) Cross-lingual Natural Language Inference (XNLI) and (2) dependency parsing."]},{"idx":9,"heading":"Zero-shot Experiments ::: Data","paragraphs":["We evaluate our approach for six target languages: French (fr), Russian (ru), Arabic (ar), Chinese (zh), Hindi (hi), and Vietnamese (vi). These languages belong to four different language families. French, Russian, and Hindi are Indo-European languages, similar to English. Arabic, Chinese, and Vietnamese belong to Afro-Asiatic, Sino-Tibetan, and Austro-Asiatic family respectively. The choice of the six languages also reflects different training conditions depending on the amount of monolingual data. French and Russian, and Arabic can be regarded as high resource languages whereas Hindi has far less data and can be considered as low resource.","For experiments that use parallel data to initialize foreign specific parameters, we use the same datasets in the work of BIBREF6. Specifically, we use United Nations Parallel Corpus BIBREF18 for en-ru, en-ar, en-zh, and en-fr. We collect en-hi parallel data from IIT Bombay corpus BIBREF19 and en-vi data from OpenSubtitles 2018. For experiments that use only monolingual data to initialize foreign parameters, instead of training word-vectors from the scratch, we use the pre-trained word vectors from fastText BIBREF14 to estimate word translation probabilities (Eq. DISPLAY_FORM13). We align these vectors into a common space using orthogonal Procrustes BIBREF20, BIBREF15, BIBREF16. We only use identical words between the two languages as the supervised signal. We use WikiExtractor to extract extract raw sentences from Wikipedias as monolingual data for fine-tuning target embeddings and bilingual LMs (§SECREF15). We do not lowercase or remove accents in our data preprocessing pipeline.","We tokenize English using the provided tokenizer from pre-trained models. For target languages, we use fastBPE to learn 30,000 BPE codes and 50,000 codes when transferring from BERT and RoBERTa respectively. We truncate the BPE vocabulary of foreign languages to match the size of the English vocabulary in the source models. Precisely, the size of foreign vocabulary is set to 32,000 when transferring from BERT and 50,000 when transferring from RoBERTa.","We use XNLI dataset BIBREF9 for classification task and Universal Dependencies v2.4 BIBREF21 for parsing task. Since a language might have more than one treebank in Universal Dependencies, we use the following treebanks: en_ewt (English), fr_gsd (French), ru_syntagrus (Russian) ar_padt (Arabic), vi_vtb (Vietnamese), hi_hdtb (Hindi), and zh_gsd (Chinese)."]},{"idx":10,"heading":"Zero-shot Experiments ::: Data ::: Remark on BPE","paragraphs":["BIBREF22 show that sharing subwords between languages improves alignments between embedding spaces. BIBREF2 observe a strong correlation between the percentage of overlapping subwords and mBERT's performances for cross-lingual zero-shot transfer. However, in our current approach, subwords between source and target are not shared. A subword that is in both English and foreign vocabulary has two different embeddings."]},{"idx":11,"heading":"Zero-shot Experiments ::: Estimating translation probabilities","paragraphs":["Since pre-trained models operate on subword level, we need to estimate subword translation probabilities. Therefore, we subsample 2M sentence pairs from each parallel corpus and tokenize the data into subwords before running fast-align BIBREF13.","Estimating subword translation probabilities from aligned word vectors requires an additional processing step since the provided vectors from fastText are not at subword level. We use the following approximation to obtain subword vectors: the vector $_s$ of subword $s$ is the weighted average of all the aligned word vectors $_{w_i}$ that have $s$ as an subword","where $p(w_j)$ is the unigram probability of word $w_j$ and $n_s = \\sum _{w_j:\\, s\\in w_j} p(w_j)$. We take the top 50,000 words in each aligned word-vectors to compute subword vectors.","In both cases, not all the words in the foreign vocabulary can be initialized from the English word-embeddings. Those words are initialized randomly from a Gaussian $\\mathcal {N}(0, {1}{d^2})$."]},{"idx":12,"heading":"Zero-shot Experiments ::: Hyper-parameters","paragraphs":["In all the experiments, we tune RAMEN$_{\\textsc {base}}$ for 175,000 updates and RAMEN$_{\\textsc {large}}$ for 275,000 updates where the first 25,000 updates are for language specific parameters. The sequence length is set to 256. The mini-batch size are 64 and 24 when tuning language specific parameters using RAMEN$_{\\textsc {base}}$ and RAMEN$_{\\textsc {large}}$ respectively. For tuning bilingual LMs, we use a mini-batch size of 64 for RAMEN$_{\\textsc {base}}$ and 24 for RAMEN$_{\\textsc {large}}$ where half of the batch are English sequences and the other half are foreign sequences. This strategy of balancing mini-batch has been used in multilingual neural machine translation BIBREF23, BIBREF24.","We optimize RAMEN$_{\\textsc {base}}$ using Lookahead optimizer BIBREF25 wrapped around Adam with the learning rate of $10^{-4}$, the number of fast weight updates $k=5$, and interpolation parameter $\\alpha =0.5$. We choose Lookahead optimizer because it has been shown to be robust to the initial parameters of the based optimizer (Adam). For Adam optimizer, we linearly increase the learning rate from $10^{-7}$ to $10^{-4}$ in the first 4000 updates and then follow an inverse square root decay. All RAMEN$_{\\textsc {large}}$ models are optimized with Adam due to memory limit.","When fine-tuning RAMEN on XNLI and UD, we use a mini-batch size of 32, Adam's learning rate of $10^{-5}$. The number of epochs are set to 4 and 50 for XNLI and UD tasks respectively. All experiments are carried out on a single Tesla V100 16GB GPU. Each RAMEN$_{\\textsc {base}}$ model is trained within a day and each RAMEN$_{\\textsc {large}}$ is trained within two days."]},{"idx":13,"heading":"Results","paragraphs":["In this section, we present the results of out models for two zero-shot cross lingual transfer tasks: XNLI and dependency parsing."]},{"idx":14,"heading":"Results ::: Cross-lingual Natural Language Inference","paragraphs":["Table TABREF32 shows the XNLI test accuracy. For reference, we also include the scores from the previous work, notably the state-of-the-art system XLM BIBREF6. Before discussing the results, we spell out that the fairest comparison in this experiment is the comparison between mBERT and RAMEN$_{\\textsc {base}}$+BERT trained with monolingual only.","We first discuss the transfer results from BERT. Initialized from fastText vectors, RAMEN$_{\\textsc {base}}$ slightly outperforms mBERT by 1.9 points on average and widen the gap of 3.3 points on Arabic. RAMEN$_{\\textsc {base}}$ gains extra 0.8 points on average when initialized from parallel data. With triple number of parameters, RAMEN$_{\\textsc {large}}$ offers an additional boost in term of accuracy and initialization with parallel data consistently improves the performance. It has been shown that BERT$_{\\textsc {large}}$ significantly outperforms BERT$_{\\textsc {base}}$ on 11 English NLP tasks BIBREF0, the strength of BERT$_{\\textsc {large}}$ also shows up when adapted to foreign languages.","Transferring from RoBERTa leads to better zero-shot accuracies. With the same initializing condition, RAMEN$_{\\textsc {base}}$+RoBERTa outperforms RAMEN$_{\\textsc {base}}$+BERT on average by 2.9 and 2.3 points when initializing from monolingual and parallel data respectively. This result show that with similar number of parameters, our approach benefits from a better English pre-trained model. When transferring from RoBERTa$_{\\textsc {large}}$, we obtain state-of-the-art results for five languages.","Currently, RAMEN only uses parallel data to initialize foreign embeddings. RAMEN can also exploit parallel data through translation objective proposed in XLM. We believe that by utilizing parallel data during the fine-tuning of RAMEN would bring additional benefits for zero-shot tasks. We leave this exploration to future work. In summary, starting from BERT$_{\\textsc {base}}$, our approach obtains competitive bilingual LMs with mBERT for zero-shot XNLI. Our approach shows the accuracy gains when adapting from a better pre-trained model."]},{"idx":15,"heading":"Results ::: Universal Dependency Parsing","paragraphs":["We build on top of RAMEN a graph-based dependency parser BIBREF27. For the purpose of evaluating the contextual representations learned by our model, we do not use part-of-speech tags. Contextualized representations are directly fed into Deep-Biaffine layers to predict arc and label scores. Table TABREF34 presents the Labeled Attachment Scores (LAS) for zero-shot dependency parsing.","We first look at the fairest comparison between mBERT and monolingually initialized RAMEN$_{\\textsc {base}}$+BERT. The latter outperforms the former on five languages except Arabic. We observe the largest gain of +5.2 LAS for French. Chinese enjoys +3.1 LAS from our approach. With similar architecture (12 or 24 layers) and initialization (using monolingual or parallel data), RAMEN+RoBERTa performs better than RAMEN+BERT for most of the languages. Arabic and Hindi benefit the most from bigger models. For the other four languages, RAMEN$_{\\textsc {large}}$ renders a modest improvement over RAMEN$_{\\textsc {base}}$."]},{"idx":16,"heading":"Analysis ::: Impact of initialization","paragraphs":["Initializing foreign embeddings is the backbone of our approach. A good initialization leads to better zero-shot transfer results and enables fast adaptation. To verify the importance of a good initialization, we train a RAMEN$_{\\textsc {base}}$+RoBERTa with foreign word-embeddings are initialized randomly from $\\mathcal {N}(0, {1}{d^2})$. For a fair comparison, we use the same hyper-parameters in §SECREF27. Table TABREF36 shows the results of XNLI and UD parsing of random initialization. In comparison to the initialization using aligned fastText vectors, random initialization decreases the zero-shot performance of RAMEN$_{\\textsc {base}}$ by 15.9% for XNLI and 27.8 points for UD parsing on average. We also see that zero-shot parsing of SOV languages (Arabic and Hindi) suffers random initialization."]},{"idx":17,"heading":"Analysis ::: Are contextual representations from RAMEN also good for supervised parsing?","paragraphs":["All the RAMEN models are built from English and tuned on English for zero-shot cross-lingual tasks. It is reasonable to expect RAMENs do well in those tasks as we have shown in our experiments. But are they also a good feature extractor for supervised tasks? We offer a partial answer to this question by evaluating our model for supervised dependency parsing on UD datasets.","We used train/dev/test splits provided in UD to train and evaluate our RAMEN-based parser. Table TABREF38 summarizes the results (LAS) of our supervised parser. For a fair comparison, we choose mBERT as the baseline and all the RAMEN models are initialized from aligned fastText vectors. With the same architecture of 12 Transformer layers, RAMEN$_{\\textsc {base}}$+BERT performs competitive to mBERT and outshines mBERT by +1.2 points for Vietnamese. The best LAS results are obtained by RAMEN$_{\\textsc {large}}$+RoBERTa with 24 Transformer layers. Overall, our results indicate the potential of using contextual representations from RAMEN for supervised tasks."]},{"idx":18,"heading":"Analysis ::: How does linguistic knowledge transfer happen through each training stages?","paragraphs":["We evaluate the performance of RAMEN+RoBERTa$_{\\textsc {base}}$ (initialized from monolingual data) at each training steps: initialization of word embeddings (0K update), fine-tuning target embeddings (25K), and fine-tuning the model on both English and target language (at each 25K updates). The results are presented in Figure FIGREF40.","Without fine-tuning, the average accuracy of XLNI is 39.7% for a three-ways classification task, and the average LAS score is 3.6 for dependency parsing. We see the biggest leap in the performance after 50K updates. While semantic similarity task profits significantly at 25K updates of the target embeddings, syntactic task benefits with further fine-tuning the encoder. This is expected since the target languages might exhibit different syntactic structures than English and fine-tuning encoder helps to capture language specific structures. We observe a substantial gain of 19-30 LAS for all languages except French after 50K updates.","Language similarities have more impact on transferring syntax than semantics. Without tuning the English encoder, French enjoys 50.3 LAS for being closely related to English, whereas Arabic and Hindi, SOV languages, modestly reach 4.2 and 6.4 points using the SVO encoder. Although Chinese has SVO order, it is often seen as head-final while English is strong head-initial. Perhaps, this explains the poor performance for Chinese."]},{"idx":19,"heading":"Limitations","paragraphs":["While we have successfully adapted autoencoding pre-trained LMs from English to other languages, the question whether our approach can also be applied for autoregressive LM such as XLNet still remains. We leave the investigation to future work."]},{"idx":20,"heading":"Conclusions","paragraphs":["In this work, we have presented a simple and effective approach for rapidly building a bilingual LM under a limited computational budget. Using BERT as the starting point, we demonstrate that our approach produces better than mBERT on two cross-lingual zero-shot sentence classification and dependency parsing. We find that the performance of our bilingual LM, RAMEN, correlates with the performance of the original pre-trained English models. We also find that RAMEN is also a powerful feature extractor in supervised dependency parsing. Finally, we hope that our work sparks of interest in developing fast and effective methods for transferring pre-trained English models to other languages."]},{"idx":21,"heading":"Figures and Tables","paragraphs":["Figure 1: Pictorial illustration of our approach. Left: fine-tune language specific parameters while keeping transformer encoder fixed. Right: jointly train a bilingual LM and update all the parameters.","Table 1: Zero-shot classification results on XNLI. ì indicates parallel data is used. RAMEN only uses parallel data for initialization. The best results are marked in bold.","Table 2: LAS scores for zero-shot dependency parsing. ì indicates parallel data is used for initialization. Punctuation are removed during the evaluation. The best results are marked in bold.","Table 3: Comparison between random initialization (rnd) of language specific parameters and initialization using aligned fastText vectors (vec).","Table 4: Evaluation in supervised UD parsing. The scores are LAS.","Figure 2: Accuracy and LAS evaluated at each checkpoints.","Table 5: UAS/LAS scores for zero-shot dependency parsing. ì indicates parallel data is used for initialization. Punctuation are removed during the evaluation.","Table 6: Comparison between b-BERT trained from scratch for 1,000,000 updates and RAMEN trained for 175,000 updates."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"2003.02249","title":"jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models","sections":[{"idx":0,"heading":"Abstract","paragraphs":["We introduce jiant, an open source toolkit for conducting multitask and transfer learning experiments on English NLU tasks. jiant enables modular and configuration-driven experimentation with state-of-the-art models and implements a broad set of tasks for probing, transfer learning, and multitask training experiments. jiant implements over 50 NLU tasks, including all GLUE and SuperGLUE benchmark tasks. We demonstrate that jiant reproduces published performance on a variety of tasks and models, including BERT and RoBERTa. jiant is available at https://jiant.info."]},{"idx":1,"heading":"Introduction","paragraphs":["This paper introduces jiant, an open source toolkit that allows researchers to quickly experiment on a wide array of NLU tasks, using state-of-the-art NLP models, and conduct experiments on probing, transfer learning, and multitask training. jiant supports many state-of-the-art Transformer-based models implemented by Huggingface's Transformers package, as well as non-Transformer models such as BiLSTMs.","Packages and libraries like HuggingFace's Transformers BIBREF0 and AllenNLP BIBREF1 have accelerated the process of experimenting and iterating on NLP models by both abstracting out implementation details, and simplifying the model training pipeline. jiant extends the capabilities of both toolkits by presenting a wrapper that implements a variety of complex experimental pipelines in a scalable and easily controllable setting. jiant contains a task bank of over 50 tasks, including all the tasks presented in GLUE BIBREF2, SuperGLUE BIBREF3, the edge-probing suite BIBREF4, and the SentEval probing suite BIBREF5, as well as other individual tasks including CCG supertagging BIBREF6, SocialIQA BIBREF7, and CommonsenseQA BIBREF8. jiant is also the official baseline codebase for the SuperGLUE benchmark.","jiant's core design principles are:","Ease of use: jiant should allow users to run a variety of experiments using state-of-the-art models via an easy to use configuration-driven interface. jiant should also provide features that support correct and reproducible experiments, including logging and saving and restoring model state.","Availability of NLU tasks: jiant should maintain and continue to grow a collection of tasks useful for NLU research, especially popular evaluation tasks and tasks commonly used in pretraining and transfer learning.","Availability of cutting-edge models: jiant should make implementations of state-of-the-art models available for experimentation.","Open source: jiant should be free to use, and easy to contribute to.","Early versions of jiant have already been used in multiple works, including probing analyses BIBREF4, BIBREF9, BIBREF10, BIBREF11, BIBREF12, BIBREF13, transfer learning experiments BIBREF14, BIBREF15, and dataset and benchmark construction BIBREF3, BIBREF2, BIBREF16."]},{"idx":2,"heading":"Background","paragraphs":["Transfer learning is an area of research that uses knowledge from pretrained models to transfer to new tasks. In recent years, Transformer-based models like BERT BIBREF17 and T5 BIBREF18 have yielded state-of-the-art results on the lion's share of benchmark tasks for language understanding through pretraining and transfer, often paired with some form of multitask learning.","jiant enables a variety of complex training pipelines through simple configuration changes, including multi-task training BIBREF19, BIBREF20 and pretraining, as well as the sequential fine-tuning approach from STILTs BIBREF15. In STILTs, intermediate task training takes a pretrained model like ELMo or BERT, and applies supplementary training on a set of intermediate tasks, before finally performing single-task training on additional downstream tasks."]},{"idx":3,"heading":"jiant System Overview ::: Requirements","paragraphs":["jiant v1.3.0 requires Python 3.5 or later. jiant can be installed via pip, or cloned and installed from GitHub. jiant's core dependencies are PyTorch BIBREF21, AllenNLP BIBREF1, and HuggingFace's Transformers BIBREF0. jiant is released under the MIT License BIBREF22."]},{"idx":4,"heading":"jiant System Overview ::: jiant Components","paragraphs":["Tasks: Tasks have references to task data, methods for processing data, references to classifier heads, and methods for calculating performance metrics, and making predictions.","Sentence Encoder: Sentence encoders map from the indexed examples to a sentence-level representation. Sentence encoders can include an input module (e.g., Transformer models, ELMo, or word embeddings), followed by an optional second layer of encoding (usually a BiLSTM). Examples of possible sentence encoder configurations include BERT, ELMo followed by a BiLSTM, BERT with a variety of pooling and aggregation metthods, or a bag of words model.","Task-Specific Output Heads: Task-specific output modules map representations from sentence encoders to outputs specific to a task, e.g. entailment/neutral/contradiction for NLI tasks, or tags for part-of-speech tagging. They also include logic for computing the corresponding loss for training (e.g. cross-entropy).","Trainer: Trainers manage the control flow for the training and validation loop for experiments. They sample batches from one or more tasks, perform forward and backward passes, calculate training metrics, evaluate on a validation set, and save checkpoints. Users can specify experiment-specific parameters such as learning rate, batch size, and more.","Config: Config files or flags are defined in HOCON format. Configs specify parameters for jiant experiments including choices of tasks, sentence encoder, and training routine.","Configs are jiant's primary user interface. Tasks and modeling components are designed to be modular, while jiant's pipeline is a monolithic, configuration-driven design intended to facilitate a number of common workflows outlined in SECREF17."]},{"idx":5,"heading":"jiant System Overview ::: jiant Pipeline Overview","paragraphs":["jiant's core pipeline consists of the five stages described below and illustrated in Figure FIGREF16:","A config or multiple configs defining an experiment are interpreted. Users can choose and configure models, tasks, and stages of training and evaluation.","The tasks and sentence encoder are prepared:","The task data is loaded, tokenized, and indexed, and (optionally) the preprocessed task objects are serialized and cached. In this process, AllenNLP is used to create the vocabulary and index the tokenized data.","The sentence encoder is constructed and (optionally) pretrained weights are loaded.","The task-specific output heads are created for each task, and task heads are attached to a common sentence encoder. Optionally, different tasks can share the same output head, as in BIBREF20.","Optionally, in the intermediate phase the trainer samples batches randomly from one or more tasks, and trains the shared model.","Optionally, in the target training phase, a copy of the model is configured and trained or fine-tuned for each target task separately.","Optionally, the model is evaluated on the validation and/or test sets of the target tasks."]},{"idx":6,"heading":"jiant System Overview ::: Task and Model resources in jiant","paragraphs":["jiant supports over 50 tasks. Task types include classification, regression, sequence generation, tagging, and span prediction. jiant focuses on NLU tasks like MNLI BIBREF24, CommonsenseQA BIBREF8, the Winograd Schema Challenge BIBREF25, and SQuAD BIBREF26. A full inventory of tasks and task variants is available in the jiant/tasks module.","jiant provides support for cutting-edge sentence encoder models, including support for Huggingface's Transformers. Supported models include: BERT BIBREF17, RoBERTa BIBREF27, XLNet BIBREF28, XLM BIBREF29, GPT BIBREF30, GPT-2 BIBREF31, ALBERT BIBREF32 and ELMo BIBREF33. jiant also supports the from-scratch training of (bidirectional) LSTMs BIBREF34 and deep bag of words models BIBREF35, as well as syntax-aware models such as PRPN BIBREF36 and ON-LSTM BIBREF37. jiant also supports word embeddings such as GloVe BIBREF38."]},{"idx":7,"heading":"jiant System Overview ::: User Interface","paragraphs":["jiant experiments can be run with a simple CLI:","jiant provides default config files that allow running many experiments without modifying source code.","jiant also provides baseline config files that can serve as a starting point for model development and evaluation against GLUE BIBREF2 and SuperGLUE BIBREF3 benchmarks.","More advanced configurations can be developed by composing multiple configurations files and overrides. Figure FIGREF29 shows a config file that overrides a default config, defining an experiment that uses BERT as the sentence encoder. This config includes an example of a task-specific configuration, which can be overridden in another config file or via a command line override.","jiant also implements the option to provide command line overrides with a flag. This option makes it easy to write scripts that launch jiant experiments over a range of parameters, for example while performing grid search across hyperparameters. jiant users have successfully run large-scale experiments launching hundreds of runs on both Kubernetes and Slurm."]},{"idx":8,"heading":"jiant System Overview ::: Example jiant Use Cases and Options","paragraphs":["Here we highlight some example use cases and key corresponding jiant config options required in these experiments:","Fine-tune BERT on SWAG BIBREF39 and SQUAD BIBREF26, then fine-tune on HellaSwag BIBREF40:","Train a probing classifier over a frozen BERT model, as in BIBREF9:","Compare performance of GloVe BIBREF38 embeddings using a BiLSTM:","Evaluate ALBERT BIBREF32 on the MNLI BIBREF24 task:"]},{"idx":9,"heading":"jiant System Overview ::: jiant Deployment Environments","paragraphs":["jiant runs on consumer-grade hardware or in cluster environments with or without CUDA GPUs. The jiant repository also contains documentation and configuration files demonstrating how to deploy jiant in Kubernetes clusters on Google Kubernetes Engine."]},{"idx":10,"heading":"jiant System Overview ::: Logging and Metric Tracking","paragraphs":["jiant generates custom log files that capture experimental configurations, training and evaluation metrics, and relevant run-time information.","jiant also generates TensorBoard event files BIBREF41 for training and evaluation metric tracking. TensorBoard event files can be visualized using the TensorBoard Scalars Dashboard."]},{"idx":11,"heading":"jiant System Overview ::: Optimizations and Other Features","paragraphs":["jiant implements features that improve run stability and efficiency:","jiant implements checkpointing options designed to offer efficient early stopping and to show consistent behavior when restarting after an interruption.","jiant caches preprocessed task data to speed up reuse across experiments which share common data resources and artifacts.","jiant implements gradient accumulation and multi-GPU, which enables training on larger batches than can fit in memory for a single GPU.","jiant supports outputting predictions in a format ready for GLUE and SuperGLUE benchmark submission."]},{"idx":12,"heading":"jiant System Overview ::: Extensibility","paragraphs":["jiant's design offers conveniences that reduce the need to modify code when making changes:","jiant's task registry makes it easy to define a new version of an existing task using different data. Once the new task is defined in the task registry, the task is available as an option in jiant's config.","jiant's sentence encoder and task output head abstractions allow for easy support of new sentence encoders.","In use cases requiring the introduction of a new task, users can use class inheritance to build on a number of available parent task types including classification, tagging, span prediction, span classification, sequence generation, regression, ranking, and multiple choice task classes. For these task types, corresponding task-specific output heads are already implemented.","More than 30 researchers and developers from more than 5 institutions have contributed code to the jiant project. jiant's maintainers welcome pull requests that introduce new tasks or sentence encoder components, and pull request are actively reviewed. The jiant repository's continuous integration system requires that all pull requests pass unit and integration tests and meet Black code formatting requirements."]},{"idx":13,"heading":"jiant System Overview ::: Limitations","paragraphs":["While jiant is quite flexible in the pipelines that can be specified through configs, and some components are highly modular (e.g., tasks, sentence encoders, and output heads), modification of the pipeline code can be difficult. For example, training in more than two phases would require modifying the trainer code. Making multi-stage training configurations more flexible is on jiant's development roadmap."]},{"idx":14,"heading":"jiant System Overview ::: jiant Development Roadmap","paragraphs":["jiant is actively being developed. The jiant project has prioritized continuing to add support for new Transformer models and adding tasks that are commonly used for pretraining and evaluation in NLU, including sequence-to-sequence tasks.","Additionally, there are plans to make jiant's training phase configuration options more flexible to allow training in more than two phases, and to continue to refactor jiant's code to keep jiant flexible to track developments in NLU research."]},{"idx":15,"heading":"Benchmark Experiments","paragraphs":["To benchmark jiant, we perform a set of experiments that reproduce external results for single fine-tuning and transfer learning experiments. jiant has been benchmarked extensively in both published and ongoing work on a majority of the implemented tasks.","We benchmark single-task fine-tuning configurations using CommonsenseQA BIBREF8 and SocialIQA BIBREF7. On CommonsenseQA with $\\mathrm {RoBERTa}_\\mathrm {LARGE}$, jiant achieves an accuracy of 0.722, comparable to 0.721 reported by BIBREF27. On SocialIQA with BERT-large, jiant achieves a dev set accuracy of 0.658, comparable to 0.66 reported in BIBREF7.","Next, we benchmark jiant's transfer learning regime. We perform transfer experiments from MNLI to BoolQ with BERT-large. In this configuration BIBREF42 demonstrated an accuracy improvement of 0.78 to 0.82 on the dev set, and jiant achieves an improvement of 0.78 to 0.80."]},{"idx":16,"heading":"Conclusion","paragraphs":["jiant provides a configuration-driven interface for defining transfer learning and representation learning experiments using a bank of over 50 NLU tasks, cutting-edge sentence encoder models, and multi-task and multi-stage training procedures. Further, jiant is shown to be able to replicate published performance on various NLU tasks.","jiant's modular design of task and sentence encoder components make it possible for users to quickly and easily experiment with a large number of tasks, models, and parameter configurations, without editing source code. jiant's design also makes it easy to add new tasks, and jiant's architecture makes it convenient to extend jiant to support new sentence encoders.","jiant code is open source, and jiant invites contributors to open issues or submit pull request to the jiant project repository: https://github.com/nyu-mll/jiant."]},{"idx":17,"heading":"Acknowledgments","paragraphs":["Katherin Yu, Jan Hula, Patrick Xia, Raghu Pappagari, Shuning Jin, R. Thomas McCoy, Roma Patel, Yinghui Huang, Edouard Grave, Najoung Kim, Thibault Févry, Berlin Chen, Nikita Nangia, Anhad Mohananey, Katharina Kann, Shikha Bordia, Nicolas Patry, David Benton, and Ellie Pavlick have contributed substantial engineering assistance to the project.","The early development of jiant took at the 2018 Frederick Jelinek Memorial Summer Workshop on Speech and Language Technologies, and was supported by Johns Hopkins University with unrestricted gifts from Amazon, Facebook, Google, Microsoft and Mitsubishi Electric Research Laboratories.","Subsequent development was possible in part by a donation to NYU from Eric and Wendy Schmidt made by recommendation of the Schmidt Futures program, by support from Intuit Inc., and by support from Samsung Research under the project Improving Deep Learning using Latent Structure. We gratefully acknowledge the support of NVIDIA Corporation with the donation of a Titan V GPU used at NYU in this work. Alex Wang's work on the project is supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE 1342536. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. Yada Pruksachatkun's work on the project is supported in part by the Moore-Sloan Data Science Environment as part of the NYU Data Science Services initiative. Sam Bowman's work on jiant during Summer 2019 took place in his capacity as a visiting researcher at Google."]},{"idx":18,"heading":"Figures and Tables","paragraphs":["Figure 1: Multi-phase jiant experiment configuration used by Wang et al. (2019a): a BERT sentence encoder is trained with an intermediate task model during jiant’s intermediate training phase, and fine-tuned with various target task models in jiant’s target training phase.","Figure 2: jiant pipeline stages using RoBERTa as the sentence encoder, ReCoRD and MNLI tasks as intermediate tasks, and MNLI and BoolQ as tasks for target training and evaluation. The diagram highlights that during target training and evaluation phases, copies are made of the sentence encoder model, and fine tuning and evaluation for each task are conducted on separate copies.","Figure 3: Example jiant experiment config file."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"2003.04642","title":"A Framework for Evaluation of Machine Reading Comprehension Gold Standards","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Machine Reading Comprehension (MRC) is the task of answering a question over a paragraph of text. While neural MRC systems gain popularity and achieve noticeable performance, issues are being raised with the methodology used to establish their performance, particularly concerning the data design of gold standards that are used to evaluate them. There is but a limited understanding of the challenges present in this data, which makes it hard to draw comparisons and formulate reliable hypotheses. As a first step towards alleviating the problem, this paper proposes a unifying framework to systematically investigate the present linguistic features, required reasoning and background knowledge and factual correctness on one hand, and the presence of lexical cues as a lower bound for the requirement of understanding on the other hand. We propose a qualitative annotation schema for the first and a set of approximative metrics for the latter. In a first application of the framework, we analyse modern MRC gold standards and present our findings: the absence of features that contribute towards lexical ambiguity, the varying factual correctness of the expected answers and the presence of lexical cues, all of which potentially lower the reading comprehension complexity and quality of the evaluation data."]},{"idx":1,"heading":"Introduction","paragraphs":["There is a recent spark of interest in the task of Question Answering (QA) over unstructured textual data, also referred to as Machine Reading Comprehension (MRC). This is mostly due to wide-spread success of advances in various facets of deep learning related research, such as novel architectures BIBREF0, BIBREF1 that allow for efficient optimisation of neural networks consisting of multiple layers, hardware designed for deep learning purposes and software frameworks BIBREF2, BIBREF3 that allow efficient development and testing of novel approaches. These factors enable researchers to produce models that are pre-trained on large scale corpora and provide contextualised word representations BIBREF4 that are shown to be a vital component towards solutions for a variety of natural language understanding tasks, including MRC BIBREF5. Another important factor that led to the recent success in MRC-related tasks is the widespread availability of various large datasets, e.g., SQuAD BIBREF6, that provide sufficient examples for optimising statistical models. The combination of these factors yields notable results, even surpassing human performance BIBREF7.","MRC is a generic task format that can be used to probe for various natural language understanding capabilities BIBREF8. Therefore it is crucially important to establish a rigorous evaluation methodology in order to be able to draw reliable conclusions from conducted experiments. While increasing effort is put into the evaluation of novel architectures, such as keeping the evaluation data from public access to prevent unintentional overfitting to test data, performing ablation and error studies and introducing novel metrics BIBREF9, surprisingly little is done to establish the quality of the data itself. Additionally, recent research arrived at worrisome findings: the data of those gold standards, which is usually gathered involving a crowd-sourcing step, suffers from flaws in design BIBREF10 or contains overly specific keywords BIBREF11. Furthermore, these gold standards contain “annotation artefacts”, cues that lead models into focusing on superficial aspects of text, such as lexical overlap and word order, instead of actual language understanding BIBREF12, BIBREF13. These weaknesses cast some doubt on whether the data can reliably evaluate the reading comprehension performance of the models they evaluate, i.e. if the models are indeed being assessed for their capability to read.","Figure FIGREF3 shows an example from HotpotQA BIBREF14, a dataset that exhibits the last kind of weakness mentioned above, i.e., the presence of unique keywords in both the question and the passage (in close proximity to the expected answer).","An evaluation methodology is vital to the fine-grained understanding of challenges associated with a single gold standard, in order to understand in greater detail which capabilities of MRC models it evaluates. More importantly, it allows to draw comparisons between multiple gold standards and between the results of respective state-of-the-art models that are evaluated on them.","In this work, we take a step back and propose a framework to systematically analyse MRC evaluation data, typically a set of questions and expected answers to be derived from accompanying passages. Concretely, we introduce a methodology to categorise the linguistic complexity of the textual data and the reasoning and potential external knowledge required to obtain the expected answer. Additionally we propose to take a closer look at the factual correctness of the expected answers, a quality dimension that appears under-explored in literature.","We demonstrate the usefulness of the proposed framework by applying it to precisely describe and compare six contemporary MRC datasets. Our findings reveal concerns about their factual correctness, the presence of lexical cues that simplify the task of reading comprehension and the lack of semantic altering grammatical modifiers. We release the raw data comprised of 300 paragraphs, questions and answers richly annotated under the proposed framework as a resource for researchers developing natural language understanding models and datasets to utilise further.","To the best of our knowledge this is the first attempt to introduce a common evaluation methodology for MRC gold standards and the first across-the-board qualitative evaluation of MRC datasets with respect to the proposed categories."]},{"idx":2,"heading":"Framework for MRC Gold Standard Analysis ::: Problem definition","paragraphs":["We define the task of machine reading comprehension, the target application of the proposed methodology as follows: Given a paragraph $P$ that consists of tokens (words) $p_1, \\ldots , p_{n_P}$ and a question $Q$ that consists of tokens $q_1 \\ldots q_{n_Q}$, the goal is to retrieve an answer $A$ with tokens $a_1 \\ldots a_{n_A}$. $A$ is commonly constrained to be one of the following cases BIBREF15, illustrated in Figure FIGREF9:","Multiple choice, where the goal is to predict $A$ from a given set of choices $\\mathcal {A}$.","Cloze-style, where $S$ is a sentence, and $A$ and $Q$ are obtained by removing a sequence of words such that $Q = S - A$. The task is to fill in the resulting gap in $Q$ with the expected answer $A$ to form $S$.","Span, where is a continuous subsequence of tokens from the paragraph ($A \\subseteq P$). Flavours include multiple spans as the correct answer or $A \\subseteq Q$.","Free form, where $A$ is an unconstrained natural language string.","A gold standard $G$ is composed of $m$ entries $(Q_i, A_i, P_i)_{i\\in \\lbrace 1,\\ldots , m\\rbrace }$.","The performance of an approach is established by comparing its answer predictions $A^*_{i}$ on the given input $(Q_i, T_i)$ (and $\\mathcal {A}_i$ for the multiple choice setting) against the expected answer $A_i$ for all $i\\in \\lbrace 1,\\ldots , m\\rbrace $ under a performance metric. Typical performance metrics are exact match (EM) or accuracy, i.e. the percentage of exactly predicted answers, and the F1 score – the harmonic mean between the precision and the recall of the predicted tokens compared to expected answer tokens. The overall F1 score can either be computed by averaging the F1 scores for every instance or by first averaging the precision and recall and then computing the F1 score from those averages (macro F1). Free-text answers, meanwhile, are evaluated by means of text generation and summarisation metrics such as BLEU BIBREF16 or ROUGE-L BIBREF17."]},{"idx":3,"heading":"Framework for MRC Gold Standard Analysis ::: Dimensions of Interest","paragraphs":["In this section we describe a methodology to categorise gold standards according to linguistic complexity, required reasoning and background knowledge, and their factual correctness. Specifically, we use those dimensions as high-level categories of a qualitative annotation schema for annotating question, expected answer and the corresponding context. We further enrich the qualitative annotations by a metric based on lexical cues in order to approximate a lower bound for the complexity of the reading comprehension task. By sampling entries from each gold standard and annotating them, we obtain measurable results and thus are able to make observations about the challenges present in that gold standard data."]},{"idx":4,"heading":"Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Problem setting","paragraphs":["We are interested in different types of the expected answer. We differentiate between Span, where an answer is a continuous span taken from the passage, Paraphrasing, where the answer is a paraphrase of a text span, Unanswerable, where there is no answer present in the context, and Generated, if it does not fall into any of the other categories. It is not sufficient for an answer to restate the question or combine multiple Span or Paraphrasing answers to be annotated as Generated. It is worth mentioning that we focus our investigations on answerable questions. For a complementary qualitative analysis that categorises unanswerable questions, the reader is referred to Yatskar2019.","Furthermore, we mark a sentence as Supporting Fact if it contains evidence required to produce the expected answer, as they are used further in the complexity analysis."]},{"idx":5,"heading":"Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Factual Correctness","paragraphs":["An important factor for the quality of a benchmark is its factual correctness, because on the one hand, the presence of factually wrong or debatable examples introduces an upper bound for the achievable performance of models on those gold standards. On the other hand, it is hard to draw conclusions about the correctness of answers produced by a model that is evaluated on partially incorrect data.","One way by which developers of modern crowd-sourced gold standards ensure quality is by having the same entry annotated by multiple workers BIBREF18 and keeping only those with high agreement. We investigate whether this method is enough to establish a sound ground truth answer that is unambiguously correct. Concretely we annotate an answer as Debatable when the passage features multiple plausible answers, when multiple expected answers contradict each other, or an answer is not specific enough with respect to the question and a more specific answer is present. We annotate an answer as Wrong when it is factually wrong and a correct answer is present in the context."]},{"idx":6,"heading":"Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Required Reasoning","paragraphs":["It is important to understand what types of reasoning the benchmark evaluates, in order to be able to accredit various reasoning capabilities to the models it evaluates. Our proposed reasoning categories are inspired by those found in scientific question answering literature BIBREF19, BIBREF20, as research in this area focuses on understanding the required reasoning capabilities. We include reasoning about the Temporal succession of events, Spatial reasoning about directions and environment, and Causal reasoning about the cause-effect relationship between events. We further annotate (multiple-choice) answers that can only be answered By Exclusion of every other alternative.","We further extend the reasoning categories by operational logic, similar to those required in semantic parsing tasks BIBREF21, as solving those tasks typically requires “multi-hop” reasoning BIBREF14, BIBREF22. When an answer can only be obtained by combining information from different sentences joined by mentioning a common entity, concept, date, fact or event (from here on called entity), we annotate it as Bridge. We further annotate the cases, when the answer is a concrete entity that satisfies a Constraint specified in the question, when it is required to draw a Comparison of multiple entities' properties or when the expected answer is an Intersection of their properties (e.g. “What do Person A and Person B have in common?”)","We are interested in the linguistic reasoning capabilities probed by a gold standard, therefore we include the appropriate category used by Wang2019. Specifically, we annotate occurrences that require understanding of Negation, Quantifiers (such as “every”, “some”, or “all”), Conditional (“if ...then”) statements and the logical implications of Con-/Disjunction (i.e. “and” and “or”) in order to derive the expected answer.","Finally, we investigate whether arithmetic reasoning requirements emerge in MRC gold standards as this can probe for reasoning that is not evaluated by simple answer retrieval BIBREF23. To this end, we annotate the presence of of Addition and Subtraction, answers that require Ordering of numerical values, Counting and Other occurrences of simple mathematical operations.","An example can exhibit multiple forms of reasoning. Notably, we do not annotate any of the categories mentioned above if the expected answer is directly stated in the passage. For example, if the question asks “How many total points were scored in the game?” and the passage contains a sentence similar to “The total score of the game was 51 points”, it does not require any reasoning, in which case we annotate it as Retrieval."]},{"idx":7,"heading":"Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Knowledge","paragraphs":["Worthwhile knowing is whether the information presented in the context is sufficient to answer the question, as there is an increase of benchmarks deliberately designed to probe a model's reliance on some sort of background knowledge BIBREF24. We seek to categorise the type of knowledge required. Similar to Wang2019, on the one hand we annotate the reliance on factual knowledge, that is (Geo)political/Legal, Cultural/Historic, Technical/Scientific and Other Domain Specific knowledge about the world that can be expressed as a set of facts. On the other hand, we denote Intuitive knowledge requirements, which is challenging to express as a set of facts, such as the knowledge that a parenthetic numerical expression next to a person's name in a biography usually denotes his life span."]},{"idx":8,"heading":"Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Linguistic Complexity","paragraphs":["Another dimension of interest is the evaluation of various linguistic capabilities of MRC models BIBREF25, BIBREF26, BIBREF27. We aim to establish which linguistic phenomena are probed by gold standards and to which degree. To that end, we draw inspiration from the annotation schema used by Wang2019, and adapt it around lexical semantics and syntax.","More specifically, we annotate features that introduce variance between the supporting facts and the question. With regard to lexical semantics, we focus on the use of redundant words that do not alter the meaning of a sentence for the task of retrieving the expected answer (Redundancy), requirements on the understanding of words' semantic fields (Lexical Entailment) and the use of Synonyms and Paraphrases with respect to the question wording. Furthermore we annotate cases where supporting facts contain Abbreviations of concepts introduced in the question (and vice versa) and when a Dative case substitutes the use of a preposition (e.g. “I bought her a gift” vs “I bought a gift for her”). Regarding syntax, we annotate changes from passive to active Voice, the substitution of a Genitive case with a preposition (e.g. “of”) and changes from nominal to verbal style and vice versa (Nominalisation).","We recognise features that add ambiguity to the supporting facts, for example when information is only expressed implicitly by using an Ellipsis. As opposed to redundant words, we annotate Restrictivity and Factivity modifiers, words and phrases whose presence does change the meaning of a sentence with regard to the expected answer, and occurrences of intra- or inter-sentence Coreference in supporting facts (that is relevant to the question). Lastly, we mark ambiguous syntactic features, when their resolution is required in order to obtain the answer. Concretely, we mark argument collection with con- and disjunctions (Listing) and ambiguous Prepositions, Coordination Scope and Relative clauses/Adverbial phrases/Appositions."]},{"idx":9,"heading":"Framework for MRC Gold Standard Analysis ::: Dimensions of Interest ::: Complexity","paragraphs":["Finally, we want to approximate the presence of lexical cues that might simplify the reading required in order to arrive at the answer. Quantifying this allows for more reliable statements about and comparison of the complexity of gold standards, particularly regarding the evaluation of comprehension that goes beyond simple lexical matching. We propose the use of coarse metrics based on lexical overlap between question and context sentences. Intuitively, we aim to quantify how much supporting facts “stand out” from their surrounding passage context. This can be used as proxy for the capability to retrieve the answer BIBREF10. Specifically, we measure (i) the number of words jointly occurring in a question and a sentence, (ii) the length of the longest n-gram shared by question and sentence and (iii) whether a word or n-gram from the question uniquely appears in a sentence.","The resulting taxonomy of the framework is shown in Figure FIGREF10. The full catalogue of features, their description, detailed annotation guideline as well as illustrating examples can be found in Appendix ."]},{"idx":10,"heading":"Application of the Framework ::: Candidate Datasets","paragraphs":["We select contemporary MRC benchmarks to represent all four commonly used problem definitions BIBREF15. In selecting relevant datasets, we do not consider those that are considered “solved”, i.e. where the state of the art performance surpasses human performance, as is the case with SQuAD BIBREF28, BIBREF7. Concretely, we selected gold standards that fit our problem definition and were published in the years 2016 to 2019, have at least $(2019 - publication\\ year) \\times 20$ citations, and bucket them according to the answer selection styles as described in Section SECREF4 We randomly draw one from each bucket and add two randomly drawn datasets from the candidate pool. This leaves us with the datasets described in Table TABREF19. For a more detailed description, we refer to Appendix ."]},{"idx":11,"heading":"Application of the Framework ::: Annotation Task","paragraphs":["We randomly select 50 distinct question, answer and passage triples from the publicly available development sets of the described datasets. Training, development and the (hidden) test set are drawn from the same distribution defined by the data collection method of the respective dataset. For those collections that contain multiple questions over a single passage, we ensure that we are sampling unique paragraphs in order to increase the variety of investigated texts.","The samples were annotated by the first author of this paper, using the proposed schema. In order to validate our findings, we further take 20% of the annotated samples and present them to a second annotator (second author). Since at its core, the annotation is a multi-label task, we report the inter-annotator agreement by computing the (micro-averaged) F1 score, where we treat the first annotator's labels as gold. Table TABREF21 reports the agreement scores, the overall (micro) average F1 score of the annotations is 0.82, which means that on average, more than two thirds of the overall annotated labels were agreed on by both annotators. We deem this satisfactory, given the complexity of the annotation schema."]},{"idx":12,"heading":"Application of the Framework ::: Qualitative Analysis","paragraphs":["We present a concise view of the annotation results in Figure FIGREF23. The full annotation results can be found in Appendix . We centre our discussion around the following main points:"]},{"idx":13,"heading":"Application of the Framework ::: Qualitative Analysis ::: Linguistic Features","paragraphs":["As observed in Figure FIGREF23 the gold standards feature a high degree of Redundancy, peaking at 76% of the annotated HotpotQA samples and synonyms and paraphrases (labelled Synonym), with ReCoRd samples containing 58% of them, likely to be attributed to the elaborating type of discourse of the dataset sources (encyclopedia and newswire). This is, however, not surprising, as it is fairly well understood in the literature that current state-of-the-art models perform well on distinguishing relevant words and phrases from redundant ones BIBREF32. Additionally, the representational capability of synonym relationships of word embeddings has been investigated and is well known BIBREF33. Finally, we observe the presence of syntactic features, such as ambiguous relative clauses, appositions and adverbial phrases, (RelAdvApp 40% in HotpotQA and ReCoRd) and those introducing variance, concretely switching between verbal and nominal styles (e.g. Nominalisation 10% in HotpotQA) and from passive to active voice (Voice, 8% in HotpotQA).","Syntactic features contributing to variety and ambiguity that we did not observe in our samples are the exploitation of verb symmetry, the use of dative and genitive cases or ambiguous prepositions and coordination scope (respectively Symmetry, Dative, Genitive, Prepositions, Scope). Therefore we cannot establish whether models are capable of dealing with those features by evaluating them on those gold standards."]},{"idx":14,"heading":"Application of the Framework ::: Qualitative Analysis ::: Factual Correctness","paragraphs":["We identify three common sources that surface in different problems regarding an answer's factual correctness, as reported in Figure FIGREF23 and illustrate their instantiations in Table TABREF31:","Design Constraints: Choosing the task design and the data collection method introduces some constraints that lead to factually debatable examples. For example, a span might have been arbitrarily selected from multiple spans that potentially answer a question, but only a single continuous answer span per question is allowed by design, as observed in the NewsQA and MsMarco samples (32% and 34% examples annotated as Debatable with 16% and 53% thereof exhibiting arbitrary selection, respectively). Sometimes, when additional passages are added after the annotation step, they can by chance contain passages that answer the question more precisely than the original span, as seen in HotpotQA (16% Debatable samples, 25% of them due to arbitrary selection). In the case of MultiRC it appears to be inconsistent, whether multiple correct answer choices are expected to be correct in isolation or in conjunction (28% Debatable with 29% of them exhibiting this problem). This might provide an explanation to its relatively weak human baseline performance of 84% F1 score BIBREF31.","Weak Quality assurance: When the (typically crowd-sourced) annotations are not appropriately validated, incorrect examples will find their way into the gold standards. This typically results in factually wrong expected answers (i.e. when a more correct answer is present in the context) or a question is expected to be Unanswerable, but is actually answerable from the provided context. The latter is observed in MsMarco (83% of examples annotated as Wrong) and NewsQA, where 60% of the examples annotated as Wrong are Unanswerable with an answer present.","Arbitrary Precision: There appears to be no clear guideline on how precise the answer is expected to be, when the passage expresses the answer in varying granularities. We annotated instances as Debatable when the expected answer was not the most precise given the context (44% and 29% of Debatable instances in NewsQA and MultiRC, respectively)."]},{"idx":15,"heading":"Application of the Framework ::: Qualitative Analysis ::: Semantics-altering grammatical modifiers","paragraphs":["We took interest in whether any of the benchmarks contain what we call distracting lexical features (or distractors): grammatical modifiers that alter the semantics of a sentence for the final task of answering the given question while preserving a similar lexical form. An example of such features are cues for (double) Negation (e.g., “no”, “not”), which when introduced in a sentence, reverse its meaning. Other examples include modifiers denoting Restrictivity, Factivity and Reasoning (such as Monotonicity and Conditional cues). Examples of question-answer pairs containing a distractor are shown in Table FIGREF37.","We posit that the presence of such distractors would allow for evaluating reading comprehension beyond potential simple word matching. However, we observe no presence of such features in the benchmarks (beyond Negation in DROP, ReCoRd and HotpotQA, with 4%, 4% and 2% respectively). This results in gold standards that clearly express the evidence required to obtain the answer, lacking more challenging, i.e., distracting, sentences that can assess whether a model can truly understand meaning."]},{"idx":16,"heading":"Application of the Framework ::: Qualitative Analysis ::: Other","paragraphs":["In the Figure FIGREF23 we observe that Operational and Arithmetic reasoning moderately (6% to 8% combined) appears “in the wild”, i.e. when not enforced by the data design as is the case with HotpotQA (80% Operations combined) or DROP (68% Arithmetic combined). Causal reasoning is (exclusively) present in MultiRC (32%), whereas Temporal and Spatial reasoning requirements seem to not naturally emerge in gold standards. In ReCoRd, a fraction of 38% questions can only be answered By Exclusion of every other candidate, due to the design choice of allowing questions where the required information to answer them is not fully expressed in the accompanying paragraph.","Therefore, it is also a little surprising to observe that ReCoRd requires external resources with regard to knowledge, as seen in Figure FIGREF23. MultiRC requires technical or more precisely basic scientific knowledge (6% Technical/Scientific), as a portion of paragraphs is extracted from elementary school science textbooks BIBREF31. Other benchmarks moderately probe for factual knowledge (0% to 4% across all categories), while Intuitive knowledge is required to derive answers in each gold standard.","It is also worth pointing out, as done in Figure FIGREF23, that although MultiRC and MsMarco are not modelled as a span selection problem, their samples still contain 50% and 66% of answers that are directly taken from the context. DROP contains the biggest fraction of generated answers (60%), due to the requirement of arithmetic operations.","To conclude our analysis, we observe similar distributions of linguistic features and reasoning patterns, except where there are constraints enforced by dataset design, annotation guidelines or source text choice. Furthermore, careful consideration of design choices (such as single-span answers) is required, to avoid impairing the factual correctness of datasets, as pure crowd-worker agreement seems not sufficient in multiple cases."]},{"idx":17,"heading":"Application of the Framework ::: Quantitative Results ::: Lexical overlap","paragraphs":["We used the scores assigned by our proposed set of metrics (discussed in Section SECREF11 Dimensions of Interest: Complexity) to predict the supporting facts in the gold standard samples (that we included in our manual annotation). Concretely, we used the following five features capturing lexical overlap: (i) the number of words occurring in sentence and question, (ii) the length of the longest n-gram shared by sentence and question, whether a (iii) uni- and (iv) bigram from the question is unique to a sentence, and (v) the sentence index, as input to a logistic regression classifier. We optimised on each sample leaving one example for evaluation. We compute the average Precision, Recall and F1 score by means of leave-one-out validation with every sample entry. The averaged results after 5 runs are reported in Table TABREF41.","We observe that even by using only our five features based lexical overlap, the simple logistic regression baseline is able to separate out the supporting facts from the context to a varying degree. This is in line with the lack of semantics-altering grammatical modifiers discussed in the qualitative analysis section above. The classifier performs best on DROP (66% F1) and MultiRC (40% F1), which means that lexical cues can considerably facilitate the search for the answer in those gold standards. On MultiRC, yadav2019quick come to a similar conclusion, by using a more sophisticated approach based on overlap between question, sentence and answer choices.","Surprisingly, the classifier is able to pick up a signal from supporting facts even on data that has been pruned against lexical overlap heuristics by populating the context with additional documents that have high overlap scores with the question. This results in significantly higher scores than when guessing randomly (HotpotQA 26% F1, and MsMarco 11% F1). We observe similar results in the case the length of the question leaves few candidates to compute overlap with $6.3$ and $7.3$ tokens on average for MsMarco and NewsQA (26% F1), compared to $16.9$ tokens on average for the remaining four dataset samples.","Finally, it is worth mentioning that although the queries in ReCoRd are explicitly independent from the passage, the linear classifier is still capable of achieving 34% F1 score in predicting the supporting facts.","However, neural networks perform significantly better than our admittedly crude baseline (e.g. 66% F1 for supporting facts classification on HotpotQA BIBREF14), albeit utilising more training examples, and a richer sentence representation. This facts implies that those neural models are capable of solving more challenging problems than simple “text matching” as performed by the logistic regression baseline. However, they still circumvent actual reading comprehension as the respective gold standards are of limited suitability to evaluate this BIBREF34, BIBREF35. This suggests an exciting future research direction, that is categorising the scale between text matching and reading comprehension more precisely and respectively positioning state-of-the-art models thereon."]},{"idx":18,"heading":"Related Work","paragraphs":["Although not as prominent as the research on novel architecture, there has been steady progress in critically investigating the data and evaluation aspects of NLP and machine learning in general and MRC in particular."]},{"idx":19,"heading":"Related Work ::: Adversarial Evaluation","paragraphs":["The authors of the AddSent algorithm BIBREF11 show that MRC models trained and evaluated on the SQuAD dataset pay too little attention to details that might change the semantics of a sentence, and propose a crowd-sourcing based method to generate adversary examples to exploit that weakness. This method was further adapted to be fully automated BIBREF36 and applied to different gold standards BIBREF35. Our proposed approach differs in that we aim to provide qualitative justifications for those quantitatively measured issues."]},{"idx":20,"heading":"Related Work ::: Sanity Baselines","paragraphs":["Another line of research establishes sane baselines to provide more meaningful context to the raw performance scores of evaluated models. When removing integral parts of the task formulation such as question, the textual passage or parts thereof BIBREF37 or restricting model complexity by design in order to suppress some required form of reasoning BIBREF38, models are still able to perform comparably to the state-of-the-art. This raises concerns about the perceived benchmark complexity and is related to our work in a broader sense as one of our goals is to estimate the complexity of benchmarks."]},{"idx":21,"heading":"Related Work ::: Benchmark evaluation in NLP","paragraphs":["Beyond MRC, efforts similar to ours that pursue the goal of analysing the evaluation of established datasets exist in Natural Language Inference BIBREF13, BIBREF12. Their analyses reveal the existence of biases in training and evaluation data that can be approximated with simple majority-based heuristics. Because of these biases, trained models fail to extract the semantics that are required for the correct inference. Furthermore, a fair share of work was done to reveal gender bias in coreference resolution datasets and models BIBREF39, BIBREF40, BIBREF41."]},{"idx":22,"heading":"Related Work ::: Annotation Taxonomies","paragraphs":["Finally, related to our framework are works that introduce annotation categories for gold standards evaluation. Concretely, we build our annotation framework around linguistic features that were introduced in the GLUE suite BIBREF42 and the reasoning categories introduced in the WorldTree dataset BIBREF19. A qualitative analysis complementary to ours, with focus on the unanswerability patterns in datasets that feature unanswerable questions was done by Yatskar2019."]},{"idx":23,"heading":"Conclusion","paragraphs":["In this paper, we introduce a novel framework to characterise machine reading comprehension gold standards. This framework has potential applications when comparing different gold standards, considering the design choices for a new gold standard and performing qualitative error analyses for a proposed approach.","Furthermore we applied the framework to analyse popular state-of-the-art gold standards for machine reading comprehension: We reveal issues with their factual correctness, show the presence of lexical cues and we observe that semantics-altering grammatical modifiers are missing in all of the investigated gold standards. Studying how to introduce those modifiers into gold standards and observing whether state-of-the-art MRC models are capable of performing reading comprehension on text containing them, is a future research goal.","A future line of research is to extend the framework to be able to identify the different types of exploitable cues such as question or entity typing and concrete overlap patterns. This will allow the framework to serve as an interpretable estimate of reading comprehension complexity of gold standards. Finally, investigating gold standards under this framework where MRC models outperform the human baseline (e.g. SQuAD) will contribute to a deeper understanding of the seemingly superb performance of deep learning approaches on them."]},{"idx":24,"heading":"Figures and Tables","paragraphs":["Figure 1: While initially this looks like a complex question that requires the synthesis of different information across multiple documents, the keyword “2010” appears in the question and only in the sentence that answers it, considerably simplifying the search. Full example with 10 passages can be seen in Appendix D.","Figure 3: The hierarchy of categories in our proposed annotation framework. Abstract higher-level categories are presented in bold while actual annotation features are shown in italics.","Table 1: Summary of selected datasets","Table 2: Inter-Annotator agreement F1 scores, averaged for each dataset","Figure 4: Annotation results","Table 4: (Average) Precision, Recall and F1 score within the 95% confidence interval of a linear classifier optimised on lexical features for the task of predicting supporting facts","Figure 5: Example of semantics altering lexical features","Table 5: Detailed Answer Type results. We calculate percentages relative to the number of examples in the sample.","Table 6: Detailed results for the annotation of factual correctness.","Table 7: Detailed results for the annotation of factual correctness. We calculate percentages relative to the number of examples that were annotated to be not unanswerable.","Table 8: Detailed reasoning results. We calculate percentages relative to the number of examples that are not unanswerable, i.e. require reasoning to obtain the answer according to our definition.","Table 9: Detailed linguistic feature results. We calculate percentages relative to the number of examples that were annotated to contain supporting facts."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"2003.04866","title":"Multi-SimLex: A Large-Scale Evaluation of Multilingual and Cross-Lingual Lexical Semantic Similarity","sections":[{"idx":0,"heading":"Abstract","paragraphs":["We introduce Multi-SimLex, a large-scale lexical resource and evaluation benchmark covering datasets for 12 typologically diverse languages, including major languages (e.g., Mandarin Chinese, Spanish, Russian) as well as less-resourced ones (e.g., Welsh, Kiswahili). Each language dataset is annotated for the lexical relation of semantic similarity and contains 1,888 semantically aligned concept pairs, providing a representative coverage of word classes (nouns, verbs, adjectives, adverbs), frequency ranks, similarity intervals, lexical fields, and concreteness levels. Additionally, owing to the alignment of concepts across languages, we provide a suite of 66 cross-lingual semantic similarity datasets. Due to its extensive size and language coverage, Multi-SimLex provides entirely novel opportunities for experimental evaluation and analysis. On its monolingual and cross-lingual benchmarks, we evaluate and analyze a wide array of recent state-of-the-art monolingual and cross-lingual representation models, including static and contextualized word embeddings (such as fastText, M-BERT and XLM), externally informed lexical representations, as well as fully unsupervised and (weakly) supervised cross-lingual word embeddings. We also present a step-by-step dataset creation protocol for creating consistent, Multi-Simlex-style resources for additional languages. We make these contributions -- the public release of Multi-SimLex datasets, their creation protocol, strong baseline results, and in-depth analyses which can be be helpful in guiding future developments in multilingual lexical semantics and representation learning -- available via a website which will encourage community effort in further expansion of Multi-Simlex to many more languages. Such a large-scale semantic resource could inspire significant further advances in NLP across languages."]},{"idx":1,"heading":"Introduction","paragraphs":["The lack of annotated training and evaluation data for many tasks and domains hinders the development of computational models for the majority of the world's languages BIBREF0, BIBREF1, BIBREF2. The necessity to guide and advance multilingual and cross-lingual NLP through annotation efforts that follow cross-lingually consistent guidelines has been recently recognized by collaborative initiatives such as the Universal Dependency (UD) project BIBREF3. The latest version of UD (as of March 2020) covers more than 70 languages. Crucially, this resource continues to steadily grow and evolve through the contributions of annotators from across the world, extending the UD's reach to a wide array of typologically diverse languages. Besides steering research in multilingual parsing BIBREF4, BIBREF5, BIBREF6 and cross-lingual parser transfer BIBREF7, BIBREF8, BIBREF9, the consistent annotations and guidelines have also enabled a range of insightful comparative studies focused on the languages' syntactic (dis)similarities BIBREF10, BIBREF11, BIBREF12.","Inspired by the UD work and its substantial impact on research in (multilingual) syntax, in this article we introduce Multi-SimLex, a suite of manually and consistently annotated semantic datasets for 12 different languages, focused on the fundamental lexical relation of semantic similarity BIBREF13, BIBREF14. For any pair of words, this relation measures whether their referents share the same (functional) features, as opposed to general cognitive association captured by co-occurrence patterns in texts (i.e., the distributional information). Datasets that quantify the strength of true semantic similarity between concept pairs such as SimLex-999 BIBREF14 or SimVerb-3500 BIBREF15 have been instrumental in improving models for distributional semantics and representation learning. Discerning between semantic similarity and relatedness/association is not only crucial for theoretical studies on lexical semantics (see §SECREF2), but has also been shown to benefit a range of language understanding tasks in NLP. Examples include dialog state tracking BIBREF16, BIBREF17, spoken language understanding BIBREF18, BIBREF19, text simplification BIBREF20, BIBREF21, BIBREF22, dictionary and thesaurus construction BIBREF23, BIBREF24.","Despite the proven usefulness of semantic similarity datasets, they are available only for a small and typologically narrow sample of resource-rich languages such as German, Italian, and Russian BIBREF25, whereas some language types and low-resource languages typically lack similar evaluation data. Even if some resources do exist, they are limited in their size (e.g., 500 pairs in Turkish BIBREF26, 500 in Farsi BIBREF27, or 300 in Finnish BIBREF28) and coverage (e.g., all datasets which originated from the original English SimLex-999 contain only high-frequent concepts, and are dominated by nouns). This is why, as our departure point, we introduce a larger and more comprehensive English word similarity dataset spanning 1,888 concept pairs (see §SECREF4).","Most importantly, semantic similarity datasets in different languages have been created using heterogeneous construction procedures with different guidelines for translation and annotation, as well as different rating scales. For instance, some datasets were obtained by directly translating the English SimLex-999 in its entirety BIBREF25, BIBREF16 or in part BIBREF28. Other datasets were created from scratch BIBREF26 and yet others sampled English concept pairs differently from SimLex-999 and then translated and reannotated them in target languages BIBREF27. This heterogeneity makes these datasets incomparable and precludes systematic cross-linguistic analyses. In this article, consolidating the lessons learned from previous dataset construction paradigms, we propose a carefully designed translation and annotation protocol for developing monolingual Multi-SimLex datasets with aligned concept pairs for typologically diverse languages. We apply this protocol to a set of 12 languages, including a mixture of major languages (e.g., Mandarin, Russian, and French) as well as several low-resource ones (e.g., Kiswahili, Welsh, and Yue Chinese). We demonstrate that our proposed dataset creation procedure yields data with high inter-annotator agreement rates (e.g., the average mean inter-annotator agreement for Welsh is 0.742).","The unified construction protocol and alignment between concept pairs enables a series of quantitative analyses. Preliminary studies on the influence that polysemy and cross-lingual variation in lexical categories (see §SECREF6) have on similarity judgments are provided in §SECREF5. Data created according to Multi-SimLex protocol also allow for probing into whether similarity judgments are universal across languages, or rather depend on linguistic affinity (in terms of linguistic features, phylogeny, and geographical location). We investigate this question in §SECREF25. Naturally, Multi-SimLex datasets can be used as an intrinsic evaluation benchmark to assess the quality of lexical representations based on monolingual, joint multilingual, and transfer learning paradigms. We conduct a systematic evaluation of several state-of-the-art representation models in §SECREF7, showing that there are large gaps between human and system performance in all languages. The proposed construction paradigm also supports the automatic creation of 66 cross-lingual Multi-SimLex datasets by interleaving the monolingual ones. We outline the construction of the cross-lingual datasets in §SECREF6, and then present a quantitative evaluation of a series of cutting-edge cross-lingual representation models on this benchmark in §SECREF8.","Contributions. We now summarize the main contributions of this work:","1) Building on lessons learned from prior work, we create a more comprehensive lexical semantic similarity dataset for the English language spanning a total of 1,888 concept pairs balanced with respect to similarity, frequency, and concreteness, and covering four word classes: nouns, verbs, adjectives and, for the first time, adverbs. This dataset serves as the main source for the creation of equivalent datasets in several other languages.","2) We present a carefully designed and rigorous language-agnostic translation and annotation protocol. These well-defined guidelines will facilitate the development of future Multi-SimLex datasets for other languages. The proposed protocol eliminates some crucial issues with prior efforts focused on the creation of multi-lingual semantic resources, namely: i) limited coverage; ii) heterogeneous annotation guidelines; and iii) concept pairs which are semantically incomparable across different languages.","3) We offer to the community manually annotated evaluation sets of 1,888 concept pairs across 12 typologically diverse languages, and 66 large cross-lingual evaluation sets. To the best of our knowledge, Multi-SimLex is the most comprehensive evaluation resource to date focused on the relation of semantic similarity.","4) We benchmark a wide array of recent state-of-the-art monolingual and cross-lingual word representation models across our sample of languages. The results can serve as strong baselines that lay the foundation for future improvements.","5) We present a first large-scale evaluation study on the ability of encoders pretrained on language modeling (such as bert BIBREF29 and xlm BIBREF30) to reason over word-level semantic similarity in different languages. To our own surprise, the results show that monolingual pretrained encoders, even when presented with word types out of context, are sometimes competitive with static word embedding models such as fastText BIBREF31 or word2vec BIBREF32. The results also reveal a huge gap in performance between massively multilingual pretrained encoders and language-specific encoders in favor of the latter: our findings support other recent empirical evidence related to the “curse of multilinguality” BIBREF33, BIBREF34 in representation learning.","6) We make all of these resources available on a website which facilitates easy creation, submission and sharing of Multi-Simlex-style datasets for a larger number of languages. We hope that this will yield an even larger repository of semantic resources that inspire future advances in NLP within and across languages.","In light of the success of Universal Dependencies BIBREF3, we hope that our initiative will instigate a collaborative public effort with established and clear-cut guidelines that will result in additional Multi-SimLex datasets in a large number of languages in the near future. Moreover, we hope that it will provide means to advance our understanding of distributional and lexical semantics across a large number of languages. All monolingual and cross-lingual Multi-SimLex datasets–along with detailed translation and annotation guidelines–are available online at: https://multisimlex.com/."]},{"idx":2,"heading":"Lexical Semantic Similarity ::: Similarity and Association","paragraphs":["The focus of the Multi-SimLex initiative is on the lexical relation of pure semantic similarity. For any pair of words, this relation measures whether their referents share the same features. For instance, graffiti and frescos are similar to the extent that they are both forms of painting and appear on walls. This relation can be contrasted with the cognitive association between two words, which often depends on how much their referents interact in the real world, or are found in the same situations. For instance, a painter is easily associated with frescos, although they lack any physical commonalities. Association is also known in the literature under other names: relatedness BIBREF13, topical similarity BIBREF35, and domain similarity BIBREF36.","Semantic similarity and association overlap to some degree, but do not coincide BIBREF37, BIBREF38. In fact, there exist plenty of pairs that are intuitively associated but not similar. Pairs where the converse is true can also be encountered, although more rarely. An example are synonyms where a word is common and the other infrequent, such as to seize and to commandeer. BIBREF14 revealed that while similarity measures based on the WordNet graph BIBREF39 and human judgments of association in the University of South Florida Free Association Database BIBREF40 do correlate, a number of pairs follow opposite trends. Several studies on human cognition also point in the same direction. For instance, semantic priming can be triggered by similar words without association BIBREF41. On the other hand, a connection with cue words is established more quickly for topically related words rather than for similar words in free association tasks BIBREF42.","A key property of semantic similarity is its gradience: pairs of words can be similar to a different degree. On the other hand, the relation of synonymy is binary: pairs of words are synonyms if they can be substituted in all contexts (or most contexts, in a looser sense), otherwise they are not. While synonyms can be conceived as lying on one extreme of the semantic similarity continuum, it is crucial to note that their definition is stated in purely relational terms, rather than invoking their referential properties BIBREF43, BIBREF44, BIBREF45. This makes behavioral studies on semantic similarity fundamentally different from lexical resources like WordNet BIBREF46, which include paradigmatic relations (such as synonymy)."]},{"idx":3,"heading":"Lexical Semantic Similarity ::: Similarity for NLP: Intrinsic Evaluation and Semantic Specialization","paragraphs":["The ramifications of the distinction between similarity and association are profound for distributional semantics. This paradigm of lexical semantics is grounded in the distributional hypothesis, formulated by BIBREF47 and BIBREF48. According to this hypothesis, the meaning of a word can be recovered empirically from the contexts in which it occurs within a collection of texts. Since both pairs of topically related words and pairs of purely similar words tend to appear in the same contexts, their associated meaning confounds the two distinct relations BIBREF14, BIBREF49, BIBREF50. As a result, distributional methods obscure a crucial facet of lexical meaning. This limitation also reflects onto word embeddings (WEs), representations of words as low-dimensional vectors that have become indispensable for a wide range of NLP applications BIBREF51, BIBREF52, BIBREF53. In particular, it involves both static WEs learned from co-occurrence patterns BIBREF32, BIBREF54, BIBREF31 and contextualized WEs learned from modeling word sequences BIBREF55, BIBREF29. As a result, in the induced representations, geometrical closeness (measured e.g. through cosine distance) conflates genuine similarity with broad relatedness. For instance, the vectors for antonyms such as sober and drunk, by definition dissimilar, might be neighbors in the semantic space under the distributional hypothesis. BIBREF36, BIBREF56, and BIBREF53 demonstrated that different choices of hyper-parameters in WE algorithms (such as context window) emphasize different relations in the resulting representations. Likewise, BIBREF57 and BIBREF54 discovered that WEs learned from texts annotated with syntactic information mirror similarity better than simple local bag-of-words neighborhoods.","The failure of WEs to capture semantic similarity, in turn, affects model performance in several NLP applications where such knowledge is crucial. In particular, Natural Language Understanding tasks such as statistical dialog modeling, text simplification, or semantic text similarity BIBREF58, BIBREF18, BIBREF59, among others, suffer the most. As a consequence, resources providing clean information on semantic similarity are key in mitigating the side effects of the distributional signal. In particular, such databases can be employed for the intrinsic evaluations of specific WE models as a proxy of their reliability for downstream applications BIBREF60, BIBREF61, BIBREF14; intuitively, the more WEs are misaligned with human judgments of similarity, the more their performance on actual tasks is expected to be degraded. Moreover, word representations can be specialized (a.k.a. retrofitted) by disentangling word relations of similarity and association. In particular, linguistic constraints sourced from external databases (such as synonyms from WordNet) can be injected into WEs BIBREF62, BIBREF63, BIBREF16, BIBREF22, BIBREF64 in order to enforce a particular relation in a distributional semantic space while preserving the original adjacency properties."]},{"idx":4,"heading":"Lexical Semantic Similarity ::: Similarity and Language Variation: Semantic Typology","paragraphs":["In this work, we tackle the concept of (true) semantic similarity from a multilingual perspective. While the same meaning representations may be shared by all human speakers at a deep cognitive level, there is no one-to-one mapping between the words in the lexicons of different languages. This makes the comparison of similarity judgments across languages difficult, since the meaning overlap of translationally equivalent words is sometimes far less than exact. This results from the fact that the way languages `partition' semantic fields is partially arbitrary BIBREF65, although constrained cross-lingually by common cognitive biases BIBREF66. For instance, consider the field of colors: English distinguishes between green and blue, whereas Murle (South Sudan) has a single word for both BIBREF67.","In general, semantic typology studies the variation in lexical semantics across the world's languages. According to BIBREF68, the ways languages categorize concepts into the lexicon follow three main axes: 1) granularity: what is the number of categories in a specific domain?; 2) boundary location: where do the lines marking different categories lie?; 3) grouping and dissection: what are the membership criteria of a category; which instances are considered to be more prototypical? Different choices with respect to these axes lead to different lexicalization patterns. For instance, distinct senses in a polysemous word in English, such as skin (referring to both the body and fruit), may be assigned separate words in other languages such as Italian pelle and buccia, respectively BIBREF70. We later analyze whether similarity scores obtained from native speakers also loosely follow the patterns described by semantic typology."]},{"idx":5,"heading":"Previous Work and Evaluation Data","paragraphs":["Word Pair Datasets. Rich expert-created resources such as WordNet BIBREF46, BIBREF71, VerbNet BIBREF72, BIBREF73, or FrameNet BIBREF74 encode a wealth of semantic and syntactic information, but are expensive and time-consuming to create. The scale of this problem gets multiplied by the number of languages in consideration. Therefore, crowd-sourcing with non-expert annotators has been adopted as a quicker alternative to produce smaller and more focused semantic resources and evaluation benchmarks. This alternative practice has had a profound impact on distributional semantics and representation learning BIBREF14. While some prominent English word pair datasets such as WordSim-353 BIBREF75, MEN BIBREF76, or Stanford Rare Words BIBREF77 did not discriminate between similarity and relatedness, the importance of this distinction was established by BIBREF14 through the creation of SimLex-999. This inspired other similar datasets which focused on different lexical properties. For instance, SimVerb-3500 BIBREF15 provided similarity ratings for 3,500 English verbs, whereas CARD-660 BIBREF78 aimed at measuring the semantic similarity of infrequent concepts.","Semantic Similarity Datasets in Other Languages. Motivated by the impact of datasets such as SimLex-999 and SimVerb-3500 on representation learning in English, a line of related work focused on creating similar resources in other languages. The dominant approach is translating and reannotating the entire original English SimLex-999 dataset, as done previously for German, Italian, and Russian BIBREF25, Hebrew and Croatian BIBREF16, and Polish BIBREF79. Venekoski:2017nodalida apply this process only to a subset of 300 concept pairs from the English SimLex-999. On the other hand, BIBREF27 sampled a new set of 500 English concept pairs to ensure wider topical coverage and balance across similarity spectra, and then translated those pairs to German, Italian, Spanish, and Farsi (SEMEVAL-500). A similar approach was followed by BIBREF26 for Turkish, by BIBREF80 for Mandarin Chinese, and by BIBREF81 for Japanese. BIBREF82 translated the concatenation of SimLex-999, WordSim-353, and the English SEMEVAL-500 into Thai and then reannotated it. Finally, BIBREF83 translated English SimLex-999 and WordSim-353 to 11 resource-rich target languages (German, French, Russian, Italian, Dutch, Chinese, Portuguese, Swedish, Spanish, Arabic, Farsi), but they did not provide details concerning the translation process and the resolution of translation disagreements. More importantly, they also did not reannotate the translated pairs in the target languages. As we discussed in § SECREF6 and reiterate later in §SECREF5, semantic differences among languages can have a profound impact on the annotation scores; particulary, we show in §SECREF25 that these differences even roughly define language clusters based on language affinity.","A core issue with the current datasets concerns a lack of one unified procedure that ensures the comparability of resources in different languages. Further, concept pairs for different languages are sourced from different corpora (e.g., direct translation of the English data versus sampling from scratch in the target language). Moreover, the previous SimLex-based multilingual datasets inherit the main deficiencies of the English original version, such as the focus on nouns and highly frequent concepts. Finally, prior work mostly focused on languages that are widely spoken and do not account for the variety of the world's languages. Our long-term goal is devising a standardized methodology to extend the coverage also to languages that are resource-lean and/or typologically diverse (e.g., Welsh, Kiswahili as in this work).","Multilingual Datasets for Natural Language Understanding. The Multi-SimLex initiative and corresponding datasets are also aligned with the recent efforts on procuring multilingual benchmarks that can help advance computational modeling of natural language understanding across different languages. For instance, pretrained multilingual language models such as multilingual bert BIBREF29 or xlm BIBREF30 are typically probed on XNLI test data BIBREF84 for cross-lingual natural language inference. XNLI was created by translating examples from the English MultiNLI dataset, and projecting its sentence labels BIBREF85. Other recent multilingual datasets target the task of question answering based on reading comprehension: i) MLQA BIBREF86 includes 7 languages ii) XQuAD BIBREF87 10 languages; iii) TyDiQA BIBREF88 9 widely spoken typologically diverse languages. While MLQA and XQuAD result from the translation from an English dataset, TyDiQA was built independently in each language. Another multilingual dataset, PAWS-X BIBREF89, focused on the paraphrase identification task and was created translating the original English PAWS BIBREF90 into 6 languages. We believe that Multi-SimLex can substantially contribute to this endeavor by offering a comprehensive multilingual benchmark for the fundamental lexical level relation of semantic similarity. In future work, Multi-SimLex also offers an opportunity to investigate the correlations between word-level semantic similarity and performance in downstream tasks such as QA and NLI across different languages."]},{"idx":6,"heading":"The Base for Multi-SimLex: Extending English SimLex-999","paragraphs":["In this section, we discuss the design principles behind the English (eng) Multi-SimLex dataset, which is the basis for all the Multi-SimLex datasets in other languages, as detailed in §SECREF5. We first argue that a new, more balanced, and more comprehensive evaluation resource for lexical semantic similarity in English is necessary. We then describe how the 1,888 word pairs contained in the eng Multi-SimLex were selected in such a way as to represent various linguistic phenomena within a single integrated resource.","Construction Criteria. The following criteria have to be satisfied by any high-quality semantic evaluation resource, as argued by previous studies focused on the creation of such resources BIBREF14, BIBREF15, BIBREF91, BIBREF27:","(C1) Representative and diverse. The resource must cover the full range of diverse concepts occurring in natural language, including different word classes (e.g., nouns, verbs, adjectives, adverbs), concrete and abstract concepts, a variety of lexical fields, and different frequency ranges.","(C2) Clearly defined. The resource must provide a clear understanding of which semantic relation exactly is annotated and measured, possibly contrasting it with other relations. For instance, the original SimLex-999 and SimVerb-3500 explicitly focus on true semantic similarity and distinguish it from broader relatedness captured by datasets such as MEN BIBREF76 or WordSim-353 BIBREF75.","(C3) Consistent and reliable. The resource must ensure consistent annotations obtained from non-expert native speakers following simple and precise annotation guidelines.","In choosing the word pairs and constructing eng Multi-SimLex, we adhere to these requirements. Moreover, we follow good practices established by the research on related resources. In particular, since the introduction of the original SimLex-999 dataset BIBREF14, follow-up works have improved its construction protocol across several aspects, including: 1) coverage of more lexical fields, e.g., by relying on a diverse set of Wikipedia categories BIBREF27, 2) infrequent/rare words BIBREF78, 3) focus on particular word classes, e.g., verbs BIBREF15, 4) annotation quality control BIBREF78. Our goal is to make use of these improvements towards a larger, more representative, and more reliable lexical similarity dataset in English and, consequently, in all other languages.","The Final Output: English Multi-SimLex. In order to ensure that the criterion C1 is satisfied, we consolidate and integrate the data already carefully sampled in prior work into a single, comprehensive, and representative dataset. This way, we can control for diversity, frequency, and other properties while avoiding to perform this time-consuming selection process from scratch. Note that, on the other hand, the word pairs chosen for English are scored from scratch as part of the entire Multi-SimLex annotation process, introduced later in §SECREF5. We now describe the external data sources for the final set of word pairs:","1) Source: SimLex-999. BIBREF14. The English Multi-SimLex has been initially conceived as an extension of the original SimLex-999 dataset. Therefore, we include all 999 word pairs from SimLex, which span 666 noun pairs, 222 verb pairs, and 111 adjective pairs. While SimLex-999 already provides examples representing different POS classes, it does not have a sufficient coverage of different linguistic phenomena: for instance, it contains only very frequent concepts, and it does not provide a representative set of verbs BIBREF15.","2) Source: SemEval-17: Task 2 BIBREF92. We start from the full dataset of 500 concept pairs to extract a total of 334 concept pairs for English Multi-SimLex a) which contain only single-word concepts, b) which are not named entities, c) where POS tags of the two concepts are the same, d) where both concepts occur in the top 250K most frequent word types in the English Wikipedia, and e) do not already occur in SimLex-999. The original concepts were sampled as to span all the 34 domains available as part of BabelDomains BIBREF93, which roughly correspond to the main high-level Wikipedia categories. This ensures topical diversity in our sub-sample.","3) Source: CARD-660 BIBREF78. 67 word pairs are taken from this dataset focused on rare word similarity, applying the same selection criteria a) to e) employed for SEMEVAL-500. Words are controlled for frequency based on their occurrence counts from the Google News data and the ukWaC corpus BIBREF94. CARD-660 contains some words that are very rare (logboat), domain-specific (erythroleukemia) and slang (2mrw), which might be difficult to translate and annotate across a wide array of languages. Hence, we opt for retaining only the concept pairs above the threshold of top 250K most frequent Wikipedia concepts, as above.","4) Source: SimVerb-3500 BIBREF15 Since both CARD-660 and SEMEVAL-500 are heavily skewed towards noun pairs, and nouns also dominate the original SimLex-999, we also extract additional verb pairs from the verb-specific similarity dataset SimVerb-3500. We randomly sample 244 verb pairs from SimVerb-3500 that represent all similarity spectra. In particular, we add 61 verb pairs for each of the similarity intervals: $[0,1.5), [1.5,3), [3,4.5), [4.5, 6]$. Since verbs in SimVerb-3500 were originally chosen from VerbNet BIBREF95, BIBREF73, they cover a wide range of verb classes and their related linguistic phenomena.","5) Source: University of South Florida BIBREF96 norms, the largest database of free association for English. In order to improve the representation of different POS classes, we sample additional adjectives and adverbs from the USF norms following the procedure established by BIBREF14, BIBREF15. This yields additional 122 adjective pairs, but only a limited number of adverb pairs (e.g., later – never, now – here, once – twice). Therefore, we also create a set of adverb pairs semi-automatically by sampling adjectives that can be derivationally transformed into adverbs (e.g. adding the suffix -ly) from the USF, and assessing the correctness of such derivation in WordNet. The resulting pairs include, for instance, primarily – mainly, softly – firmly, roughly – reliably, etc. We include a total of 123 adverb pairs into the final English Multi-SimLex. Note that this is the first time adverbs are included into any semantic similarity dataset.","Fulfillment of Construction Criteria. The final eng Multi-SimLex dataset spans 1,051 noun pairs, 469 verb pairs, 245 adjective pairs, and 123 adverb pairs. As mentioned above, the criterion C1 has been fulfilled by relying only on word pairs that already underwent meticulous sampling processes in prior work, integrating them into a single resource. As a consequence, Multi-SimLex allows for fine-grained analyses over different POS classes, concreteness levels, similarity spectra, frequency intervals, relation types, morphology, lexical fields, and it also includes some challenging orthographically similar examples (e.g., infection – inflection). We ensure that the criteria C2 and C3 are satisfied by using similar annotation guidelines as Simlex-999, SimVerb-3500, and SEMEVAL-500 that explicitly target semantic similarity. In what follows, we outline the carefully tailored process of translating and annotating Multi-SimLex datasets in all target languages."]},{"idx":7,"heading":"Multi-SimLex: Translation and Annotation","paragraphs":["We now detail the development of the final Multi-SimLex resource, describing our language selection process, as well as translation and annotation of the resource, including the steps taken to ensure and measure the quality of this resource. We also provide key data statistics and preliminary cross-lingual comparative analyses.","Language Selection. Multi-SimLex comprises eleven languages in addition to English. The main objective for our inclusion criteria has been to balance language prominence (by number of speakers of the language) for maximum impact of the resource, while simultaneously having a diverse suite of languages based on their typological features (such as morphological type and language family). Table TABREF10 summarizes key information about the languages currently included in Multi-SimLex. We have included a mixture of fusional, agglutinative, isolating, and introflexive languages that come from eight different language families. This includes languages that are very widely used such as Chinese Mandarin and Spanish, and low-resource languages such as Welsh and Kiswahili. We hope to further include additional languages and inspire other researchers to contribute to the effort over the lifetime of this project.","The work on data collection can be divided into two crucial phases: 1) a translation phase where the extended English language dataset with 1,888 pairs (described in §SECREF4) is translated into eleven target languages, and 2) an annotation phase where human raters scored each pair in the translated set as well as the English set. Detailed guidelines for both phases are available online at: https://multisimlex.com."]},{"idx":8,"heading":"Multi-SimLex: Translation and Annotation ::: Word Pair Translation","paragraphs":["Translators for each target language were instructed to find direct or approximate translations for the 1,888 word pairs that satisfy the following rules. (1) All pairs in the translated set must be unique (i.e., no duplicate pairs); (2) Translating two words from the same English pair into the same word in the target language is not allowed (e.g., it is not allowed to translate car and automobile to the same Spanish word coche). (3) The translated pairs must preserve the semantic relations between the two words when possible. This means that, when multiple translations are possible, the translation that best conveys the semantic relation between the two words found in the original English pair is selected. (4) If it is not possible to use a single-word translation in the target language, then a multi-word expression (MWE) can be used to convey the nearest possible semantics given the above points (e.g., the English word homework is translated into the Polish MWE praca domowa).","Satisfying the above rules when finding appropriate translations for each pair–while keeping to the spirit of the intended semantic relation in the English version–is not always straightforward. For instance, kinship terminology in Sinitic languages (Mandarin and Yue) uses different terms depending on whether the family member is older or younger, and whether the family member comes from the mother’s side or the father’s side. UTF8gbsn In Mandarin, brother has no direct translation and can be translated as either: 哥哥 (older brother) or 弟弟 (younger brother). Therefore, in such cases, the translators are asked to choose the best option given the semantic context (relation) expressed by the pair in English, otherwise select one of the translations arbitrarily. This is also used to remove duplicate pairs in the translated set, by differentiating the duplicates using a variant at each instance. Further, many translation instances were resolved using near-synonymous terms in the translation. For example, the words in the pair: wood – timber can only be directly translated in Estonian to puit, and are not distinguishable. Therefore, the translators approximated the translation for timber to the compound noun puitmaterjal (literally: wood material) in order to produce a valid pair in the target language. In some cases, a direct transliteration from English is used. For example, the pair: physician and doctor both translate to the same word in Estonian (arst); the less formal word doktor is used as a translation of doctor to generate a valid pair.","We measure the quality of the translated pairs by using a random sample set of 100 pairs (from the 1,888 pairs) to be translated by an independent translator for each target language. The sample is proportionally stratified according to the part-of-speech categories. The independent translator is given identical instructions to the main translator; we then measure the percentage of matched translated words between the two translations of the sample set. Table TABREF12 summarizes the inter-translator agreement results for all languages and by part-of-speech subsets. Overall across all languages, the agreement is 84.8%, which is similar to prior work BIBREF27, BIBREF97."]},{"idx":9,"heading":"Multi-SimLex: Translation and Annotation ::: Guidelines and Word Pair Scoring","paragraphs":["Across all languages, 145 human annotators were asked to score all 1,888 pairs (in their given language). We finally collect at least ten valid annotations for each word pair in each language. All annotators were required to abide by the following instructions:","1. Each annotator must assign an integer score between 0 and 6 (inclusive) indicating how semantically similar the two words in a given pair are. A score of 6 indicates very high similarity (i.e., perfect synonymy), while zero indicates no similarity.","2. Each annotator must score the entire set of 1,888 pairs in the dataset. The pairs must not be shared between different annotators.","3. Annotators are able to break the workload over a period of approximately 2-3 weeks, and are able to use external sources (e.g. dictionaries, thesauri, WordNet) if required.","4. Annotators are kept anonymous, and are not able to communicate with each other during the annotation process.","The selection criteria for the annotators required that all annotators must be native speakers of the target language. Preference to annotators with university education was given, but not required. Annotators were asked to complete a spreadsheet containing the translated pairs of words, as well as the part-of-speech, and a column to enter the score. The annotators did not have access to the original pairs in English.","To ensure the quality of the collected ratings, we have employed an adjudication protocol similar to the one proposed and validated by Pilehvar:2018emnlp. It consists of the following three rounds:","Round 1: All annotators are asked to follow the instructions outlined above, and to rate all 1,888 pairs with integer scores between 0 and 6.","Round 2: We compare the scores of all annotators and identify the pairs for each annotator that have shown the most disagreement. We ask the annotators to reconsider the assigned scores for those pairs only. The annotators may chose to either change or keep the scores. As in the case with Round 1, the annotators have no access to the scores of the other annotators, and the process is anonymous. This process gives a chance for annotators to correct errors or reconsider their judgments, and has been shown to be very effective in reaching consensus, as reported by BIBREF78. We used a very similar procedure as BIBREF78 to identify the pairs with the most disagreement; for each annotator, we marked the $i$th pair if the rated score $s_i$ falls within: $s_i \\ge \\mu _i + 1.5$ or $s_i \\le \\mu _i - 1.5$, where $\\mu _i$ is the mean of the other annotators' scores.","Round 3: We compute the average agreement for each annotator (with the other annotators), by measuring the average Spearman's correlation against all other annotators. We discard the scores of annotators that have shown the least average agreement with all other annotators, while we maintain at least ten annotators per language by the end of this round. The actual process is done in multiple iterations: (S1) we measure the average agreement for each annotator with every other annotator (this corresponds to the APIAA measure, see later); (S2) if we still have more than 10 valid annotators and the lowest average score is higher than in the previous iteration, we remove the lowest one, and rerun S1. Table TABREF14 shows the number of annotators at both the start (Round 1) and end (Round 3) of our process for each language.","We measure the agreement between annotators using two metrics, average pairwise inter-annotator agreement (APIAA), and average mean inter-annotator agreement (AMIAA). Both of these use Spearman's correlation ($\\rho $) between annotators scores, the only difference is how they are averaged. They are computed as follows:","where $\\rho (s_i,s_j)$ is the Spearman's correlation between annotators $i$ and $j$'s scores ($s_i$,$s_j$) for all pairs in the dataset, and $N$ is the number of annotators. APIAA has been used widely as the standard measure for inter-annotator agreement, including in the original SimLex paper BIBREF14. It simply averages the pairwise Spearman's correlation between all annotators. On the other hand, AMIAA compares the average Spearman's correlation of one held-out annotator with the average of all the other $N-1$ annotators, and then averages across all $N$ `held-out' annotators. It smooths individual annotator effects and arguably serves as a better upper bound than APIAA BIBREF15, BIBREF91, BIBREF78.","We present the respective APIAA and AMIAA scores in Table TABREF16 and Table TABREF17 for all part-of-speech subsets, as well as the agreement for the full datasets. As reported in prior work BIBREF15, BIBREF91, AMIAA scores are typically higher than APIAA scores. Crucially, the results indicate `strong agreement' (across all languages) using both measurements. The languages with the highest annotator agreement were French (fra) and Yue Chinese (yue), while Russian (rus) had the lowest overall IAA scores. These scores, however, are still considered to be `moderately strong agreement'."]},{"idx":10,"heading":"Multi-SimLex: Translation and Annotation ::: Data Analysis","paragraphs":["Similarity Score Distributions. Across all languages, the average score (mean $=1.61$, median$=1.1$) is on the lower side of the similarity scale. However, looking closer at the scores of each language in Table TABREF19, we indicate notable differences in both the averages and the spread of scores. Notably, French has the highest average of similarity scores (mean$=2.61$, median$=2.5$), while Kiswahili has the lowest average (mean$=1.28$, median$=0.5$). Russian has the lowest spread ($\\sigma =1.37$), while Polish has the largest ($\\sigma =1.62$). All of the languages are strongly correlated with each other, as shown in Figure FIGREF20, where all of the Spearman's correlation coefficients are greater than 0.6 for all language pairs. Languages that share the same language family are highly correlated (e.g, cmn-yue, rus-pol, est-fin). In addition, we observe high correlations between English and most other languages, as expected. This is due to the effect of using English as the base/anchor language to create the dataset. In simple words, if one translates to two languages $L_1$ and $L_2$ starting from the same set of pairs in English, it is higly likely that $L_1$ and $L_2$ will diverge from English in different ways. Therefore, the similarity between $L_1$-eng and $L_2$-eng is expected to be higher than between $L_1$-$L_2$, especially if $L_1$ and $L_2$ are typologically dissimilar languages (e.g., heb-cmn, see Figure FIGREF20). This phenomenon is well documented in related prior work BIBREF25, BIBREF27, BIBREF16, BIBREF97. While we acknowledge this as a slight artifact of the dataset design, it would otherwise be impossible to construct a semantically aligned and comprehensive dataset across a large number of languages.","We also report differences in the distribution of the frequency of words among the languages in Multi-SimLex. Figure FIGREF22 shows six example languages, where each bar segment shows the proportion of words in each language that occur in the given frequency range. For example, the 10K-20K segment of the bars represents the proportion of words in the dataset that occur in the list of most frequent words between the frequency rank of 10,000 and 20,000 in that language; likewise with other intervals. Frequency lists for the presented languages are derived from Wikipedia and Common Crawl corpora. While many concept pairs are direct or approximate translations of English pairs, we can see that the frequency distribution does vary across different languages, and is also related to inherent language properties. For instance, in Finnish and Russian, while we use infinitive forms of all verbs, conjugated verb inflections are often more frequent in raw corpora than the corresponding infinitive forms. The variance can also be partially explained by the difference in monolingual corpora size used to derive the frequency rankings in the first place: absolute vocabulary sizes are expected to fluctuate across different languages. However, it is also important to note that the datasets also contain subsets of lower-frequency and rare words, which can be used for rare word evaluations in multiple languages, in the spirit of Pilehvar:2018emnlp's English rare word dataset.","Cross-Linguistic Differences. Table TABREF23 shows some examples of average similarity scores of English, Spanish, Kiswahili and Welsh concept pairs. Remember that the scores range from 0 to 6: the higher the score, the more similar the participants found the concepts in the pair. The examples from Table TABREF23 show evidence of both the stability of average similarity scores across languages (unlikely – friendly, book – literature, and vanish – disappear), as well as language-specific differences (care – caution). Some differences in similarity scores seem to group languages into clusters. For example, the word pair regular – average has an average similarity score of 4.0 and 4.1 in English and Spanish, respectively, whereas in Kiswahili and Welsh the average similarity score of this pair is 0.5 and 0.8. We analyze this phenomenon in more detail in §SECREF25.","There are also examples for each of the four languages having a notably higher or lower similarity score for the same concept pair than the three other languages. For example, large – big in English has an average similarity score of 5.9, whereas Spanish, Kiswahili and Welsh speakers rate the closest concept pair in their native language to have a similarity score between 2.7 and 3.8. What is more, woman – wife receives an average similarity of 0.9 in English, 2.9 in Spanish, and greater than 4.0 in Kiswahili and Welsh. The examples from Spanish include banco – asiento (bank – seat) which receives an average similarity score 5.1, while in the other three languages the similarity score for this word pair does not exceed 0.1. At the same time, the average similarity score of espantosamente – fantásticamente (amazingly – fantastically) is much lower in Spanish (0.4) than in other languages (4.1 – 5.1). In Kiswahili, an example of a word pair with a higher similarity score than the rest would be machweo – jioni (sunset – evening), having an average score of 5.5, while the other languages receive 2.8 or less, and a notably lower similarity score is given to wa ajabu - mkubwa sana (wonderful – terrific), getting 0.9, while the other languages receive 5.3 or more. Welsh examples include yn llwyr - yn gyfan gwbl (purely – completely), which scores 5.4 among Welsh speakers but 2.3 or less in other languages, while addo – tyngu (promise – swear) is rated as 0 by all Welsh annotators, but in the other three languages 4.3 or more on average.","There can be several explanations for the differences in similarity scores across languages, including but not limited to cultural context, polysemy, metonymy, translation, regional and generational differences, and most commonly, the fact that words and meanings do not exactly map onto each other across languages. For example, it is likely that the other three languages do not have two separate words for describing the concepts in the concept pair: big – large, and the translators had to opt for similar lexical items that were more distant in meaning, explaining why in English the concept pair received a much higher average similarity score than in other languages. A similar issue related to the mapping problem across languages arose in the Welsh concept pair yn llwye – yn gyfan gwbl, where Welsh speakers agreed that the two concepts are very similar. When asked, bilingual speakers considered the two Welsh concepts more similar than English equivalents purely – completely, potentially explaining why a higher average similarity score was reached in Welsh. The example of woman – wife can illustrate cultural differences or another translation-related issue where the word `wife' did not exist in some languages (for example, Estonian), and therefore had to be described using other words, affecting the comparability of the similarity scores. This was also the case with the football – soccer concept pair. The pair bank – seat demonstrates the effect of the polysemy mismatch across languages: while `bank' has two different meanings in English, neither of them is similar to the word `seat', but in Spanish, `banco' can mean `bank', but it can also mean `bench'. Quite naturally, Spanish speakers gave the pair banco – asiento a higher similarity score than the speakers of languages where this polysemy did not occur.","An example of metonymy affecting the average similarity score can be seen in the Kiswahili version of the word pair: sunset – evening (machweo – jioni). The average similarity score for this pair is much higher in Kiswahili, likely because the word `sunset' can act as a metonym of `evening'. The low similarity score of wonderful – terrific in Kiswahili (wa ajabu - mkubwa sana) can be explained by the fact that while `mkubwa sana' can be used as `terrific' in Kiswahili, it technically means `very big', adding to the examples of translation- and mapping-related effects. The word pair amazingly – fantastically (espantosamente – fantásticamente) brings out another translation-related problem: the accuracy of the translation. While `espantosamente' could arguably be translated to `amazingly', more common meanings include: `frightfully', `terrifyingly', and `shockingly', explaining why the average similarity score differs from the rest of the languages. Another problem was brought out by addo – tyngu (promise – swear) in Welsh, where the `tyngu' may not have been a commonly used or even a known word choice for annotators, pointing out potential regional or generational differences in language use.","Table TABREF24 presents examples of concept pairs from English, Spanish, Kiswahili, and Welsh on which the participants agreed the most. For example, in English all participants rated the similarity of trial – test to be 4 or 5. In Spanish and Welsh, all participants rated start – begin to correspond to a score of 5 or 6. In Kiswahili, money – cash received a similarity rating of 6 from every participant. While there are numerous examples of concept pairs in these languages where the participants agreed on a similarity score of 4 or higher, it is worth noting that none of these languages had a single pair where all participants agreed on either 1-2, 2-3, or 3-4 similarity rating. Interestingly, in English all pairs where all the participants agreed on a 5-6 similarity score were adjectives."]},{"idx":11,"heading":"Multi-SimLex: Translation and Annotation ::: Effect of Language Affinity on Similarity Scores","paragraphs":["Based on the analysis in Figure FIGREF20 and inspecting the anecdotal examples in the previous section, it is evident that the correlation between similarity scores across languages is not random. To corroborate this intuition, we visualize the vectors of similarity scores for each single language by reducing their dimensionality to 2 via Principal Component Analysis BIBREF98. The resulting scatter plot in Figure FIGREF26 reveals that languages from the same family or branch have similar patterns in the scores. In particular, Russian and Polish (both Slavic), Finnish and Estonian (both Uralic), Cantonese and Mandarin Chinese (both Sinitic), and Spanish and French (both Romance) are all neighbors.","In order to quantify exactly the effect of language affinity on the similarity scores, we run correlation analyses between these and language features. In particular, we extract feature vectors from URIEL BIBREF99, a massively multilingual typological database that collects and normalizes information compiled by grammarians and field linguists about the world's languages. In particular, we focus on information about geography (the areas where the language speakers are concentrated), family (the phylogenetic tree each language belongs to), and typology (including syntax, phonological inventory, and phonology). Moreover, we consider typological representations of languages that are not manually crafted by experts, but rather learned from texts. BIBREF100 proposed to construct such representations by training language-identifying vectors end-to-end as part of neural machine translation models.","The vector for similarity judgments and the vector of linguistic features for a given language have different dimensionality. Hence, we first construct a distance matrix for each vector space, such that both columns and rows are language indices, and each cell value is the cosine distance between the vectors of the corresponding language pair. Given a set of L languages, each resulting matrix $S$ has dimensionality of $\\mathbb {R}^{|L| \\times |L|}$ and is symmetrical. To estimate the correlation between the matrix for similarity judgments and each of the matrices for linguistic features, we run a Mantel test BIBREF101, a non-parametric statistical test based on matrix permutations that takes into account inter-dependencies among pairwise distances.","The results of the Mantel test reported in Table FIGREF26 show that there exist statistically significant correlations between similarity judgments and geography, family, and syntax, given that $p < 0.05$ and $z > 1.96$. The correlation coefficient is particularly strong for geography ($r = 0.647$) and syntax ($r = 0.649$). The former result is intuitive, because languages in contact easily borrow and loan lexical units, and cultural interactions may result in similar cognitive categorizations. The result for syntax, instead, cannot be explained so easily, as formal properties of language do not affect lexical semantics. Instead, we conjecture that, while no causal relation is present, both syntactic features and similarity judgments might be linked to a common explanatory variable (such as geography). In fact, several syntactic properties are not uniformly spread across the globe. For instance, verbs with Verb–Object–Subject word order are mostly concentrated in Oceania BIBREF102. In turn, geographical proximity leads to similar judgment patterns, as mentioned above. On the other hand, we find no correlation with phonology and inventory, as expected, nor with the bottom-up typological features from BIBREF100."]},{"idx":12,"heading":"Cross-Lingual Multi-SimLex Datasets","paragraphs":["A crucial advantage of having semantically aligned monolingual datasets across different languages is the potential to create cross-lingual semantic similarity datasets. Such datasets allow for probing the quality of cross-lingual representation learning algorithms BIBREF27, BIBREF103, BIBREF104, BIBREF105, BIBREF106, BIBREF30, BIBREF107 as an intrinsic evaluation task. However, the cross-lingual datasets previous work relied upon BIBREF27 were limited to a homogeneous set of high-resource languages (e.g., English, German, Italian, Spanish) and a small number of concept pairs (all less than 1K pairs). We address both problems by 1) using a typologically more diverse language sample, and 2) relying on a substantially larger English dataset as a source for the cross-lingual datasets: 1,888 pairs in this work versus 500 pairs in the work of Camacho:2017semeval. As a result, each of our cross-lingual datasets contains a substantially larger number of concept pairs, as shown in Table TABREF30. The cross-lingual Multi-Simlex datasets are constructed automatically, leveraging word pair translations and annotations collected in all 12 languages. This yields a total of 66 cross-lingual datasets, one for each possible combination of languages. Table TABREF30 provides the final number of concept pairs, which lie between 2,031 and 3,480 pairs for each cross-lingual dataset, whereas Table TABREF29 shows some sample pairs with their corresponding similarity scores.","The automatic creation and verification of cross-lingual datasets closely follows the procedure first outlined by Camacho:2015acl and later adopted by Camacho:2017semeval (for semantic similarity) and Vulic:2019acl (for graded lexical entailment). First, given two languages, we intersect their aligned concept pairs obtained through translation. For instance, starting from the aligned pairs attroupement – foule in French and rahvasumm – rahvahulk in Estonian, we construct two cross-lingual pairs attroupement – rahvaluk and rahvasumm – foule. The scores of cross-lingual pairs are then computed as averages of the two corresponding monolingual scores. Finally, in order to filter out concept pairs whose semantic meaning was not preserved during this operation, we retain only cross-lingual pairs for which the corresponding monolingual scores $(s_s, s_t)$ differ at most by one fifth of the full scale (i.e., $\\mid s_s - s_t \\mid \\le 1.2$). This heuristic mitigates the noise due to cross-lingual semantic shifts BIBREF27, BIBREF97. We refer the reader to the work of Camacho:2015acl for a detailed technical description of the procedure. UTF8gbsn","To assess the quality of the resulting cross-lingual datasets, we have conducted a verification experiment similar to Vulic:2019acl. We randomly sampled 300 concept pairs in the English-Spanish, English-French, and English-Mandarin cross-lingual datasets. Subsequently, we asked bilingual native speakers to provide similarity judgments of each pair. The Spearman's correlation score $\\rho $ between automatically induced and manually collected ratings achieves $\\rho \\ge 0.90$ on all samples, which confirms the viability of the automatic construction procedure.","Score and Class Distributions. The summary of score and class distributions across all 66 cross-lingual datasets are provided in Figure FIGREF31 and Figure FIGREF31, respectively. First, it is obvious that the distribution over the four POS classes largely adheres to that of the original monolingual Multi-SimLex datasets, and that the variance is quite low: e.g., the eng-fra dataset contains the lowest proportion of nouns (49.21%) and the highest proportion of verbs (27.1%), adjectives (15.28%), and adverbs (8.41%). On the other hand, the distribution over similarity intervals in Figure FIGREF31 shows a much greater variance. This is again expected as this pattern resurfaces in monolingual datasets (see Table TABREF19). It is also evident that the data are skewed towards lower-similarity concept pairs. However, due to the joint size of all cross-lingual datasets (see Table TABREF30), even the least represented intervals contain a substantial number of concept pairs. For instance, the rus-yue dataset contains the least highly similar concept pairs (in the interval $[4,6]$) of all 66 cross-lingual datasets. Nonetheless, the absolute number of pairs (138) in that interval for rus-yue is still substantial. If needed, this makes it possible to create smaller datasets which are balanced across the similarity spectra through sub-sampling."]},{"idx":13,"heading":"Monolingual Evaluation of Representation Learning Models","paragraphs":["After the numerical and qualitative analyses of the Multi-SimLex datasets provided in §§ SECREF18–SECREF25, we now benchmark a series of representation learning models on the new evaluation data. We evaluate standard static word embedding algorithms such as fastText BIBREF31, as well as a range of more recent text encoders pretrained on language modeling such as multilingual BERT BIBREF29. These experiments provide strong baseline scores on the new Multi-SimLex datasets and offer a first large-scale analysis of pretrained encoders on word-level semantic similarity across diverse languages. In addition, the experiments now enabled by Multi-SimLex aim to answer several important questions. (Q1) Is it viable to extract high-quality word-level representations from pretrained encoders receiving subword-level tokens as input? Are such representations competitive with standard static word-level embeddings? (Q2) What are the implications of monolingual pretraining versus (massively) multilingual pretraining for performance? (Q3) Do lightweight unsupervised post-processing techniques improve word representations consistently across different languages? (Q4) Can we effectively transfer available external lexical knowledge from resource-rich languages to resource-lean languages in order to learn word representations that distinguish between true similarity and conceptual relatedness (see the discussion in §SECREF6)?"]},{"idx":14,"heading":"Monolingual Evaluation of Representation Learning Models ::: Models in Comparison","paragraphs":["Static Word Embeddings in Different Languages. First, we evaluate a standard method for inducing non-contextualized (i.e., static) word embeddings across a plethora of different languages: fastText (ft) vectors BIBREF31 are currently the most popular and robust choice given 1) the availability of pretrained vectors in a large number of languages BIBREF108 trained on large Common Crawl (CC) plus Wikipedia (Wiki) data, and 2) their superior performance across a range of NLP tasks BIBREF109. In fact, fastText is an extension of the standard word-level CBOW and skip-gram word2vec models BIBREF32 that takes into account subword-level information, i.e. the contituent character n-grams of each word BIBREF110. For this reason, fastText is also more suited for modeling rare words and morphologically rich languages.","We rely on 300-dimensional ft word vectors trained on CC+Wiki and available online for 157 languages. The word vectors for all languages are obtained by CBOW with position-weights, with character n-grams of length 5, a window of size 5, 10 negative examples, and 10 training epochs. We also probe another (older) collection of ft vectors, pretrained on full Wikipedia dumps of each language.. The vectors are 300-dimensional, trained with the skip-gram objective for 5 epochs, with 5 negative examples, a window size set to 5, and relying on all character n-grams from length 3 to 6. Following prior work, we trim the vocabularies for all languages to the 200K most frequent words and compute representations for multi-word expressions by averaging the vectors of their constituent words.","Unsupervised Post-Processing. Further, we consider a variety of unsupervised post-processing steps that can be applied post-training on top of any pretrained input word embedding space without any external lexical semantic resource. So far, the usefulness of such methods has been verified only on the English language through benchmarks for lexical semantics and sentence-level tasks BIBREF113. In this paper, we assess if unsupervised post-processing is beneficial also in other languages. To this end, we apply the following post-hoc transformations on the initial word embeddings:","1) Mean centering (mc) is applied after unit length normalization to ensure that all vectors have a zero mean, and is commonly applied in data mining and analysis BIBREF114, BIBREF115.","2) All-but-the-top (abtt) BIBREF113, BIBREF116 eliminates the common mean vector and a few top dominating directions (according to PCA) from the input distributional word vectors, since they do not contribute towards distinguishing the actual semantic meaning of different words. The method contains a single (tunable) hyper-parameter $dd_{A}$ which denotes the number of the dominating directions to remove from the initial representations. Previous work has verified the usefulness of abtt in several English lexical semantic tasks such as semantic similarity, word analogies, and concept categorization, as well as in sentence-level text classification tasks BIBREF113.","3) uncovec BIBREF117 adjusts the similarity order of an arbitrary input word embedding space, and can emphasize either syntactic or semantic information in the transformed vectors. In short, it transforms the input space $\\mathbf {X}$ into an adjusted space $\\mathbf {X}\\mathbf {W}_{\\alpha }$ through a linear map $\\mathbf {W}_{\\alpha }$ controlled by a single hyper-parameter $\\alpha $. The $n^{\\text{th}}$-order similarity transformation of the input word vector space $\\mathbf {X}$ (for which $n=1$) can be obtained as $\\mathbf {M}_{n}(\\mathbf {X}) = \\mathbf {M}_1(\\mathbf {X}\\mathbf {W}_{(n-1)/2})$, with $\\mathbf {W}_{\\alpha }=\\mathbf {Q}\\mathbf {\\Gamma }^{\\alpha }$, where $\\mathbf {Q}$ and $\\mathbf {\\Gamma }$ are the matrices obtained via eigendecomposition of $\\mathbf {X}^T\\mathbf {X}=\\mathbf {Q}\\mathbf {\\Gamma }\\mathbf {Q}^T$. $\\mathbf {\\Gamma }$ is a diagonal matrix containing eigenvalues of $\\mathbf {X}^T\\mathbf {X}$; $\\mathbf {Q}$ is an orthogonal matrix with eigenvectors of $\\mathbf {X}^T\\mathbf {X}$ as columns. While the motivation for the uncovec methods does originate from adjusting discrete similarity orders, note that $\\alpha $ is in fact a continuous real-valued hyper-parameter which can be carefully tuned. For more technical details we refer the reader to the original work of BIBREF117.","As mentioned, all post-processing methods can be seen as unsupervised retrofitting methods that, given an arbitrary input vector space $\\mathbf {X}$, produce a perturbed/transformed output vector space $\\mathbf {X}^{\\prime }$, but unlike common retrofitting methods BIBREF62, BIBREF16, the perturbation is completely unsupervised (i.e., self-contained) and does not inject any external (semantic similarity oriented) knowledge into the vector space. Note that different perturbations can also be stacked: e.g., we can apply uncovec and then use abtt on top the output uncovec vectors. When using uncovec and abtt we always length-normalize and mean-center the data first (i.e., we apply the simple mc normalization). Finally, we tune the two hyper-parameters $d_A$ (for abtt) and $\\alpha $ (uncovec) on the English Multi-SimLex and use the same values on the datasets of all other languages; we report results with $dd_A = 3$ or $dd_A = 10$, and $\\alpha =-0.3$.","Contextualized Word Embeddings. We also evaluate the capacity of unsupervised pretraining architectures based on language modeling objectives to reason over lexical semantic similarity. To the best of our knowledge, our article is the first study performing such analyses. State-of-the-art models such as bert BIBREF29, xlm BIBREF30, or roberta BIBREF118 are typically very deep neural networks based on the Transformer architecture BIBREF119. They receive subword-level tokens as inputs (such as WordPieces BIBREF120) to tackle data sparsity. In output, they return contextualized embeddings, dynamic representations for words in context.","To represent words or multi-word expressions through a pretrained model, we follow prior work BIBREF121 and compute an input item's representation by 1) feeding it to a pretrained model in isolation; then 2) averaging the $H$ last hidden representations for each of the item’s constituent subwords; and then finally 3) averaging the resulting subword representations to produce the final $d$-dimensional representation, where $d$ is the embedding and hidden-layer dimensionality (e.g., $d=768$ with bert). We opt for this approach due to its proven viability and simplicity BIBREF121, as it does not require any additional corpora to condition the induction of contextualized embeddings. Other ways to extract the representations from pretrained models BIBREF122, BIBREF123, BIBREF124 are beyond the scope of this work, and we will experiment with them in the future.","In other words, we treat each pretrained encoder enc as a black-box function to encode a single word or a multi-word expression $x$ in each language into a $d$-dimensional contextualized representation $\\mathbf {x}_{\\textsc {enc}} \\in \\mathbb {R}^d = \\textsc {enc}(x)$ (e.g., $d=768$ with bert). As multilingual pretrained encoders, we experiment with the multilingual bert model (m-bert) BIBREF29 and xlm BIBREF30. m-bert is pretrained on monolingual Wikipedia corpora of 102 languages (comprising all Multi-SimLex languages) with a 12-layer Transformer network, and yields 768-dimensional representations. Since the concept pairs in Multi-SimLex are lowercased, we use the uncased version of m-bert. m-bert comprises all Multi-SimLex languages, and its evident ability to perform cross-lingual transfer BIBREF12, BIBREF125, BIBREF126 also makes it a convenient baseline model for cross-lingual experiments later in §SECREF8. The second multilingual model we consider, xlm-100, is pretrained on Wikipedia dumps of 100 languages, and encodes each concept into a $1,280$-dimensional representation. In contrast to m-bert, xlm-100 drops the next-sentence prediction objective and adds a cross-lingual masked language modeling objective. For both encoders, the representations of each concept are computed as averages over the last $H=4$ hidden layers in all experiments, as suggested by Wu:2019arxiv.","Besides m-bert and xlm, covering multiple languages, we also analyze the performance of “language-specific” bert and xlm models for the languages where they are available: Finnish, Spanish, English, Mandarin Chinese, and French. The main goal of this comparison is to study the differences in performance between multilingual “one-size-fits-all” encoders and language-specific encoders. For all experiments, we rely on the pretrained models released in the Transformers repository BIBREF127.","Unsupervised post-processing steps devised for static word embeddings (i.e., mean-centering, abtt, uncovec) can also be applied on top of contextualized embeddings if we predefine a vocabulary of word types $V$ that will be represented in a word vector space $\\mathbf {X}$. We construct such $V$ for each language as the intersection of word types covered by the corresponding CC+Wiki fastText vectors and the (single-word or multi-word) expressions appearing in the corresponding Multi-SimLex dataset.","Finally, note that it is not feasible to evaluate a full range of available pretrained encoders within the scope of this work. Our main intention is to provide the first set of baseline results on Multi-SimLex by benchmarking a sample of most popular encoders, at the same time also investigating other important questions such as performance of static versus contextualized word embeddings, or multilingual versus language-specific pretraining. Another purpose of the experiments is to outline the wide potential and applicability of the Multi-SimLex datasets for multilingual and cross-lingual representation learning evaluation."]},{"idx":15,"heading":"Monolingual Evaluation of Representation Learning Models ::: Results and Discussion","paragraphs":["The results we report are Spearman's $\\rho $ coefficients of the correlation between the ranks derived from the scores of the evaluated models and the human scores provided in each Multi-SimLex dataset. The main results with static and contextualized word vectors for all test languages are summarized in Table TABREF43. The scores reveal several interesting patterns, and also pinpoint the main challenges for future work.","State-of-the-Art Representation Models. The absolute scores of CC+Wiki ft, Wiki ft, and m-bert are not directly comparable, because these models have different coverage. In particular, Multi-SimLex contains some out-of-vocabulary (OOV) words whose static ft embeddings are not available. On the other hand, m-bert has perfect coverage. A general comparison between CC+Wiki and Wiki ft vectors, however, supports the intuition that larger corpora (such as CC+Wiki) yield higher correlations. Another finding is that a single massively multilingual model such as m-bert cannot produce semantically rich word-level representations. Whether this actually happens because the training objective is different—or because the need to represent 100+ languages reduces its language-specific capacity—is investigated further below.","The overall results also clearly indicate that (i) there are differences in performance across different monolingual Multi-SimLex datasets, and (ii) unsupervised post-processing is universally useful, and can lead to huge improvements in correlation scores for many languages. In what follows, we also delve deeper into these analyses.","Impact of Unsupervised Post-Processing. First, the results in Table TABREF43 suggest that applying dimension-wise mean centering to the initial vector spaces has positive impact on word similarity scores in all test languages and for all models, both static and contextualized (see the +mc rows in Table TABREF43). BIBREF128 show that distributional word vectors have a tendency towards narrow clusters in the vector space (i.e., they occupy a narrow cone in the vector space and are therefore anisotropic BIBREF113, BIBREF129), and are prone to the undesired effect of hubness BIBREF130, BIBREF131. Applying dimension-wise mean centering has the effect of spreading the vectors across the hyper-plane and mitigating the hubness issue, which consequently improves word-level similarity, as it emerges from the reported results. Previous work has already validated the importance of mean centering for clustering-based tasks BIBREF132, bilingual lexicon induction with cross-lingual word embeddings BIBREF133, BIBREF134, BIBREF135, and for modeling lexical semantic change BIBREF136. However, to the best of our knowledge, the results summarized in Table TABREF43 are the first evidence that also confirms its importance for semantic similarity in a wide array of languages. In sum, as a general rule of thumb, we suggest to always mean-center representations for semantic tasks.","The results further indicate that additional post-processing methods such as abtt and uncovec on top of mean-centered vector spaces can lead to further gains in most languages. The gains are even visible for languages which start from high correlation scores: for instance., cmn with CC+Wiki ft increases from 0.534 to 0.583, from 0.315 to 0.526 with Wiki ft, and from 0.408 to 0.487 with m-bert. Similarly, for rus with CC+Wiki ft we can improve from 0.422 to 0.500, and for fra the scores improve from 0.578 to 0.613. There are additional similar cases reported in Table TABREF43.","Overall, the unsupervised post-processing techniques seem universally useful across languages, but their efficacy and relative performance does vary across different languages. Note that we have not carefully fine-tuned the hyper-parameters of the evaluated post-processing methods, so additional small improvements can be expected for some languages. The main finding, however, is that these post-processing techniques are robust to semantic similarity computations beyond English, and are truly language independent. For instance, removing dominant latent (PCA-based) components from word vectors emphasizes semantic differences between different concepts, as only shared non-informative latent semantic knowledge is removed from the representations.","In summary, pretrained word embeddings do contain more information pertaining to semantic similarity than revealed in the initial vectors. This way, we have corroborated the hypotheses from prior work BIBREF113, BIBREF117 which were not previously empirically verified on other languages due to a shortage of evaluation data; this gap has now been filled with the introduction of the Multi-SimLex datasets. In all follow-up experiments, we always explicitly denote which post-processing configuration is used in evaluation.","POS-Specific Subsets. We present the results for subsets of word pairs grouped by POS class in Table TABREF46. Prior work based on English data showed that representations for nouns are typically of higher quality than those for the other POS classes BIBREF49, BIBREF137, BIBREF50. We observe a similar trend in other languages as well. This pattern is consistent across different representation models and can be attributed to several reasons. First, verb representations need to express a rich range of syntactic and semantic behaviors rather than purely referential features BIBREF138, BIBREF139, BIBREF73. Second, low correlation scores on the adjective and adverb subsets in some languages (e.g., pol, cym, swa) might be due to their low frequency in monolingual texts, which yields unreliable representations. In general, the variance in performance across different word classes warrants further research in class-specific representation learning BIBREF140, BIBREF50. The scores further attest the usefulness of unsupervised post-processing as almost all class-specific correlation scores are improved by applying mean-centering and abtt. Finally, the results for m-bert and xlm-100 in Table TABREF46 further confirm that massively multilingual pretraining cannot yield reasonable semantic representations for many languages: in fact, for some classes they display no correlation with human ratings at all.","Differences across Languages. Naturally, the results from Tables TABREF43 and TABREF46 also reveal that there is variation in performance of both static word embeddings and pretrained encoders across different languages. Among other causes, the lowest absolute scores with ft are reported for languages with least resources available to train monolingual word embeddings, such as Kiswahili, Welsh, and Estonian. The low performance on Welsh is especially indicative: Figure FIGREF20 shows that the ratings in the Welsh dataset match up very well with the English ratings, but we cannot achieve the same level of correlation in Welsh with Welsh ft word embeddings. Difference in performance between two closely related languages, est (low-resource) and fin (high-resource), provides additional evidence in this respect.","The highest reported scores with m-bert and xlm-100 are obtained for Mandarin Chinese and Yue Chinese: this effectively points to the weaknesses of massively multilingual training with a joint subword vocabulary spanning 102 and 100 languages. Due to the difference in scripts, “language-specific” subwords for yue and cmn do not need to be shared across a vast amount of languages and the quality of their representation remains unscathed. This effectively means that m-bert's subword vocabulary contains plenty of cmn-specific and yue-specific subwords which are exploited by the encoder when producing m-bert-based representations. Simultaneously, higher scores with m-bert (and xlm in Table TABREF46) are reported for resource-rich languages such as French, Spanish, and English, which are better represented in m-bert's training data. We also observe lower absolute scores (and a larger number of OOVs) for languages with very rich and productive morphological systems such as the two Slavic languages (Polish and Russian) and Finnish. Since Polish and Russian are known to have large Wikipedias and Common Crawl data BIBREF33 (e.g., their Wikipedias are in the top 10 largest Wikipedias worldwide), the problem with coverage can be attributed exactly to the proliferation of morphological forms in those languages.","Finally, while Table TABREF43 does reveal that unsupervised post-processing is useful for all languages, it also demonstrates that peak scores are achieved with different post-processing configurations. This finding suggests that a more careful language-specific fine-tuning is indeed needed to refine word embeddings towards semantic similarity. We plan to inspect the relationship between post-processing techniques and linguistic properties in more depth in future work.","Multilingual vs. Language-Specific Contextualized Embeddings. Recent work has shown that—despite the usefulness of massively multilingual models such as m-bert and xlm-100 for zero-shot cross-lingual transfer BIBREF12, BIBREF125—stronger results in downstream tasks for a particular language can be achieved by pretraining language-specific models on language-specific data. In this experiment, motivated by the low results of m-bert and xlm-100 (see again Table TABREF46), we assess if monolingual pretrained encoders can produce higher-quality word-level representations than multilingual models. Therefore, we evaluate language-specific bert and xlm models for a subset of the Multi-SimLex languages for which such models are currently available: Finnish BIBREF141 (bert-base architecture, uncased), French BIBREF142 (the FlauBERT model based on xlm), English (bert-base, uncased), Mandarin Chinese (bert-base) BIBREF29 and Spanish (bert-base, uncased). In addition, we also evaluate a series of pretrained encoders available for English: (i) bert-base, bert-large, and bert-large with whole word masking (wwm) from the original work on BERT BIBREF29, (ii) monolingual “English-specific” xlm BIBREF30, and (iii) two models which employ parameter reduction techniques to build more compact encoders: albert-b uses a configuration similar to bert-base, while albert-l is similar to bert-large, but with an $18\\times $ reduction in the number of parameters BIBREF143.","From the results in Table FIGREF49, it is clear that monolingual pretrained encoders yield much more reliable word-level representations. The gains are visible even for languages such as cmn which showed reasonable performance with m-bert and are substantial on all test languages. This further confirms the validity of language-specific pretraining in lieu of multilingual training, if sufficient monolingual data are available. Moreover, a comparison of pretrained English encoders in Figure FIGREF49 largely follows the intuition: the larger bert-large model yields slight improvements over bert-base, and we can improve a bit more by relying on word-level (i.e., lexical-level) masking.Finally, light-weight albert model variants are quite competitive with the original bert models, with only modest drops reported, and albert-l again outperforms albert-b. Overall, it is interesting to note that the scores obtained with monolingual pretrained encoders are on a par with or even outperform static ft word embeddings: this is a very intriguing finding per se as it shows that such subword-level models trained on large corpora can implicitly capture rich lexical semantic knowledge.","Similarity-Specialized Word Embeddings. Conflating distinct lexico-semantic relations is a well-known property of distributional representations BIBREF144, BIBREF53. Semantic specialization fine-tunes distributional spaces to emphasize a particular lexico-semantic relation in the transformed space by injecting external lexical knowledge BIBREF145. Explicitly discerning between true semantic similarity (as captured in Multi-SimLex) and broad conceptual relatedness benefits a number of tasks, as discussed in §SECREF4. Since most languages lack dedicated lexical resources, however, one viable strategy to steer monolingual word vector spaces to emphasize semantic similarity is through cross-lingual transfer of lexical knowledge, usually through a shared cross-lingual word vector space BIBREF106. Therefore, we evaluate the effectiveness of specialization transfer methods using Multi-SimLex as our multilingual test bed.","We evaluate a current state-of-the-art cross-lingual specialization transfer method with minimal requirements, put forth recently by Ponti:2019emnlp. In a nutshell, their li-postspec method is a multi-step procedure that operates as follows. First, the knowledge about semantic similarity is extracted from WordNet in the form of triplets, that is, linguistic constraints $(w_1, w_2, r)$, where $w_1$ and $w_2$ are two concepts, and $r$ is a relation between them obtained from WordNet (e.g., synonymy or antonymy). The goal is to “attract” synonyms closer to each other in the transformed vector space as they reflect true semantic similarity, and “repel” antonyms further apart. In the second step, the linguistic constraints are translated from English to the target language via a shared cross-lingual word vector space. To this end, following Ponti:2019emnlp we rely on cross-lingual word embeddings (CLWEs) BIBREF146 available online, which are based on Wiki ft vectors. Following that, a constraint refinement step is applied in the target language which aims to eliminate the noise inserted during the translation process. This is done by training a relation classification tool: it is trained again on the English linguistic constraints and then used on the translated target language constraints, where the transfer is again enabled via a shared cross-lingual word vector space. Finally, a state-of-the-art monolingual specialization procedure from Ponti:2018emnlp injects the (now target language) linguistic constraints into the target language distributional space.","The scores are summarized in Table TABREF56. Semantic specialization with li-postspec leads to substantial improvements in correlation scores for the majority of the target languages, demonstrating the importance of external semantic similarity knowledge for semantic similarity reasoning. However, we also observe deteriorated performance for the three target languages which can be considered the lowest-resource ones in our set: cym, swa, yue. We hypothesize that this occurs due to the inferior quality of the underlying monolingual Wikipedia word embeddings, which generates a chain of error accumulations. In particular, poor distributional word estimates compromise the alignment of the embedding spaces, which in turn results in increased translation noise, and reduced refinement ability of the relation classifier. On a high level, this “poor get poorer” observation again points to the fact that one of the primary causes of low performance of resource-low languages in semantic tasks is the sheer lack of even unlabeled data for distributional training. On the other hand, as we see from Table TABREF46, typological dissimilarity between the source and the target does not deteriorate the effectiveness of semantic specialization. In fact, li-postspec does yield substantial gains also for the typologically distant targets such as heb, cmn, and est. The critical problem indeed seems to be insufficient raw data for monolingual distributional training."]},{"idx":16,"heading":"Cross-Lingual Evaluation","paragraphs":["Similar to monolingual evaluation in §SECREF7, we now evaluate several state-of-the-art cross-lingual representation models on the suite of 66 automatically constructed cross-lingual Multi-SimLex datasets. Again, note that evaluating a full range of cross-lingual models available in the rich prior work on cross-lingual representation learning is well beyond the scope of this article. We therefore focus our cross-lingual analyses on several well-established and indicative state-of-the-art cross-lingual models, again spanning both static and contextualized cross-lingual word embeddings."]},{"idx":17,"heading":"Cross-Lingual Evaluation ::: Models in Comparison","paragraphs":["Static Word Embeddings. We rely on a state-of-the-art mapping-based method for the induction of cross-lingual word embeddings (CLWEs): vecmap BIBREF148. The core idea behind such mapping-based or projection-based approaches is to learn a post-hoc alignment of independently trained monolingual word embeddings BIBREF106. Such methods have gained popularity due to their conceptual simplicity and competitive performance coupled with reduced bilingual supervision requirements: they support CLWE induction with only as much as a few thousand word translation pairs as the bilingual supervision BIBREF149, BIBREF150, BIBREF151, BIBREF107. More recent work has shown that CLWEs can be induced with even weaker supervision from small dictionaries spanning several hundred pairs BIBREF152, BIBREF135, identical strings BIBREF153, or even only shared numerals BIBREF154. In the extreme, fully unsupervised projection-based CLWEs extract such seed bilingual lexicons from scratch on the basis of monolingual data only BIBREF103, BIBREF148, BIBREF155, BIBREF156, BIBREF104, BIBREF157.","Recent empirical studies BIBREF158, BIBREF135, BIBREF159 have compared a variety of unsupervised and weakly supervised mapping-based CLWE methods, and vecmap emerged as the most robust and very competitive choice. Therefore, we focus on 1) its fully unsupervised variant (unsuper) in our comparisons. For several language pairs, we also report scores with two other vecmap model variants: 2) a supervised variant which learns a mapping based on an available seed lexicon (super), and 3) a supervised variant with self-learning (super+sl) which iteratively increases the seed lexicon and improves the mapping gradually. For a detailed description of these variants, we refer the reader to recent work BIBREF148, BIBREF135. We again use CC+Wiki ft vectors as initial monolingual word vectors, except for yue where Wiki ft is used. The seed dictionaries of two different sizes (1k and 5k translation pairs) are based on PanLex BIBREF160, and are taken directly from prior work BIBREF135, or extracted from PanLex following the same procedure as in the prior work.","Contextualized Cross-Lingual Word Embeddings. We again evaluate the capacity of (massively) multilingual pretrained language models, m-bert and xlm-100, to reason over cross-lingual lexical similarity. Implicitly, such an evaluation also evaluates “the intrinsic quality” of shared cross-lingual word-level vector spaces induced by these methods, and their ability to boost cross-lingual transfer between different language pairs. We rely on the same procedure of aggregating the models' subword-level parameters into word-level representations, already described in §SECREF34.","As in monolingual settings, we can apply unsupervised post-processing steps such as abtt to both static and contextualized cross-lingual word embeddings."]},{"idx":18,"heading":"Cross-Lingual Evaluation ::: Results and Discussion","paragraphs":["Main Results and Differences across Language Pairs. A summary of the results on the 66 cross-lingual Multi-SimLex datasets are provided in Table TABREF59 and Figure FIGREF60. The findings confirm several interesting findings from our previous monolingual experiments (§SECREF44), and also corroborate several hypotheses and findings from prior work, now on a large sample of language pairs and for the task of cross-lingual semantic similarity.","First, we observe that the fully unsupervised vecmap model, despite being the most robust fully unsupervised method at present, fails to produce a meaningful cross-lingual word vector space for a large number of language pairs (see the bottom triangle of Table TABREF59): many correlation scores are in fact no-correlation results, accentuating the problem of fully unsupervised cross-lingual learning for typologically diverse languages and with fewer amounts of monolingual data BIBREF135. The scores are particularly low across the board for lower-resource languages such as Welsh and Kiswahili. It also seems that the lack of monolingual data is a larger problem than typological dissimilarity between language pairs, as we do observe reasonably high correlation scores with vecmap for language pairs such as cmn-spa, heb-est, and rus-fin. However, typological differences (e.g., morphological richness) still play an important role as we observe very low scores when pairing cmn with morphologically rich languages such fin, est, pol, and rus. Similar to prior work of Vulic:2019we and doval2019onthe, given the fact that unsupervised vecmap is the most robust unsupervised CLWE method at present BIBREF158, our results again question the usefulness of fully unsupervised approaches for a large number of languages, and call for further developments in the area of unsupervised and weakly supervised cross-lingual representation learning.","The scores of m-bert and xlm-100 lead to similar conclusions as in the monolingual settings. Reasonable correlation scores are achieved only for a small subset of resource-rich language pairs (e.g., eng, fra, spa, cmn) which dominate the multilingual m-bert training. Interestingly, the scores indicate a much higher performance of language pairs where yue is one of the languages when we use m-bert instead of vecmap. This boils down again to the fact that yue, due to its specific language script, has a good representation of its words and subwords in the shared m-bert vocabulary. At the same time, a reliable vecmap mapping between yue and other languages cannot be found due to a small monolingual yue corpus. In cases when vecmap does not yield a degenerate cross-lingual vector space starting from two monolingual ones, the final correlation scores seem substantially higher than the ones obtained by the single massively multilingual m-bert model.","Finally, the results in Figure FIGREF60 again verify the usefulness of unsupervised post-processing also in cross-lingual settings. We observe improved performance with both m-bert and xlm-100 when mean centering (+mc) is applied, and further gains can be achieved by using abtt on the mean-centered vector spaces. A similar finding also holds for static cross-lingual word embeddings, where applying abbt (-10) yields higher scores on 61/66 language pairs.","Fully Unsupervised vs. Weakly Supervised Cross-Lingual Embeddings. The results in Table TABREF59 indicate that fully unsupervised cross-lingual learning fails for a large number of language pairs. However, recent work BIBREF135 has noted that these sub-optimal non-alignment solutions with the unsuper model can be avoided by relying on (weak) cross-lingual supervision spanning only several thousands or even hundreds of word translation pairs. Therefore, we examine 1) if we can further improve the results on cross-lingual Multi-SimLex resorting to (at least some) cross-lingual supervision for resource-rich language pairs; and 2) if such available word-level supervision can also be useful for a range of languages which displayed near-zero performance in Table TABREF59. In other words, we test if recent “tricks of the trade” used in the rich literature on CLWE learning reflect in gains on cross-lingual Multi-SimLex datasets.","First, we reassess the findings established on the bilingual lexicon induction task BIBREF161, BIBREF135: using at least some cross-lingual supervision is always beneficial compared to using no supervision at all. We report improvements over the unsuper model for all 10 language pairs in Table TABREF66, even though the unsuper method initially produced strong correlation scores. The importance of self-learning increases with decreasing available seed dictionary size, and the +sl model always outperforms unsuper with 1k seed pairs; we observe the same patterns also with even smaller dictionary sizes than reported in Table TABREF66 (250 and 500 seed pairs). Along the same line, the results in Table TABREF67 indicate that at least some supervision is crucial for the success of static CLWEs on resource-leaner language pairs. We note substantial improvements on all language pairs; in fact, the vecmap model is able to learn a more reliable mapping starting from clean supervision. We again note large gains with self-learning.","Multilingual vs. Bilingual Contextualized Embeddings. Similar to the monolingual settings, we also inspect if massively multilingual training in fact dilutes the knowledge necessary for cross-lingual reasoning on a particular language pair. Therefore, we compare the 100-language xlm-100 model with i) a variant of the same model trained on a smaller set of 17 languages (xlm-17); ii) a variant of the same model trained specifically for the particular language pair (xlm-2); and iii) a variant of the bilingual xlm-2 model that also leverages bilingual knowledge from parallel data during joint training (xlm-2++). We again use the pretrained models made available by Conneau:2019nips, and we refer to the original work for further technical details.","The results are summarized in Figure FIGREF60, and they confirm the intuition that massively multilingual pretraining can damage performance even on resource-rich languages and language pairs. We observe a steep rise in performance when the multilingual model is trained on a much smaller set of languages (17 versus 100), and further improvements can be achieved by training a dedicated bilingual model. Finally, leveraging bilingual parallel data seems to offer additional slight gains, but a tiny difference between xlm-2 and xlm-2++ also suggests that this rich bilingual information is not used in the optimal way within the xlm architecture for semantic similarity.","In summary, these results indicate that, in order to improve performance in cross-lingual transfer tasks, more work should be invested into 1) pretraining dedicated language pair-specific models, and 2) creative ways of leveraging available cross-lingual supervision (e.g., word translation pairs, parallel or comparable corpora) BIBREF121, BIBREF123, BIBREF124 with pretraining paradigms such as bert and xlm. Using such cross-lingual supervision could lead to similar benefits as indicated by the results obtained with static cross-lingual word embeddings (see Table TABREF66 and Table TABREF67). We believe that Multi-SimLex can serve as a valuable means to track and guide future progress in this research area."]},{"idx":19,"heading":"Conclusion and Future Work","paragraphs":["We have presented Multi-SimLex, a resource containing human judgments on the semantic similarity of word pairs for 12 monolingual and 66 cross-lingual datasets. The languages covered are typologically diverse and include also under-resourced ones, such as Welsh and Kiswahili. The resource covers an unprecedented amount of 1,888 word pairs, carefully balanced according to their similarity score, frequency, concreteness, part-of-speech class, and lexical field. In addition to Multi-Simlex, we release the detailed protocol we followed to create this resource. We hope that our consistent guidelines will encourage researchers to translate and annotate Multi-Simlex -style datasets for additional languages. This can help and create a hugely valuable, large-scale semantic resource for multilingual NLP research.","The core Multi-SimLex we release with this paper already enables researchers to carry out novel linguistic analysis as well as establishes a benchmark for evaluating representation learning models. Based on our preliminary analyses, we found that speakers of closely related languages tend to express equivalent similarity judgments. In particular, geographical proximity seems to play a greater role than family membership in determining the similarity of judgments across languages. Moreover, we tested several state-of-the-art word embedding models, both static and contextualized representations, as well as several (supervised and unsupervised) post-processing techniques, on the newly released Multi-SimLex. This enables future endeavors to improve multilingual representation learning with challenging baselines. In addition, our results provide several important insights for research on both monolingual and cross-lingual word representations:","1) Unsupervised post-processing techniques (mean centering, elimination of top principal components, adjusting similarity orders) are always beneficial independently of the language, although the combination leading to the best scores is language-specific and hence needs to be tuned.","2) Similarity rankings obtained from word embeddings for nouns are better aligned with human judgments than all the other part-of-speech classes considered here (verbs, adjectives, and, for the first time, adverbs). This confirms previous generalizations based on experiments on English.","3) The factor having the greatest impact on the quality of word representations is the availability of raw texts to train them in the first place, rather than language properties (such as family, geographical area, typological features).","4) Massively multilingual pretrained encoders such as m-bert BIBREF29 and xlm-100 BIBREF30 fare quite poorly on our benchmark, whereas pretrained encoders dedicated to a single language are more competitive with static word embeddings such as fastText BIBREF31. Moreover, for language-specific encoders, parameter reduction techniques reduce performance only marginally.","5) Techniques to inject clean lexical semantic knowledge from external resources into distributional word representations were proven to be effective in emphasizing the relation of semantic similarity. In particular, methods capable of transferring such knowledge from resource-rich to resource-lean languages BIBREF59 increased the correlation with human judgments for most languages, except for those with limited unlabelled data.","Future work can expand our preliminary, yet large-scale study on the ability of pretrained encoders to reason over word-level semantic similarity in different languages. For instance, we have highlighted how sharing the same encoder parameters across multiple languages may harm performance. However, it remains unclear if, and to what extent, the input language embeddings present in xlm-100 but absent in m-bert help mitigate this issue. In addition, pretrained language embeddings can be obtained both from typological databases BIBREF99 and from neural architectures BIBREF100. Plugging these embeddings into the encoders in lieu of embeddings trained end-to-end as suggested by prior work BIBREF162, BIBREF163, BIBREF164 might extend the coverage to more resource-lean languages.","Another important follow-up analysis might involve the comparison of the performance of representation learning models on multilingual datasets for both word-level semantic similarity and sentence-level Natural Language Understanding. In particular, Multi-SimLex fills a gap in available resources for multilingual NLP and might help understand how lexical and compositional semantics interact if put alongside existing resources such as XNLI BIBREF84 for natural language inference or PAWS-X BIBREF89 for cross-lingual paraphrase identification. Finally, the Multi-SimLex annotation could turn out to be a unique source of evidence to study the effects of polysemy in human judgments on semantic similarity: for equivalent word pairs in multiple languages, are the similarity scores affected by how many senses the two words (or multi-word expressions) incorporate?","In light of the success of initiatives like Universal Dependencies for multilingual treebanks, we hope that making Multi-SimLex and its guidelines available will encourage other researchers to expand our current sample of languages. We particularly encourage creation and submission of comparable Multi-SimLex datasets for under-resourced and typologically diverse languages in future work. In particular, we have made a Multi-Simlex community website available to facilitate easy creation, gathering, dissemination, and use of annotated datasets: https://multisimlex.com/.","This work is supported by the ERC Consolidator Grant LEXICAL: Lexical Acquisition Across Languages (no 648909). Thierry Poibeau is partly supported by a PRAIRIE 3IA Institute fellowship (\"Investissements d'avenir\" program, reference ANR-19-P3IA-0001)."]},{"idx":20,"heading":"Figures and Tables","paragraphs":["Table 1: The list of 12 languages in the Multi-SimLex multilingual suite along with their corresponding language family (IE = Indo-European), broad morphological type, and their ISO 639-3 code. The number of speakers is based on the total count of L1 and L2 speakers, according to ethnologue.com.","Table 2: Inter-translator agreement (% of matched translated words) by independent translators using a randomly selected 100-pair English sample from the Multi-SimLex dataset, and the corresponding 100-pair samples from the other datasets.","Table 3: Number of human annotators. R1 = Annotation Round 1, R3 = Round 3.","Table 4: Average pairwise inter-annotator agreement (APIAA). A score of 0.6 and above indicates strong agreement.","Table 5: Average mean inter-annotator agreement (AMIAA). A score of 0.6 and above indicates strong agreement.","Table 6: Fine-grained distribution of concept pairs over different rating intervals in each Multi-SimLex language, reported as percentages. The total number of concept pairs in each dataset is 1,888.","Figure 1: Spearman’s correlation coefficient (ρ) of the similarity scores for all languages in Multi-SimLex.","Figure 2: A distribution over different frequency ranges for words from Multi-SimLex datasets for selected languages. Multi-word expressions are excluded from the analysis.","Table 7: Examples of concept pairs with their similarity scores from four languages. For brevity, only the original English concept pair is included, but note that the pair is translated to all target languages, see §5.1.","Table 8: Examples of concept pairs with their similarity scores from four languages where all participants show strong agreement in their rating.","Figure 3: PCA of the language vectors resulting from the concatenation of similarity judgments for all pairs.","Table 9: Mantel test on the correlation between similarity judgments from Multi-SimLex and linguistic features from typological databases.","Table 10: Example concept pairs with their scores from a selection of cross-lingual MultiSimLex datasets.","Table 11: The sizes of all monolingual (main diagonal) and cross-lingual datasets.","Figure 4: (a) Rating distribution and (b) distribution of pairs over the four POS classes in cross-lingual Multi-SimLex datasets averaged across each of the 66 language pairs (y axes plot percentages as the total number of concept pairs varies across different cross-lingual datasets). Minimum and maximum percentages for each rating interval and POS class are also plotted.","Table 12: A summary of results (Spearman’s ρ correlation scores) on the full monolingual Multi-SimLex datasets for 12 languages. We benchmark fastText word embeddings trained on two different corpora (CC+Wiki and only Wiki) as well the multilingual MBERT model (see §7.1). Results with the initial word vectors are reported (i.e., without any unsupervised post-processing), as well as with different unsupervised post-processing methods, described in §7.1. The language codes are provided in Table 1. The numbers in the parentheses (gray rows) refer to the number of OOV concepts excluded from the computation. The highest scores for each language and per model are in bold.","Table 13: Spearman’s ρ correlation scores over the four POS classes represented in MultiSimLex datasets. In addition to the word vectors considered earlier in Table 12, we also report scores for another contextualized model, XLM-100. The numbers in parentheses refer to the total number of POS-class pairs in the original ENG dataset and, consequently, in all other monolingual datasets.","Figure 5: (a) A performance comparison between monolingual pretrained language encoders and massively multilingual encoders. For four languages (CMN, ENG, FIN, SPA), we report the scores with monolingual uncased BERT-BASE architectures and multilingual uncased M-BERT model, while for FRA we report the results of the multilingual XLM-100 architecture and a monolingual French FlauBERT model (Le et al. 2019), which is based on the same architecture as XLM-100. (b) A comparison of various pretrained encoders available for English. All these models are post-processed via ABTT (-3).","Table 14: The impact of vector space specialization for semantic similarity. The scores are reported using the current state-of-the-art specialization transfer LI-POSTSPEC method of Ponti et al. (2019c), relying on English as a resource-rich source language and the external lexical semantic knowledge from the English WordNet.","Table 15: Spearman’s ρ correlation scores on all 66 cross-lingual datasets. 1) The scores below the main diagonal are computed based on cross-lingual word embeddings (CLWEs) induced by aligning CC+Wiki FT in all languages (except for YUE where we use Wiki FT) in a fully unsupervised way (i.e., without any bilingual supervision). We rely on a standard CLWE mapping-based (i.e., alignment) approach: VECMAP (Artetxe, Labaka, and Agirre 2018b). 2) The scores above the main diagonal are computed by obtaining 768-dimensional word-level vectors from pretrained multilingual BERT (MBERT) following the procedure described in §7.1. For both fully unsupervised VECMAP and M-BERT, we report the results with unsupervised postprocessing enabled: all 2× 66 reported scores are obtained using the +ABBT (-10) variant.","Figure 6: Further performance analyses of cross-lingual Multi-SimLex datasets. (a) Spearman’s ρ correlation scores averaged over all 66 cross-lingual Multi-SimLex datasets for two pretrained multilingual encoders (M-BERT and XLM). The scores are obtained with different configurations that exclude (INIT) or enable unsupervised post-processing. (b) A comparison of various pretrained encoders available for the English-French language pair, see the main text for a short description of each benchmarked pretrained encoder.","Table 16: Results on a selection of cross-lingual Multi-SimLex datasets where the fully unsupervised (UNSUPER) CLWE variant yields reasonable performance. We also show the results with supervised VECMAP without self-learning (SUPER) and with self-learning (+SL), with two seed dictionary sizes: 1k and 5k pairs; see §8.1 for more detail. Highest scores for each language pair are in bold.","Table 17: Results on a selection of cross-lingual Multi-SimLex datasets where the fully unsupervised (UNSUPER) CLWE variant fails to learn a coherent shared cross-lingual space. See also the caption of Table 16."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"2003.09244","title":"Language Technology Programme for Icelandic 2019-2023","sections":[{"idx":0,"heading":"Abstract","paragraphs":["In this paper, we describe a new national language technology programme for Icelandic. The programme, which spans a period of five years, aims at making Icelandic usable in communication and interactions in the digital world, by developing accessible, open-source language resources and software. The research and development work within the programme is carried out by a consortium of universities, institutions, and private companies, with a strong emphasis on cooperation between academia and industries. Five core projects will be the main content of the programme: language resources, speech recognition, speech synthesis, machine translation, and spell and grammar checking. We also describe other national language technology programmes and give an overview over the history of language technology in Iceland."]},{"idx":1,"heading":"Introduction","paragraphs":["During the last decade, we have witnessed enormous advances in language technology (LT). Applications that allow users to interact with technology via spoken or written natural language are emerging in all areas, and access to language resources and open-source software libraries enables faster development for new domains and languages.","However, LT is highly language dependent and it takes considerable resources to develop LT for new languages. The recent LT development has focused on languages that have both a large number of speakers and huge amounts of digitized language resources, like English, German, Spanish, Japanese, etc. Other languages, that have few speakers and/or lack digitized language resources, run the risk of being left behind.","Icelandic is an example of a language with almost a negligible number of speakers, in terms of market size, since only about 350,000 people speak Icelandic as their native language. Icelandic is therefore seldom on the list of supported languages in LT software and applications.","The Icelandic Government decided in 2017 to fund a five-year programme for Icelandic LT, based on a report written by a group of LT experts BIBREF0. After more than two years of preparation, a consortium consisting of universities, institutions, associations, and private companies started the work on the programme on the 1st of October 2019. The goal of the programme is to ensure that Icelandic can be made available in LT applications, and thus will be usable in all areas of communication. Furthermore, that access to information and other language-based communication and interaction in Icelandic will be accessible to all, e.g. via speech synthesis or speech-to-text systems.","The focus of the programme will be on the development of text and speech-based language resources, on the development of core natural language processing (NLP) tools like tokenisers, taggers and parsers, and finally, to publish open-source software in the areas of speech recognition, speech synthesis, machine translation, and spell and grammar checking. All deliverables of the programme will be published under open licenses, to encourage use of resources and software in commercial products.","While the government-funded programme for the development of resources and infrastructure software builds the backbone of the Icelandic LT programme, another branch is a competitive fund for research and development. This Strategic Research and Development Programme for Language Technology is managed by the Icelandic Centre for Research, Rannís, which publishes calls for applications on a regular basis.","The third pillar of the programme is the revival of the joint Master's programme in LT at Reykjavik University (RU) and the University of Iceland (UI). The goal is further to increase the number of PhD students and to build strong knowledge centres for sustainable LT development in Iceland.","The budget estimation for the programme, including the competitive fund, education plan and infrastructure costs, is around 14 million euros. Additionally, around 3.6 million euros is expected to be the contribution of the industry through the competitive fund.","This paper is structured as follows: In Section SECREF2 we discuss national LT programmes that have been run in other European countries and helped developing the Icelandic project plan. Section SECREF3 gives an overview over the 20 years of LT development in Iceland. Section SECREF4 shows the organisation of the new programme, and in Section SECREF5 we describe the core projects that have been defined for it. Finally, a conclusion is presented in Section SECREF6."]},{"idx":2,"heading":"Other European LT Programmes","paragraphs":["In recent years, there has been much international discussion on how the future of languages depends on them being usable in the digital world. This concern has led to a number of national LT programmes. We studied three of these national programmes: the STEVIN programme in the Netherlands which ran between 2004 and 2011, the Plan for the Advancement of Language Technology in Spain, and, in particular, the Estonian LT programmes that have been running since 2006."]},{"idx":3,"heading":"Other European LT Programmes ::: The Netherlands","paragraphs":["The STEVIN programme was launched in 2004 to strengthen the position of Dutch in LT by building essential resources for the language. Its objectives were to raise awareness of LT in order to stimulate demand for LT products, to promote strategic research in the field and develop essential resources, and to organise the management, maintenance and distribution of language resources that have been developed BIBREF1. The programme was based on cooperation between government, academia and industry, both in Flanders and the Netherlands. It encompassed a range of projects from basic resources to applications for language users, and attention was paid to distribution, dissemination and valorisation of project results by means of the HLT Agency, which also had a role in clearing intellectual property rights (IPRs) and issuing licence agreements BIBREF2.","The general targets of the STEVIN programme were reached to a large extent. According to a report on the results of the programme BIBREF3, it resulted in a network with strong ties between academia and industry, beneficial for future utilisation of the STEVIN results. The evaluators of the programme qualified it as successful, but had recommendations for a future programme, if initiated. They suggested more interaction with other similar (inter)national R&D programmes, asserted that the complexity of IPR issues had been seriously underestimated and called for a better clarification of the role of open-source. The total cost of the STEVIN programme was over 10 million euros, of which well over 80% was spent on R&D projects."]},{"idx":4,"heading":"Other European LT Programmes ::: Spain","paragraphs":["The Spanish LT programme Plan for Advancement of Language Technology started in 2016, and is scheduled to finish in 2020. Its aims are to develop infrastructure for LT in Spain, specifically for Spanish and the co-official languages, Basque, Catalan, Galician and Aranese. Furthermore, to promote the LT industry by boosting knowledge transfer between research and industry actors, and to improve the quality and capacity of public services by employing NLP and machine translation (MT) technology. Government should be the leading participant in LT with high-profile projects in healthcare, as well as in the judicial and educational systems, and in tourism BIBREF4.","The plan was to facilitate the development of tools and linguistic resources. Examples of tools are named entity recognisers, word-sense disambiguation, tools for computing semantic similarity and text classification, automatic summarisation and MT. Examples of linguistic resources to be developed in the programme are parallel corpora, lists of proper nouns, terminology lists and dictionaries.","The estimated total cost of the programme was 90 million euros. As the programme had just recently started when the Icelandic programme was being planned, we did not have any information on what went well and what could have been done better."]},{"idx":5,"heading":"Other European LT Programmes ::: Estonia","paragraphs":["Regarding LT, the Estonian situation is, in many ways, similar to that of Iceland: It has too few users for companies to see opportunities in embarking on development of (costly) LT, but on the other hand society is technologically advanced – people use, or want to be able to use, LT software. In Estonia, the general public wants Estonian to maintain its status, and like Icelandic, the language has a complex inflection system and very active word generation. The problems faced by Estonia are therefore not unlike those that Iceland faces.","In Estonia, three consecutive national programmes have been launched. The third national programme, Estonian Language Technology 2018–2027, is currently under way. While the Estonian Ministry of Education and Research has been responsible for the programmes, the universities in Tallinn and Tartu, together with the Institute of the Estonian Language, led the implementation.","The National Programme for Estonian Language Technology was launched in 2006. The first phase ran from 2006 to 2010. All results of this first phase, language resources and software prototypes, were released as public domain. All such resources and tools are preserved long term and available from the Center of Estonian Language Resources. 33 projects were funded, which included the creation of reusable language resources and development of essential linguistic software, as well as bringing the relevant infrastructure up to date BIBREF5. The programme managed to significantly improve upon existing Estonian language resources, both in size, annotation and standardisation. In creating software, most noticeable results were in speech technology. Reporting on the results of the programme BIBREF5 stress that the first phase of the programme created favourable conditions for LT development in Estonia. According to an evaluation of the success of the programme, at least 84% of the projects had satisfactory results. The total budged for this first phase was 3.4 million euros.","The second phase of the programme ran from 2011 to 2017 with a total budget of approx. 5.5 million euros. It focused on the implementation and integration of existing resources and software prototypes in public services. Project proposals were called for, funding several types of actions in an open competition. The main drawback of this method is that it does not fully cover the objectives, and LT support for Estonian is thus not systematically developed. Researchers were also often mostly interested in results using prototypes rather than stable applications. As most of the projects were instigated at public institutes, relation to IT business was weak. Furthermore, the programme does not deal explicitly with LT education. On the other hand, the state of LT in Estonia soon become relatively good compared to languages with similar number of speakers BIBREF6."]},{"idx":6,"heading":"History of Icelandic LT","paragraphs":["The history of Icelandic LT is usually considered to have begun around the turn of the century, even though a couple of LT resources and products were developed in the years leading up to that. Following the report of an expert group appointed by the Minister of Education, Science and Culture BIBREF7, the Icelandic Government launched a special LT Programme in the year 2000, with the aim of supporting institutions and companies to create basic resources for Icelandic LT work. This initiative resulted in a few projects which laid the ground for future work in the field. The most important of these were a 25 million token, balanced, tagged corpus, a full-form database of Icelandic inflections, a training model for PoS taggers, an improved speech synthesiser, and an isolated word speech recogniser BIBREF8.","After the LT Programme ended in 2004, researchers from three institutions, UI, RU, and the Árni Magnússon Institute for Icelandic Studies (AMI), joined forces in a consortium called the Icelandic Centre for Language Technology (ICLT), in order to follow up on the tasks of the Programme. In the following years, these researchers developed a few more tools and resources with support from The Icelandic Research Fund, notably a rule-based tagger, a shallow parser, a lemmatiser, and a historical treebank BIBREF9.","In 2011–2012, researchers from the ICLT also participated in two speech technology projects initiated by others: A new speech synthesiser for Icelandic which was developed by the Polish company Ivona, now a subsidiary of Amazon, for the Icelandic Association for the Visually Impaired, and a speech recogniser for Icelandic developed by Google BIBREF9.","Iceland was an active participant in the META-NORD project, a subproject of META-NET, from 2011 to 2013. Within that project, a number of language resources for Icelandic were collected, enhanced, and made available, both through META-SHARE and through a local website, málföng.is (málföng being a neologism for `language resources'). Among the main deliveries of META-NET were the Language White Papers BIBREF10. The paper on Icelandic BIBREF11 highlighted the alarming status of Icelandic LT. Icelandic was among four languages that received the lowest score, “support is weak or non-existent” in all four areas that were evaluated.","The White Paper received considerable attention in Icelandic media and its results were discussed in the Icelandic Parliament. In 2014, the Parliament unanimously accepted a resolution where the Minister of Education, Science and Culture was given mandate to appoint an expert group which should come up with a long-term LT plan for Icelandic. The group delivered its report to the Minister in December 2014. The result was that a small LT Fund was established in 2015.","During the last years, a strong centre for speech technology has been established at RU, where development in speech recognition and synthesis has been ongoing since 2011. Acoustic data for speech recognition was collected and curated at RU BIBREF12, BIBREF13, BIBREF14 and a baseline speech recognition system for Icelandic was developed BIBREF15. Specialised speech recognisers have also been developed at RU for the National University Hospital and Althingi BIBREF16, BIBREF17, BIBREF18. A work on a baseline speech synthesis system for Icelandic has also been carried out at RU BIBREF19, BIBREF20.","The AMI has built a 1.3-billion-word corpus, the Icelandic Gigaword Corpus (IGC) BIBREF21, partially funded by the Icelandic Infrastructure Fund. Further, a private company, Miðeind Ltd., has developed a context-free parser BIBREF22 partially funded by the LT Fund.","In October 2016, the Minister of Education, Science and Culture appointed a special LT steering group, consisting of representatives from the Ministry, from academia, and from the Confederation of Icelandic Enterprise (CIE). The steering group commissioned three LT experts to work out a detailed five-year Project Plan for Icelandic LT. The experts delivered their proposals, Language Technology for Icelandic 2018–2022 – Project Plan BIBREF0 to the Minister in June 2017."]},{"idx":7,"heading":"Organisation of the Icelandic LT Programme 2019–2023","paragraphs":["The Icelandic Government decided soon after the publication of the report Language Technology for Icelandic 2018–2022 – Project Plan to use the report as a base for a five-year government funded LT programme for Icelandic. The self-owned foundation Almannarómur, founded in 2014 to support the development of Icelandic LT, was to be prepared to take over a role as a Centre of Icelandic LT and to elaborate on how the programme could be organised and executed to meet the goals defined in the report.","The Icelandic Ministry of Education, Science and Culture signed an agreement with Almannarómur in August 2018, giving Almannarómur officially the function of organising the execution of the LT programme for Icelandic. Following a European Tender published in March 2019, Almannarómur decided to make an agreement with a consortium of universities, institutions, associations, and private companies (nine in total) in Iceland (listed in Table TABREF6) to perform the research and development part of the programme. This Consortium for Icelandic LT (Samstarf um íslenska máltækni – SÍM) is a joint effort of LT experts in Iceland from academia and industry. SÍM is not a legal entity but builds the cooperation on a consortium agreement signed by all members. During the preparation of the project, an expert panel of three experienced researchers from Denmark, the Netherlands, and Estonia was established to oversee the project planning and to evaluate deliverables at predefined milestones during the project.","SÍM has created teams across the member organisations, each taking charge of a core project and/or defined subtasks. This way the best use of resources is ensured, since the team building is not restricted to one organisation per project. One project manager coordinates the work and handles communication and reporting to Almannarómur and the expert panel.","Besides the role of the executive of the research and development programme itself, Almannarómur will conduct communication between the executing parties and the local industry, as well as foreign companies and institutions. Together with the executing parties, Almannarómur will also host conferences and events to promote the programme and bring together interested parties."]},{"idx":8,"heading":"Core Projects","paragraphs":["In this section, we describe the five core projects that have been defined in the Icelandic LT programme."]},{"idx":9,"heading":"Core Projects ::: Language Resources","paragraphs":["As mentioned above, a number of language resources have been made available at the repository málföng. Most of these are now also available at the CLARIN-IS website and will be integrated into the CLARIN Virtual Language Observatory. Below we give a brief and non-exhaustive overview of language resources for Icelandic which will be developed in the programme.","Tagged corpora. The IGC BIBREF21 contains 1.3 billion running words, tagged and lemmatised. It is much bigger than previous tagged corpora, most notably the Icelandic Frequency Dictionary (IFD; Pind et al., 1991), which was the first morphologically tagged corpus of Icelandic texts, containing half a million words tokens from various texts, and the Tagged Icelandic Corpus (MÍM; Helgadóttir et al,. 2012), a balanced corpus of texts from the first decade of the 21st century, containing around 25 million tokens. A gold standard tagged corpus was created from a subset of MÍM BIBREF23. Some revisions of the morphosyntactic tagset used for tagging Icelandic texts will be done in the programme, and the gold standard updated accordingly.","We will update the IGC with new data from more sources and continue collecting data from rights holders who have given their permission for using their material. A new version will be released each year during the five-year programme.","Treebanks. The largest of the syntactically parsed treebanks that exist is the Icelandic Parsed Historical Corpus (IcePaHC; Wallenberg et al., 2011; Rögnvaldsson et al., 2011, 2012), which contains one million words from the 12th to the 21st century. The scheme used for the syntactic annotation is based on the Penn Parsed Corpora of Historical English BIBREF24, BIBREF25. On the other hand, no Universal Dependencies (UD)-treebanks are available for Icelandic. Within the programme, a UD-treebank will by built, based on IcePaHC, and extended with new material.","Morphological database. The Database of Icelandic Morphology (DIM; Bjarnadóttir et al., 2019) contains inflectional paradigms of about 287,000 lemmas. A part of the database, DMII-Core, only includes data in a prescriptive context and is suited for language learners, creating teaching material and other prescriptive uses. It consists of the inflection of approx. 50,000 words. We will extend it by reviewing ambiguous inflection forms. We will define format for data publication as the core will be available for use by a third party. For the sake of simplifying the process of adding material to the database and its maintenance, we will take advantage of the lexicon acquisition tool described in Section SECREF16 and adapt it for DIM.","Hyphenation tool. Hyphenation from one language to another often seems rather idiosyncratic but within one and the same language, such as Icelandic, such rules are often reasonably clear. A list of more than 200,000 Icelandic words with permissible hyphenations is available in the language resources repository. It will be expanded based on words from the DIM. A new hyphenation tool, trained on the extended list, will be built in the programme. The tool makes a suggestion for correct hyphenation possibilities of words that are not found on the hyphenation list.","Icelandic wordnet. The Icelandic wordnet BIBREF26, which contains 200,000 phrasemes of various kinds and about 100,000 compounds, is not a traditional dictionary as it analyses internal connections semantically and syntactically within Icelandic vocabulary. We will define a more appropriate data format and convert the wordnet data to that format. In addition, we will work on improving the wordnet itself by filling in gaps in various categories."]},{"idx":10,"heading":"Core Projects ::: NLP Tools","paragraphs":["A wide variety of NLP tools are to be developed or improved upon within the programme. It is of vital importance to develop quality NLP tools, as many tools often form a pipeline that analyses data and delivers the results to tools used by end users, and, in the pipeline, errors can accumulate and perpetuate.","When the programme started, there were a few available tools for Icelandic. IceNLP BIBREF27 is a suite of NLP tools containing modules for tokenisation, PoS-tagging, lemmatising, parsing and named entity recognition. Greynir BIBREF22 is a full parser which also includes a tokeniser and recognises some types of named entities. Nefnir BIBREF28 is a lemmatiser which uses suffix substitution rules, derived from the Database of Icelandic Morphology BIBREF29, giving results that outperform IceNLP. ABLTagger BIBREF30 is a PoS tagger that outperforms other taggers that have been trained for tagging Icelandic texts.","Some of these tools give good results, but can be improved upon. For other tasks, new tools need to be built. As part of the release process care will be taken to ensure all resulting software are up to high quality standards, and well documented to facilitate use by third parties. Where applicable, RESTful APIs will also be set up to further promote the usage of the products.","Tokeniser. A basic step in NLP is to segment text into units, normally sentences and tokens. Since any errors made at this stage will cascade through the process, it is important that the tokeniser is as accurate as possible. A tokeniser for Icelandic needs to be able to correctly recognises abbreviations, time units, dates, etc. It must also be adjustable and able to run using different settings, since its output must be adaptable to different projects and different uses.","Previously, two tokenisers have been built for Icelandic, one is a part of IceNLP and the other a part of Greynir. As Greynir is still in active development, it will be used as a base for the LT project's development. In order to be able to test the tokenisers' accuracy, a test set that takes different tokeniser settings into account will be developed.","PoS tagger. Precise PoS-tagging is important in many LT projects because information on word class or morphological features is often needed in later stages of an NLP pipeline. Improved tagging accuracy, thus often results in an improvement in the overall quality of LT software.","A number of PoS-taggers have been developed for Icelandic, with the best results achieved by a recent bidirectional LSTM tagging model BIBREF30. While developing PoS taggers for Icelandic further using state-of-the-art methods, we will also study and try to estimate how much accuracy can theoretically be reached in tagging a variety of Icelandic text styles, using the tag set chosen for the LT programme (see Section SECREF7).","Lemmatiser. A new lemmatiser for Icelandic, Nefnir, has recently been published BIBREF28. It has been shown to be quite accurate, although a standardised test set is not available to compare it to other lemmatisers, like Lemmald BIBREF31. Its main weakness is in lemmatising unknown words, which is a hard problem for inflected languages. We will study if its accuracy can be improved in that regard.","Parser. Three parsers have previously been developed for Icelandic. IceNLP includes a shallow parser based on a cascade of finite-state transducers BIBREF32. Greynir, on the other hand, fully parses sentences according to a hand-crafted context-free grammar. A parsing pipeline for Icelandic based on the IcePaHC corpus and the Berkeley-parser has also been released BIBREF33. No Universal Dependencies (UD) parser is available for Icelandic and no UD treebank, but in a project that started in 2019, independent of the LT programme, IcePaHC BIBREF34 will be converted to a UD treebank.","The IceNLP and Greynir parsers will be evaluated and either one of them or both developed further. We will also adapt a UD-parser to Icelandic UD-grammar.","Named entity recogniser. Some work has been carried out on named entity recognition for Icelandic. IceNLP contains a rule-based module that has achieved 71-79% accuracy and a recent tool based on a bidirectional LSTM BIBREF35 obtained an F1 score of 81.3%. There is also a named entity recogniser for proper names in Greynir, but its accuracy has not yet been evaluated. Within the programme, different training methods will be experimented with and evaluated, and the most promising tools evaluated further.","Semantic analysis. A variety of different tasks involve semantic analysis, including word-sense disambiguation (WSD), anaphora resolution, identifying co-references, analysing semantic similarity between compound verbs and phrases, and more.","We will work on these four aspects of semantic analysis listed above. In recent years, not much work has been carried out in this field for Icelandic. This part of the LT programme will thus start with researching the current state-of-the-art and defining realistic goals.","Lexicon acquisition tool. When constructing and maintaining lexical databases, such as DIM, the Icelandic wordnet or other related resources, it is vital to be able to systematically add neologies and words that are missing from the datasets, especially those commonly used in the language. Within the LT programme a flexible lexicon acquisition tool will be developed. It will be able to identify and collect unknown words and word forms, together with statistics, through structured lexical acquisition from the Icelandic Gigaword Corpus, which is constantly being updated, and other data sources in the same format."]},{"idx":11,"heading":"Core Projects ::: Automatic Speech Recognition (ASR)","paragraphs":["The main aim of the automatic speech recognition (ASR) project is to gather all necessary language and software resources to implement and build standard speech recognition systems for Icelandic. The project should enable developers to either research, develop or implement ASR without having to gather language resources. To achieve this goal, the project is divided into data gathering, recipe development, and software implementation and research.","Data gathering. The data gathering part of the project encompasses a wide variety of speech and transcript resources. A continuation of the Málrómur project BIBREF14 has already been implemented using Mozilla Common Voice. Here the aim is to double the size of the existing data set, get a more even distribution of speakers across geographic locations and age groups, and gather data from second language speakers. Additionally, radio and television transcripts are being gathered on a large scale and prepared for publication for ASR development. Conversations, queries and lectures will also be transcribed and published, and large open historical data sets will be aligned and prepared for publication.","Recipe development. ASR recipes for Icelandic will be developed further using more language resources BIBREF15 and specific application areas such as conversations, question answering and voice commands will be given a special attention. ASR systems that focus on teenagers, children and second language speakers are also within the scope of the project. These recipes are then used to create resources for smart-phone and web-based integration of ASR for Icelandic.","Software implementation and research. The research areas are chosen so to enhance the language resource development for Icelandic. A punctuation system for Icelandic will be analysed and implemented. Compound words are common in Icelandic and the language also has a relatively rich inflection structure so it is important to address those features for language modeling. Pronunciation analysis, speaker diarization and speech analysis will also be addressed especially for Icelandic, and acoustic modelling for children and teenagers receive attention in the project."]},{"idx":12,"heading":"Core Projects ::: Speech Synthesis (TTS)","paragraphs":[". The text-to-speech project will produce language resources that enable voice building for Icelandic.","Unit selection. Eight voices for unit-selection TTS will be recorded, with the aim of attaining diversity in age and dialect, with an equal number of male and female voices. The reason why unit-selection is chosen is to increase the likelihood that the project will produce useful and viable voices that can be used in addition to the two unit-selection voices that already exist for Icelandic.","Statistical parametric speech synthesis. Forty voices for statistical parametric speech synthesis (SPSS) will be recorded during the project. The plan is to publish open-source unit-selection and SPSS recipes with all necessary language resources so that programmers and researchers can continue to develop voices for Icelandic.","Suitable TTS voices for web-reading and smartphones will be developed within an open-source paradigm. This will allow the industry to use the voices developed within the project.","Research. The targeted research part of the project will facilitate the recipe development and software implementation. Quality assessment systems will be set up, text normalization for Icelandic will be developed fully, and intonation analysis for Icelandic will be implemented and applied to TTS."]},{"idx":13,"heading":"Core Projects ::: Spell and Grammar Checking","paragraphs":["The Spell and Grammar Checking project will develop and make freely available, under open-source licensing, important data sets and tools for further establishment of automated text correction systems for Icelandic. The project makes extensive use of other resources that have been developed independently, or will be developed within the larger framework of the current LT Programme for Icelandic, including the Database of Icelandic Morphology BIBREF29, the Greynir system BIBREF22, and the Icelandic Gigaword corpus BIBREF21. On the one hand, the project focuses on developing error corpora for Icelandic, and on the other, it focuses on creating a set of correction tools. Challenges associated with richly inflected languages continue to be a matter of central interest in this project, like previous work on Icelandic spelling correction BIBREF36.","Text correction data. The data construction aspect of the project will develop three error corpora that can be used for quantitative analysis of errors in written Icelandic text. The error corpora will also serve as a foundation for training data-driven training correction systems. One corpus will focus on the written language of Icelandic speakers who are not known to have unusual language properties. Another corpus will focus on speakers who are in the process of learning Icelandic as a second language, and a third one will include data from dyslexic speakers.","Software development. The software development tasks of the spell and grammar checking project will build a working open source correction system whose development is informed by the analysis of the data sets created within the project. The spell and grammar checker will be based on the foundation for processing Icelandic text provided by the Greynir system."]},{"idx":14,"heading":"Core Projects ::: Machine Translation","paragraphs":["The purpose of the MT project is to build open-source systems capable of translating between Icelandic and English, in both directions, is$\\rightarrow $en and en$\\rightarrow $is. The goal is that the translation quality will be good enough to be useful for translators in specific domains. A part of the MT project is indeed to define in which translation domain most value can be gained with the systems.","Very limited work on MT for Icelandic has been carried out since the turn of the century. A prototype of an open-source is$\\rightarrow $en rule-based MT system has been developed using the Apertium platform BIBREF37, but this system is not currently in public use.","The AMI has recently compiled an English-Icelandic parallel corpus, ParIce, the first parallel corpus built for the purposes of MT research and development for Icelandic BIBREF38. The primary goal of the compilation of ParIce was to build a corpus large enough and of good enough quality for training useful MT systems. ParIce currently consists of 39 million Icelandic words in 3.5 million segment pairs. The largest parts of ParIce consists of film and TV subtitles from the Opus corpus BIBREF39, and texts from the European Medicines Agency document portal, included in the Tilde MODEL corpus BIBREF40.","Google Translate supports translations between Icelandic and various languages and is currently used widely by Icelanders and foreigners for obtaining understandable translations of given texts (the task of assimilation). The problem with Google's system is, however, that neither the source code nor the training data is publicly available. Moreover, the system is a general translation engine, but not developed specifically for translating texts in a particular domain.","Our MT project in the new LT programme consists of the following sub-parts:","Parallel data. Icelandic's rich morphology and relatively free word order is likely to demand large amount of training data in order to develop MT systems that produce adequate and fluent translations. The ParIce corpus currently consists of only 3.5 million sentence pairs which is rather small in relation to parallel corpora in general. The goal of this phase is to create an aligned and filtered parallel corpus of translated documents from the European Economic Area (EEA) domain (e.g. regulations and directives). As of 2017, around 7,000 documents were available in Icelandic with corresponding documents in English. The aim is to pair all accessible documents in the course of the project.","Back-translation. In order to augment the training data, back-translated texts will be used. Monolingual Icelandic texts will be selected and translated to English with one of the baseline system (see below). By doing so, more training data can be obtained for the en$\\rightarrow $is direction. An important part of using back-translated texts during training is filtering out translations that may otherwise lead to poor quality of the augmented part.","Baseline system. In this part, three baseline MT systems will be developed. First, a statistical phrase-based MT system based on Moses BIBREF41, second, a bidirectional LSTM model using the neural translation system OpenNMT BIBREF42, and, third, a system based on an attention-based neural network BIBREF43 using Tensor2Tensor. All the three systems will be trained on ParIce, and the additional data from tasks 1 and 2 above. Eventually, the goal is to choose the best performing MT-system for further development of MT for Icelandic.","MT interface. An API and a web user interface for the three baseline systems, mentioned in item 3 above, will be developed to give interested parties access to the systems under development, and to establish a testing environment in which members of the public can submit their own text. Thus, results from the three systems can be compared directly, as well as to the translations produced by Google Translate. Moreover, in this part, a crowd-sourcing mechanism will be developed, i.e. a functionality to allow users to submit improved translations back to the system for inclusion in the training corpus.","Pre- and postprocessing. Preprocessing in MT is the task of changing the training corpus/source text in some manner for the purpose of making the translation task easier or mark particular words/phrases that should not be translated. Postprocessing is then the task of restoring the generated target language to its normal form. An example of pre- and postprocessing in our project is the handling of named entities (NEs). NEs are found and matched within source and target sentence pairs in the training corpus, and replaced by placeholders with information about case and singular/plural number. NE-to-placeholder substitution is implemented in the input and placeholder-to-NE substitution in the output pipelines of the translation system."]},{"idx":15,"heading":"Conclusion","paragraphs":["We have described a five-year, national LT programme for Icelandic. The goal is to make Icelandic useable in communication and interactions in the digital world. Further, to establish graduate and post-graduate education in LT in Iceland to enable the building of strong knowledge centres in LT in the country.","After studying somewhat similar national programmes in other European countries, we have defined the most important factors that in our opinion will help lead to the success of the programme: First, we have defined core projects that comprise the most important language resources and software tools necessary for various LT applications. Second, all deliverables will be published under as open licenses as possible and all resources and software will be easily accessible. The deliverables will be packaged and published for use in commercial applications, where applicable. Third, from the beginning of the programme, we encourage innovation projects from academia and industry through a competitive R&D fund, and fourth, constant communication with users and industry through conferences, events and direct interaction will be maintained, with the aim of putting deliverables to use in products as soon as possible. The cooperation between academia and industry is also reflected in the consortium of universities, institutions, associations, and private companies, performing the R&D work for all core projects.","The described plan is tied in with 20 years of LT history in Iceland, and despite the steep path to getting where we are, we have every reason to be optimistic about the future of Icelandic LT."]},{"idx":16,"heading":"Figures and Tables","paragraphs":["Table 1: Members of the SÍM consortium for Icelandic LT"]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"2003.13028","title":"Abstractive Summarization with Combination of Pre-trained Sequence-to-Sequence and Saliency Models","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Pre-trained sequence-to-sequence (seq-to-seq) models have significantly improved the accuracy of several language generation tasks, including abstractive summarization. Although the fluency of abstractive summarization has been greatly improved by fine-tuning these models, it is not clear whether they can also identify the important parts of the source text to be included in the summary. In this study, we investigated the effectiveness of combining saliency models that identify the important parts of the source text with the pre-trained seq-to-seq models through extensive experiments. We also proposed a new combination model consisting of a saliency model that extracts a token sequence from a source text and a seq-to-seq model that takes the sequence as an additional input text. Experimental results showed that most of the combination models outperformed a simple fine-tuned seq-to-seq model on both the CNN/DM and XSum datasets even if the seq-to-seq model is pre-trained on large-scale corpora. Moreover, for the CNN/DM dataset, the proposed combination model exceeded the previous best-performed model by 1.33 points on ROUGE-L."]},{"idx":1,"heading":"Introduction","paragraphs":["Pre-trained language models such as BERT BIBREF0 have significantly improved the accuracy of various language processing tasks. However, we cannot apply BERT to language generation tasks as is because its model structure is not suitable for language generation. Several pre-trained seq-to-seq models for language generation BIBREF1, BIBREF2 based on an encoder-decoder Transformer model, which is a standard model for language generation, have recently been proposed. These models have achieved blackstate-of-the-art results in various language generation tasks, including abstractive summarization.","However, when generating a summary, it is essential to correctly predict which part of the source text should be included in the summary. Some previous studies without pre-training have examined combining extractive summarization with abstractive summarization BIBREF3, BIBREF4. Although pre-trained seq-to-seq models have achieved higher accuracy compared to previous models, it is not clear whether modeling “Which part of the source text is important?” can be learned through pre-training.","blackThe purpose of this study is to clarify the blackeffectiveness of combining saliency models that identify the important part of the source text with a pre-trained seq-to-seq model in the abstractive summarization task. Our main contributions are as follows:","We investigated nine combinations of pre-trained seq-to-seq and token-level saliency models, where the saliency models share the parameters with the encoder of the seq-to-seq model or extract important tokens independently of the encoder.","We proposed a new combination model, the conditional summarization model with important tokens (CIT), in which a token sequence extracted by a saliency model is explicitly given to a seq-to-seq model as an additional input text.","We evaluated the combination models on the CNN/DM BIBREF5 and XSum BIBREF6 datasets. Our CIT model outperformed a simple fine-tuned model in terms of ROUGE scores on both datasets."]},{"idx":2,"heading":"Task Definition","paragraphs":["Our study focuses on two tasks: abstractive summarization and blacksaliency detection. The main task is abstractive summarization and the sub task is blacksaliency detection, which is the prediction of important parts of the source text. The problem formulations of each task are described below.","Task 1 (Abstractive summarization) Given the source text $X$, the output is an abstractive summary $Y$ = $(y_1,\\ldots ,y_T)$.","Task 2 (Saliency detection) Given the source text $X$ with $L$ words $X$= $(x_1,\\dots ,x_L)$, the output is the saliency score $S = \\lbrace S_1, S_2, ... S_L \\rbrace $.","In this study, we investigate several combinations of models for these two tasks."]},{"idx":3,"heading":"Pre-trained seq-to-seq Model","paragraphs":["There are several pre-trained seq-to-seq models applied for abstractive summarization BIBREF7, BIBREF8, BIBREF2. The models use a simple Transformer-based encoder-decoder model BIBREF9 in which the encoder-decoder model is pre-trained on large unlabeled data."]},{"idx":4,"heading":"Pre-trained seq-to-seq Model ::: Transformer-based Encoder-Decoder","paragraphs":["In this work, we define the Transformer-based encoder-decoder model as follows."]},{"idx":5,"heading":"Pre-trained seq-to-seq Model ::: Transformer-based Encoder-Decoder ::: Encoder","paragraphs":["The encoder consists of $M$ layer encoder blocks. The input of the encoder is $X = \\lbrace x_i, x_2, ... x_L \\rbrace $. The output through the $M$ layer encoder blocks is defined as","The encoder block consists of a self-attention module and a two-layer feed-forward network."]},{"idx":6,"heading":"Pre-trained seq-to-seq Model ::: Transformer-based Encoder-Decoder ::: Decoder","paragraphs":["The decoder consists of $M$ layer decoder blocks. The inputs of the decoder are the output of the encoder $H_e^M$ and the output of the previous step of the decoder $\\lbrace y_1,...,y_{t-1} \\rbrace $. The output through the $M$ layer Transformer decoder blocks is defined as","In each step $t$, the $h_{dt}^M$ is projected to blackthe vocabulary space and the decoder outputs the highest probability token as the next token. The Transformer decoder block consists of a self-attention module, a context-attention module, and a two-layer feed-forward network."]},{"idx":7,"heading":"Pre-trained seq-to-seq Model ::: Transformer-based Encoder-Decoder ::: Multi-head Attention","paragraphs":["The encoder and decoder blocks use multi-head attention, which consists of a combination of $K$ attention heads and is denoted as $\\mathrm {Multihead}(Q, K, V) = \\mathrm {Concat}(\\mathrm {head}_1, ...,\\mathrm {head}_k)W^o$, where each head is $\\mathrm {head}_i = \\mathrm {Attention}(QW_i^Q, KW_i^K , VW_i^V)$.","The weight matrix $A$ in each attention-head $\\mathrm {Attention}(\\tilde{Q}, \\tilde{K}, \\tilde{V}) = A \\tilde{V}$ is defined as","where $d_k = d / k$, $\\tilde{Q} \\in \\mathbb {R}^{I \\times d}$, $\\tilde{K}, \\tilde{V} \\in \\mathbb {R}^{J \\times d}$.","In the $m$-th layer of self-attention, the same representation $H^m_{\\cdot }$ is given to $Q$, $K$, and $V$. In the context-attention, we give $H^m_d$ to $Q$ and $H^M_e$ to $K$ and $V$."]},{"idx":8,"heading":"Pre-trained seq-to-seq Model ::: Summary Loss Function","paragraphs":["To fine-tune the seq-to-seq model for abstractive summarization, we use cross entropy loss as","where $N$ is the number of training samples."]},{"idx":9,"heading":"Saliency Models","paragraphs":["Several studies have proposed the combination of a token-level saliency model and a seq-to-seq model, blackwhich is not pre-trained, and reported its effectiveness BIBREF3, BIBREF10. We also use a simple token-level saliency model blackas a basic model in this study."]},{"idx":10,"heading":"Saliency Models ::: Basic Saliency Model","paragraphs":["A basic saliency model consists of $M$-layer Transformer encoder blocks ($\\mathrm {Encoder}_\\mathrm {sal}$) and a single-layer feed-forward network. We define the saliency score of the $l$-th token ($1 \\le l \\le L$) in the source text as","where ${\\rm Encoder_{sal}()}$ represents the output of the last layer of black$\\rm Encoder_{sal}$, $W_1 \\in \\mathbb {R}^{d}$ and $b_1$ are learnable parameters, and $\\sigma $ represents a sigmoid function."]},{"idx":11,"heading":"Saliency Models ::: Two Types of Saliency Model for Combination","paragraphs":["In this study, we use two types of saliency model for combination: a shared encoder and an extractor. Each model structure is based on the basic saliency model. We describe them below."]},{"idx":12,"heading":"Saliency Models ::: Two Types of Saliency Model for Combination ::: Shared encoder","paragraphs":["The shared encoder blackshares the parameters of black$\\rm Encoder_{sal}$ and the encoder of the seq-to-seq model. This model is jointly trained with the seq-to-seq model and the saliency score is used to bias the representations of the seq-to-seq model."]},{"idx":13,"heading":"Saliency Models ::: Two Types of Saliency Model for Combination ::: Extractor","paragraphs":["The extractor extracts the important tokens or sentences from the source text on the basis of the saliency score. The extractor is separated with the seq-to-seq model, and each model is trained independently."]},{"idx":14,"heading":"Saliency Models ::: Pseudo Reference Label","paragraphs":["The saliency model predicts blackthe saliency score $S_l$ for each token $x_l$. If there is a reference label $r_l$ $\\in \\lbrace 0, 1\\rbrace $ for each $x_l$, we can train the saliency model in a supervised manner. However, the reference label for each token is typically not given, since the training data for the summarization consists of only the source text and its reference summary. Although there are no reference saliency labels, we can make pseudo reference labels by aligning both source and summary token sequences and extracting common tokens BIBREF3. blackWe used pseudo labels when we train the saliency model in a supervised manner."]},{"idx":15,"heading":"Saliency Models ::: Saliency Loss Function","paragraphs":["To train the saliency model in a supervised way blackwith pseudo reference labels, we use binary cross entropy loss as","where $r_l^n$ is a pseudo reference label of token $x_l$ in the $n$-th sample."]},{"idx":16,"heading":"Combined Models","paragraphs":["This section describes nine combinations of the pre-trained seq-to-seq model and saliency models."]},{"idx":17,"heading":"Combined Models ::: Combination types","paragraphs":["We roughly categorize the combinations into three types. Figure FIGREF23 shows an image of each combination.","The first type uses the shared encoder (§SECREF26). These models consist of the shared encoder and the decoder, where the shared encoder module blackplays two roles: saliency detection and the encoding of the seq-to-seq model. blackThe saliency scores are used to bias the representation of the seq-to-seq model for several models in this type.","The second type uses the extractor (§SECREF34, §SECREF37). These models consist of the extractor, encoder, and decoder and follow two steps: first, blackthe extractor blackextracts the important tokens or sentences from the source text, and second, blackthe encoder uses them as an input of the seq-to-seq models. Our proposed model (CIT) belongs to this type.","The third type uses both the shared encoder and the extractor (§SECREF39). These models consist of the extractor, shared encoder, and decoder and also follow two steps: first, blackthe extractor extracts the important tokens from the source text, and second, blackthe shared encoder uses them as an input of the seq-to-seq model."]},{"idx":18,"heading":"Combined Models ::: Loss function","paragraphs":["From the viewpoint of the loss function, there are two major types of model: those that use the saliency loss (§SECREF21) and those that do not. We also denote the loss function for the seq-to-seq model as $L_\\mathrm {abs}$ and the loss function for the extractor as $L_\\mathrm {ext}$. black$L_\\mathrm {ext}$ is trained with $L_\\mathrm {sal}$, and $L_\\mathrm {abs}$ is trained with $L_\\mathrm {sum}$ or $L_\\mathrm {sum} + L_\\mathrm {sal}$."]},{"idx":19,"heading":"Combined Models ::: Using Shared Encoder to Combine the Saliency Model and the Seq-to-seq Model ::: Multi-Task (MT)","paragraphs":["This model trains the shared encoder and the decoder by minimizing both the summary and saliency losses. The loss function of this model is $L_\\mathrm {abs} = L_\\mathrm {sum} + L_\\mathrm {sal}$."]},{"idx":20,"heading":"Combined Models ::: Using Shared Encoder to Combine the Saliency Model and the Seq-to-seq Model ::: Selective Encoding (SE)","paragraphs":["This model uses the saliency score to weight the shared encoder output. Specifically, the final output $h_{el}^M$ of the shared encoder is weighted as","Then, we replace the input of the decoder $h_{el}^M$ with $\\tilde{h}_{el}^{M}$. Although BIBREF10 used BiGRU, we use Transformer for fair comparison. The loss function of this model is $L_\\mathrm {abs} = L_\\mathrm {sum}.$"]},{"idx":21,"heading":"Combined Models ::: Using Shared Encoder to Combine the Saliency Model and the Seq-to-seq Model ::: Combination of SE and MT","paragraphs":["This model has the same structure as the SE. The loss function of this model is $L_\\mathrm {abs} = L_\\mathrm {sum} + L_\\mathrm {sal}$."]},{"idx":22,"heading":"Combined Models ::: Using Shared Encoder to Combine the Saliency Model and the Seq-to-seq Model ::: Selective Attention (SA)","paragraphs":["This model weights the attention scores of the decoder side, unlike the SE model. Specifically, the attention score $a_{i}^{t} \\in \\mathbb {R}^L$ in each step $t$ is weighted by $S_l$. $a_{i}^{t}$ is a $t$-th row of $A_i \\in \\mathbb {R}^{T \\times L}$, which is a weight matrix of the $i$-th attention head in the context-attention (Eq. (DISPLAY_FORM12)).","BIBREF3 took a similar approach in that their model weights the copy probability of a pointer-generator model. However, as the pre-trained seq-to-seq model does not have a copy mechanism, we weight the context-attention for all Transformer decoder blocks. The loss function of this model is $L_\\mathrm {abs} = L_\\mathrm {sum}$."]},{"idx":23,"heading":"Combined Models ::: Using Shared Encoder to Combine the Saliency Model and the Seq-to-seq Model ::: Combination of SA and MT","paragraphs":["This model has the same structure as the SA. The loss function of this model is $L_\\mathrm {abs} = L_\\mathrm {sum} + L_\\mathrm {sal}$."]},{"idx":24,"heading":"Combined Models ::: Using the Extractor to Refine the Input Text ::: Sentence Extraction then Generation (SEG)","paragraphs":["This model first extracts the saliency sentences on the basis of a sentence-level saliency score $S_j$. $S_j$ is calculated by using the token level saliency score of the extractor, $S_l$, as","where $N_j$ and $X_j$ are the number of tokens and the set of tokens within the $j$-th sentence. Top $P$ sentences are extracted according to the sentence-level saliency score and then concatenated as one text $X_s$. These extracted sentences are then used as the input of the seq-to-seq model.","In the training, we extracted $X_s$, which maximizes the ROUGE-L scores with the reference summary text. In the test, we used the average number of sentences in $X_{s}$ in the training set as $P$. The loss function of the extractor is $L_\\mathrm {ext} = L_\\mathrm {sal}$, and that of the seq-to-seq model is $L_\\mathrm {abs} = L_\\mathrm {sum}$."]},{"idx":25,"heading":"Combined Models ::: Proposed: Using Extractor to Extract an Additional Input Text ::: Conditional Summarization Model with Important Tokens","paragraphs":["We propose a new combination of the extractor and the seq-to-seq model, CIT, which can consider important tokens explicitly. Although the SE and SA models softly weight the representations of the source text or attention scores, they cannot select salient tokens explicitly. SEG explicitly extracts the salient sentences from the source text, but it cannot give token-level information to the seq-to-seq model, and it sometimes drops important information when extracting sentences. In contrast, CIT uses the tokens extracted according to saliency scores as an additional input of the seq-to-seq model. By adding token-level information, CIT can effectively guide the abstractive summary without dropping any important information.","Specifically, $K$ tokens $C = \\lbrace c_1, ..., c_K \\rbrace $ are extracted in descending order of the saliency score $S$. $S$ is obtained by inputting $X$ to the extractor. The order of $C$ retains the order of the source text $X$. A combined text $\\tilde{X} = \\mathrm {Concat}(C, X)$ is given to the seq-to-seq model as the input text. The loss function of the extractor is $L_\\mathrm {ext} = L_\\mathrm {sal}$, and that of the seq-to-seq model is $L_\\mathrm {abs} = L_\\mathrm {sum}$."]},{"idx":26,"heading":"Combined Models ::: Proposed: Combination of Extractor and Shared Encoder ::: Combination of CIT and SE","paragraphs":["This model combines the CIT and SE, so CIT uses an extractor for extracting important tokens, and SE is trained by using a shared encoder in the seq-to-seq model. The SE model is trained in an unsupervised way. The output $H^M_e \\in \\mathbb {R}^{L+K}$ of the shared encoder is weighted by saliency score $S \\in \\mathbb {R}^{L+K}$ with Eq. (DISPLAY_FORM29), where $S$ is estimated by using the output of the shared encoder with Eq. (DISPLAY_FORM16). The loss function of the extractor is $L_\\mathrm {ext} = L_\\mathrm {sal}$, and that of the seq-to-seq model is $L_\\mathrm {abs} = L_\\mathrm {sum}$."]},{"idx":27,"heading":"Combined Models ::: Proposed: Combination of Extractor and Shared Encoder ::: Combination of CIT and SA","paragraphs":["This model combines the CIT and SA, so we also train two saliency models. The SA model is trained in an unsupervised way, the same as the CIT + SE model. The attention score $a_i^t \\in \\mathbb {R}^{L+K}$ is weighted by $S \\in \\mathbb {R}^{L+K}$ with Eq. (DISPLAY_FORM32). The loss function of the extractor is $L_\\mathrm {ext} = L_\\mathrm {sal}$, and that of the seq-to-seq model is $L_\\mathrm {abs} = L_\\mathrm {sum}$."]},{"idx":28,"heading":"Experiments ::: Dataset","paragraphs":["We used the CNN/DM dataset BIBREF5 and the XSum dataset BIBREF6, which are both standard datasets for news summarization. The details of the two datasets are listed in Table TABREF48. The CNN/DM is a highly extractive summarization dataset and the XSum is a highly abstractive summarization dataset."]},{"idx":29,"heading":"Experiments ::: Dataset ::: Model Configurations","paragraphs":["blackWe used BART$_{\\mathrm {LARGE}}$ BIBREF1, which is one of the state-of-the-art models, as the pre-trained seq-to-seq model and RoBERTa$_\\mathrm {BASE}$ BIBREF11 as the initial model of the extractor. In the extractor of CIT, stop words and duplicate tokens are ignored for the XSum dataset.","We used fairseq for the implementation of the seq-to-seq model. For fine-tuning of BART$_\\mathrm {LARGE}$ and the combination models, we used the same parameters as the official code. For fine-tuning of RoBERTa$_\\mathrm {BASE}$, we used Transformers. We set the learning rate to 0.00005 and the batch size to 32."]},{"idx":30,"heading":"Experiments ::: Evaluation Metrics","paragraphs":["We used ROUGE scores (F1), including ROUGE-1 (R-1), ROUGE-2 (R-2), and ROUGE-L (R-L), as the evaluation metrics BIBREF12. ROUGE scores were calculated using the files2rouge toolkit."]},{"idx":31,"heading":"Experiments ::: Results ::: Do saliency models improve summarization accuracy in highly extractive datasets?","paragraphs":["Rouge scores of the combined models on the CNN/DM dataset are shown in Table TABREF51. We can see that all combined models outperformed the simple fine-tuned BART. This indicates that the saliency detection is effective in highly extractive datasets. One of the proposed models, CIT + SE, achieved the highest accuracy. The CIT model alone also outperformed other saliency models. This indicates that the CIT model effectively guides the abstractive summarization by combining explicitly extracted tokens."]},{"idx":32,"heading":"Experiments ::: Results ::: Do saliency models improve summarization accuracy in highly abstractive datasets?","paragraphs":["Rouge scores of the combined models on the XSum dataset are shown in Table TABREF52. The CIT model performed the best, although its improvement was smaller than on the CNN/DM dataset. Moreover, the accuracy of the MT, SE + MT, and SEG models decreased on the XSum dataset. These results were very different from those on the CNN/DM dataset.","One reason for the difference can be traced to the quality of the pseudo saliency labels. CNN/DM is a highly extractive dataset, so it is relatively easy to create token alignments for generating pseudo saliency labels, while in contrast, a summary in XSum is highly abstractive and short, which makes it difficult to create pseudo labels with high quality by simple token alignment. To improve the accuracy of summarization in this dataset, we have to improve the quality of the pseudo saliency labels and the accuracy of the saliency model."]},{"idx":33,"heading":"Experiments ::: Results ::: How accurate are the outputs of the extractors?","paragraphs":["We analyzed the quality of the tokens extracted by the extractor in CIT. The results are summarized in Table TABREF55. On the CNN/DM dataset, the ROUGE-1 and ROUGE-2 scores of our extractor (Top-$K$ tokens) were higher than other models, while the ROUGE-L score was lower than the other sentence-based extraction method. This is because that our token-level extractor finds the important tokens whereas the seq-to-seq model learns how to generate a fluent summary incorporating these important tokens.","On the other hand, the extractive result on the XSum dataset was lower. For highly abstractive datasets, there is little overlap between the tokens. We need to consider how to make the high-quality pseudo saliency labels and how to evaluate the similarity of these two sequences."]},{"idx":34,"heading":"Experiments ::: Results ::: Does the CIT model outperform other fine-tuned models?","paragraphs":["Our study focuses on the combinations of saliency models and the pre-trained seq-to-seq model. However, there are several studies that focus more on the pre-training strategy. We compared the CIT model with those models. Their ROUGE scores are shown in Tables TABREF57 and TABREF58. From Table TABREF57, we can see that our model outperformed the recent pre-trained models on the CNN/DM dataset. Even though PEGASUS$_\\mathrm {HugeNews}$ was pre-trained on the largest corpus comprised of news-like articles, the accuracy of abstractive summarization was not improved much. Our model improved the accuracy without any additional pre-training. This result indicates that it is more effective to combine saliency models with the seq-to-seq model for generating a highly extractive summary.","On the other hand, on the XSum dataset, PEGASUS$_\\mathrm {HugeNews}$ improved the ROUGE scores and achieved the best results. In the XSum dataset, summaries often include the expressions that are not written in the source text. Therefore, increasing the pre-training data and learning more patterns were effective. However, by improving the quality of the pseudo saliency labels, we should be able to improve the accuracy of the CIT model."]},{"idx":35,"heading":"Related Work and Discussion ::: Pre-trained Language Models for Abstractive Summarization","paragraphs":["BIBREF18 used BERT for their sentence-level extractive summarization model. BIBREF19 proposed a new pre-trained model that considers document-level information for sentence-level extractive summarization. Several researchers have published pre-trained encoder-decoder models very recently BIBREF20, BIBREF1, BIBREF2. BIBREF20 pre-trained a Transformer-based pointer-generator model. BIBREF1 pre-trained a standard Transformer-based encoder-decoder model using large unlabeled data and achieved state-of-the-art results. BIBREF8 and BIBREF16 extended the BERT structure to handle seq-to-seq tasks.","All the studies above focused on how to learn a universal pre-trained model; they did not consider the combination of pre-trained and saliency models for an abstractive summarization model."]},{"idx":36,"heading":"Related Work and Discussion ::: Abstractive Summarization with Saliency Models","paragraphs":["BIBREF4, BIBREF3, and BIBREF21 incorporated a sentence- and word-level extractive model in the pointer-generator model. Their models weight the copy probability for the source text by using an extractive model and guide the pointer-generator model to copy important words. BIBREF22 proposed a keyword guided abstractive summarization model. BIBREF23 proposed a sentence extraction and re-writing model that is trained in an end-to-end manner by using reinforcement learning. BIBREF24 proposed a search and rewrite model. BIBREF25 proposed a combination of sentence-level extraction and compression. None of these models are based on a pre-trained model. In contrast, our purpose is to clarify whether combined models are effective or not, and we are the first to investigate the combination of pre-trained seq-to-seq and saliency models. We compared a variety of combinations and clarified which combination is the most effective."]},{"idx":37,"heading":"Conclusion","paragraphs":["This is the first study that has conducted extensive experiments to investigate the effectiveness of incorporating saliency models into the pre-trained seq-to-seq model. From the results, we found that saliency models were effective in finding important parts of the source text, even if the seq-to-seq model is pre-trained on large-scale corpora, especially for generating an highly extractive summary. We also proposed a new combination model, CIT, that outperformed simple fine-tuning and other combination models. Our combination model improved the summarization accuracy without any additional pre-training data and can be applied to any pre-trained model. While recent studies have been conducted to improve summarization accuracy by increasing the amount of pre-training data and developing new pre-training strategies, this study sheds light on the importance of saliency models in abstractive summarization."]},{"idx":38,"heading":"Figures and Tables","paragraphs":["Figure 1: Combinations of seq-to-seq and saliency models. Purple: Encoder. Blue: Decoder. Red: Shared encoder, which is a shared model for saliency detection and encoding, used in (a), (b), and (e). Yellow: Extractor, which is an independent saliency model to extract important (c) sentences Xs or (d), (e) tokens C from the source text X . Each of these colored blocks represents M -layer Transformer blocks. Gray: Linear transformation. Green: Context attention. Pink: Output trained in a supervised manner, where S is the saliency score and Y is the summary.","Table 1: Details of the datasets used in this paper.","Table 2: Results of BART and combined models on CNN/DM dataset. Five row-groups are the models described in §3, §5.1, §5.2, §5.3, and §5.4 in order from top to bottom.","Table 3: Results of BART and combined models on XSum dataset. The underlined result represents the best result among the models that outperformed our simple fine-tuning result.","Table 5: Results of state-of-the-art models and the proposed model on CNN/DM dataset. We also report the size of pre-training data and parameters utilized for each model. 1(Song et al., 2019); 2(Liu and Lapata, 2019); 3(Dong et al., 2019); 4(Raffel et al., 2019); 5(Lewis et al., 2019); 6(Zhang et al., 2019a) 7(Yan et al., 2020); 8(Xiao et al., 2020); 9(Bao et al., 2020)","Table 4: Results of saliency models on CNN/DM and XSum datasets. CIT extracted the top-K tokens or top3 sentences from the source text. 1(Liu and Lapata, 2019). The summaries in XSum are highly extractive, so the result of BertSumExt for XSum was not reported.","Table 6: Results of state-of-the-art models and the proposed model on XSum dataset. 1(Song et al., 2019); 2(Liu and Lapata, 2019); 3(Lewis et al., 2019); 4(Zhang et al., 2019a); 5(Bao et al., 2020)"]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"2004.01853","title":"STEP: Sequence-to-Sequence Transformer Pre-training for Document Summarization","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Abstractive summarization aims to rewrite a long document to its shorter form, which is usually modeled as a sequence-to-sequence (Seq2Seq) learning problem. Seq2Seq Transformers are powerful models for this problem. Unfortunately, training large Seq2Seq Transformers on limited supervised summarization data is challenging. We, therefore, propose STEP (as shorthand for Sequence-to-Sequence Transformer Pre-training), which can be trained on large scale unlabeled documents. Specifically, STEP is pre-trained using three different tasks, namely sentence reordering, next sentence generation, and masked document generation. Experiments on two summarization datasets show that all three tasks can improve performance upon a heavily tuned large Seq2Seq Transformer which already includes a strong pre-trained encoder by a large margin. By using our best task to pre-train STEP, we outperform the best published abstractive model on CNN/DailyMail by 0.8 ROUGE-2 and New York Times by 2.4 ROUGE-2."]},{"idx":1,"heading":"Introduction","paragraphs":["Large pre-trained language models BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4 improved the state-of-the-art of various natural language understanding (NLU) tasks such as question answering (e.g., SQuAD; BIBREF5), natural language inference (e.g., MNLI; BIBREF6) as well as text classification BIBREF7. These models (i.e., large LSTMs; BIBREF8 or Transformers; BIBREF9) are pre-trained on large scale unlabeled text with language modeling BIBREF0, BIBREF1, masked language modeling BIBREF2, BIBREF4 and permutation language modeling BIBREF3 objectives. In NLU tasks, pre-trained language models are mostly used as text encoders.","Abstractive document summarization aims to rewrite a long document to its shorter form while still retaining its important information. Different from extractive document summarization that extacts important sentences, abstractive document summarization may paraphrase original sentences or delete contents from them. For more details on differences between abstractive and extractive document summary, we refer the interested readers to Nenkova:McKeown:2011 and Section SECREF2. This task is usually framed as a sequence-to-sequence learning problem BIBREF10, BIBREF11. In this paper, we adopt the sequence-to-sequence (seq2seq) Transformer BIBREF9, which has been demonstrated to be the state-of-the-art for seq2seq modeling BIBREF9, BIBREF12. Unfortunately, training large seq2seq Transformers on limited supervised summarization data is challenging BIBREF12 (refer to Section SECREF5). The seq2seq Transformer has an encoder and a decoder Transformer. Abstractive summarization requires both encoding of an input document and generation of a summary usually containing multiple sentences. As mentioned earlier, we can take advantage of recent pre-trained Transformer encoders for the document encoding part as in liu2019text. However, liu2019text leave the decoder randomly initialized. In this paper, we aim to pre-train both the encoder (i.e., the encoding part) and decoder (i.e., the generation part) of a seq2seq Transformer , which is able to improve abstractive summarization performance.","Based on the above observations, we propose Step (as shorthand for Sequence-to-Sequence TransformEr Pre-training), which can be pre-trained on large scale unlabeled documents. Specifically, we design three tasks for seq2seq model pre-training, namely Sentence Reordering (SR), Next Sentence Generation (NSG), and Masked Document Generation (MDG). SR learns to recover a document with randomly shuffled sentences. NSG generates the next segment of a document based on its preceding segment. MDG recovers a masked document to its original form. After pre-trianing Step using the three tasks on unlabeled documents, we fine-tune it on supervised summarization datasets.","We evaluate our methods on two summarization datasets (i.e., the CNN/DailyMail and the New York Times datasets). Experiments show that all three tasks we propose can improve upon a heavily tuned large seq2seq Transformer which already includes a strong pre-trained encoder by a large margin. Compared to the best published abstractive models, Step improves the ROUGE-2 by 0.8 on the CNN/DailyMail dataset and by 2.4 on the New York Times dataset using our best performing task for pre-training. Human experiments also show that Step can produce significantly better summaries in comparison with recent strong abstractive models."]},{"idx":2,"heading":"Related Work","paragraphs":["This section introduces extractive and abstractive document summarization as well as pre-training methods for natural language processing tasks."]},{"idx":3,"heading":"Related Work ::: Extractive Summarization","paragraphs":["Extractive summarization systems learn to find the informative sentences in a document as its summary. This task is usually viewed as a sentence ranking problem BIBREF13, BIBREF14 using scores from a binary (sequence) classification model, which predicts whether a sentence is in the summary or not. Extractive neural models employ hierarchical LSTMs/CNNs as the feature learning part of the binary (sequence) classifier BIBREF15, BIBREF16, BIBREF17, BIBREF18, which largely outperforms discrete feature based models BIBREF19, BIBREF20, BIBREF21. Very recently, the feature learning part was replaced again with pre-trained transformers BIBREF22, BIBREF23 that lead to another huge improvement of summarization performance. However, extractive models have their own limitations. For example, the extracted sentences might be too long and redundant. Besides, human written summaries in their nature are abstractive. Therefore, we focus on abstractive summarization in this paper."]},{"idx":4,"heading":"Related Work ::: Abstractive Summarization","paragraphs":["The goal of abstractive summarization is to generate summaries by rewriting a document, which is a sequence-to-sequence learning problem. seq2seq attentive LSTMs BIBREF8, BIBREF24 are employed in nallapati2016abstractive. Even these models are extended with copy mechanism BIBREF25, coverage model BIBREF11 and reinforcement learning BIBREF26, their results are still very close to that of Lead3 which selects the leading three sentences of a document as its summary. One possible reason is that LSTMs without pre-training are not powerful enough. liu2019text used a seq2seq Transformer model with its encoder initialized with a pre-trained Transformer (i.e., BERT; BIBREF2) and achieved the state-of-the-art performance. Our work goes one step further, we propose a method to pre-train the decoder together with the encoder and then initialize both the encoder and decoder of a summarization model with the pre-trained Transformers.","There is also a line of work that bridges extractive and abstractive models with reinforcement learning BIBREF27, attention fusion BIBREF28 and bottom-up attention BIBREF29, while our model is conceptually simpler."]},{"idx":5,"heading":"Related Work ::: Pre-training","paragraphs":["Pre-training methods draw a lot of attention recently. peters2018deep and radford:2019:arxiv pre-trained LSTM and Transformer encoders using language modeling objectives. To leverage the context in both directions, BIBREF2 proposed BERT, which is trained with the mask language modeling objective. XLNet BIBREF3 is trained with permutation language modeling objective, which removes the independence assumption of masked tokens in BERT. RoBERTa BIBREF4 extends BERT with more training data and better training strategies. All the methods above focus on pre-training an encoder, while we propose methods to pre-train both the encoder and decoder of a seq2seq model.","dong2019unified proposed a Transformer language model that can be used for both natural language understanding and generation tasks, which is pre-trained using masked, unidirectional and seq2seq language modeling objectives. Their method tries to pre-train a seq2seq Transformer with its encoder and decoder parameters shared. Differently, we pre-train a seq2seq Transformer with separate parameters for the encoder and decoder. song2019mass proposed a method to pre-train a seq2seq Transformer by masking a span of text and then predicting the original text with masked tokens at other positions. Their pre-training task is similar to our Masked Document Generation task, but we apply a different masking strategy and predict the original text without masked tokens. Besides, we propose another two tasks for seq2seq model pre-training. BIBREF30 tested their model on sentence-level tasks (e.g., machine translation and sentence compression), while we aim to solve document-level tasks (e.g., abstractive document summarization)."]},{"idx":6,"heading":"Sequence-to-Sequence Transformer Pre-training","paragraphs":["This section first introduces the backbone architecture of our abstractive summarization model Step. We then describe methods to pre-train Step and finally move on to the fine-tuning on summarization datasets."]},{"idx":7,"heading":"Sequence-to-Sequence Transformer Pre-training ::: Architecture","paragraphs":["In this work, the task of abstractive document summarization is modeled as a sequence-to-sequence learning problem, where a document is viewed as a sequence of tokens and its corresponding summary as another sequence of tokens. We adopt the seq2seq Transformer architecture BIBREF9, which includes an encoder Transformer and a decoder Transformer. Both the encoder and decoder Transformers have multiple layers and each layer contains a multi-head attentive sub-layer followed by a fully connected sub-layer with residual connections BIBREF31 and layer normalization BIBREF32.","Let us use $X = (x_1, x_2, \\dots , x_{|X|})$ to denote a document and use $Y = (y_1, y_2, \\dots , y_{|Y|})$ to denote its summary. The encoder takes the document $X$ as input and transforms it to its contextual representations. The decoder learns to generate the summary $Y$ one token at a time based on the contextual representations and all preceding tokens that have been generated so far:","where $y_{<t}$ stands for all tokens before position $t$ (i.e., $y_{<t}=(y_1, y_2, \\dots , y_{t-1})$). This model can be trained by minimizing the negative log-likelihood of the training document-summary pairs."]},{"idx":8,"heading":"Sequence-to-Sequence Transformer Pre-training ::: Pre-training Tasks","paragraphs":["Training a seq2seq Transformer model on a summarization dataset from scratch is difficult due to the limited number of document-summary pairs. Pre-trained Transformer encoders such as BERT BIBREF2 and RoBERTa BIBREF4 have achieved great success in many natural language understanding tasks. Therefore, we first initialize the encoder of our seq2seq Transformer summarization model Step with an existing pre-trained Transformer encoder (i.e., RoBERTa) to enhance its language understanding capabilities. To help Step gain language generation capabilities and the abilities of associating generated text with encoder outputs, we continue to pre-train it on unlabeled text. In the following, we describe our pre-training tasks."]},{"idx":9,"heading":"Sequence-to-Sequence Transformer Pre-training ::: Pre-training Tasks ::: Sentence Reordering","paragraphs":["A document is typically composed of multiple sentences separated by full stops. In this task, we first shuffle the document by sentences and then recover the original document. There are several reasons why we design this task. First, a summary of a document usually consists of multiple sentences. We expect that Step learns to generate long and coherent summaries (across sentences). The output of the task (i.e., the original document) also contains multiple sentences. Second, sentence reordering (or content reordering) is necessary for summarization. According to the statistics on training sets of our summarization datasets, contents of the original documents are reordered in their summaries for 40% of cases. We define content reordering as follows. For each document-summary pair, we first map each sentence in the summary to one sentence in its paired document by maximizing the ROUGE score. If the sequence of sentences in the summary is different from the sequence of their mapped sentences in the original document, we count this as one content reordering. Thirdly, abstractive summary requires reproducing factual details (e.g., named entities, figures) from source text. We also expect Step to learn to copy tokens. Here is a formal definition of this task. Let us change the notation of a document slightly in this paragraph. Let $X=(S_1, S_2, \\dots , S_m)$ denote a document, where $S_i = (w^i_1, w^i_2, \\dots , w^i_{|S_i|})$ is a sentence in it, $w^i_j$ is a word in $S_i$ and $m$ is the number of sentences. $X$ is still a sequence of tokens (by concatenating tokens in all sentences). Let $A=\\text{\\tt permutation}(m)=(a_1,a_2,\\dots , a_m)$ denote a permuted range of $(1, 2, \\dots , m)$ and therefore $\\hat{X}_S=(S_{a_1}, S_{a_2}, \\dots , S_{a_m})$ is the shuffled document. Note that $\\hat{X}_S$ is a sequence of tokens by concatenating all shuffled sentences. Step can be trained on $\\langle \\hat{X}_S, X \\rangle $ pairs constructed from unlabeled documents, as demonstrated in Figure FIGREF5.","Note that document rotation is a special case of sentence reordering with significant amount of partially ordered sentences, which we believe is a simpler task. In this work, we thus only consider the general case of sentence reordering."]},{"idx":10,"heading":"Sequence-to-Sequence Transformer Pre-training ::: Pre-training Tasks ::: Next Sentence Generation","paragraphs":["The second pre-training task leverages the natural order of text. Next Sentence Generation (NSG) uses one span of text in a document to predict its next span of text, as shown in Figure FIGREF5. Specifically, we split a document into two segments (i.e., $G_1$ and $G_2$). Note that each segment might contain multiple sentences, which fits the document summarization task very well, since either a document or its summary usually includes multiple sentences. Intuitively, in a document, sentences are highly correlated with their preceding sentences due to the context dependent nature of documents or language. We intend our model to learn to generate multiple sentences and also learn to focus on preceding context.","We have at least two options for the splitting position of the two segments. Option one: the position right after a full-stop symbol (such as period, question mark, etc.) is selected as the splitting point, which ensures full sentences for each segment. Option two: the splitting point can be at any position within the document. We choose the second option, which may lead to incomplete sentences in segments. We intend to force the encoder and decoder to understand input text without complete information, which we believe is more challenging compared to option one. Besides, as a common wisdom in abstractive summarization, documents are truncated to a fixed number of tokens, which may also contain incomplete sentences. We use option two to reduce the pre-training and fine-tuning input mismatch. In this task, we train the model Step on large amount of $\\langle G_1, G_2\\rangle $ pairs constructed following the option two splitting strategy.","Next sentence prediction has been used in skip-thought vectors BIBREF33. There are two differences. First, each segment in their model only has one sentence; second, they use this task to pre-train an encoder rather than an entire seq2seq model. BIBREF2 introduced a task named next sentence prediction (NSP), which is different from this task. NSP is a classification task, but NSG is a generation task, which intends to pre-train a generation model."]},{"idx":11,"heading":"Sequence-to-Sequence Transformer Pre-training ::: Pre-training Tasks ::: Masked Document Generation","paragraphs":["The third task we consider is Masked Document Generation (MDG) that learns to recover a document with a masked span of tokens (see Figure FIGREF5). For simplicity, a document consisting of a sequence of tokens is denoted as $X=(x_1, x_2, \\cdots , x_{|X|})$. We randomly sample the length of the span $l$ from a discrete uniform distribution $\\mathcal {U}(a, b)$ and the span start position $k$ from another discrete uniform distribution $\\mathcal {U}(1, |X|-l+1)$ (see Section SECREF4 for more details). Thus, $\\mathcal {M}=(x_k, x_{k+1}, \\cdots , x_{k+l-1})$ is the text span to be masked.","One straightforward masking strategy is to replace each token residing in $\\mathcal {M}$ with a special [MASK] token. However, we refrain from doing so because of the following three reasons. Usually, [MASK] tokens will not appear in downstream tasks. Second, entirely masking a continuous sub-sequence of $X$ may make the whole document incomprehensible, which might be too challenging for our model to learn. Third, similar to SR, avoiding replacing every token with [MASK] also helps our model learn the ability of copying tokens from the input while preserving the ability of generating novel tokens.","In the sub-sequence $\\mathcal {M}$, each token is processed with one of the three strategies: 1) replaced with the [MASK] token; 2) replaced with a random token; 3) remain unchanged. Inspired by BERT BIBREF2, for 80% tokens, we follow strategy 1). In 10% of cases, we employ strategy 2) and we use strategy 3) for the remaining 10% of cases. Let $\\hat{X}_M$ denote the document after the application of our masking strategy. We could create infinite amount of $\\langle \\hat{X}_M,X\\rangle $ pairs to train Step.","During pre-training, we could also employ all the three tasks (i.e., SR, NSG, MDG) together. For each training batch, we randomly choose one task and each task is used for $1/3$ of the time."]},{"idx":12,"heading":"Sequence-to-Sequence Transformer Pre-training ::: Fine-tuning","paragraphs":["After pre-training Step with the three tasks introduced in Section SECREF9, we fine-tune the model on abstractive document summarization datasets. The fine-tuning process is straightforward. We simply continue to train Step on the supervised document-summary pairs. Similar to other seq2seq summarization models, we do beam search during the generation of summaries."]},{"idx":13,"heading":"Experimental Setup","paragraphs":["In this section, we present the experimental setup for evaluating our summarization models. We first introduce the datasets used for our experiments. Then we describe training details of our models as well as our evaluation protocols."]},{"idx":14,"heading":"Experimental Setup ::: Datasets","paragraphs":["We assess the summarization performance of our models on two benchmark datasets: the CNN/DailyMail (CNNDM) dataset BIBREF34, BIBREF11 and the New York Times (NYT) dataset BIBREF35. We pre-train our models on the GIGA-CM dataset introduced in zhang-etal-2019-hibert."]},{"idx":15,"heading":"Experimental Setup ::: Datasets ::: CNNDM","paragraphs":["CNNDM contains news articles and their associated highlights (i.e., summaries) collected from the CNN and Daily Mail Online websites. Following previous work BIBREF11, BIBREF22, BIBREF23, we use the non-anonymized version of CNNDM. Specifically, we preprocess the dataset with the publicly available scripts provided by see2017get and obtain 287,226 document-summary pairs for training, 13,368 for validation and 11,490 for test."]},{"idx":16,"heading":"Experimental Setup ::: Datasets ::: NYT","paragraphs":["The NYT dataset is a collection of articles along with multi-sentence summaries written by library scientists. We closely follow the preprocessing procedures described in durrett2016learning and liu2019text. The test set is constructed by including all articles published on January 1, 2017 or later, which contains 9,076 articles. The remaining 100,834 articles are split into a training set of 96,834 examples and a validation set of 4,000 examples. As in BIBREF36, we also remove articles whose summaries contain less than 50 words from the test set, and the resulting test set contains 3,452 examples."]},{"idx":17,"heading":"Experimental Setup ::: Datasets ::: GIGA-CM","paragraphs":["To pre-train our model with the tasks introduced in Section SECREF9, following the procedures in BIBREF22, we created the GIGA-CM dataset, which contains only unlabeled documents. The training set of GIGA-CM is composed of 6,521,658 documents sampled from the English Gigaword dataset and the training documents in CNNDM. We used the 13,368 documents in the validation split of CNNDM as its validation set. Note that the Gigaword dataset overlaps with the NYT dataset and we therefore exclude the test set of NYT from the training set of GIGA-CM.","For CNNDM, NYT and GIGA-CM datasets, we segment and tokenize documents and/or summaries (GIGA-CM only contains documents) using the Stanford CoreNLP toolkit BIBREF37. To reduce the vocabulary size, we further apply the UTF8 based BPE BIBREF38 introduced in GPT-2 BIBREF39 to all datasets. As a common wisdom in abstractive summarization, documents and summaries in CNNDM and NYT are usually truncated to 512 and 256 tokens, respectively.","We leverage unlabeled documents differently for different pre-training tasks (see Section SECREF9). We first split each document into 512 token segments if it contains more than 512 tokens (segments or documents with less than 512 tokens are removed). In Sentence Reordering (SR) and Masked Document Generation (MDG), we use the segment after transformation to predict the original segment. We set the minimum masked length $a=100$ and the maximum masked length $b=256$ in MDG. In Next Sentence Generation (NSG), each segment is used to predict its next 256 tokens."]},{"idx":18,"heading":"Experimental Setup ::: Implementation Details","paragraphs":["As mentioned in Section SECREF3, our model is a Seq2Seq Transformer model BIBREF9. The encoder is initialized with the $\\text{RoBERTa}_{\\text{LARGE}}$ model BIBREF4, and therefore they share the same architecture. Specifically, the encoder is a 24-layer Transformer. Each layer has 16 attention heads and its hidden size and feed-forward filter size are 1,024 and 4,096, respectively. The decoder is shallower with 6 layers. The hidden size and number of attention head of the decoder are identical to these of the encoder, but the feed-forward filter size is 2,048. We use a smaller filter size in the decoder to reduce the computational and memory cost. The dropout rates of all layers in the encoder are set to 0.1 and all dropout rates in the decoder are set to 0.3. Our models are optimized using Adam BIBREF40 with $\\beta _1=0.9$, $\\beta _2=0.98$. The other optimization hyper-parameters for pre-training and fine-tuning are different. In the pre-training stage, the encoder is initialized with a pre-trained model while the decoder is randomly initialized. Therefore, we used two separate optimizers for the encoder and decoder with a smaller learning rate for the encoder optimizer. Learning rates of the encoder and decoder are set to $2e-5$ and $1e-4$ with 10,000 warmup steps, respectively. We also adopted the same learning rate schedule strategies as BIBREF9. We used smaller batch sizes for datasets with less examples (i.e., 1,024 for GIGA-CM, 256 for CNNDM and 128 for NYT) to ensure each epoch has sufficient number of model updates. We trained our models until their convergence of validation perplexities (around 30 epochs on GIGA-CM, 60 epochs on CNNDM and 40 epochs on NYT). One epoch on GIGA-CM takes around 24 hours with 8 Nvidia Tesla V100 GPUs. The time costs for different pre-training tasks are close.","Most of the hyper-parameters in the fine-tuning stage are the same as these in the pre-training stage. The differences are as follows. The learning rates for both the encoder and decoder are set to $2e-5$ with 4,000 warmup steps, since both the encoder and decoder are already pre-trained. We trained our models for 50 epochs (saved per epoch) and selected the best model w.r.t. ROUGE score on the validation set . During decoding, we applied beam search with beam size of 5. Following BIBREF26, we also blocked repeating trigrams during beam search and tuned the minimum summary length on the validation set. Similar to the pre-training process, the datasets with less instances were fine-tuned with smaller batch size (i.e., 768 for CNNDM and 64 for NYT)."]},{"idx":19,"heading":"Experimental Setup ::: Evaluations","paragraphs":["We used ROUGE BIBREF41 to measure the quality of different summarization model outputs. We reported full-length F1 based ROUGE-1, ROUGE-2 and ROUGE-L scores on CNNDM, while we used the limited-length recall based ROUGE-1, ROUGE-2 and ROUGE-L on NYT following BIBREF36. The ROUGE scores are computed using the ROUGE-1.5.5.pl script.","Since summaries generated by abstractive models may produce disfluent or ungrammatical outputs, we also evaluated abstractive systems by eliciting human judgements. Following previous work BIBREF15, BIBREF17, 20 documents are randomly sampled from the test split of CNNDM. Participants are presented with a document and a list of outputs generated by different abstractive summarization systems. Then they are asked to rank the outputs according to informativeness (does the summary capture the informative part of the document?), fluency (is the summary grammatical?), and succinctness (does the summary express the document clearly in a few words?)"]},{"idx":20,"heading":"Results ::: Automatic Evaluation","paragraphs":["The results on the CNNDM are summarized in Table TABREF25. The first and second blocks show results of previous extractive and abstractive models, respectively. Results of Step are all listed in the third block. Lead3 is a baseline which simply takes the first three sentences of a document as its summary. BERTExt BIBREF23 is an extractive model fine-tuning on BERT BIBREF2 that outperforms other extractive systems. PTGen BIBREF11, DRM BIBREF26, and DCA BIBREF42 are sequence-to-sequence learning based models extended with copy and coverage mechanism, reinforcement learning, and deep communicating agents individually. BottomUp BIBREF29 assisted summary generation with a word prediction model. BERTAbs BIBREF23 and UniLM BIBREF43 are both pre-training based seq2seq summarization models. We also implemented three abstractive models as our baselines. Transformer-S2S is 6-layer seq2seq Transformer BIBREF9 with random initialization. When we replaced the encoder of Transformer-S2S with $\\text{RoBERTa}_\\text{BASE}$ BIBREF4, $\\text{RoBERTa}_\\text{BASE}$-S2S outperforms Transformer-S2S by nearly 2 ROUGE, which demonstrates the effectiveness of pre-trained models. With even larger pre-trained model $\\text{RoBERTa}_\\text{LARGE}$, $\\text{RoBERTa}$-S2S is comparable with the best published abstractive model UniLM BIBREF43.","Based on $\\text{RoBERTa}$-S2S (the sizes of Step and $\\text{RoBERTa}$-S2S are identical), we study the effect of different pre-training tasks (see Section SECREF9). We first pre-train Step on unlabeled documents of CNNDM training split to get quick feedback, denoted as Step (in-domain). From the top part of the third block in Table TABREF25, we can see that Sentence Reordering (SR), Next Sentence Generation (NSG) and Masked Document Generation (MDG) can all improve $\\text{RoBERTa}$-S2S significantly measured by the ROUGE script. Note that according to the ROUGE script, $\\pm 0.22$ ROUGE almost always means a significant difference with $p < 0.05$. Interesting, even Step is pre-trained on 230 million words, it outperforms UniLM that is pre-trained on 3,000 million words BIBREF43. When we pre-train Step on even larger dataset (i.e., GIGA-CM), the results are further improved and Step outperforms all models in comparison, as listed in the bottom part of Table TABREF25.","Table TABREF26 presents results on NYT dataset. Following the same evaluation protocol as BIBREF36, we adopted the limited-length recall based ROUGE, where we truncated the predicted summaries to the length of the gold ones. Again, the first and second blocks show results of previous extractive and abstractive models, respectively. Results of Step are listed in the third block. Similar to the trends in CNNDM, Step leads significant performance gains (with $p<0.05$) compared to all other models in Table TABREF26.","Among all three pre-training tasks, SR works slightly better than the other two tasks (i.e., NSG and MDG). We also tried to randomly use all the three tasks during training with 1/3 probability each (indicated as ALL). Interesting, we observed that, in general, All outperforms all three tasks when employing unlabeled documents of training splits of CNNDM or NYT, which might be due to limited number of unlabeled documents of the training splits. After adding more data (i.e., GIAG-CM) to pre-training, SR consistently achieves highest ROUGE-2 on both CNNDM and NYT. We conclude that SR is the most effective task for pre-training since sentence reordering task requires comprehensively understanding a document in a wide coverage, going beyond individual words and sentences, which is highly close to the essense of abstractive document summarization."]},{"idx":21,"heading":"Results ::: Human Evaluation","paragraphs":["We also conducted human evaluation with 20 documents randomly sampled from the test split of CNNDM. We compared the best preforming Step model (i.e., pre-training on the GIGA-CM dataset using SR task) with human references (denoted as Gold), $\\text{RoBERTa}$-S2S, and two pre-training based models, BERTAbs BIBREF23 and UniLM BIBREF43. Participants were asked to rank the outputs of these systems from best to worst. We report the proportions of system rankings and mean rank (lower is better) in Table TABREF29. The output of Step is selected as the best for the 25% of cases and we obtained lower mean rank than all systems except for Gold, which shows the participants' preference for our model. Then we converted ranking numbers into ratings (i.e., rank $i$ is converted into $6-i$) and applied the student $t$-test on the ratings. Step is significantly better than all other systems in comparison with $p<0.05$. But it still lags behind human. One possible reason is that Step (as well as other systems) only takes the first 512 tokens of a long document as input and thus may lose information residing in the following tokens."]},{"idx":22,"heading":"Conclusion","paragraphs":["We proposed Step, a seq2seq transformer pre-training approach, for abstractive document summarization. Specifically, three pre-training tasks are designed, sentence reordering, next sentence generation, and masked document generation. When we only employ the unlabeled documents in the training splits of summarization datasets to pre-training Step with our proposed tasks, the summarization model based on the pre-trained Step outperforms the best published abstractive system. Involving large scale data to pre-training leads to larger performance gains. By using the best performing pre-training task, Step achieves 0.8 absolute ROUGE-2 improvements on CNN/DailyMail and 2.4 absolute ROUGE-2 improvements on New York Times. In the future, we would like to investigate other tasks to pre-train the seq2seq transformer model. Pre-training for unsupervised abstractive summarization is also an interesting direction and worth exploration."]},{"idx":23,"heading":"Figures and Tables","paragraphs":["Figure 1: Pre-training of STEP. The document (x1, x2, · · · , x8) contains three sentences (i.e., SENT. 1, SENT. 2 and SENT. 3). STEP adopts the SEQ2SEQ Transformer architecture. It takes the transformed document (i.e., a shuffled document, the first segment of a document, or a masked document) as input and learns to recover the original document (or part of the original document) by generation. SR: Sentence Reordering; NSG: Next Sentence Generation; MDG: Masked Document Generation.","Table 1: Results on the test split of CNNDM using fulllength F1 based ROUGE-1 (R-1), ROUGE-2 (R-2) and ROUGE-L (R-L). ∗ indicates significant improvements (p < 0.05 measured with the ROUGE script) compared to models in the first two blocks.","Table 2: Results on the test set of NYT dataset using limited-length recall based ROUGE. ∗ indicates significant improvements (p < 0.05 measured with the ROUGE script) to models in the first two blocks.","Table 3: Human evaluation results: proportions of system rankings. MR: mean rank (the lower the better)."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"2004.01970","title":"BAE: BERT-based Adversarial Examples for Text Classification","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Modern text classification models are susceptible to adversarial examples, perturbed versions of the original text indiscernible by humans but which get misclassified by the model. We present BAE, a powerful black box attack for generating grammatically correct and semantically coherent adversarial examples. BAE replaces and inserts tokens in the original text by masking a portion of the text and leveraging a language model to generate alternatives for the masked tokens. Compared to prior work, we show that BAE performs a stronger attack on three widely used models for seven text classification datasets."]},{"idx":1,"heading":"Introduction","paragraphs":["Recent studies have shown the vulnerability of ML models to adversarial attacks, small perturbations which lead to misclassification of inputs. Adversarial example generation in NLP BIBREF0 is more challenging than in common computer vision tasks BIBREF1, BIBREF2, BIBREF3 due to two main reasons: the discrete nature of input space and ensuring semantic coherence with the original sentence. A major bottleneck in applying gradient based BIBREF4 or generator model BIBREF5 based approaches to generate adversarial examples in NLP is the backward propagation of the perturbations from the continuous embedding space to the discrete token space.","Recent works for attacking text models rely on introducing errors at the character level in words BIBREF6, BIBREF7 or adding and deleting words BIBREF8, BIBREF9, BIBREF10, etc. for creating adversarial examples. These techniques often result in adversarial examples which are unnatural looking and lack grammatical correctness, and thus can be easily identified by humans.","TextFooler BIBREF11 is a black-box attack, that uses rule based synonym replacement from a fixed word embedding space to generate adversarial examples. These adversarial examples do not account for the overall semantics of the sentence, and consider only the token level similarity using word embeddings. This can lead to out-of-context and unnaturally complex replacements (see Table ), which can be easily identifiable by humans.","The recent advent of powerful language models BIBREF12, BIBREF13 in NLP has paved the way for using them in various downstream applications. In this paper, we present a simple yet novel technique: BAE (BERT-based Adversarial Examples), which uses a language model (LM) for token replacement to best fit the overall context. We perturb an input sentence by either replacing a token or inserting a new token in the sentence, by means of masking a part of the input and using a LM to fill in the mask (See Figure FIGREF1). BAE relies on the powerful BERT masked LM for ensuring grammatical correctness of the adversarial examples. Our attack beats the previous baselines by a large margin and confirms the inherent vulnerabilities of modern text classification models to adversarial attacks. Moreover, BAE produces more richer and natural looking adversarial examples as it uses the semantics learned by a LM.","To the best of our knowledge, we are the first to use a LM for adversarial example generation. We summarize our major contributions as follows:","We propose BAE, a novel strategy for generating natural looking adversarial examples using a masked language model.","We introduce 4 BAE attack modes, all of which are almost always stronger than previous baselines on 7 text classification datasets.","We show that, surprisingly, just a few replace/insert operations can reduce the accuracy of even a powerful BERT-based classifier by over $80\\%$ on some datasets."]},{"idx":2,"heading":"Methodology","paragraphs":["Problem Definition We are given a dataset $(S,Y) = \\lbrace (\\mathbb {S}_1,y_1),(\\mathbb {S}_2,y_2)\\dots (\\mathbb {S}_m,y_m)\\rbrace $ and a trained classification model $C:\\mathbb {S}\\rightarrow Y$. We assume the soft-label black-box setting where the attacker can only query the classifier for output probabilities on a given input, and does not have access to the model parameters, gradients or training data. For an input pair $(\\mathbb {S},y)$, we want to generate an adversarial example $\\mathbb {S}_{adv}$ such that $C(\\mathbb {S}_{adv}){\\ne }y$ where $\\mathbb {S}_{adv}$ is natural looking, grammatically correct and semantically similar to $\\mathbb {S}$ (by some pre-defined definition","of similarity).","BAE For generating adversarial example $\\mathbb {S}_{adv}$, we define two perturbations on the input $\\mathbb {S}$:","Replace a token $t \\in \\mathbb {S}$ with another","Insert a new token $t^{\\prime }$ in $\\mathbb {S}$","Some tokens in the input are more attended to by $C$ than others, and therefore contribute more towards the final prediction. Replacing these tokens or inserting a new token adjacent to them can thus have a stronger effect on altering the classifier prediction. We estimate the token importance $I_i$ of each token $t_i \\in \\mathbb {S}=[t_1, \\dots , t_n]$, by deleting $t_i$ from $\\mathbb {S}$ and computing the decrease in probability of predicting the correct label $y$, similar to BIBREF11.","While the motivation for replacing tokens in decreasing order of importance is clear, we conjecture that adjacent insertions in this same order can lead to a powerful attack. This intuition stems from the fact that the inserted token changes the local context around the original token.","The Replace (R) and Insert (I) operations are performed on a token $t$ by masking it and inserting a mask token adjacent to it in $\\mathbb {S}$ respectively. The pre-trained BERT masked language model (MLM) is used to predict the mask tokens (See Figure FIGREF1).","BERT is a powerful LM trained on a large training corpus ($\\sim $ 2 billion words), and hence the predicted mask tokens fit well grammatically in $\\mathbb {S}$. The BERT-MLM does not however guarantee semantic coherence to the original text $\\mathbb {S}$ as demonstrated by the following simple example. Consider the sentence: `the food was good'. For replacing the token `good', BERT-MLM may predict the tokens `nice' and `bad', both of which fit well into the context of the sentence. However, replacing `good' with `bad' changes the original sentiment of the sentence.","To ensure semantic similarity on introducing perturbations in the input text, we filter the set of top K masked tokens (K is a pre-defined constant) predicted by BERT-MLM using a Universal Sentence Encoder (USE) BIBREF14 based sentence similarity scorer. For the R operations we add an additional check for grammatical correctness of the generated adversarial example by filtering out predicted tokens that do not form the same part of speech (POS) as the original token $t_i$ in the sentence.","To choose the token for a perturbation (R/I) that best attacks the model from the filtered set of predicted tokens:","If there are multiple tokens can cause $C$ to misclassify $\\mathbb {S}$ when they replace the mask, we choose the token which makes $\\mathbb {S}_{adv}$ most similar to the original $\\mathbb {S}$ based on the USE score.","If no token causes misclassification, we choose the perturbation that decreases the prediction probability $P(C(\\mathbb {S}_{adv}){=}y)$ the most.","The perturbations are applied iteratively to the input tokens in decreasing order of importance, until either $C(\\mathbb {S}_{adv}){\\ne }y$ (successful attack) or all the tokens of $\\mathbb {S}$ have been perturbed (failed attack).","We present 4 attack modes for BAE based on the R and I operations, where for each token $t$ in $\\mathbb {S}$:","BAE-R: Replace token $t$ (See Algorithm )","BAE-I: Insert a token to the left or right of $t$","BAE-R/I: Either replace token $t$ or insert a token to the left or right of $t$","BAE-R+I: First replace token $t$, then insert a token to the left or right of $t$"]},{"idx":3,"heading":"Experiments","paragraphs":["Datasets and Models We evaluate our adversarial attacks on different text classification datasets from tasks such as sentiment classification, subjectivity detection and question type classification. Amazon, Yelp, IMDB are sentence-level sentiment classification datasets which have been used in recent work BIBREF15 while MR BIBREF16 contains movie reviews based on sentiment polarity. MPQA BIBREF17 is a dataset for opinion polarity detection, Subj BIBREF18 for classifying a sentence as subjective or objective and TREC BIBREF19 is a dataset for question type classification.","We use 3 popular text classification models: word-LSTM BIBREF20, word-CNN BIBREF21 and a fine-tuned BERT BIBREF12 base-uncased classifier. For each dataset we train the model on the training data and perform the adversarial attack on the test data. For complete model details refer to Appendix.","As a baseline, we consider TextFooler BIBREF11 which performs synonym replacement using a fixed word embedding space BIBREF22. We only consider the top $K{=}50$ synonyms from the MLM predictions and set a threshold of 0.8 for the cosine similarity between USE based embeddings of the adversarial and input text.","Results We perform the 4 modes of our attack and summarize the results in Table . Across datasets and models, our BAE attacks are almost always more effective than the baseline attack, achieving significant drops of 40-80% in test accuracies, with higher average semantic similarities as shown in parentheses. BAE-R+I is the strongest attack since it allows both replacement and insertion at the same token position, with just one exception. We observe a general trend that the BAE-R and BAE-I attacks often perform comparably, while the BAE-R/I and BAE-R+I attacks are much stronger. We observe that the BERT-based classifier is more robust to the BAE and TextFooler attacks than the word-LSTM and word-CNN models which can be attributed to its large size and pre-training on a large corpus.","The baseline attack is often stronger than the BAE-R and BAE-I attacks for the BERT based classifier. We attribute this to the shared parameter space between the BERT-MLM and the BERT classifier before fine-tuning. The predicted tokens from BERT-MLM may not drastically change the internal representations learned by the BERT classifier, hindering their ability to adversarially affect the classifier prediction.","Effectiveness We study the effectiveness of BAE on limiting the number of R/I operations permitted on the original text. We plot the attack performance as a function of maximum $\\%$ perturbation (ratio of number of word replacements and insertions to the length of the original text) for the TREC dataset. From Figure , we clearly observe that the BAE attacks are consistently stronger than TextFooler. The classifier models are relatively robust to perturbations up to 20$\\%$, while the effectiveness saturates at 40-50$\\%$. Surprisingly, a 50$\\%$ perturbation for the TREC dataset translates to replacing or inserting just 3-4 words, due to the short text lengths.","Qualitative Examples We present adversarial examples generated by the attacks on a sentence from the IMDB and Yelp datasets in Table . BAE produces more natural looking examples than TextFooler as tokens predicted by the BERT-MLM fit well in the sentence context. TextFooler tends to replace words with complex synonyms, which can be easily detected. Moreover, BAE's additional degree of freedom to insert tokens allows for a successful attack with fewer perturbations.","Human Evaluation We consider successful adversarial examples generated from the Amazon and IMDB datasets and verify their sentiment and grammatical correctness. Human evaluators annotated the sentiment and the grammar (Likert scale of 1-5) of randomly shuffled adversarial examples and original texts. From Table , BAE and TextFooler have inferior accuracies compared to the Original, showing they are not always perfect. However, BAE has much better grammar scores, suggesting more natural looking adversarial examples.","Ablation Study We analyze the benefits of R/I operations in BAE in Table . From the table, the splits $\\mathbb {A}$ and $\\mathbb {B}$ are the $\\%$ of test points which compulsorily need I and R operations respectively for a successful attack. We can observe that the split $\\mathbb {A}$ is larger than $\\mathbb {B}$ thereby indicating the importance of the I operation over R. Test points in split require both R and I operations for a successful attack. Interestingly, split is largest for Subj, which is the most robust to attack (Table ) and hence needs both R/I operations. Thus, this study gives positive insights towards the importance of having the flexibility to both replace and insert words.","Refer to the Appendix for additional results, effectiveness graphs and details of human evaluation."]},{"idx":4,"heading":"Conclusion","paragraphs":["In this paper, we have presented a novel technique for generating adversarial examples (BAE) based on a language model. The results obtained on several text classification datasets demonstrate the strength and effectiveness of our attack."]},{"idx":5,"heading":"Figures and Tables","paragraphs":["Figure 1: We use BERT-MLM to predict masked tokens in a given text for generating adversarial examples. The MASK token replaces a given word (BAE-R attack) or is inserted to the left/right of the word (BAE-I attack). Actual predictions made by the BERT-MLM are shown (they fit well into the sentence context)","Table 2: Test accuracies on adversarial attack across models and datasets. The average semantic similarity between the original and adversarial examples are mentioned in parentheses for the baseline and the strongest BAE attack.","Table 1: Summary statistics for the datasets","Figure 2: Graphs comparing the performance of the attacks on the TREC question classification dataset, as a function of maximum % perturbation to the input. All BAE attacks outperform the baseline TEXTFOOLER .","Table 3: Qualitative examples of each attack on the BERT classifier (Replacements: Red, Inserts: Blue)","Table 5: A denotes the % of test instances which are successfully attacked by BAE-R/I, but not BAE-R , i.e, ((R/I)∩ R) ; B : (R/I)∩ I ; C : (R/I)∩ R∩ I","Table 4: Human evaluation of accuracies and grammaticality scores. TF refers to TextFooler.","Table 1: Test accuracies on adversarial attack across models and datasets. The average semantic similarity between the original and adversarial examples are mentioned in parentheses.","Figure 4: Yelp","Figure 8: Subj"]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"2004.04060","title":"Self-Attention Gazetteer Embeddings for Named-Entity Recognition","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Recent attempts to ingest external knowledge into neural models for named-entity recognition (NER) have exhibited mixed results. In this work, we present GazSelfAttn, a novel gazetteer embedding approach that uses self-attention and match span encoding to build enhanced gazetteer embeddings. In addition, we demonstrate how to build gazetteer resources from the open source Wikidata knowledge base. Evaluations on CoNLL-03 and Ontonotes 5 datasets, show F1 improvements over baseline model from 92.34 to 92.86 and 89.11 to 89.32 respectively, achieving performance comparable to large state-of-the-art models."]},{"idx":1,"heading":"Introduction","paragraphs":["Named-entity recognition (NER) is the task of tagging relevant entities such as person, location and organization in unstructured text. Modern NER has been dominated by neural models BIBREF0, BIBREF1 combined with contextual embeddings from language models (LMs) BIBREF2, BIBREF3, BIBREF4. The LMs are pre-trained on large amounts of unlabeled text which allows the NER model to use the syntactic and semantic information captured by the LM embeddings. On the popular benchmark datasets CoNLL-03 BIBREF5 and Ontonotes 5 BIBREF6, neural models with LMs achieved state-of-the-art results without gazetteers features, unlike earlier approaches that heavily relied on them BIBREF7. Gazetteers are lists that contain entities such as cities, countries, and person names. The gazetteers are matched against unstructured text to provide additional features to the model. Data for building gazetteers is available for multiple language from structured data resources such as Wikipedia, DBpedia BIBREF8 and Wikidata BIBREF9.","In this paper, we propose GazSelfAttn, a novel gazetteer embedding approach that uses self-attention and match span encoding to build enhanced gazetteer representation. GazSelfAttn embeddings are concatenated with the input to a LSTM BIBREF10 or CNN BIBREF11 sequence layer and are trained end-to-end with the model. In addition, we show how to extract general gazetteers from the Wikidata, a structured knowledge-base which is part of the Wikipedia project.","Our contributions are the following:","[topsep=1pt, leftmargin=15pt, itemsep=-1pt]","We propose novel gazetteer embeddings that use self-attention combined with match span encoding.","We enhance gazetteer matching with multi-token and single-token matches in the same representation.","We demonstrate how to use Wikidata with entity popularity filtering as a resource for building gazetteers.","GazSelfAttn evaluations on CoNLL-03 and Ontonotes 5 datasets show F$_1$ score improvement over baseline model from 92.34 to 92.86 and from 89.11 to 89.32 respectively. Moreover, we perform ablation experiments to study the contribution of the different model components."]},{"idx":2,"heading":"Related Work","paragraphs":["Recently, researchers added gazetteers to neural sequence models. BIBREF12 demonstrated small improvements on large datasets and bigger improvements on small datasets. BIBREF13 proposed to train a gazetteer attentive network to learn name regularities and spans of NER entities. BIBREF14 demonstrated that trained gazetteers scoring models combined with hybrid semi-Markov conditional random field (HSCRF) layer improve overall performance. The HSCRF layer predicts a set of candidate spans that are rescored using a gazetteer classifier model. The HSCRF approach differs from the common approach of including gazetteers as an embedding in the model. Unlike the work of BIBREF14, our GazSelfAttn does not require training a separate gazetteer classifier and the HSCRF layer, thus our approach works with any standard output layer such as conditional random field (CRF) BIBREF15.","BIBREF16 proposed an auto-encoding loss with hand-crafted features, including gazetteers, to improve accuracy. However, they did not find that gazetteer features significantly improve accuracy.","Extracting gazetteers from structure knowledge sources was investigated by BIBREF17 and BIBREF18. They used Wikipedia's instance of relationship as a resource for building gazetteers with classical machine learning models. Compared to Wikidata, the data extracted from Wikipedia is smaller and noisier.","Similar to this paper, BIBREF19 used Wikidata as a gazetteer resource. However, they did not use entity popularity to filter ambiguous entities and their gazetteer model features use simple one-hot encoding."]},{"idx":3,"heading":"Approach ::: Model Architecture","paragraphs":["We add GazSelfAttn embeddings to the popular Neural CRF model architecture with ELMo LM embeddings from BIBREF2. Figure FIGREF5 depicts the model, which consists of Glove word embeddings BIBREF20, Char-CNN BIBREF21, BIBREF1, ELMo embeddings, Bi-LSTM, and output CRF layer with BILOU (Beginning Inside Last Outside Unit) labels encoding BIBREF22. Note that, we concatenate the gazetteer embeddings to the Bi-LSTM input."]},{"idx":4,"heading":"Approach ::: Gazetteers","paragraphs":["In this section, we address the issue of building a high-quality gazetteer dictionary $M$ that maps entities to types, e.g., $M$[Andy Murray] $\\rightarrow $ Person. In this work, we use Wikidata, an open source structured knowledge-base, as the source of gazetteers. Although, Wikidata and DBpedia are similar knowledge bases, we choose Wikidata because, as of 2019, it provides data on around 45 million entities compared to around 5 million in DBpedia.","Wikidata is organized as entities and properties. Entities can be concrete (Boston, NATO, Michael Jordan) and abstract (City, Organization, Person). Properties describe an entity relations. For example, Boston instance_of City and Boston part_of Massachusetts; both instance_of and part_of are properties. Also, each entity is associated with sitelink count which tacks mentions of the entity on Wikimedia website and can be used as proxy for popularity.","To extract gazetteers from Wikidata, we process the official dumps into tuples of entity and type based only on the left and right part of the instance_of triplet, example resulting tuples are Boston $\\rightarrow $ City and Massachusetts $\\rightarrow $ State. Each entity is associated with a set of aliases, we keep only the aliases that are less than seven tokens long. Example aliases for Boston are “Beantown” and “The Cradle of Liberty”. If there are multiple types per alias, we use the sitelink count to keep the six most popular types. The sitelink filtering is important to reduce the infrequent meanings of an entity in the gazetteer data.","The Wikidata types that we obtain after processing the Wikidata dumps are fine-grained. However, certain NER tasks require coarse-grained types. For instance, CoNLL-03 task has a single Location label that consists of cities, states, countries, and other geographic location. To move from fine-grained to coarse-grained types, we use the Wikidata hierarchical structure induced by the subclass_of property. Examples of subclass_of hierarchies in Wikidata are: City $\\rightarrow $ Human Settlement $\\rightarrow $ Geographic Location, and Artist $\\rightarrow $ Creator $\\rightarrow $ Person. We change the types granularity depending on the NER task by traversing up, from fine-grained types to the target coarse-grained types. For instance, we merge the Artist and Painter types to Person, and the River and Mountain types to Location."]},{"idx":5,"heading":"Approach ::: Gazetteer Matching","paragraphs":["Gazetteer matching is the process of assigning gazetteer features to sentence tokens. Formally, given a gazetteer dictionary $M$ that maps entities to types, and a sentence $S = (t_1, t_2, ..., t_n)$ with tokens $t_i$, we have to find the $m$ gazetteer types $\\lbrace g^1_i, g^2_i,..,g^m_i\\rbrace $ and spans $\\lbrace s^1_i, s^2_i,..,s^m_i\\rbrace $ for every token $t_i$. The set notation $\\lbrace $} indicates that multiple $m$ matches are allowed per token. The match span $\\lbrace s^j_i\\rbrace $ represents positional information which encodes multi-token matches. The match spans are encoded using a BILU (Beginning Inside Last Unit) tags, similar to the BILOU tags that we use to encode the NER labels.","In general, there are two methods for gazetteer matching: multi-token and single-token. Multi-token matching is searching for the longest segments of the sentence that are in $M$. For instance, given $M$[New York] $\\rightarrow $ State, $M$[New York City] $\\rightarrow $ City and the sentence “Yesterday in New York City”, the multi-token matcher assigns the City gazetteer type to the longest segment “New York City”. Single-token matching is searching to match any vocabulary word from a gazetteer type. In the earlier example, each word from the sentence is individually matched to the tokens in $M$, thus “New” and “York” are individually matched to both City and State, and “City” is matched only to City.","BIBREF12 research shows that both multi-token and single-token matching perform better on certain datasets. We propose to combine both methods: we tag the multi-token matches with BILU tags, and the single-token matches with a Single (S) tag. The single-token matches are used only if multi-token matches are not present. We consider that the single-token matches are high-recall low-precision, and multi-token matches are low-recall and high-precision. Thus, a combination of both works better than individually. Example sentences are: “Yesterday in New(City-B) York(City-I) City(City-L)”, and “Yesterday in York(City-S) City(City-S)” York City is marked with singles tag since $M$ does not have entities for “York City”, “York”, and “City”.","Note that gazetteer matching is unsupervised, i.e., we do not have a ground truth of correctly matched sentences for $M$. Furthermore, it is a hard task because of the many variations in writing and ambiguity of entities."]},{"idx":6,"heading":"Approach ::: Gazetteer Embeddings","paragraphs":["px","Equations DISPLAY_FORM11- shows the gazetteer embedding $\\mathbf {g}_i$ computation for a token $t_i$. To compute $\\mathbf {g}_i$, given a set of $m$ gazetteer types $\\lbrace g^m_i\\rbrace $ and spans $\\lbrace s^m_i\\rbrace $, we execute the following procedure:","[topsep=1pt, leftmargin=15pt, itemsep=-1pt]","Equation DISPLAY_FORM11. We embed the sets $\\lbrace g^m_i\\rbrace $ and $\\lbrace s^m_i\\rbrace $ using the embedding matrices $\\mathbf {G}$ and $\\mathbf {S}$. Then, we do an element-wise addition, denoted $\\oplus $, of the corresponding types and spans embeddings to get a matrix $\\mathbf {E}_i$.","Equation . We compute $\\mathbf {A}_i$ using scaled dot-product self-attention BIBREF23, where $d$ is the dimensionality of the gazetteer embeddings. The attention contextualizes the embeddings with multiple gazetteer matches per token $t_i$.","Equation . To add model flexibility, we compute $\\mathbf {H}_i$ with a position-wise feed-forward layer and GELU activation BIBREF24.","Equation . Finally, we perform max pooling across the embeddings $\\mathbf {H}_i$ to obtain the final gazetteer embedding $\\mathbf {g}_i$."]},{"idx":7,"heading":"Approach ::: Gazetteer Dropout","paragraphs":["To prevent the neural NER model from overfitting on the gazetteers, we use gazetteers dropout BIBREF25. We randomly set to zero gazetteer embeddings $\\mathbf {g}_i$, so the gazetteer vectors that are input to the LSTM become zero. We tune the gazetteer dropout hyperparameter on the validation set."]},{"idx":8,"heading":"Experiments ::: Setup","paragraphs":["Datasets. We evaluate on the English language versions of CoNLL-03 dataset BIBREF5 and the human annotated portion of the Ontonotes 5 BIBREF6 dataset. CoNLL-03 labels cover 4 entity types: person, location, organization, and miscellaneous. The Onotonotes 5 dataset is larger and its labels cover 18 types: person, NORP, facility, organization, GPE, location, product, event, work of art, law, language, date, time, percent, money, quantity, ordinal, cardinal.","px","Gazetteers. We use the Wikidata gazetteers with types merged to the granularity of the CoNLL-03 and Ononotes 5 datasets. We filter non-relevant types (e.g., genome names, disease) and get a total of one million records. For CoNLL-03 and Ontonotes 5, the percentage of entities covered by gazetteers are 96% and 78% respectively, and percentage of gazetteers wrongly assigned to non-entity tokens are 41% and 41.5% respectively.","Evaluation. We use the standard CoNLL evaluation script which reports entity F1 scores. The F1 scores are averages over 5 runs.","Configuration. We use the Bi-LSTM-CNN-CRF model architecture with ELMo language model embeddings from BIBREF2, which consist of 50 dim pre-trained Glove word embeddings BIBREF20, 128 dim Char-CNN BIBREF21, BIBREF1 embeddings with filter size of 3 and randomly initialized 16 dim char embeddings, 1024 pre-trained ELMo pre-trained embeddings, two layer 200 dim Bi-LSTM, and output CRF layer with BILOU (Beginning Inside Last Outside Unit) spans BIBREF22.","For the gazetteer embeddings, we use 128 dim for the embedding matrices $\\mathbf {G}$ and $\\mathbf {S}$, 128 dim output for $\\mathbf {W}$, which yields a gazetteer embedding $\\mathbf {g}_i$ with 128 dim. The parameters are randomly initialized and trained. We apply gazetteer dropout of 0.1 which we tuned on the development set; we tried values form 0.05 to 0.6.","All parameters except the ELMo embeddings are trained. We train using the Adam BIBREF26 optimizer with learning rate of 0.001 for 100 epochs. We use early stopping with patience 25 on the development set. Batch size of 64, dropout rate of 0.5 and L2 regularization of 0.1."]},{"idx":9,"heading":"Experiments ::: Results","paragraphs":["The experimental results for NER are summarized in Table TABREF20. The top part of the table shows recently published results. BIBREF14's work is using gazetteers with HSCRF and BIBREF4's work is using the Flair language model which is much larger than ELMo. BIBREF27 is the current state-of-the-art language model that uses cloze-driven pretraining. The bottom part of the table is shows our baseline models and results with included gazetteers. We experiment with the Neural CRF model with and without ELMo embeddings. Including ELMo embeddings the CoNLL-03 and Ontonotes 5, F$_1$ score improves from 92.34 to 92.86 and 89.11 to 89.32 respectively. Without ELMo embeddings the F$_1$ score improves from 90.42 to 91.12 and 86.63 to 87 respectively. We observe that GazSelfAttn relative improvements are similar with and without ELMo embeddings. We obtain slightly better CoNLL-03 F$_1$ score compared to BIBREF14 work that uses the HSCRF model, and we match the Ononotes 5 F$_1$ scores of BIBREF4 that uses a much bigger model. BIBREF14 Ononotes 5 results use subset of the dataset labels and are not comparable. Note that because of computation constrains, we did not perform extensive hyperparameter tuning except for the gazetteer dropout rate."]},{"idx":10,"heading":"Experiments ::: Ablation study","paragraphs":["Table TABREF22 shows ablation experiments. We remove components of the gazetteer embedding model from the Neural CRF model. In each experiment, we removed only the specified component. Ablations show decreased F$_1$ score on the development and test set if any of the components is removed. The highest degradation is when single matches are removed which underscores the importance of the combining the gazetteer matching techniques for NER. We observe that match span encoding is more important for the CoNLL-02 compared to Ononotes 5 because the former has more entities with multiple tokens. Removing the self-attention shows that self-attention is effective at combining information form multiple gazetteers.","In addition, we tried moving the gazetteer embeddings to the CRF layer and using the LSTM token embeddings as attention keys but the F$_1$ degraded significantly. We experimented with adding auto-encoding loss similar to BIBREF16 and multi-head self-attention. However, we did not observe F$_1$ score improvements and sometimes small degradations."]},{"idx":11,"heading":"Conclusion","paragraphs":["We presented GazSelfAttn, a novel approach for gazetteer embeddings that uses self-attention and match span positions. Evaluation results of GazSelfAttn show improvement compared to competitive baselines and state-of-the-art models on multiple datasets.","For future work we would like to evaluate GazSelfAttn on non-English language datasets and improve the multi-token gazetteer matching with fuzzy string matching. Also, we would like to explore transfer learning of gazetteer embeddings from high-resource to low-resource setting."]},{"idx":12,"heading":"Figures and Tables","paragraphs":["Figure 1: Model architecture with gazetteer embeddings.","Table 1: Dataset sizes in number of sentences.","Table 2: Results on CoNLL-03 and OntoNotes 5.","Table 3: Ablation study results on CoNLL-03 and OntoNotes 5. “- span encoding” removes the BILU match span encoding leaving only the gazetteer types. “- self attention” removes the self-attention. “- uncased matches” removes the uncased matches."]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"2004.04124","title":"LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression","sections":[{"idx":0,"heading":"Abstract","paragraphs":["BERT is a cutting-edge language representation model pre-trained by a large corpus, which achieves superior performances on various natural language understanding tasks. However, a major blocking issue of applying BERT to online services is that it is memory-intensive and leads to unsatisfactory latency of user requests, raising the necessity of model compression. Existing solutions leverage the knowledge distillation framework to learn a smaller model that imitates the behaviors of BERT. However, the training procedure of knowledge distillation is expensive itself as it requires sufficient training data to imitate the teacher model. In this paper, we address this issue by proposing a hybrid solution named LadaBERT (Lightweight adaptation of BERT through hybrid model compression), which combines the advantages of different model compression methods, including weight pruning, matrix factorization and knowledge distillation. LadaBERT achieves state-of-the-art accuracy on various public datasets while the training overheads can be reduced by an order of magnitude."]},{"idx":1,"heading":"Introduction","paragraphs":["The pre-trained language model, BERT BIBREF0 has led to a big breakthrough in various kinds of natural language understanding tasks. Ideally, people can start from a pre-trained BERT checkpoint and fine-tune it on a specific downstream task. However, the original BERT models are memory-exhaustive and latency-prohibitive to be served in embedded devices or CPU-based online environments. As the memory and latency constraints vary in different scenarios, the pre-trained BERT model should be adaptive to different requirements with accuracy retained to the largest extent. Existing BERT-oriented model compression solutions largely depend on knowledge distillation BIBREF1, which is inefficient and resource-consuming because a large training corpus is required to learn the behaviors of a teacher. For example, DistilBERT BIBREF2 is re-trained on the same corpus as pre-training a vanilla BERT from scratch; and TinyBERT BIBREF3 utilizes expensive data augmentation to fit the distillation target. The costs of these model compression methods are as large as pre-training and unaffordable for low-resource settings. Therefore, it is straight-forward to ask, can we design a lightweight method to generate adaptive models with comparable accuracy using significantly less time and resource consumption? In this paper, we propose LadaBERT (Lightweight adaptation of BERT through hybrid model compression) to tackle the raised questions. Specifically, LadaBERT is based on an iterative hybrid model compression framework consisting of weighting pruning, matrix factorization and knowledge distillation. Initially, the architecture and weights of student model are inherited from the BERT teacher. In each iteration, the student model is first compressed by a small ratio based on weight pruning and matrix factorization, and is then fine-tuned under the guidance of teacher model through knowledge distillation. Because weight pruning and matrix factorization help to generate better initial and intermediate status","in the knowledge distillation iterations, the accuracy and efficiency of model compression can be greatly improved.","We conduct extensive experiments on five public datasets of natural language understanding. As an example, the performance comparison of LadaBERT and state-of-the-art models on MNLI-m dataset is illustrated in Figure FIGREF1. We can see that LadaBERT outperforms other BERT-oriented model compression baselines at various model compression ratios. Especially, LadaBERT-1 outperforms BERT-PKD significantly under $2.5\\times $ compression ratio, and LadaBERT-3 outperforms TinyBERT under $7.5\\times $ compression ratio while the training speed is accelerated by an order of magnitude.","The rest of this paper is organized as follows. First, we summarizes the related works of model compression and their applications to BERT in Section SECREF2. Then, the methodology of LadaBERT is introduced in Section SECREF3, and experimental results are presented in Section SECREF4. At last, we conclude this work and discuss future works in Section SECREF5."]},{"idx":2,"heading":"Related Work","paragraphs":["Deep Neural Networks (DNNs) have achieved great success in many areas in recent years, but the memory consumption and computational cost expand greatly with the growing complexity of models. Therefore, model compression has become an indispensable technique for practice, especially in low-resource settings. In this section, we review the current progresses of model compression techniques briefly, which can be divided into four categories, namely weight pruning, matrix factorization, weight quantization and knowledge distillation. We also present hybrid approaches and the applications of model compression to pre-trained BERT models."]},{"idx":3,"heading":"Related Work ::: Weight pruning","paragraphs":["Numerous researches have shown that removing a large portion of connections or neurons does not cause significant performance drop in deep neural network models BIBREF4, BIBREF5, BIBREF6, BIBREF7. For example, Han et al. BIBREF4 proposed a method to reduce the storage and computation of neural networks by removing unimportant connections, resulting in sparse networks without affecting the model accuracy. Li et al. BIBREF5 presented an acceleration method for convolution neural network by pruning whole filters together with their connecting filter maps. This approach does not generate sparse connectivity patterns and brings much larger acceleration ratio with existing BLAS libraries for dense matrix multiplications. Ye et al. BIBREF8 argued that small weights are in fact important for preserving the performance of a model, and Hu et al. BIBREF6 alleviated this problem by a data-driven approach that pruned zero-activation neurons iteratively based on intermediate feature maps. Zhu and Gupta BIBREF7 empirically compared large-sparse models with smaller dense models of similar parameter sizes and found that large sparse models performed better consistently. In addition, sparsity-induced models BIBREF9, BIBREF10, BIBREF11 can be regarded as similar methods as pruning. For example, Wen et al. BIBREF9 applied group lasso as a regularizer at training time, and Louizos et al. BIBREF10 learned sparse neural networks through $l_0$ regularization."]},{"idx":4,"heading":"Related Work ::: Matrix factorization","paragraphs":["The goal of matrix factorization is to decompose a matrix into the product of two matrices in lower dimensions, and Singular Value Decomposition (SVD) is a popular way of matrix factorization that generalizes the eigendecomposition of a square normal matrix to a $m \\times n$ matrix. It has been proved that SVD is the best approximation of a matrix given the rank $r$ under Frobenius norm BIBREF12. Matrix factorization was widely studied in the deep learning domain for model compression and acceleration BIBREF13, BIBREF14, BIBREF15. Sainath et al BIBREF13 explored a low-rank matrix factorization method of DNN layers for acoustic modeling. Xu et al. BIBREF14, BIBREF15 applied singular value decomposition to deep neural network acoustic models and achieved comparable performances with state-of-the-art models through much fewer parameters. GroupReduce BIBREF16 focused on the compression of neural language models and applied low-rank matrix approximation to vocabulary-partition. Acharya et al. BIBREF17 compressed the word embedding layer via matrix factorization and achieved promising results in text classification. Winata et al. BIBREF18 carried out experiments for low-rank matrix factorization on different NLP tasks and demonstrated that it was more effective in general than weight pruning."]},{"idx":5,"heading":"Related Work ::: Weight quantization","paragraphs":["Weight quantization is a common technique for compressing deep neural networks, which aims to reduce the number of bits to represent every weight in the model. In a neural network, parameters are stacked into clusters, and the parameters in the same cluster share the same value. With weight quantization, the weights can be reduced to at most 1-bit binary value from 32-bits floating point numbers. Zhou et al. BIBREF19 showed that quantizing weights to 8-bits does not hurt the performance, and Binarized Neural Networks BIBREF20 contained binary weights and activations of only one bit. Incremental Network Quantization BIBREF21 converted a pre-trained full-precision neural network into low-precision counterpart through three interdependent operations: weight partition, groupwise quantization and re-training. Variational Network Quantization BIBREF22 formulated the problem of network quantization as a variational inference problem. Moreover, Choi et al. BIBREF23 investigated the drawbacks of conventional quantization methods based on k-means and proposed a Hessian-weighted k-means clustering algorithm as the solution."]},{"idx":6,"heading":"Related Work ::: Knowledge distillation","paragraphs":["Knowledge distillation is first proposed by BIBREF1, which trains a compact or smaller model to approximate the function learned by a large and complex model. A preliminary step of knowledge distillation is to train a deep network (the teacher model) that automatically generates soft labels for training instances. This “synthetic\" label is then used to train a smaller network (the student model), which assimilates the function that is learned by the teacher model. Chen et al. BIBREF24 successfully applied knowledge distillation to object detection tasks by introducing several modifications, including a weighted cross-entropy loss, a teacher bounded loss, and adaptation layers to model intermediate teacher distributions. Li et al. BIBREF25 developed a framework to learn from noisy labels, where the knowledge learned from a clean dataset and semantic knowledge graph were leveraged to correct the wrong labels. Anil et al. BIBREF26 proposed online distillation, a variant of knowledge distillation which enabled extra parallelism for training large-scale data. In addition, knowledge distillation is also useful for aggregating model ensembles into a single model by treating the ensemble model as a teacher."]},{"idx":7,"heading":"Related Work ::: Hybrid approach","paragraphs":["To improve the performance of model compression, there are many attempts to conduct hybrid model compression method that combines more than one category of algorithms. Han et al. BIBREF27 combined quantization, hamming coding and weight pruning to conduct model compression on image classification tasks. Yu et al. BIBREF28 proposed a unified framework for low-rank and sparse decomposition of weight matrices with feature map reconstructions. Polino et al. BIBREF29 advocated a combination of distillation and quantization techniques and proposed two hybrid models, i.e., quantified distillation and differentiable quantization to address this problem. Li et al., BIBREF30 compressed DNN-based acoustic model through knowledge distillation and pruning. NNCF BIBREF31 provided a neural network compression framework that supported an integration of various model compression methods to generate more lightweight networks and achieved state-of-the-art performances in terms of a trade-off between accuracy and efficiency. In BIBREF32, an AutoML pipeline was adopted for model compression. It leveraged reinforcement learning to search for the best model compression strategy among multiple combinatorial configurations."]},{"idx":8,"heading":"Related Work ::: BERT model compression","paragraphs":["In the natural language processing community, there is a growing interest recently to study BERT-oriented model compression for shipping its performance gain into latency-critical or low-resource scenarios. Most existing works focus on knowledge distillation. For instance, BERT-PKD BIBREF33 is a patient knowledge distillation approach that compresses the original BERT model into a lightweight shallow network. Different from traditional knowledge distillation methods, BERT-PKD enables an exploitation of rich information in the teacher's hidden layers by utilizing a layer-wise distillation constraint. DistillBERT BIBREF2 pre-trains a smaller general-purpose language model on the same corpus as vanilla BERT. Distilled BiLSTM BIBREF34 adopts a single-layer BiLSTM as the student model and achieves comparable results with ELMo BIBREF35 through much fewer parameters and less inference time. TinyBERT BIBREF3 reports the best-ever performance on BERT model compression, which exploits a novel attention-based distillation schema that encourages the linguistic knowledge in teacher to be well transferred into the student model. It adopts a two-stage learning framework, including general distillation (pre-training from scratch via distillation loss) and task-specific distillation with data augmentation. Both procedures require huge resources and long training times (from several days to weeks), which is cumbersome for industrial applications. Therefore, we are aiming to explore more lightweight solutions in this paper."]},{"idx":9,"heading":"Lightweight Adaptation of BERT ::: Overview","paragraphs":["The overall pipeline of LadaBERT (Lightweight Adaptation of BERT) is illustrated in Figure FIGREF8. As shown in the figure, the pre-trained BERT model (e.g., BERT-Base) is served as the teacher as well as the initial status of the student model. Then, the student model is compressed towards smaller parameter size through a hybrid model compression framework in an iterative manner until the target compression ratio is reached. Concretely, in each iteration, the parameter size of student model is first reduced by $1-\\Delta $ based on weight pruning and matrix factorization, and then the parameters are fine-tuned by the loss function of knowledge distillation. The motivation behind is that matrix factorization and weight pruning are complementary with each other. Matrix factorization calculates the optimal approximation under a certain rank, while weight pruning introduces additional sparsity to the decomposed matrices. Moreover, weight pruning and matrix factorization generates better initial and intermediate status of the student model, which improve the efficiency and effectiveness of knowledge distillation. In the following subsections, we will introduce the algorithms in detail."]},{"idx":10,"heading":"Lightweight Adaptation of BERT ::: Overview ::: Matrix factorization","paragraphs":["We use Singular Value Decomposition (SVD) for matrix factorization. Each parameter matrix, including the embedding layer are compressed by SVD. Without loss generality, we assume a matrix of parameters ${W} \\in \\mathbb {R}^{m\\times n}$, the singular value decomposition of which can be written as:","where ${U} \\in \\mathbb {R}^{m \\times p}$ and ${V} \\in \\mathbb {R}^{p \\times n}$. ${\\Sigma } =diag(\\sigma _1,\\sigma _2,\\ldots ,\\sigma _p)$ is a diagonal matrix composed of singular values and $p$ is the full rank of $W$ satisfying $p \\le min(m, n)$.","To compress this weight matrix, we select a lower rank $r$. The diagonal matrix ${\\Sigma }$ is truncated by selecting the top $r$ singular values. i.e., ${\\Sigma }_r =diag(\\sigma _1, \\sigma _2,\\ldots ,\\sigma _r)$, while ${U}$ and ${V}$ are also truncated by selecting the top $r$ columns and rows respectively, resulting in ${U}_r \\in \\mathbb {R}^{m\\times r}$ and ${V}_r \\in \\mathbb {R}^{r\\times n}$.","Thus, low-rank matrix approximation of ${W}$ can be formulated as:","In this way, the original weight matrix $W$ is decomposed by the multiplication of two smaller matrices, where ${A}={U}_r\\sqrt{{\\Sigma }_r} \\in \\mathbb {R}^{n\\times r}$ and ${B}={V}_r\\sqrt{{\\Sigma }_r} \\in \\mathbb {R}^{m\\times r}$. These two matrices are initialized by SVD and will be further tuned during training.","Given a rank $r \\le min(m, n)$, the compression ratio of matrix factorization is defined as:","Therefore, for a target model compression ratio $P_{svd}$, the desired rank $r$ can be calculated by:"]},{"idx":11,"heading":"Lightweight Adaptation of BERT ::: Overview ::: Weight pruning","paragraphs":["Weight pruning BIBREF4 is an unstructured compression method that induces desirable sparsity for a neural network model. For a neural network $f({x; \\theta })$ with parameters $\\theta $, weight pruning finds a binary mask ${M} \\in \\lbrace 0, 1\\rbrace ^{|\\theta |}$ subject to a given sparsity ratio, $P_{weight}$. The neural network after pruning will be $f({x; M \\cdot \\theta })$, where the non-zero parameter size is $||{M}||_1 = P_{weight}\\cdot |\\theta |$, where $|\\theta |$ is the number of parameters in $\\theta $. For example, when $P_m = 0.3$, there are 70% zeros and 30% ones in the mask ${m}$. We adopt a simple pruning strategy in our implementation: the binary mask is generated by setting the smallest weights to zeros BIBREF36.","To combine the benefits of weight pruning with matrix factorization, we leverage a hybrid approach that applies weight pruning on the basis of decomposed matrices generated by SVD. Following Equation (DISPLAY_FORM12), SVD-based matrix factorization for any weight matrix ${W}$ can be written as: ${W}_{svd}={A}_{m\\times r}{B}_{n\\times r}^T$. Then, weight pruning is applied on the decomposed matrices ${A} \\in \\mathbb {R}^{m \\times r}$ and ${B} \\in \\mathbb {R}^{n \\times r}$ separately. The weight matrix after hybrid compression is denoted by:","where ${M_A}$ and ${M_B}$ are binary masks derived by the weight pruning algorithm with compression ratio $P_{weight}$. The compression ratio of this hybrid approach can be calculated by:","In LadaBERT, the hybrid compression produce is applied to each layer of the pre-trained BERT model. Given an overall model compression target $P$, the following constraint should be satisfied:","where $|\\theta |$ is the total number of model parameters and $P$ is the target compression ratio; $|\\theta _{embd}|$ denotes the parameter number of embedding layer, which has a relative compression ratio of $P_embd$, and $|\\theta _{encd}|$ denotes the number of parameters of all layers in BERT encoder, which have a compression ratio of $P_{hybrid}$. The classification layer (often MLP layer with Softmax activation) has a small parameter size ($|\\theta _{cls}|$), so it is not modified in the model compression procedure. In the experiments, these fine-grained compression ratios can be optimized by random search on the validation data."]},{"idx":12,"heading":"Lightweight Adaptation of BERT ::: Knowledge distillation","paragraphs":["Knowledge distillation (KD) has been widely used to transfer knowledge from a large teacher model to a smaller student model. In other words, the student model mimics the behavior of the teacher model by minimize the knowledge distillation loss functions. Various types of knowledge distillation can be employed at different sub-layers. Generally, all types of knowledge distillation can be modeled as minimizing the following loss function:","Where $x$ indicates a sample input and $\\mathcal {X}$ is the training dataset. $f^{(s)}({x})$ and $f^{(t)}({x})$ represent intermediate outputs or weight matrices for the student model and teacher model correspondingly. $L(\\cdot )$ represents for a loss function which can be carefully defined for different types of knowledge distillation. We follow the recent technique proposed by TinyBERT BIBREF3, which applies knowledge distillation constraints upon embedding, self-attention, hidden representation and prediction levels. Concretely, there are four types of knowledge distillation constraints as follows:","Embedding-layer distillation is performed upon the embedding layer. $f({x}) \\in \\mathbb {R}^{n \\times d}$ represents for the word embedding output for input $x$, where $n$ is the input word length and $d$ is the dimension of word embedding. Mean Squared Error (MSE) is adopted as the loss function $L(\\cdot )$.","Attention-layer distillation is performed upon the self-attention sub-layer. $f({x}) = \\lbrace a_{ij}\\rbrace \\in \\mathbb {R}^{n \\times n}$ represents the attention output for each self-attention sub-layer, and $L(\\cdot )$ denotes MSE loss function.","Hidden-layer Distillation is performed at each fully-connected sub-layer in the Transformer architectures. $f({x})$ denotes the output representation of the corresponding sub-layer, and $L(\\cdot )$ also adopts MSE loss function.","Prediction-layer distillation makes the student model to learns the predictions from a teacher model directly. It is identical to the vanilla form of knowledge distillation BIBREF1. It takes the soft cross-entropy loss function, which is formulated as:","where $f^t({x})$ and $f^s({x})$ are the predictive logits of teacher and student models respectively."]},{"idx":13,"heading":"Experiments ::: Datasets & Baselines","paragraphs":["We compare LadaBERT with state-of-the-art model compression approaches on five public datasets of different tasks of natural language understanding, including sentiment classification (SST-2), natural language inference (MNLI-m, MNLI-mm, QNLI) and pairwise semantic equivalence (QQP). The statistics of these datasets are described in Table TABREF27.","The baseline approaches are summarized below.","Weight pruning and matrix factorization are two simple baselines described in Section SECREF2. We evaluate both pruning methods in an iterative manner until the target compression ratio is reached.","Hybrid pruning is a combination of matrix factorization and weight pruning, which conducts iterative weight pruning on the basis of SVD-based matrix factorization. It is performed iteratively until the desired compression ratio is achieved.","BERT-FT, BERT-KD and BERT-PKD are reported in BIBREF33, where BERT-FT directly fine-tunes the model via supervision labels, BERT-KD is the vanilla knowledge distillation algorithm BIBREF1, and BERT-PKD stands for Patient Knowledge Distillation proposed in BIBREF33. The student model is composed of 3 Transformer layers, resulting in a $2.5\\times $ compression ratio. Each layer has the same hidden size as the pre-trained teacher, so the initial parameters of student model can be inherited from the corresponding teacher.","TinyBERT BIBREF3 instantiates a tiny student model, which has totally 14.5M parameters ($7.5\\times $ compression ratio) composed of 4 layers, 312 hidden units, 1200 intermediate size and 12 heads. For a fair comparison, we reproduce the TinyBERT pipeline without general distillation and data augmentation, which is time-exhaustive and resource-consuming.","BERT-SMALL has the same model architecture as TinyBERT, but is directly pre-trained by the official BERT pipeline. The performance values are inherited from BIBREF3 for reference.","Distilled-BiLSTM BIBREF34 leverages a single-layer bidirectional-LSTM as the student model, where the hidden units and intermediate size are set to be 300 and 400 respectively, resulting in a $10.8 \\times $ compression ratio. This model requires a expensive pre-training process using the knowledge distillation constraints."]},{"idx":14,"heading":"Experiments ::: Setup","paragraphs":["We leverage the pre-trained checkpoint of base-bert-uncased as the initial model for compression, which contains 12 layers, 12 heads, 110M parameters, and 768 hidden units per layer. Hyper-parameter selection is conducted on the validation data for each dataset. After training, the prediction results are submitted to the GLUE-benchmark evaluation platform to get the evaluation performance on test data.","For a comprehensive evaluation, we experiment with four settings of LadaBERT, namely LadaBERT-1, -2, -3 and -4, which reduce the model parameters of BERT-Base by 2.5, 5, 7.5 and 10 times respectively. In our experiment, we take the batch size as 32, learning rate as 2e-5. The optimizer is BertAdam with default setting. Fine-grained compression ratios are optimized by random search and shown in Table TABREF38."]},{"idx":15,"heading":"Experiments ::: Performance Comparison","paragraphs":["The evaluation results of LadaBERT and state-of-the-art approaches are listed in Table TABREF40, where the models are ranked by parameter sizes for feasible comparison. As shown in the table, LadaBERT consistently outperforms the strongest baselines under similar model sizes. In addition, the performance of LadaBERT demonstrates the superiority of hybrid combination of SVD-based matrix factorization, weight pruning and knowledge distillation.","With model size of $2.5\\times $ reduction, LadaBERT-1 performs significantly better than BERT-PKD, boosting the performance by relative 8.9, 8.1, 6.1, 3.8 and 5.8 percentages on MNLI-m, MNLI-mm, SST-2, QQP and QNLI datasets respectively. Recall that BERT-PKD initializes the student model by selecting 3 of 12 layers in the pre-trained BERT-Base model. It turns out that the discarded layers have huge impact on the model performance, which is hard to be recovered by knowledge distillation. On the other hand, LadaBERT generates the student model by iterative pruning on the pre-trained teacher. In this way, the original knowledge in the teacher model can be preserved to the largest extent, and the benefit of which is complementary to knowledge distillation.","LadaBERT-3 has a comparable size as TinyBERT with a $7.5 \\times $ compression ratio. As shown in the results, TinyBERT does not work well without expensive data augmentation and general distillation, hindering its application to low-resource settings. The reason is that the student model of TinyBERT is distilled from scratch, so it requires much more data to mimic the teacher's behaviors. Instead, LadaBERT has better initial and intermediate status calculated by hybrid model compression, which is much more light-weighted and achieves competitive performances with much faster learning speed (learning curve comparison is shown in Section SECREF41). Moreover, LadaBERT-3 also outperforms BERT-SMALL on most of the datasets, which is pre-trained from scratch by the official BERT pipeline on a $7.5 \\times $ smaller architecture. This indicates that LadaBERT can quickly adapt to a smaller model size and achieve competitive performance without expansive re-training on a large corpus.","Moreover, Distilled-BiLSTM performs well on SST-2 dataset with more than $10 \\times $ compression ratio, perhaps owing to its advantage of generalization on small datasets. Nevertheless, the performance of LadaBERT-4 is competitive on larger datasets such as MNLI and QQP. This is impressive as LadaBERT is much more efficient without exhaustive re-training on a large corpus. In addition, the inference speed of BiLSTM is usually slower than transformer-based models with similar parameter sizes."]},{"idx":16,"heading":"Experiments ::: Learning curve comparison","paragraphs":["To further demonstrate the efficiency of LadaBERT, we visualize the learning curves on MNLI-m and QQP datasets in Figure FIGREF42 and FIGREF42, where LadaBERT-3 is compared to the strongest baseline, TinyBERT, under $7.5 \\times $ compression ratio. As shown in the figures, LadaBERT-3 achieves good performances much faster and results in a better convergence point. After training $2 \\times 10^4$ steps (batches) on MNLI-m dataset, the performance of LadaBERT-3 is already comparable to TinyBERT after convergence (approximately $2 \\times 10^5$ steps), achieving nearly $10 \\times $ acceleration. And on QQP dataset, both performance improvement and training speed acceleration is very significant. This clearly shows the superiority of combining matrix factorization, weight pruning and knowledge distillation in a reinforce manner. Instead, TinyBERT is based on pure knowledge distillation, so the learning speed is much slower."]},{"idx":17,"heading":"Experiments ::: Effect of low-rank + sparsity","paragraphs":["In this paper, we demonstrate that a combination of matrix factorization and weight pruning is better than single solutions for BERT-oriented model compression. Similar phenomena has been reported in the computer vision scenarios BIBREF28, which shows that low-rank and sparsity are complementary to each other. Here we provide another explanation to support this observation.","In Figure FIGREF44, we visualize the distribution of errors for a weight matrix in the neural network after pruning to 20% of its original parameter size. The errors can be calculated by $\\mathop {Error}=||\\hat{{M}}-{M}||_1$, where $\\hat{{M}}$ denotes the weight matrix after pruning.","The yellow line in Figure FIGREF44 shows the distribution of errors generated by pure weight pruning, which has a sudden drop at the pruning threshold. The orange line represents for pure SVD pruning, which turns out to be smoother and aligned with Gaussian distribution. The blue line shows the result of hybrid pruning, which conducts weight pruning on the decomposed matrices. First, we apply SVD-based matrix factorization to reduce 60% of total parameters. Then, weight pruning is applied on the decomposed matrices by 50%, resulting in only 20% parameters while the error distribution changes slightly. As a result, it has smaller mean and deviation than pure matrix factorization. In addition, a smoother distribution is more appropriate for the knowledge distillation procedure to fine-tune the weights, so it is advantageous than pure weight pruning."]},{"idx":18,"heading":"Conclusion","paragraphs":["Model compression is a common way to deal with latency-critical or memory-intensive scenarios. Existing model compression methods for BERT need to be re-trained on a large corpus to reserve its original performance, which is inapplicable in low-resource settings. In this paper, we propose LadaBERT to address this problem. LadaBERT is a lightweight model compression pipeline that generates adaptive BERT model efficiently based on a given task and specific constraint. It is based on a hybrid solution, which conducts matrix factorization, weight pruning and knowledge distillation in a reinforce manner. The experimental results verify that EAdaBERT is able to achieve comparable performance with other state-of-the-art solutions using much less training data and time budget. Therefore, LadaBERT can be easily plugged into various applications with competitive performances and little training overheads. In the future, we would like to apply LadaBERT to large-scale industrial applications, such as search relevance and query recommendation."]},{"idx":19,"heading":"Figures and Tables","paragraphs":["Figure 1: Accuracy comparison on MNLI-m dataset","Figure 2: Overview of LadaBERT framework","Table 1: Dataset Statistics","Table 2: Fine-grained compression ratios","Table 3: Performance comparison on various model sizes","Figure 3: Learning curve on MNLI-m dataset. Figure 4: Learning curve on QQP dataset.","Figure 5: Distribution of pruning errors"]}],"source":"qasper","license":"CC-BY-4.0"}
{"doc_id":"2004.04721","title":"Translation Artifacts in Cross-lingual Transfer Learning","sections":[{"idx":0,"heading":"Abstract","paragraphs":["Both human and machine translation play a central role in cross-lingual transfer learning: many multilingual datasets have been created through professional translation services, and using machine translation to translate either the test set or the training set is a widely used transfer technique. In this paper, we show that such translation process can introduce subtle artifacts that have a notable impact in existing cross-lingual models. For instance, in natural language inference, translating the premise and the hypothesis independently can reduce the lexical overlap between them, which current models are highly sensitive to. We show that some previous findings in cross-lingual transfer learning need to be reconsidered in the light of this phenomenon. Based on the gained insights, we also improve the state-of-the-art in XNLI for the translate-test and zero-shot approaches by 4.3 and 2.8 points, respectively."]},{"idx":1,"heading":"Introduction","paragraphs":["While most NLP resources are English-specific, there have been several recent efforts to build multilingual benchmarks. One possibility is to collect and annotate data in multiple languages separately BIBREF0, but most existing datasets have been created through translation BIBREF1, BIBREF2. This approach has two desirable properties: it relies on existing professional translation services rather than requiring expertise in multiple languages, and it results in parallel evaluation sets that offer a meaningful measure of the cross-lingual transfer gap of different models. The resulting multilingual datasets are generally used for evaluation only, relying on existing English datasets for training.","Closely related to that, cross-lingual transfer learning aims to leverage large datasets available in one language—typically English—to build multilingual models that can generalize to other languages. Previous work has explored 3 main approaches to that end: machine translating the test set into English and using a monolingual English model (Translate-Test), machine translating the training set into each target language and training the models on their respective languages (Translate-Train), or using English data to fine-tune a multilingual model that is then transferred to the rest of languages (Zero-Shot).","The dataset creation and transfer procedures described above result in a mixture of original, human translated and machine translated data when dealing with cross-lingual models. In fact, the type of text a system is trained on does not typically match the type of text it is exposed to at test time: Translate-Test systems are trained on original data and evaluated on machine translated test sets, Zero-Shot systems are trained on original data and evaluated on human translated test sets, and Translate-Train systems are trained on machine translated data and evaluated on human translated test sets.","Despite overlooked to date, we show that such mismatch has a notable impact in the performance of existing cross-lingual models. By using back-translation BIBREF3 to paraphrase each training instance, we obtain another English version of the training set that better resembles the test set, obtaining substantial improvements for the Translate-Test and Zero-Shot approaches in cross-lingual Natural Language Inference (NLI). While improvements brought by machine translation have previously been attributed to data augmentation BIBREF4, we reject this hypothesis and show that the phenomenon is only present in translated test sets, but not in original ones. Instead, our analysis reveals that this behavior is caused by subtle artifacts arising from the translation process itself. In particular, we show that translating different parts of each instance separately (e.g. the premise and the hypothesis in NLI) can alter superficial patterns in the data (e.g. the degree of lexical overlap between them), which severely affects the generalization ability of current models. Based on the gained insights, we improve the state-of-the-art in XNLI, and show that some previous findings need to be reconsidered in the light of this phenomenon."]},{"idx":2,"heading":"Related work ::: Cross-lingual transfer learning.","paragraphs":["Current cross-lingual models work by pre-training multilingual representations using some form of language modeling, which are then fine-tuned on the relevant task and transferred to different languages. Some authors leverage parallel data to that end BIBREF5, BIBREF6, but training a model akin to BERT BIBREF7 on the combination of monolingual corpora in multiple languages is also effective BIBREF8. Closely related to our work, BIBREF4 showed that replacing segments of the training data with their translation during fine-tuning is helpful. However, they attribute this behavior to a data augmentation effect, which we believe should be reconsidered given the new evidence we provide."]},{"idx":3,"heading":"Related work ::: Multilingual benchmarks.","paragraphs":["Most benchmarks covering a wide set of languages have been created through translation, as it is the case of XNLI BIBREF1 for NLI, PAWS-X BIBREF9 for adversarial paraphrase identification, and XQuAD BIBREF2 and MLQA BIBREF10 for Question Answering (QA). A notable exception is TyDi QA BIBREF0, a contemporaneous QA dataset that was separately annotated in 11 languages. Other cross-lingual datasets leverage existing multilingual resources, as it is the case of MLDoc BIBREF11 for document classification and Wikiann BIBREF12 for named entity recognition. Concurrent to our work, BIBREF13 combine some of these datasets into a single multilingual benchmark, and evaluate some well-known methods on it."]},{"idx":4,"heading":"Related work ::: Annotation artifacts.","paragraphs":["Several studies have shown that NLI datasets like SNLI BIBREF14 and MultiNLI BIBREF15 contain spurious patterns that can be exploited to obtain strong results without making real inferential decisions. For instance, BIBREF16 and BIBREF17 showed that a hypothesis-only baseline performs better than chance due to cues on their lexical choice and sentence length. Similarly, BIBREF18 showed that NLI models tend to predict entailment for sentence pairs with a high lexical overlap. Several authors have worked on adversarial datasets to diagnose these issues and provide a more challenging benchmark BIBREF19, BIBREF20, BIBREF21. Besides NLI, other tasks like QA have also been found to be susceptible to annotation artifacts BIBREF22, BIBREF23. While previous work has focused on the monolingual scenario, we show that translation can interfere with these artifacts in multilingual settings."]},{"idx":5,"heading":"Related work ::: Translationese.","paragraphs":["Translated texts are known to have unique features like simplification, explicitation, normalization and interference, which are refer to as translationese BIBREF24. This phenomenon has been reported to have a notable impact in machine translation evaluation BIBREF25, BIBREF26. For instance, back-translation brings large BLEU gains for reversed test sets (i.e. when translationese is on the source side and original text is used as reference), but its effect diminishes in the natural direction BIBREF27. While connected, the phenomenon we analyze is different in that it arises from translation inconsistencies due to the lack of context, and affects cross-lingual transfer learning rather than machine translation."]},{"idx":6,"heading":"Experimental design","paragraphs":["Our goal is to analyze the effect of both human and machine translation in cross-lingual models. For that purpose, the core idea of our work is to (i) use machine translation to either translate the training set into other languages, or generate English paraphrases of it through back-translation, and (ii) evaluate the resulting systems on original, human translated and machine translated test sets in comparison with systems trained on original data. We next describe the models used in our experiments (§SECREF6), the specific training variants explored (§SECREF8), and the evaluation procedure followed (§SECREF10)."]},{"idx":7,"heading":"Experimental design ::: Models and transfer methods","paragraphs":["We experiment with two models that are representative of the state-of-the-art in monolingual and cross-lingual pre-training: (i) Roberta BIBREF28, which is an improved version of BERT that uses masked language modeling to pre-train an English Transformer model, and (ii) XLM-R BIBREF8, which is a multilingual extension of the former pre-trained on 100 languages. In both cases, we use the large models released by the authors under the fairseq repository. As discussed next, we explore different variants of the training set to fine-tune each model on different tasks. At test time, we try both machine translating the test set into English (Translate-Test) and, in the case of XLM-R, using the actual test set in the target language (Zero-Shot)."]},{"idx":8,"heading":"Experimental design ::: Training variants","paragraphs":["We try 3 variants of each training set to fine-tune our models: (i) the original one in English (Orig), (ii) an English paraphrase of it generated through back-translation using Spanish or Finnish as pivot (BT-ES and BT-FI), and (iii) a machine translated version in Spanish or Finnish (MT-ES and MT-FI). For sentences occurring multiple times in the training set (e.g. premises repeated for multiple hypotheses), we use the exact same translation for all occurrences, as our goal is to understand the inherent effect of translation rather than its potential application as a data augmentation method.","In order to train the machine translation systems for MT-XX and BT-XX, we use the big Transformer model BIBREF29 with the same settings as BIBREF30 and SentencePiece tokenization BIBREF31 with a joint vocabulary of 32k subwords. For English-Spanish, we train for 10 epochs on all parallel data from WMT 2013 BIBREF32 and ParaCrawl v5.0 BIBREF33. For English-Finnish, we train for 40 epochs on Europarl and Wiki Titles from WMT 2019 BIBREF34, ParaCrawl v5.0, and DGT, EUbookshop and TildeMODEL from OPUS BIBREF35. In both cases, we remove sentences longer than 250 tokens, with a source/target ratio exceeding 1.5, or for which langid.py BIBREF36 predicts a different language, resulting in a final corpus size of 48M and 7M sentence pairs, respectively. We use sampling decoding with a temperature of 0.5 for inference, which produces more diverse translations than beam search BIBREF37 and performed better in our preliminary experiments."]},{"idx":9,"heading":"Experimental design ::: Tasks and evaluation procedure","paragraphs":["We use the following tasks for our experiments:"]},{"idx":10,"heading":"Experimental design ::: Tasks and evaluation procedure ::: Natural Language Inference (NLI).","paragraphs":["Given a premise and a hypothesis, the task is to determine whether there is an entailment, neutral or contradiction relation between them. We fine-tune our models on MultiNLI BIBREF15 for 10 epochs using the same settings as BIBREF28. In most of our experiments, we evaluate on XNLI BIBREF1, which comprises 2490 development and 5010 test instances in 15 languages. These were originally annotated in English, and the resulting premises and hypotheses were independently translated into the rest of the languages by professional translators. For the Translate-Test approach, we use the machine translated versions from the authors. Following BIBREF8, we select the best epoch checkpoint according to the average accuracy in the development set."]},{"idx":11,"heading":"Experimental design ::: Tasks and evaluation procedure ::: Question Answering (QA).","paragraphs":["Given a context paragraph and a question, the task is to identify the span answering the question in the context. We fine-tune our models on SQuAD v1.1 BIBREF38 for 2 epochs using the same settings as BIBREF28, and report test results for the last epoch. We use two datasets for evaluation: XQuAD BIBREF2, a subset of the SQuAD development set translated into 10 other languages, and MLQA BIBREF10 a dataset consisting of parallel context paragraphs plus the corresponding questions annotated in English and translated into 6 other languages. In both cases, the translation was done by professional translators at the document level (i.e. when translating a question, the text answering it was also shown). For our BT-XX and MT-XX variants, we translate the context paragraph and the questions independently, and map the answer spans using the same procedure as BIBREF39. For the Translate-Test approach, we use the official machine translated versions of MLQA, run inference over them, and map the predicted answer spans back to the target language.","Both for NLI and QA, we run each system 5 times with different random seeds and report the average results. Space permitting, we also report the standard deviation across the 5 runs."]},{"idx":12,"heading":"NLI experiments","paragraphs":["We next discuss our main results in the XNLI development set (§SECREF15, §SECREF16), run additional experiments to better understand the behavior of our different variants (§SECREF17, §SECREF22, §SECREF25), and compare our results to previous work in the XNLI test set (§SECREF30)."]},{"idx":13,"heading":"NLI experiments ::: Translate-Test results","paragraphs":["We start by analyzing XNLI development results for Translate-Test. Recall that, in this approach, the test set is machine translated into English, but training is typically done on original English data. Our BT-ES and BT-FI variants close this gap by training on a machine translated English version of the training set generated through back-translation. As shown in Table TABREF9, this brings substantial gains for both Roberta and XLM-R, with an average improvement of 4.6 points in the best case. Quite remarkably, MT-ES and MT-FI also outperform Orig by a substantial margin, and are only 0.8 points below their BT-ES and BT-FI counterparts. Recall that, for these two systems, training is done in machine translated Spanish or Finnish, while inference is done in machine translated English. This shows that the loss of performance when generalizing from original data to machine translated data is substantially larger than the loss of performance when generalizing from one language to another."]},{"idx":14,"heading":"NLI experiments ::: Zero-Shot results","paragraphs":["We next analyze the results for the Zero-Shot approach. In this case, inference is done in the test set in each target language which, in the case of XNLI, was human translated from English. As such, different from the Translate-Test approach, neither training on original data (Orig) nor training on machine translated data (BT-XX and MT-XX) makes use of the exact same type of text that the system is exposed to at test time. However, as shown in Table TABREF9, both BT-XX and MT-XX outperform Orig by approximately 2 points, which suggests that our (back-)translated versions of the training set are more similar to the human translated test sets than the original one. This also provides a new perspective on the Translate-Train approach, which was reported to outperform Orig in previous work BIBREF5: while the original motivation was to train the model on the same language that it is tested on, our results show that machine translating the training set is beneficial even when the target language is different."]},{"idx":15,"heading":"NLI experiments ::: Original vs. translated test sets","paragraphs":["So as to understand whether the improvements observed so far are limited to translated test sets or apply more generally, we conduct additional experiments comparing translated test sets to original ones. However, to the best of our knowledge, all existing non-English NLI benchmarks were created through translation. For that reason, we build a new test set that mimics XNLI, but is annotated in Spanish rather than English. We first collect the premises from a filtered version of CommonCrawl BIBREF42, taking a subset of 5 websites that represent a diverse set of genres: a newspaper, an economy forum, a celebrity magazine, a literature blog, and a consumer magazine. We then ask native Spanish annotators to generate an entailment, a neutral and a contradiction hypothesis for each premise. We collect a total of 2490 examples using this procedure, which is the same size as the XNLI development set. Finally, we create a human translated and a machine translated English version of the dataset using professional translators from Gengo and our machine translation system described in §SECREF8, respectively. We report results for the best epoch checkpoint on each set.","As shown in Table TABREF18, both BT-XX and MT-XX clearly outperform Orig in all test sets created through translation, which is consistent with our previous results. In contrast, the best results on the original English set are obtained by Orig, and neither BT-XX nor MT-XX obtain any clear improvement on the one in Spanish either. This confirms that the underlying phenomenon is limited to translated test sets. In addition, it is worth mentioning that the results for the machine translated test set in English are slightly better than those for the human translated one, which suggests that the difficulty of the task does not only depend on the translation quality. Finally, it is also interesting that MT-ES is only marginally better than MT-FI in both Spanish test sets, even if it corresponds to the Translate-Train approach, whereas MT-FI needs to Zero-Shot transfer from Finnish into Spanish. This reinforces the idea that it is training on translated data rather than training on the target language that is key in Translate-Train."]},{"idx":16,"heading":"NLI experiments ::: Stress tests","paragraphs":["In order to better understand how systems trained on original and translated data differ, we run additional experiments on the NLI Stress Tests BIBREF19, which were designed to test the robustness of NLI models to specific linguistic phenomena in English. The benchmark consists of a competence test, which evaluates the ability to understand antonymy relation and perform numerical reasoning, a distraction test, which evaluates the robustness to shallow patterns like lexical overlap and the presence of negation words, and a noise test, which evaluates robustness to spelling errors. Just as with previous experiments, we report results for the best epoch checkpoint in each test set.","As shown in Table TABREF23, Orig outperforms BT-FI and MT-FI on the competence test by a large margin, but the opposite is true on the distraction test. In particular, our results show that BT-FI and MT-FI are less reliant on lexical overlap and the presence of negative words. This feels intuitive, as translating the premise and hypothesis independently—as BT-FI and MT-FI do—is likely to reduce the lexical overlap between them. More generally, the translation process can alter similar superficial patterns in the data, which NLI models are sensitive to (§SECREF2). This would explain why the resulting models have a different behavior on different stress tests."]},{"idx":17,"heading":"NLI experiments ::: Output class distribution","paragraphs":["With the aim to understand the effect of the previous phenomenon in cross-lingual settings, we look at the output class distribution of our different models in the XNLI development set. As shown in Table TABREF28, the predictions of all systems are close to the true class distribution in the case of English. Nevertheless, Orig is strongly biased for the rest of languages, and tends to underpredict entailment and overpredict neutral. This can again be attributed to the fact that the English test set is original, whereas the rest are human translated. In particular, it is well-known that NLI models tend to predict entailment when there is a high lexical overlap between the premise and the hypothesis (§SECREF2). However, the degree of overlap will be smaller in the human translated test sets given that the premise and the hypothesis were translated independently, which explains why entailment is underpredicted. In contrast, BT-FI and MT-FI are exposed to the exact same phenomenon during training, which explains why they are not that heavily affected.","So as to measure the impact of this phenomenon, we explore a simple approach to correct this bias: having fine-tuned each model, we adjust the bias term added to the logit of each class so the model predictions match the true class distribution for each language. As shown in Table TABREF29, this brings large improvements for Orig, but is less effective for BT-FI and MT-FI. This shows that the performance of Orig was considerably hindered by this bias, which BT-FI and MT-FI effectively mitigate."]},{"idx":18,"heading":"NLI experiments ::: Comparison with the state-of-the-art","paragraphs":["So as to put our results into perspective, we compare our best variant to previous work on the XNLI test set. As shown in Table TABREF31, our method improves the state-of-the-art for both the Translate-Test and the Zero-Shot approaches by 4.3 and 2.8 points, respectively. It also obtains the best overall results published to date, with the additional advantage that the previous state-of-the-art required a machine translation system between English and each of the 14 target languages, whereas our method uses a single machine translation system between English and Finnish (which is not one of the target languages). While the main goal of our work is not to design better cross-lingual models, but to analyze their behavior in connection to translation, this shows that the phenomenon under study is highly relevant, to the extent that it can be exploited to improve the state-of-the-art."]},{"idx":19,"heading":"QA experiments","paragraphs":["So as to understand whether our previous findings apply to other tasks besides NLI, we run additional experiments on QA. As shown in Table TABREF32, BT-FI and BT-ES do indeed outperform Orig for the Translate-Test approach on MLQA. The improvement is modest, but very consistent across different languages, models and runs. The results for MT-ES and MT-FI are less conclusive, presumably because mapping the answer spans across languages might introduce some noise. In contrast, we do not observe any clear improvement for the Zero-Shot approach on this dataset. Our XQuAD results in Table TABREF33 are more positive, but still inconclusive.","These results can partly be explained by the translation procedure used to create the different benchmarks: the premises and hypotheses of XNLI were translated independently, whereas the questions and context paragraphs of XQuAD were translated together. Similarly, MLQA made use of parallel contexts, and translators were shown the sentence containing each answer when translating the corresponding question. As a result, one can expect both QA benchmarks to have more consistent translations than XNLI, which would in turn diminish this phenomenon. In contrast, the questions and context paragraphs are independently translated when using machine translation, which explains why BT-ES and BT-FI outperform Orig for the Translate-Test approach. We conclude that the translation artifacts revealed by our analysis are not exclusive to NLI, as they also show up on QA for the Translate-Test approach, but their actual impact can be highly dependent on the translation procedure used and the nature of the task."]},{"idx":20,"heading":"Discussion","paragraphs":["Our analysis prompts to reconsider previous findings in cross-lingual transfer learning as follows:"]},{"idx":21,"heading":"Discussion ::: The cross-lingual transfer gap on XNLI was overestimated.","paragraphs":["Given the parallel nature of XNLI, accuracy differences across languages are commonly interpreted as the loss of performance when generalizing from English to the rest of languages. However, our work shows that there is another factor that can have a much larger impact: the loss of performance when generalizing from original to translated data. Our results suggest that the real cross-lingual generalization ability of XLM-R is considerably better than what the accuracy numbers in XNLI reflect."]},{"idx":22,"heading":"Discussion ::: Overcoming the cross-lingual gap is not what makes Translate-Train work.","paragraphs":["The original motivation for Translate-Train was to train the model on the same language it is tested on. However, we show that it is training on translated data, rather than training on the target language, that is key for this approach to outperform Zero-Shot as reported by previous authors."]},{"idx":23,"heading":"Discussion ::: Improvements previously attributed to data augmentation should be reconsidered.","paragraphs":["The method by BIBREF4 combines machine translated premises and hypotheses in different languages (§SECREF2), resulting in an effect similar to BT-XX and MT-XX. As such, we believe that this method should be analyzed from the point of view of dataset artifacts rather than data augmentation, as the authors do. From this perspective, having the premise and the hypotheses in different languages can reduce the superficial patterns between them, which would explain why this approach is better than using examples in a single language."]},{"idx":24,"heading":"Discussion ::: The potential of Translate-Test was underestimated.","paragraphs":["The previous best results for Translate-Test on XNLI lagged behind the state-of-the-art by 4.6 points. Our work reduces this gap to only 0.8 points by addressing the underlying translation artifacts. The reason why Translate-Test is more severely affected by this phenomenon is twofold: (i) the effect is doubled by first using human translation to create the test set and then machine translation to translate it back to English, and (ii) Translate-Train was inadvertently mitigating this issue (see above), but equivalent techniques were never applied to Translate-Test."]},{"idx":25,"heading":"Discussion ::: Future evaluation should better account for translation artifacts.","paragraphs":["The evaluation issues raised by our analysis do not have a simple solution. In fact, while we use the term translation artifacts to highlight that they are an unintended effect of translation that impacts final evaluation, one could also argue that it is the original datasets that contain the artifacts, which translation simply alters or even mitigates. In any case, this is a more general issue that falls beyond the scope of cross-lingual transfer learning, so we argue that it should be carefully controlled when evaluating cross-lingual models. In the absence of more robust datasets, we recommend that future multilingual benchmarks should at least provide consistent test sets for English and the rest of languages. This can be achieved by (i) using original annotations in all languages, (ii) using original annotations in a non-English language and translating them into English and other languages, or (iii) if translating from English, doing so at the document level to minimize translation inconsistencies."]},{"idx":26,"heading":"Conclusions","paragraphs":["In this paper, we have shown that both human and machine translation can alter superficial patterns in data, which requires reconsidering previous findings in cross-lingual transfer learning. Based on the gained insights, we have improved the state-of-the-art in XNLI for the Translate-Test and Zero-Shot approaches by a substantial margin. Finally, we have shown that the phenomenon is not specific to NLI but also affects QA, although it is less pronounced there thanks to the translation procedure used in the corresponding benchmarks. So as to facilitate similar studies in the future, we release our NLI dataset, which, unlike previous benchmarks, was annotated in a non-English language and human translated into English."]},{"idx":27,"heading":"Acknowledgments","paragraphs":["We thank Nora Aranberri and Uxoa Iñurrieta for helpful discussion during the development of this work, as well as the rest of our colleagues from the IXA group that worked as annotators for our NLI dataset.","This research was partially funded by a Facebook Fellowship, the Basque Government excellence research group (IT1343-19), the Spanish MINECO (UnsupMT TIN2017‐91692‐EXP MCIU/AEI/FEDER, UE), Project BigKnowledge (Ayudas Fundación BBVA a equipos de investigación científica 2018), and the NVIDIA GPU grant program."]},{"idx":28,"heading":"Figures and Tables","paragraphs":["Table 1: XNLI dev results (acc). BT-XX and MT-XX consistently outperform ORIG in all cases.","Table 2: NLI results on original (OR), human translated (HT) and machine translated (MT) sets (acc). BT-XX and MT-XX outperform ORIG in translated sets, but do not get any clear improvement in original ones.","Table 3: NLI Stress Test results (combined matched & mismatched acc). AT = antonymy, NR = numerical reasoning, WO = word overlap, NG = negation, LN = length mismatch, SE = spelling error. BT-FI and MT-FI are considerably weaker than ORIG in the competence test, but substantially stronger in the distraction test.","Table 4: Output class distribution on XNLI dev. All systems are close to the true distribution in English, but ORIG is biased toward neu and con in the transfer languages. BT-FI and MT-FI alleviate this issue.","Table 5: XNLI dev results with class distribution unbiasing (average acc across all languages). Adjusting the bias term of the classifier to match the true class distribution brings large improvements for ORIG, but is less effective for BT-FI and MT-FI.","Table 6: XNLI test results (acc). Results for other methods are taken from their respective papers or, if not provided, from Conneau et al. (2019). For those with multiple variants, we select the one with the best results.","Table 7: MLQA test results (F1 / exact match).","Table 8: XQuAD results (F1). Results for the exact match metric are similar."]}],"source":"qasper","license":"CC-BY-4.0"}