8)   $C$ is an attention matrix. Note that $\hat{H^q}$ and $\hat{H^p}$ is transformed from $H^q$ and $H^p$ by one layer neural network $ReLU(W_3x)$ , respectively. Next, we gather all the information on passages by a simple concatenation of its contextual information $H^p$ and its question-aware representation $H^q \cdot C$ :  $$U^p = concat(H^p, H^qC) \in \mathbb {R}^{4d \times n}$$   (Eq. 9)  Typically, a passage may contain hundred of tokens, making it hard to learn the long dependencies within it. Inspired by  , we apply a self-attended layer to rearrange the information $U^p$ as:  $$\hat{U}^p = U^p drop_{diag}(f_{attention}(U^p, U^p)).$$   (Eq.


Predicting political views	Parties change their policies and positions in the political spectrum. More reliable categories for political bias are party independent labels for political views, see sec:data. A separate suite of experiments was run to train and test the prediction performance of the text classifiers models described in sec:model. As there was no out-of-domain evaluation set available in this setting only evaluation error on in-domain data is reported. Note however that also in this experiment the evaluation data was never seen by any model during training time. In tab:resultsavgpoliticalview results for the best and worst classes, in terms of predictability, are listed along with the average performance metrics on all classes. Precision and recall values of close to 0.5 on average can be considered rather high considering the large number of labels.


Joint-BiLSTM: Different from S2VT-based models, we employ a joint bidirectional LSTM networks to encode video sequence and decode description applying another LSTM respectively rather than sharing the common one. We stack two layers of LSTM networks to encode video and pre-train language model as in S2VT-based models. Similarly, unidirectional LSTM, bidirectional LSTM and reinforced BiLSTM are executed to investigate the performance of each structure. We set 1024 hidden units of the first LSTM in unidirectional encoder so that the output could pass to the second encoder directly, and the memory cell and hidden state of the last time point are applied to initialize description decoder. Bidirectional structure and reinforced BiLSTM in encoder are implemented similarly to the corresponding type structure in S2VT-based models, respectively, and then feed the video representation into description generator as the unidirectional model aforementioned.


Small-Footprint Keyword Spotting on Raw Audio Data with Sinc-Convolutions	Keyword Spotting (KWS) enables speech-based user interaction on smart devices. Always-on and battery-powered application scenarios for smart devices put constraints on hardware resources and power consumption, while also demanding high accuracy as well as real-time capability. Previous architectures first extracted acoustic features and then applied a neural network to classify keyword probabilities, optimizing towards memory footprint and execution time. Compared to previous publications, we took additional steps to reduce power and memory consumption without reducing classification accuracy. Power-consuming audio preprocessing and data transfer steps are eliminated by directly classifying from raw audio. For this, our end-to-end architecture extracts spectral features using parametrized Sinc-convolutions. Its memory footprint is further reduced by grouping depthwise separable convolutions. Our network achieves the competitive accuracy of 96.4% on Google's Speech Commands test set with only 62k parameters.


You can notice that we increased the number of languages from the first review of our paper. In addition, the table shows the comparison between our baseline method (Least Squares) and our final one (Normalized vectors, with cosine similarity), which shows a slight improvement with respect to the first one.


Discussion & Conclusions	We established that our Siamese recurrent networks (with SRN, GRU or LSTM cells) are able to recognize logical entailment relations without any a priori cues about syntax or semantics of the input expressions. Indeed, some of the recurrent set-ups even outperform tree-shaped networks, whose topology is specifically designed to deal with such tasks. This indicates that recurrent networks can develop representations that can adequately process a formal language with a nontrivial hierarchical structure. The formal language we defined did not exploit the full expressive power of first-order predicate logic; nevertheless by using standard first-order predicate logic, a standard theorem prover, and a set-up where the training set only covers a tiny fraction of the space of possible logical expressions, our experiments avoid the problems observed in earlier attempts to demonstrate logical reasoning in recurrent networks. The experiments performed in the last few sections moreover show that the GRU and LSTM architectures exhibit at least basic forms of compositional generalization. In particular, the results of the zero-shot generalization experiments with novel lengths and novel words cannot be explained with a `memorize-and-interpolate' account, i.e.


However when performing the same comparison between our large models (see Table TABREF24), this performance gap becomes significantly smaller. As expected, not using the multi-span head causes the multi-span performance to plummet. Note that for this ablation test the single-span heads were permitted to train on multi-span questions. Compared to using greedy decoding in the prediction of multi-span questions, using beam search results in a small improvement. We used a beam with of 5, and didn't perform extensive tuning of the beam width.


Corpus Annotation ::: Extraction of PIE Candidates	To annotate the corpus completely manually would require annotators to read the whole corpus, and cross-reference each sentence to a list of almost 600 PIEs, to check whether one of those PIEs occurs in a sentence. We do not consider this a feasible annotation settings, due to both the difficulty of recognising literal usages of idioms and the time cost needed to find enough PIEs, given their low overall frequency. As such, we use a pre-extraction step to present candidates for annotation to the human annotators. Given the corpus and the set of PIEs, we heuristically extract the PIE candidates as follows: given an idiomatic expression, extract every sentence which contains all the defining words of the idiom, in any form. This ensures that all possibly matching sentences get extracted, while greatly pruning the amount of sentences for annotators to look at. In addition, it allows us to present the heuristically matched PIE type and corresponding words to the annotators, which makes it much easier to judge whether something is a PIE or not. This also means that annotators never have to go through the full list of PIEs during the annotation process.


Experiments and Results ::: Experimental Settings	Data Splits: We used the same splits of train, develop, and test sets used by Karlekar and Bansal , with 7201, 990 and 1701 stories, respectively. In this study, we only considered single label classifications. Baseline Models: CNN and BiLSTM models that perform classification and extraction separately were used as baseline models. In classification, we also experimented with BiLSTM with the attention layer. To demonstrate that the improvement came from joint learning structure rather the two layer structure in J-CNN, we investigated the same model structure without training on key element extraction. We use J-CNN* to denote it. Preprocess: All the texts were converted to lowercase and preprocessed by removing non-alphanumeric characters, excluding “. ! ? ” . The word embeddings were pre-trained using fastText  with dimension equaling 100. Hyperparameters: For the CNN model, the filter size was chosen to be (1,2,3,4), with 50 filters per filter size. Batch size was set to 50 and the dropout rate was 0.5.


Proposed System ::: The Crawler ::: Decide	A decider has three main decisions to take. First, based on the metadata associated with an URL, should it be visited? In practice, we visit only new URLs, but the tool is designed in a way such that a recrawl is possible if the page is detected as highly dynamic. The second decision arises at the end of the processing, where the page can be either saved or blacklisted. To favor recall, we currently keep any URL with at least one GSW sentence. Finally, the decider can choose to visit the outgoing links or not. After some trials, we found that following links from pages with more than two new GSW sentences is a reasonable choice, as pages with less sentences are often quotes or false positives.


It is the same with the second example, where a human selects “UTF8gbsn专家” (expert) and Dual-Train selects “UTF8gbsn工作者” (worker), while the baseline selects “UTF8gbsn钻研” (research) and fails to generate a grammatical sentence later. In the third one, the reference and the baseline use the same word, while Dual-Train chooses a word of the same meaning. It can be concluded that Dual-Train indeed learns better word relations that could generalize to the test set, and good word relations can guide the decoder to generate semantically consistent summaries. To show why the generated text of the proposed method is more related to the source content, we further analyze the label distribution, i.e., the word distribution, generated by the (first) output layer, from which the output word is selected. To illustrate the relationship, we calculate a representation for each word based on the label distributions. Each representation is associated with a specific label (word), denoted by INLINEFORM0 , and each dimension INLINEFORM1 shows how likely the label indexed by INLINEFORM2 will be generated instead of the label INLINEFORM3 .


We mask (exclude) all words of the vocabulary except the current source sentence tokens, relation tokens, separator tokens (`;', `$\vert $'), UNK, and EOS tokens in the softmax operation. To mask (exclude) some word from softmax, we set the corresponding value in $\hat{\mathbf {o}}_t$ at $-\infty $ and the corresponding softmax score will be zero. This ensures the copying of entities from the source sentence only. We include the UNK token in the softmax operation to make sure that the model generates new entities during inference. If the decoder predicts an UNK token, we replace it with the corresponding source word which has the highest attention score. During inference, after decoding is finished, we extract all tuples based on the special tokens, remove duplicate tuples and tuples in which both entities are the same or tuples where the relation token is not from the relation set. This model is referred to as WordDecoding (WDec) henceforth.


We denote this categorical variable as the "head predictor". Numerically-aware BERT (NABERT+)  introduced two main improvements over NAQANET. The first was to replace the QANET encoder with BERT. This change alone resulted in an absolute improvement of more than eight points in both EM and F1 metrics. The second improvement was to the arithmetic head, consisting of the addition of "standard numbers" and "templates". Standard numbers were predefined numbers which were added as additional inputs to the arithmetic head, regardless of their occurrence in the passage. Templates were an attempt to enrich the head's arithmetic capabilities, by adding the ability of doing simple multiplications and divisions between up to three numbers. MTMSN  is the first, and only model so far, that specifically tried to tackle the multi-span questions of DROP. Their approach consisted of two parts. The first was to train a dedicated categorical variable to predict the number of spans to extract. The second was to generalize the single-span head method of extracting a span, by utilizing the non-maximum suppression (NMS) algorithm  to find the most probable set of non-overlapping spans.


Acknowledgments	This research was supported by the Swiss National Science Foundation (SNSF) grant number 407540_167176 under the project "Conversational Agent for Interactive Access to Information".


In this way, we filter out i) articles with excessively long content to reduce memory consumption; ii) very short leading sentences with little information which are unlikely to be a good summary. To encourage the model to generate abstrative summaries, we also remove articles where any of the top three sentences is exactly repeated in the rest of the article. Third, we try to remove articles whose top three sentences may not form a relevant summary. For this purpose, we utilize a simple metric: overlapping words. We compute the portion of non-stopping words in the top three sentences that are also in the rest of an article. A higher portion implies that the summary is representative and has a higher chance of being inferred by the model using the rest of the article. To verify, we compute the overlapping ratio of non-stopping words between human-edited summary and the article in CNN/DailyMail dataset, which has a median value of 0.87. Therefore, in pretraining, we keep articles with an overlapping word ratio higher than 0.65.


Modelos propuestos ::: Modelo generativo estocástico usando cadenas de Markov	Este modelo generativo, que llamaremos Modelo de Markov, está basado en el algoritmo de Viterbi y las cadenas de Markov , donde se selecciona una etiqueta POS con la máxima probabilidad de ocurrencia, para ser agregada al final de la secuencia actual. Utilizamos el corpus de frases literarias 8KF (ver Sección SECREF5), que fue convenientemente filtrado para eliminar tokens indeseables: números, siglas, horas y fechas. El corpus filtrado se analizó usando Freeling, que recibe en entrada una cadena de texto y entrega el texto con una etiqueta POS para cada palabra. El corpus es analizado frase a frase, reemplazando cada palabra por su respectiva etiqueta POS. Al final del análisis, se obtiene un nuevo corpus 8KPOS con $s = 7~679$ secuencias de etiquetas POS, correspondientes al mismo número de frases del corpus 8KF.


Our classifier features: In the following sections, we discuss related work on hate speech classification, followed by a description of the datasets, methods and results of our study.


Experiments ::: Baselines ::: CVAE.	An RNN-based conditional variational autoencoder for dialogue response generation , which uses a multivariate Gaussian latent variable to model the response and concatenate it with the last hidden state of the encoder as the initial state of the decoder. KL annealing, early stopping strategy and bag-of-word auxiliary loss are applied during the training. We use the implementation released by .


Type Classification	We employ a multi-class Naive Bayes classifier as the second stage classification mechanism, for categorizing tweets appropriately, depending on the type of emergencies they indicate. This multi-class classifier is trained on data manually labeled with classes. We tokenize the training data using “NgramTokenizer” and then, apply a filter to create word vectors of strings before training. We use “trigrams” as features to build a model which, later, classifies tweets into appropriate categories, in real time. We then perform cross validation using standard techniques to calculate the results, which are shown under the label “Stage 2”, in table TABREF20 .


Methodology ::: Interval Bound Propagation ::: Verifiable Training.	The upper bound in (DISPLAY_FORM17) is fast to compute (only requires two forward passes for upper and lower bounds through the network). Hence, we can define a loss to optimise models such that the models are trained to be verifiable. Solving (DISPLAY_FORM17) is equivalent to finding the worst-case logit difference, and this is achieved when the logit of the true class is equal to its lower bound, and all other logits equal to their upper bounds. Concretely, for each class $y \ne y_\textrm {true} $: $\hat{\mathbf {z}}_{K,y}(\delta ) = \overline{\mathbf {z}}_{K,y} (\delta ) $, and $\hat{\mathbf {z}}_{K,y_\textrm {true}}(\delta ) = \underline{\mathbf {z}}_{K,y_\textrm {true}} (\delta ) $.


Interesting directions of future work include: tightening IBP bounds to allow applicability to deeper models, investigating bound propagation in other types of neural architectures (e.g. those based on recurrent networks or self-attention), and exploring other forms of specifications in NLP.


Methods ::: Incorporating Commonsense Rationales	We explore how to utilize additional commonsense knowledge (i.e. rationales) as the input to the task. Like we mentioned in Section SECREF6, we search relevant sentences from the OMCS corpus as the additional distant rationales, and ground truth rationale sentences for dev/test data. The inputs are no longer the concept-sets themselves, but in a form of “[rationales$|$concept-set]” (i.e. concatenating the rationale sentences and original concept-set strings).


Queries, on the other hand, are not expressed in natural language, but instead consist of tuples $q$2 where the object entity is unknown and it has to be inferred by reading the support documents. Therefore, answering a query corresponds to finding the entity $q$3 that is the object of a tuple in the KB with subject $q$4 and relation $q$5 among the provided set of candidate answers $q$6 . The goal is to learn a model that can identify the correct answer $a^\star $ from the set of supporting documents $S_q$ . To that end, we exploit the available supervision to train a neural network that computes scores for candidates in $C_q$ . We estimate the parameters of the architecture by maximizing the likelihood of observations. For prediction, we then output the candidate that achieves the highest probability. In the following, we present our model discussing the design decisions that enable multi-step reasoning and an efficient computation.


Approach and Contributions	Our approach to generating question-answer pairs from text is a two-stage process: in the first stage we select the most relevant and appropriate candidate answer, i.e., the pivotal answer, using an answer selection module, and in the second stage we encode the answer span in the sentence and use a sequence to sequence model with a rich set of linguistic features to generate questions for the pivotal answer. Our sentence encoder transforms the input sentence into a list of fixed-length continuous vector word representation, each input symbol being represented as a vector. The question decoder takes in the output from the sentence encoder and produces one symbol at a time and stops at the EOS (end of sentence) marker. To focus on certain important words while generating questions (decoding) we use a global attention mechanism. The attention module is connected to both the sentence encoder as well as the question decoder, thus allowing the question decoder to focus on appropriate segments of the sentence while generating the next word of the question. We include linguistic features for words so that the model can learn more generalized syntactic transformations. We provide a detailed description of these modules in the following sections.


Conclusions	We introduced the problem of parsing AMR structures, annotated for English, from sentences written in other languages as a way to test the cross-lingual properties of AMR. We provided evidence that AMR can be indeed shared across the languages tested and that it is possible to overcome translational divergences. We further proposed a novel way to evaluate the target parsers that does not require manual annotations of the target language. The full-cycle procedure is not limited to AMR parsing and could be used for other cross-lingual problems in NLP. The results of the projection-based AMR parsers indicate that there is a vast room for improvements, especially in terms of generating better alignments. We encourage further work in this direction by releasing professional translations of the AMR test set into four languages.


Approximate inference	Given a dataset INLINEFORM0 , the goal of inference is to compute the posterior distribution of the per-document topic proportions INLINEFORM1 , the per-word topic assignments INLINEFORM2 , the per-topic distribution over words INLINEFORM3 , the per-document latent true class INLINEFORM4 , and the per-annotator confusion parameters INLINEFORM5 . As with LDA, computing the exact posterior distribution of the latent variables is computationally intractable. Hence, we employ mean-field variational inference to perform approximate Bayesian inference. Variational inference methods seek to minimize the KL divergence between the variational and the true posterior distribution. We assume a fully-factorized (mean-field) variational distribution of the form DISPLAYFORM0   where INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 and INLINEFORM4 are variational parameters. Table TABREF23 shows the correspondence between variational parameters and the original parameters. Let INLINEFORM0 denote the model parameters. Following  , the KL minimization can be equivalently formulated as maximizing the following lower bound on the log marginal likelihood DISPLAYFORM0   which we maximize using coordinate ascent. Optimizing INLINEFORM0 w.r.t.


We obtain samples from $\mathbf {z}|\mathbf {x},\mathbf {y}$ by performing collapsed Gibbs sampling as described in section 5.1 of journals/jasa/Teh06: we repeatedly sample $\mathbf {z}$ from its conditional distribution, with $\mathbf {\phi }$ integrated out:  $$z^\textbf {n}_i | \mathbf {x}, \mathbf {y}, z^\textbf {n}_{-i} =


Introduction	The collection and analysis of historical document images is a key component in the preservation of culture and heritage. Given its importance, a number of active research efforts exist across the world , , , , , . In this paper, we focus on palm-leaf and early paper documents from the Indian sub-continent. In contrast with modern or recent era documents, such manuscripts are considerably more fragile, prone to degradation from elements of nature and tend to have a short shelf life , , . More worryingly, the domain experts who can decipher such content are small in number and dwindling. Therefore, it is essential to access the content within these documents before it is lost forever. Surprisingly, no large-scale annotated Indic manuscript image datasets exist for the benefit of researchers in the community. In this paper, we take a significant step to address this gap by creating such a dataset. Given the large diversity in language, script and non-textual regional elements in these manuscripts, spatial layout parsing is crucial in enabling downstream applications such as OCR, word-spotting, style-and-content based retrieval and clustering. For this reason, we first tackle the problem of creating a diverse, annotated spatial layout dataset.


Users open some existing documents in a file system, make some changes (e.g. addition, deletion or replacement), and save them as separate documents. These documents become revisions of the original documents. We started from an initial corpus that did not have revisions, and kept adding new documents and revising existing documents. Similar to a file system, at any moment new documents could be added and/or some of the current documents could be revised. The revision operations we used were deletion, addition and replacement of words, sentences, paragraphs, section names and document titles. The addition of words, ..., section names, and new documents were pulled from the Wikipedia abstracts. This corpus generation process had five time periods INLINEFORM0 . Figure FIGREF42 illustrates this simulation. We set a Poisson distribution with rate INLINEFORM1 (the number of documents in the initial corpus) to control the number of new documents added in each time period, and a Poisson distribution with rate INLINEFORM2 to control the number of documents revised in each time period. We generated six data sets using different random seeds, and each data set contained six corpora (Corpus 0 - 5).


As we train with more data, we obtain better validation losses. This was expected. The second trend is that by using less than 50% of the data available the model tends to overfit the data, as indicated by the consistent increase in the validation loss after about 15 epochs (check dashed lines in right side of Figure FIGREF28 ). This suggests that for the future we should not try any drastic reduction of the training data to save training time. Finally, when not overfitting, the validation loss seems to stabilize after around 20 epochs. We observed no phase-transition effects (the model seems simple enough for not showing that type of behavior). This indicates we have a practical way of safely deciding when to stop training the model.


Introduction	Currently, social networks are so popular. Some of the biggest ones include Facebook, Twitter, Youtube,... with extremely number of users. Thus, controlling content of those platforms is essential. For years, social media companies such as Twitter, Facebook, and YouTube have been investing hundreds of millions euros on this task , . However, their effort is not enough since such efforts are primarily based on manual moderation to identify and delete offensive materials. The process is labour intensive, time consuming, and not sustainable or scalable in reality , , . In the sixth international workshop on Vietnamese Language and Speech Processing (VLSP 2019), the Hate Speech Detection (HSD) task is proposed as one of the shared-tasks to handle the problem related to controlling content in SNSs. HSD is required to build a multi-class classification model that is capable of classifying an item to one of 3 classes (hate, offensive, clean).


We suspect that the less drastic the change in an attention dimension over time, the higher the chances of that dimension being part of multiple cross-view dynamics. Thus more attentions activate this important dimension. O4: Some attentions focus on cross-view dynamics that involve only two modalities. For example, in $a^3$ , the audio modality has no dark blue dimensions, while in $a^1$ all the modalities have dark blue dimensions. The attentions seem to have residual effects. $a^1$ shows activations over a broad set of variables while $a^4$ shows activation for fewer sets, indicating that attentions could learn to act in a complementary way.


Reinforcement Learning	In the last few years, reinforcement learning has emerged as a powerful tool for solving complex sequential decision-making problems. It is well known for its great success in the game field, such as Go  and Atari games  . Recently, reinforcement learning has also been successfully applied to many natural language processing tasks and achieved good performance  ,  ,  . Feng et al.  used reinforcement learning for relation classification task by filtering out the noisy data from the sentence bag and they achieved huge improvements compared with traditional classifiers. Zhang et al.  applied the reinforcement learning on sentence representation by automatically discovering task-relevant structures. To automatic taxonomy induction from a set of terms, Han et al.  designed an end-to-end reinforcement learning model to determine which term to select and where to place it on the taxonomy, which effectively reduced the error propagation between two phases. Inspired by the above works, we also add reinforcement learning to our framework.


In multitask learning, a set of related tasks are learned (e.g., emotional activation), along with a primary task (e.g., emotional valence); both tasks share parts of the network topology and are hence jointly trained, as depicted in Figure FIGREF4 . It is expected that data for the secondary task models information, which would also be discriminative in learning the primary task. In fact, this approach has been shown to improve generalizability across corpora  . The remainder of this paper is organized as follows: First we introduce the DCGAN model and discuss prior work, in Section SECREF2 . Then we describe our specific multitask DCGAN model in Section SECREF3 , introduce the datasets in Section SECREF4 , and describe our experimental design in Section SECREF5 . Finally, we report our results in Section SECREF6 and discuss our findings in Section SECREF7 .


Introduction	In recent years, word embeddings , ,  have been proven to be very useful for training downstream natural language processing (NLP) tasks. Moreover, contextualized embeddings ,  have been shown to further improve the performance of NLP tasks such as named entity recognition, question answering, or text classification when used as word features because they are able to resolve ambiguities of word representations when they appear in different contexts. Different deep learning architectures such as multilingual BERT , LASER  and XLM  have proved successful in the multilingual setting. All these architectures learn the semantic representations from unannotated text, making them cheap given the availability of texts in online multilingual resources such as Wikipedia. However, the evaluation of such resources is usually done for the high-resourced languages, where one has a smorgasbord of tasks and test sets to evaluate on. This is the best-case scenario, languages with tones of data for training that generate high-quality models. For low-resourced languages, the evaluation is more difficult and therefore normally ignored simply because of the lack of resources.


Performing curriculum learning strategy offers a considerable improvement, take the topmost generating hierarchy in Table TABREF13 for example, this method yields a 102.07% improvement in BLEU, 48.26% in ROUGE-1, 144.8% in ROUGE-2, and 39.18% in ROUGE-L. Despite that applying repeat-input mechanism alone does not offer benefit, combining these two strategies together further achieves the best performance. Note that these methods do not require any additional parameters. Unfortunately, even some of the attentional hierarchical decoders achieve the best results in the generating hierarchies (Table TABREF18 ). Mostly, the additional attention mechanisms are not capable of bringing benefit for model performance. The reason may be that the decoding process is designed for gradually importing words in the specific set of linguistic patterns to the output sequence, each decoder layer is responsible of copying the output tokens from the previous layer and insert new words into the sequence precisely. Because of this nature, a decoder needs explicit information of the structure of a sentence rather than implicit high-level latent information.


Nonetheless, few attempts have been made to distinguish both categories from another perspective. Potha and Stamatatos  , for example, categorize AV methods as either intrinsic or extrinsic (emphasis by us): “Intrinsic verification models view it [i. e., the verification task] as a one-class classification task and are based exclusively on analysing the similarity between [ INLINEFORM0 ] and [ INLINEFORM1 ]. [...] Such methods [...] do not require any external resources.”   “On the other hand, extrinsic verification models attempt to transform the verification task to a pair classification task by considering external documents to be used as samples of the negative class.”   While we agree with statement (2), the former statement (1) is unsatisfactory, as intrinsic verification models are not necessarily unary. For example, the AV approach GLAD proposed by Hürlimann et al.  directly contradicts statement (1). Here, the authors “decided to cast the problem as a binary classification task where class values are Y [ INLINEFORM0 ] and N [ INLINEFORM1 ]. [...] We do not introduce any negative examples by means of external documents, thus adhering to an intrinsic approach.”  .


This variation is due to the different ways in which annotators were recruited: student-assistants were recruited for several days, while participants recruited through the Radboud Research Participation System could only join for the duration of an hour. We calculated inter-annotator agreement by Krippendorff's Alpha , which accounts for different annotator pairs and empty values. To also zoom in on the particular agreement by category, we calculated mutual F-scores for each of the categories. This metric is typically used to evaluate system performance by category on gold standard data, but could also be applied to annotation pairs by alternating the roles of the two annotators between classifier and ground truth. A summary of the agreement by categorization is given in Table TABREF10. While both the Relevance and Subject categorizations are annotated at a percent agreement of $0.71$ and $0.70$, their agreement scores are only fair, at $\alpha =0.27$ and $\alpha =0.29$. The percent agreement on Stance and Sentiment, which carry more categories than the former two, is $0.54$ for both.


Results and Discussion ::: Performance analysis	In Figure FIGREF14, we show some of the improvements in performance obtained when including a fixed amount of virtual edges using GloVe as embedding method. In each subpanel, we show the relative improvement in performance obtained as a function of the fraction of additional edges. In this section, we considered the traditional co-occurrence as starting point. In other words, the network construction disregarded stopwords. The list of stopwords considered in this paper is available in the Supplementary Information. We also considered the global approach to filter edges. The relative improvement in performance is given by $\Gamma _+{(p)}/\Gamma _0$, where $\Gamma _+{(p)}$ is the accuracy rate obtained when $p\%$ additional edges are included and $\Gamma _0 = \Gamma _+{(p=0)}$, i.e. $\Gamma _0$ is the accuracy rate measured from the traditional co-occurrence model. We only show the highest relative improvements in performance for each classifier. In our analysis, we considered also samples of text with distinct length, since the performance of network-based methods is sensitive to text length .


Proposed Framework ::: Captioning Module	In this section we explain the captioning based for an example modality, namely, visual. Given a video $V$ and a set of proposals $P_V$ from the event localization module, the task of the captioning module is to provide a caption for each proposal in $P_V$. In order to extract features from a video $V$, we employ I3D network  pre-trained on the Kinetics dataset which produces 1024-d features. The gap between the extracted features and the generated captions is filled with Transformer  architecture which was proven to effectively encode and decode the information in a sequence-to-sequence setting.


In practice, given the size of our dataset, calculating the full denominator of the loss for a query, image pair would involve iterating through each image for each query, which is $O(n^2)$ in the number of training examples. To remedy this, we calculated the loss within each batch separately. That is, the denominator of the loss only involved summing over images in the same batch as the query. We used a batch size of 1000 for all experiments. In principle, the negative sampling approach used by  could be used instead to prevent quadratic time complexity. We can interpret this loss function as producing a softmax classification task for queries and images: given a query, the model needs to predict the image relevant to that query. The cosine similarity between the image representation $I_i$ and the query representation $Q_q$ is normalized under softmax to produce a “belief” that the image $i$ is the image relevant to the query $q$ . This is analogous to the skip-gram model proposed by  , although we use cosine similarity instead of dot product.


The problem could arise from the fact that words as “UTF8gbsn期货业” rarely occur in the training data, and their relatedness is not reflected in the data. Another issue is that there are some particles, e.g., “UTF8gbsn的” (DE) in the most related words. A possible explanation is that particles show up too often in the contexts of the word, and it is hard for the models to distinguish them from the real semantically-related words. As our proposed approach is based on regularization of the less common correspondence, it is reasonable that such kind of relation cannot be eliminated. The first case can be categorized into data sparsity, which usually needs the aid of knowledge bases to solve. The second case is due to the characteristics of natural language. However, as such words are often closed class words, the case can be resolved by manually restricting the relatedness of these words.


Experiments ::: Model Analysis ::: Examples with Different Learning Frequencies	As shown in Table TABREF30, the most frequently learnt examples are comprehensively far better than those seldom learnt examples, which exhibits the effectiveness of the adaptive multi-curricula learning framework.


Acknowledgments	We would like to thank the anonymous reviewers for their comments to improve this paper. We would also like to thank Sebastian Padó for his help training the Clark clusters. This work has been supported by the European projects NewsReader, EC/FP7/316404 and QTLeap - EC/FP7/610516, and by the Spanish Ministry for Science and Innovation (MICINN) SKATER, Grant No. TIN2012-38584-C06-01 and TUNER, TIN2015-65308-C5-1-R.


Analysis of training datasets ::: Annotation of training datasets ::: Identity of the annotators	The data statements framework given by Bender and Friedman emphasises the importance of understanding who has completed annotations. Knowing who the annotators are is important because `their own “social address" influences their experience with language and thus their perception of what they are annotating.'  In the context of online abuse, Binns et al. show that the gender of annotators systematically influences what annotations they provide . No annotator will be well-versed in all of the slang or coded meanings used to construct abusive language. Indeed, many of these coded meanings are deliberately covert and obfuscated . To help mitigate these challenges, annotators should be (a) well-qualified and (b) diverse. A homogeneous group of annotators will be poorly equipped to catch all instances of abuse in a corpus. Recruiting an intentionally mixed groups of annotators is likely to yield better recall of abuse and thus a more precise dataset . Information about annotators is unfortunately scarce.


Analogously to the work of Pelevina:16, we construct a semantic relatedness graph from a list of nearest neighbours, but we filter this list using the following procedure: Extract a list $\mathcal {N}$ = {$w_{1}$, $w_{2}$, ..., $w_{N}$} of $N$ nearest neighbours for the target (ego) word vector $w$. Compute a list $\Delta $ = {$\delta _{1}$, $\delta _{2}$, ..., $\delta _{N}$} for each $w_{i}$ in $\mathcal {N}$, where $\delta _{i}~=~w-w_{i}$. The vectors in $\delta $ contain the components of sense of $w$ which are not related to the corresponding nearest neighbours from $\mathcal {N}$. Compute a list $\overline{\mathcal {N}}$ = {$\overline{w_{1}}$, $\overline{w_{2}}$, ..., $\overline{w_{N}}$}, such that $\overline{w_{i}}$ is in the top nearest neighbours of $\delta _{i}$ in the embedding space.


Challenge Details	A dedicated website for the competition was set up. The website included instructions, the registration form, schedule, and other relevant details. Following registration, participants were able to download the training datasets. The label distribution of emotions in our data are highly unbalanced, as can be seen in Figure FIGREF6. Due to the small number of three of the labels, participants were instructed to use only four emotions for labels: joy, sadness, anger, and neutral. Evaluation of submissions was done using only utterances with these four labels. Utterances with labels other than the above four (i.e., surprise, disgust, fear or non-neutral) were discarded and not used in the evaluation. Scripts for verifying and evaluating the submissions were made available online. We used micro-F1 as the comparison metric.


The quality of our model—the conversion tool—comes from how well we encode prior knowledge about the relationship between the UD and UniMorph corpora.


MT performed the best on the test examples which according to the annotation require a correct understanding of the vocabulary (81 $\text{F}_1$ on binary /54 $\text{F}_1$ on 4-class), with Blse (79/48) slightly worse. Muse (76/23), VecMap (70/35), and Barista (67/41) perform worse. This suggests that Blse is better than Muse, VecMap and Barista at transferring sentiment of the most important sentiment bearing words. Negation: Negation is a well-studied phenomenon in sentiment analysis Pang2002,Wiegand2010,Zhu2014,Reitan2015 . Therefore, we are interested in how these four models perform on phrases that include the negation of a key element, for example “In general, this hotel isn't bad". We would like our models to recognize that the combination of two negative elements “isn't" and “bad" lead to a Positive label. Given the simple classification strategy, all models perform relatively well on phrases with negation (all reach nearly 60 $\text{F}_1$ in the binary setting).


Related Work ::: Training on synthetic data	The lack of training data is an issue for many NLP applications. There have been attempts to generate and use synthetic data for training question answering systems  and SQL2text systems . In  synthetic hashtags are generated by removing whitespace characters from frequent n-grams, while in  German compounds are synthesized for further machine translation.


Introduction	Machine translation (MT) research is biased towards language pairs including English due to the ease of collecting parallel corpora. Translation between non-English languages, e.g., French$\rightarrow $German, is usually done with pivoting through English, i.e., translating French (source) input to English (pivot) first with a French$\rightarrow $English model which is later translated to German (target) with a English$\rightarrow $German model , , . However, pivoting requires doubled decoding time and the translation errors are propagated or expanded via the two-step process. Therefore, it is more beneficial to build a single source$\rightarrow $target model directly for both efficiency and adequacy. Since non-English language pairs often have little or no parallel text, common choices to avoid pivoting in NMT are generating pivot-based synthetic data ,  or training multilingual systems , . In this work, we present novel transfer learning techniques to effectively train a single, direct NMT model for a non-English language pair. We pre-train NMT models for source$\rightarrow $pivot and pivot$\rightarrow $target, which are transferred to a source$\rightarrow $target model.


Such a model concatenates the hidden state from LSTM and a vector constructed by self-attention at each position, and feeds them into CRF as features. The model attempts to capture rich implicit structures in the input space, but it does not put effort on explicit structures in the output space. E-I is a weaker version of EI. Such a model removes the BMES sub-tags in the E tag, causing the model to learn less explicit structural information in the output space. EI- is a weaker version of EI. Such a model removes the self-attention from EI, causing the model to learn less expressive implicit structures in the input space.


Conclusion	Named entities being an open set which keeps expanding are difficult to represent through a closed NE dictionary. This work mitigates significant defects in previous closed NE definitions and proposes a new open definition for NEs by modeling their embedding distributions with least parameters. We visualize NE distributions in monolingual case and perform an effective isomorphism spaces mapping in cross-lingual case. According to our work, we demonstrate that common named entity types (PER, LOC, ORG) tend to be densely distributed in a hypersphere and it is possible to build a mapping between the NE distributions in embedding spaces to help cross-lingual NE recognition. Experimental results show that the distribution of named entities via mapping can be used as a good enough replacement for the original distribution. Then the discovery is used to build an NE dictionary for Indonesian being a truly low-resource language, which also gives satisfactory precision.


Evaluation of Representation Models ::: Results ::: The unsupervised scenario.	Table TABREF26 reports the results using the item representations directly. We report results over train and dev partitions for completeness, but note that none of them was used to tune the models. As it can be seen, multimodal representations consistently outperform their text-only counterparts. This confirms that, overall, visual information is helpful in the semantic textual similarity task and that image and sentence representation are complementary. For example, the bert model improves more than 13 points when visual information provided by the resnet is concatenated. glove shows a similar or even larger improvement, with similar trends for use and vse++(text). Although vse++(img) shows better performance than resnet when applying them alone, further experimentation showed lower complementarity when combining with textual representation (e.g. $0.807\rho $ in test combining textual and visual modalities of vse++). This is something expected as vse++(img) is pre-trained along with the textual part of the vse++ model on the same task. We do not show the combinations with vse++(img) due to the lack of space.


Analogy Datasets	Table 3 compares Chinese word analogy datasets. Most analogies in existing datasets are morphological (morph.) or named entity (entity) relations. For example, CA8-Morphological  uses 21 shared prefix characters, e.g., UTF8bkai第, to form 2,553 analogies, e.g., UTF8bkai一 : UTF8bkai第一 = UTF8bkai二 : UTF8bkai第二 (one : first = two : second). As for named entities, some 20 word pairs of the capital-country relation can be permuted to form 190 analogies, which require a knowledge base but not commonsense to solve. Only the nature part of CA8 and the man-woman part of CA-Google  contains a handful of relations that requires commonsense world knowledge. In constrast, CA-EHN extracts 85K linguist-checked analogies covering 6,490 concept pairs, e.g., (wood $\vert $ UTF8bkai木, UTF8bkai馬 $\vert $ horse).


The strongest coalitions on the left-to-center part of the axis are those between GUE-NGL, Greens-EFA, and S&D, and between S&D, ALDE, and EPP. These results reaffirm the role of Greens-EFA and ALDE as intermediaries, not only in the European Parliament but also in the debates on social media. Last, but not least, with the ERGM methodology we measure the extent to which the retweet network can explain the co-voting activities in the European Parliament. We compute this for each policy area separately and also over all RCVs. We conclude that the retweet network indeed matches the co-voting behavior, with the exception of one specific policy area. In the area Economic and monetary system, the links in the (overall) retweet network do not match the links in the co-voting network. Moreover, the negative coefficients imply a radically different formation of coalitions in the European Parliament. This is consistent with the results in Figs FIGREF36 and FIGREF37 (the left-hand panels), and is also observed in Fig FIGREF42 (the top charts).


Our Framework ::: Entity-Consistency Augmented Decoder ::: Decoder with Retrieved Entity	After the row selection and column selection, we can define the final retrieved KB entity score as the element-wise dot between the row retriever result and the column selection score, which can be calculated as where the $v^{t}$ indicates the final KB retrieved entity score. Finally, we follow eric:2017:SIGDial to use copy mechanism to incorporate the retrieved entity, which can be defined as where $\mathbf {o}_t$’s dimensionality is $ |\mathcal {V}|$ +$|\mathcal {E}|$. In $\mathbf {v}^t$ , lower $ |\mathcal {V}|$ is zero and the rest$|\mathcal {E}|$ is retrieved entity scores.


12)  Under assumption $\sigma \approx 0$ sampling the row of weights from such posterior means putting all the weights from this row either to 0 (drop the corresponding input neuron) or to some learned values. Thus this model is a probabilistic analog of binary dropout with dropout rates $p^x$ and $p^h$ . After unfolding the recurrence in the network, the maximization of the variational lower bound for such model looks as follows:  $$\sum _{i=1}^\ell \int q(\omega |M) \log \Bigl (y^i\big |f_y\bigl (f_h(x^i_T, f_h(\dots f_h (x^i_1, h^i_0))\bigr )\Bigr ) d \omega - \\


This suggests that the methods with better sample efficiency are needed to act in more realistic problem settings with huge action spaces. Experiments also show that a larger memory slot size always helps. Intuitively, with a memory mechanism (either implicit or explicit), an agent could make the environment closer to fully observed by exploring and memorizing observations. Presumably, a larger memory may further improve QA-DQN's performance, but considering the average number of sentences in each iSQuAD game is 5, a memory with more than 5 slots will defeat the purpose of our study of partially observable text environments. Not surprisingly, QA-DQN performs worse in general on iNewsQA, in all experiments. As shown in Table TABREF16, the average number of sentences per document in iNewsQA is about 6 times more than in iSQuAD. This is analogous to games with larger maps in the RL literature, where the environment is partially observable. A better exploration (in our case, jumping) strategy may help QA-DQN to master such harder games.


\mathbf {o}_t &= \sigma (\mathbf {W}_o \mathbf {x}_t + \mathbf {U}_o^L \mathbf {h}_{t-1}^L + \mathbf {U}_o^R \mathbf {h}_{t-1}^R),


Collision Study	Before diving into the actual collision studies, it is important to understand what the properties of good projections are. For instance, good projections should be as separate as possible, while still capturing the inherent n-gram features. Words with similar character n-gram feature vectors should be closer to each other i.e. cat and cats, but yet separate from each other so that the network can learn that cat and cats are related, but yet different. Such observations are not evident from the projections. One way to understand them is by looking at the collision rates. For instance, if there are too many projection collisions, this means that the network is fundamentally incapable of learning and it will not be able to generalize. For the purpose, we test how spread out the projections are for word and sentence representations. We take a large corpus enwik9 and analyze the average hamming distance of the words and sentences in the corpus. Intuitively, good projections should have less collisions. Our study shows that there is almost no collision. On an average the Hamming distances between words are 557 bits, which is around 50% of the projection dimension.


ERGM's concept of chance is based on the number of expected pair-wise co-votes between MEPs belonging to a group on a given RCV, knowing the network-related properties of the co-voting network on that particular RCV. The main difference between INLINEFORM2 and ERGM, though, is the treatment of non-voting and abstained MEPs. INLINEFORM3 considers only the yes/no votes, and consequently, agreements by the voting MEPs of the same groups are considerably higher than co-voting by chance. ERGM, on the other hand, always considers all MEPs, and non-voting and abstained MEPs are treated as disconnected nodes. The level of co-voting by chance is therefore considerably lower, since there is often a large fraction of MEPs that do not attand or abstain. As with INLINEFORM0 , Greens-EFA, S&D, and EPP exhibit the highest cohesion, even though their ranking is permuted when compared to the ranking obtained with INLINEFORM1 . At the other end of the scale, we observe the same situation as with INLINEFORM2 . The non-aligned members NI have the lowest cohesion, followed by EFDD and ENL.


Our second contribution is a copy mechanism which allows us to use much smaller decoder vocabulary sizes resulting in much faster decoding. Our copy mechanism also allows us to construct a better representation of out-of-vocabulary words. We demonstrate the effectiveness of our approach in the challenging Gigaword dataset and DUC competition showing state-of-the-art performance.


Experiments ::: Effectiveness of Paired Common Words	We further analyse the necessity and effectiveness of modifying sentences with paired common words. We consider another version that replaces one single word independently at each step without using paired common words, namely the unpaired version. Firstly, for matched adversarial examples that can be semantically different from original sentences, the unpaired version is inapplicable, because the matched label can be easily broken if common words from two sentences are changed into other words independently. And for the unmatched case, we show that the unpaired version is much less effective. For a more fair comparison, we double the step number limit for the unpaired version. As shown in Table TABREF41, the performance of target models on unmatched examples generated by the unpaired version, particularly that of BERT, is mostly much higher than those by our full algorithm, except for BiMPM on MRPC but its accuracies have almost reached 0 (0.0% for unpaired and 0.2% for paired). This demonstrates that our algorithm using paired common words are more effective in generating adversarial examples, on which the performance of the target model is generally much lower.


Here we consider five baselines to compare with GraLap: (i) Uniform: assign 3 to all the references assuming equal intensity, (ii) SVR+W: recently proposed Support Vector Regression (SVR) with the feature set mentioned in  , (iii) SVR+O: SVR model with our feature set, (iv) C4.5SSL: C4.5 semi-supervised algorithm with our feature set  , and (v) GLM: the traditional graph-based LP model with our feature set  . Three metrics are used to compare the results of the competing models with the annotated labels: Root Mean Square Error (RMSE), Pearson's correlation coefficient ( INLINEFORM0 ), and coefficient of determination ( INLINEFORM1 ). Table TABREF28 shows the performance of the competing models. We incrementally include each feature set into GraLap greedily on the basis of ranking shown in Figure FIGREF27 (a). We observe that GraLap with only FF outperforms SVR+O with 41% improvement of INLINEFORM0 . As expected, the inclusion of PF into the model improves the model marginally.


In this work we explore the automatic processing of ultrasound tongue images in order to assist SLTs, who currently largely rely on manual processing when using articulatory imaging in speech therapy. One task that could assist SLTs is the automatic classification of tongue shapes from raw ultrasound. This can facilitate the diagnosis and treatment of speech sound disorders, by allowing SLTs to automatically identify incorrect articulations, or by quantifying patient progress in therapy. In addition to being directly useful for speech therapy, the classification of tongue shapes enables further understanding of phonetic variability in ultrasound tongue images. Much of the previous work in this area has focused on speaker-dependent models. In this work we investigate how automatic processing of ultrasound tongue imaging is affected by speaker variation, and how severe degradations in performance can be avoided when applying systems to data from previously unseen speakers through the use of speaker adaptation and speaker normalization approaches. Below, we present the main challenges associated with the automatic processing of ultrasound data, together with a review of speaker-independent models applied to UTI. Following this, we present the experiments that we have performed (Section SECREF2 ), and discuss the results obtained (Section SECREF3 ).


Results	In this section, we empirically compare automatic differentiation (AD, our implementation based on Clad) and numerical differentiation (ND, based on finite difference method) in ROOT. We show that AD can drastically improve accuracy and performance of derivative evaluation, compared to ND.


The incidents are wide ranging: from patient accidents to lost forms or referrals; from delays in admission or discharge to serious untoward incidents, such as retained foreign objects after operations. The review and analysis of such data provides critical insight into complex processes in healthcare with a view towards service improvement. Although statistical analyses are routinely performed on the structured data (dates, locations, hand-coded categories, etc), free text is typically read manually and often ignored in practice, unless a detailed review of a case is undertaken because of the severity of harm that resulted. These limitations are due to a lack of methodologies that can provide content-based groupings across the large volume of reports submitted nationally for organisational learning. Automatic categorisation of incidents from free text would sidestep human error and difficulties in assigning incidents to a priori pre-defined lists in the reporting system. Such tools can also offer unbiased insight into the root cause analysis of incidents that could improve the safety and quality of care and efficiency of healthcare services. In this work, we showcase an algorithmic methodology that detects content-based groups of records in an unsupervised manner, based only on the free (unstructured) textual descriptions of the incidents.


Experiments ::: Baselines ::: HSSC @!START@@!END@.	This model adopts encoder parameter sharing for jointly sentiment classification and summarization. It predicts the sentiment label using a highway layer, concatenating the hidden state in summary decoder and the original text representation in encoder.


And finally, by comparing $\text{CMD}^{\dagger \dagger }$ with $\text{CMD}$, or comparing $\text{DANN}^{\dagger \dagger }$ with $\text{DANN}$, we can know the general effectiveness of our proposed solution.


Precisely, the Softword technique augments the representation of a character with the embedding of its corresponding segmentation label: Here, $seg(c_j) \in \mathcal {Y}_{seg}$ denotes the segmentation label of the character $c_j$ predicted by the word segmentor, $\mathbf {e}^{seg}$ denotes the segmentation label embedding lookup table, and commonly $\mathcal {Y}_{seg}=\lbrace \text{B}, \text{M}, \text{E}, \text{S}\rbrace $ with B, M, E indicating that the character is the beginning, middle, and end of a word, respectively, and S indicating that the character itself forms a single-character word. The first idea we come out based on the Softword technique is to construct a word segmenter using the lexicon and allow a character to have multiple segmentation labels. Take the sentence $s=\lbrace c_1, c_2, c_3, c_4, c_5\rbrace $ as an example.


We also relabeled participants and events from the first annotation phase with ScrEv_other and ScrPart_other, if they did not meet the frequency requirements. The event label air_bathroom (the event of letting fresh air into the room after the bath), for example, was only used once in the stories, so we relabeled that instance to ScrEv_other. Additionally, we looked at the DeScript corpus  , which contains manually clustered event paraphrase sets for the 10 scenarios that are also covered by InScript (see Section "Comparison to the DeScript Corpus" ). Every such set contains event descriptions that describe a certain event type. We extended our templates with additional labels for these events, if they were not yet part of the template.


Results and Discussion	We present the results in Table TABREF20 . In Zhang et al. zhang2015character it was noted that the performance of ngram features degrades for larger datasets. However, we have seen in our baseline experiments that this effect can be reduced by using suitable frequency cut-offs. We have also seen that in many cases, the ngram features can solely outperform the neural approaches. For the two smaller datasets, linear classifiers tend to perform better, while for the larger datasets performance increases with increasing the non-linear layers even for hand-crafted representations. Combining hand-crafted and learned features is often beneficial, but not always, especially with the linear classifier. What is clear is that different datasets benefit from different representations and model parameters and it is difficult to find a representation that consistently performs well across all datasets. This necessitates repeated experimentation to understand which approaches and parameters would provide more consistent improvements.


Document Visualization	Word2vec embeddings are known to exhibit linear regularities representing semantic relationships between words  . We explore if these regularities can be transferred to a document representation, when using as a document vector a linear combination of word2vec embeddings. As a weighting scheme we employ LRP or SA scores, with the classifier's predicted class as the target class for the relevance estimation. For comparison we perform uniform weighting, where we simply sum up the word embeddings of the document words (SUM). For SA we use either the $l_2$ -norm or squared $l_2$ -norm for pooling word gradient values along the word2vec dimensions, i.e. in addition to the standard SA word relevance defined in Eq. 15 , we use as an alternative $R_{\mathrm {SA}(l_2)}(w_t) = \Vert  \nabla _{{w_t}} f({d}) \Vert _2$ and denote this relevance model by SA $(l_2)$ . For both LRP and SA, we employ different variations of the weighting scheme.


Here, $S$ could be used as gold standard for the back-translated paragraph $S_{T_s}$. In this way, we could turn the unsupervised translation into a supervised task by maximizing the similarity between $S$ and $S_{T_S}$. The same also applies to using poem $T$ as gold standard for its corresponding back-translation $T_{S_T}$. We define the following loss: Note that $\mathcal {L}^{bt}$ does not back propagate through the generation of $T_S$ and $S_T$ as we observe no improvement in doing so. When training the model, we minimize the composite loss: where $\alpha _1$ and $\alpha _2$ are scaling factors.


We build an evidence extraction model to predict the most important sub-spans from the passages as evidence, and then develop an answer synthesis model which takes the evidence as additional features along with the question and passage to further elaborate the final answers. Specifically, we develop the answer extraction model with state-of-the-art attention based neural networks which predict the start and end positions of evidence snippets. As multiple passages are provided for each question in the MS-MARCO dataset, we propose incorporating passage ranking as an additional task to improve the results of evidence extraction under a multi-task learning framework. We use the bidirectional recurrent neural networks (RNN) for the word-level representation, and then apply the attention mechanism  to incorporate matching information from question to passage at the word level. Next, we predict start and end positions of the evidence snippet by pointer networks  . Moreover, we aggregate the word-level matching information of each passage using the attention pooling, and use the passage-level representation to rank all candidate passages as an additional task. For the answer synthesis, we apply the sequence-to-sequence model to synthesize the final answer based on the extracted evidence.


Second, we remove from the text of the tweets user names, links, punctuation, tabs, leading and lagging blanks, general spaces and “RT" - the text that points that a tweet is in fact a retweet. As shown in previous works, emojis are correlated with sentiment . Moreover, as we think that communities will express different sentiment during discussion, it is forseeable that emojis will play an important role as separators of tweets that differentiate between the two sides. Accordingly, we decide to add them to the train-set by translating each emoji into a different word. For example, the emoji :) will be translated into happy and :( into sad. Relations between emojis and words are defined in the R library textclean. Finally, we group tweets by user concatenating them in one string and labeling them with the user's community, namely with tags C1 and C2, corresponding respectively to the biggest and second biggest groups. It is important to note that we take the same number of users of each community to prevent bias in the model. Thus, we use the number of users of the smallest principal community. The train-set built that way is used to feed the model.


QAGS ::: The QAGS Score	Given these components, we obtain the QAGS score of a generation by (1) generating $K$ questions conditioned on the summary, (2) answering the questions using both the source article and the summary to get two sets of answers, (3) comparing corresponding answers using the answer similarity metric, and (4) averaging the answer similarity metric over all questions. We depict this process in Figure FIGREF3.


Granularity size is mentioned in terms of durations of event actions  and granularity levels of event participants, time and locations. Granularity is given in terms of partonomic relations or through the part-of relation, between entities and events, using the taxonomy of meronymic relations by  . Granularity levels of the human participant slot are contained within WinstonÕs et al. Member-Collection relations. The temporal granularity levels make part of WinstonÕs Portion-Mass relationships and locational levels are in line with Place-Area relations in WinstonÕs taxonomy. Cybulska and Vossen experimented with a decision-tree supervised pairwise binary classifier to determine coreference of pairs of event mentions. They also ran experiments with a linear SVM and a multinomial Naive Bayes classifier but the decision-tree classifier outperformed both of them. For the experiments, Cybulska and Vossen use the ECB+ dataset  . The ECB+ corpus contains a new corpus component, consisting of 502 texts, describing different instances of event types.


Methodology ::: Machine Learning Models ::: Multi-label Classification	RAkEL (RAndom k labELsets): RAkEL  is a multi-label classification algorithm that uses labeled powerset (LP) transformation: it basically creates a single binary classifier for every label combination and then uses multiple LP classifiers, each trained on a random subset of the actual labels, for classification.


Method ::: Defining bias in embeddings	Bias can be thought of as “prejudice in favor or against a person, group, or thing that is considered to be unfair" . Because definitions of fairness have changed over time, algorithms which are trained on “real-world" data may pick up associations which existed historically (or still exist), but which are considered undesirable. In the word embedding literature, one common idea is to analyse relationships which embeddings encode between professions and gender, race, ethnicity or nationality. We follow this approach in this paper, though note that our method is equally applicable to measuring the encoded relationship between any set of entities in a KG..


The mechanism merge each spatial location INLINEFORM0 in the annotation sequence INLINEFORM1 with the initial decoder state INLINEFORM2 obtained in equation EQREF7 with non-linearity : DISPLAYFORM0   where INLINEFORM0 is INLINEFORM1 function. The new annotations go through a L2 normalization layer followed by two INLINEFORM2 convolutional layers (of size INLINEFORM3 respectively) to obtain INLINEFORM4 weights, one for each spatial location. We normalize the weights with a softmax to obtain a soft attention map INLINEFORM5 . Each annotation INLINEFORM6 is then weighted according to its corresponding INLINEFORM7 : DISPLAYFORM0   This method can be seen as the removal of unnecessary information in the image annotations according to the source sentence. This attention is used on top of the others - before decoding - and is referred as "grounded image" in Table TABREF41 .


3)  Note that if we want the RDSN to be causal, we can simplify it to  $$P(y_t | x_t, x_{t-1}, ..., x_{t-k}, y_t, y_{t-1}, ..., y_{t-k})$$   (Eq. 4)  where $k$ 's in the above formula represent the number of recurrent frames. Figure 1 shows the framework of RDSN. Adding $y_t, y_{t-1}, ..., y_{t-k}$ as additional inputs, we transferred the pure acoustic model into a hybrid model consisted of an acoustic model and a phoneme-level N-gram model representing the relation between current phone and previous phones. The phoneme-level N-gram (or as in the formula above, k-gram) model provides additional information on phoneme-level, trying to make the output of current frame more accurate and robust with respect to noises and reverberations.