On the other hand, LC-QuaD 2.0 contains an average of 2 hops (equivalent to two joins in relational databases) per query, which lies in the nature of graph database queries that are optimized for handling queries that range over multiple triple patterns.

lC-QuaD 2.0 lacks complexity when considering more complex components (e.g., Group By, Set-Operation, etc.).


Table 2 is a comparison of existing English MWP corpora.

these existing corpora are either limited in terms of the diversity of the associated problem types (as well as lexicon usage patterns), or lacking information such as difficulty levels.


Likewise, we could also enlarge the training-set by duplicating MWPs without affecting the CLD value against the test-set.

it would be also meaningless as no new information would be provided.


Because this type of evaluation is typically task-specific, it can be conducted in multilingual settings.

training a range of task-specific multilingual models might require significant resources, namely, training time and computational power.


Typelevel probing tasks have the advantage of containing less bias (domain, annotator, and majority class); whereas token-level tests might be sensitive to the domain biases from the underlying full-text data.

token-level tests have the advantage of being more lexically diverse; whereas type-level tasks can be less diverse for some languages like Spanish, French, and English.


Unlike previously introduced embedding models, ELMo provides contextualized embeddings, that is, the same words would have different representations when used in different contexts.

our probing tests are type-level (as opposed to token-level), thus we only use the representations generated independently per each token both for the intrinsic and extrinsic experiments.


Another potential reason for the difference in ranking is the domain of the data underlying the respective data sets: For the majority of the languages, POS, DEP, and SRL data originates from the same treebanks and has gold (expert) annotations.

nER and XnLI data sets are generally compiled from a different, and often diverse set of resources.


The abstract syntax can be vastly different in domains ranging from mathematics to tourist phrasebooks.

the linguistic mechanisms needed in the concrete syntax-morphology, agreement, word order-are largely the same in all areas of discourse.


For this purpose, it is enough to express all desired content in one way: One does not need to cover all possible ways to express things.

in wide-coverage parsing, this is a serious limitation.


This led to a series of extensions as described in Section 4.5, first meant to cover the missing English structures.

if the ultimate goal is to build an interlingual grammar, the structures designed for English are not necessarily adequate for other languages-in particular, they might not allow for compositional linearizations.


There, every synset corresponds to one abstract function and then the function's linearization in each language produces all words in the language as variants.

a more detailed analysis shows that this is not ideal.


Our focus in this paper has been on recognizing valid chains of reasoning, assuming a retrieval step that retrieves a reasonable pool of candidates to start with (Section 3.2).

the retrieval step itself is not perfect: For QASC, designed so that at least one valid chain always exists, the retrieved pool of 10 contains no valid chains for 24% of the questions (upper bound in Table 2), capping the overall system's performance.


Multitask learning (Caruana, 1997;Collobert and Weston, 2008) seeks to learn a single model that can solve multiple tasks simultaneously, similar to our framework that seeks to learn a model that can solve many tasks.

in multitask learning each task is learned from examples, and the model is not able to generalize to unseen tasks.


Attribution of natural disasters/collective misfortune is a widely-studied political science problem.

such studies typically rely on surveys, expert opinions, or external signals such as voting outcomes.


For instance, the most-recent PEW survey (Pew) focused on India was conducted in 2018 on only 2,521 users.

our data set consists of comments from 43,859 users.


We observe that, on the detection task, all the BERT based models perform similarly.

on the resolution task, the F1 score substantially improves as we keep adding sophistication to our model architecture.


Third, a more comprehensive evaluation methodology would consider both the exact-match accuracy and the execution-match accuracy, because two logic forms can be semantically equivalent yet do not match precisely in their surface forms.

as shown in Table 1, most existing work is only evaluated with the exact-match accuracy.


Consequently, the execution engines of domain-specific MRs need to be significantly customized for different domains, requiring plenty of manual efforts.

sQL is a domain-general MR for querying relational databases.


There is a predicate tomorrow in all three domainspecific MRs, and this predicate can directly align to the description in the utterance.

one needs to explicitly express the concrete date values in the SQL query; this requirement can be a heavy burden for neural approaches, especially when the values will change over time.


Unfortunately, this integral is intractable due to the complex relationship between X and Z.

related latent variable models like variational autoencoders (VAEs; Kingma and Welling (2013)) learn by optimizing a variational lower bound on the log marginal likelihood.


For instance, we could feed both sentences into the semantic encoder and pool their representations.

in practice we find that alternating works well and also can be used to obtain sentence embeddings for text that is not part of a translation pair.


This model is similar to Infersent (Conneau et al., 2017) in that it is trained on natural language inference data, SNLI (Bowman et al., 2015).

instead of using pretrained word embeddings, they fine-tune BERT in a way to induce sentence embeddings.


Since BGT W/O LANGVARS also has significantly better performance on these tasks, most of this gain seems to be due to the prior having a regularizing effect.

bGT outperforms bGT W/O LANGVARS overall, and we hypothesize that the gap in performance between these two models is due to bGT being able to strip away the language-specific information in the representations with its languagespecific variables, allowing for the semantics of the sentences to be more directly compared.


Japanese is a very distant language to English both in its writing system and in its sentence structure (it is an SOV language, where English is an SVO language).

despite these difference, the semantic encoder strongly outperforms the English language-specific encoder, suggesting that the underlying meaning of the sentence is much better captured by the semantic encoder.


We can observe that the inherent data imbalance problem also exists in MAVEN.

as MAVEN is large-scale, 41% and 82% event types have more than 500 and 100 instances respectively.


Most recently, continuous improvements have been achieved by combining multiple kinds of information in KGs or using more sophisticated embedding models.

the performances of most approaches are still not satisfactory.


To generate the PCG of two KGs, we can first pair all the entities from two KGs as nodes, and then use Equation 1 to generate edges between nodes.

kGs usually contain large number of entities, the PCG of two large-scale kGs will contain huge number of nodes.


Recent studies on single-document summarization (SDS) benefit from the advances in neural sequence learning (Nallapati et al., 2016;See et al., 2017;Chen and Bansal, 2018;Narayan et al., 2018) as well as pretrained language models (Liu and Lapata, 2019;Lewis et al., 2019;Zhang et al., 2020) and make great progress.

in multi-document summarization (MDS) tasks, neural models are still facing challenges and often underperform classical statistical methods built upon handcrafted features (Kulesza and Taskar, 2012).


One extension (Cho et al., 2019) of these studies uses capsule networks (Hinton et al., 2018) to improve redundancy measures.

its capsule networks are pre-trained on SDS and fixed as feature inputs of classical methods without end-to-end representation learning.


Compared to hard cutoff, our soft attention favors top-ranked candidates of the sentence ranker (MMR).

it does not discard low-ranked ones, as the ranker is imperfect, and those sentences ranked low may also contribute to a high-quality summary.


RL-MMR has a more salient and non-redundant summary, as it is end-to-end trained with advances in SDS for sentence representation learning while maintaining the benefits of classical MDS approaches.

mmR alone only considers lexical similarity; The redundancy mea-sure in DPP-Caps-Comb is pre-trained on one SDS dataset with weak supervision and fixed during the training of DPP.


The authors considered several predefined probe architectures and picked one of them based on a manually defined criterion.

the variational code gives probe architecture as a byproduct of training and does not need human guidance.


To this day, scientific publications still serve as a fundamental fixed-domain benchmark for neural KPE methods (Meng et al., 2017;Alzaidy et al., 2019;Sahrawat et al., 2019) due to the availability of ample data of this kind.

experiments have revealed that KPE methods trained directly on such corpora do not generalize well to other web-related genres or other types of documents (Chen et al., 2018;Xiong et al., 2019), where there may be far more heterogeneity in topics, content and structure, and there may be more variation in terms of where a key phrase may appear.


Case #2 shows a similar situation where the model with visual features finds the proper keyphrases that are much larger in font size, while the text-only model selects nouns elsewhere.

case #3 demonstrates a typical kind of web page where visual features can be misleading: an indexing page.


As a result, the relationships between entities are not captured.

since KB is naturally a graph structure (nodes are entities and edges are relations between entities).


Moreover, structural knowledge such as dependency relationships has recently been investigated on some tasks (e.g., relation extraction) (Peng et al., 2017;Song et al., 2018) and shown to be effective in the model's generalizability.

such dependency relationships (essentially also graph structure) have not been explored in dialogue systems, again missing great potential for improvements.


different predecessors in H ) should have different impacts on the output hidden state h t , and we expect our model to capture that.

the inputs may have different number of predecessors at different timesteps.


We can observe that our model without the graph encoder has a 1.6% absolute value loss (over 25% in ratio) in BLEU score and a 1.1% absolute value loss (9.8% in ratio) in entity F1 on MultiWOZ 2.1, which suggests that the overall quality of the generated sentences are better improved by our graph encoder.

ours without knowledge graph means that we do not use the graph structure to store and retrieve the external knowledge data.


These systems rely on offline (batch) training and have drawn recent criticism due to their inability to adapt to new contexts (Linzen, 2020).

humans acquire language from evolving environments, require a small memory footprint (Mc-Clelland et al., 1995), and can generalize their knowledge to newer tasks (Sprouse et al., 2013).


However, these works are mainly designed for image classification tasks where the training data has "clear" task boundaries-i.e., training stream are partitioned into disjoint subsequences.

task boundaries in VisCOLL are unknown and "smooth" (i.e., with gradual transitions between tasks)-a setting that is closer to real-world situations.


In particular, Li et al. (2020) study a closely related task of continual learning of sequence prediction for synthetic instruction following

their techniques for separating semantics and syntax is restricted to text-only case.


Prior work uses only the article content and metadata including title, date, domain, and authors.

news articles often contain photos and captions as well.


Despite the fact that leveraging metadata significantly improves the performance of Grover, it also appears that the accuracy does not vary much with the exclusion of different types of metadata.

we observe a surprising observation that leveraging all metadata causes the detection accuracy to decrease.


Todd Frazier's sacrifice fly accounted for the first run before Jose Bautista drove in the next two with a line drive RBI single to right, and a bases-loaded single by Todd Frazier also scored a run.

deJong and Bader homered off Bobby Wahl to begin the Cardinals' comeback.


Each sub-problem is worthy of being standardized and continually studied given a well defined objective and data sets so that the performance could be fairly evaluated and the progress can be continually made.

it is not easy in the current methodology, since each pipeline's strategies are closely bonded to own implementation.


Finetuning a pretrained language model (Dai and Le, 2015;Howard and Ruder, 2018) often delivers competitive performance partly because pretraining leads to a better initialization across various downstream tasks than training from scratch (Hao et al., 2019).

finetuning on individual NLP tasks is not parameter-efficient.


Mallya et al. (2018) explicitly update weights in a task-specific classifier layer.

we show that end-to-end learning of selective masks, consistently for both the pretrained language model and a randomly initialized classifier layer, achieves good performance.


Large-scale training datasets lie at the core of the recent success of neural machine translation (NMT) models.

the complex patterns and potential noises in the large-scale data make training NMT models difficult.


Neural machine translation (NMT) is a data-hungry approach, which requires a large amount of data to train a well-performing NMT model (Koehn and Knowles, 2017).


Another stream is to schedule the order of training examples according to their difficulty, e.g., curriculum learning which has been applied to the training of NMT models successfully (Kocmi and Bojar, 2017; Zhang et al., 2018; Platanios et al., 2019; Liu et al., 2020b).

we explore strategies to simplify the difficult (i.e., inactive) examples without changing the model architecture and model training strategy.


For the latter, NMT models tend to prefer a more typical alternative to a relatively rare but correct one (e.g., French "Il" is often wrongly translated to the more common "it" than "he" ).

However, these seemingly trivial errors can erode translation to the extent that they can be easily distinguishable from human-translated texts (Laubli et al. ¨ , 2018).


Both of the discriminative losses essentially promote the probability of the positive (i.e., correct) sample.

the intuition behind using the additional loss over the standard loss is that the fine-tuning here focuses on improving the positive sample over the negative sample that the model has learnt to produce, rather than over the entire probability distribution over the full vocabulary.


In the second example, we observe a biased anticipation  case where the NMT system had to emit a wrong translation chien ('dog') before seeing the noun 'bird'.

the multimodal model successfully leveraged the visual context for anticipation and correctly handled the adjective-noun placement phenomenon.


Our approach is most similar to Bjerva et al. (2019a), as they build a generative model from typological features and use language embeddings, extracted from factored language modelling at character-level, as a prior of the model to extend the language coverage.

our method primarily differs as it is mainly based in linear algebra, encodes information from both sources since the beginning, and can deal with a small number of shared entries (e.g. 23 from LW) to compute robust representations.


We work with 53 languages pre-processed by (Qi et al., 2018), from where we mapped the ISO 639-1 codes to the ISO 693-2 standard.

we need to manually correct the mapping of some codes to identify the correct language vector in the URIEL (Littell et al., 2017) library: • zh (zho , Chinese macro-language) mapped to cmn (Mandarin Chinese).


In German, we reach the maximum 0.41 when the number of words in each topic equals 2, and the minimum when it equals 100.

we observe the most noticeable changes when we vary the number of topics in French (Ousidhoum et al., 2019) such that B 1 = 0.34 when |T| = 2 versus 0.21 when |T| = 7 and back to 0.37 when |T| = 100.


On the other hand, we observe the most noticeable changes when we vary the number of topics in French (Ousidhoum et al., 2019) such that B 1 = 0.34 when |T| = 2 versus 0.21 when |T| = 7 and back to 0.37 when |T| = 100.

we remark overall cohesion despite the change in topic numbers especially in the case of Italian and Portuguese caused by the limited numbers of search keywords, that equal 5 and 7 respectively.


Waseem and Hovy (2016), Founta et al. (2018) and Ousidhoum et al. (2019) report using different keywords and hashtags to collect tweets.

the scores shown in Table 4 indicate that the datasets might carry similar meanings, specifically because WUP relies on hypernymy rather than common vocabulary use.


On the other hand, the copy model (row 3) significantly improves the BLEU scores by 36.2-37.6 points, by learning to re-use words in input texts 4 .

it still suffers the small data size, and its outputs are worse than the original questions without any transformation (row 1).


For system-level evaluation, metrics which can use the reference translations for quality estimation, such as BLEU, generally achieved consistently high correlation with human evaluation for all language pairs.

qE models (including our qE model and submitted systems for the qE as a Metric task) are not allowed to use the reference translations for quality estimation and tend to generate more unstable results: high correlation with human evaluation for some language pairs but very low or even negative Pearson correlation with human evaluation for some other language pairs.


As a sequence-to-sequence generation task, neural machine translation (NMT) naturally contains intrinsic uncertainty, where a single sentence in one language has multiple valid counterparts in the other.

the dominant methods for NMT only observe one of them from the parallel corpora for the model training but have to deal with adequate variations under the same meaning at inference.


Recently, there are increasing number of studies investigating the effects of quantifying uncertainties in different applications Kendall and Gal, 2017;Xiao and Wang, 2018;Zhang et al., 2019b,a;Shen et al., 2019).

most work in NMT has focused on improving accuracy without much consideration for the intrinsic uncertainty of the translation task itself.


when h tends to 0 our controlled sampling method achieves lowest BLEU scores but highest edit distances.

if we increase h gradually, it can be quickly simplified to greedy search.


A sequence-to-sequence (seq2seq) learning with neural networks empirically shows to be an effective framework for grammatical error correction (GEC), which takes a sentence with errors as input and outputs the corrected one.

the performance of GEC models with the seq2seq framework heavily relies on the size and quality of the corpus on hand.


The former applies text editing operations such as substitution, deletion, insertion and shuffle, to introduce noises into original sentences, and the latter trains a clean-to-noise model for error generation.

the noise-corrupted sentences are often poorly readable, which are quite different from those made by humans.


Once a vulnerable position is determined, the token at that position is usually replaced with one of its synonyms.

generating adversarial examples through such synonym-based replacement is no longer applicable to the GEC task.


Adversarial training by means of adding the adversarial examples into the training set can effectively improve the models' robustness.

some studies show that the models tend to overfit the noises, and the accuracy of the clean data will drop if the number of adversarial examples dominates the training set.


Indeed, LDA implicitly assumes that Ψ¨ = Unif(1, .., K) deterministically-i.e., that every topic is assumed a priori to contain the same number of tokens.

the HDP model learns this distribution from the data by letting Ψ ∼ GEM(γ).


Semisupervised methods that utilize such external corpora have been successful in English STS.

the need for external corpora is a major obstacle when applying STS, a fundamental technology, to low-resource languages.


One particularly promising usage of BERT-based models for unsupervised STS is BERTScore , which was originally proposed as an automatic evaluation metric.

our preliminary experiments 17 show that BERTScore performs poorly on unsupervised STS.


From the figure, we find that overall the average accuracy raises when K increases from 2 to 8, which suggests the importance of disentangling components.

when K grows larger than 8, the performance starts to decline.


As shown in Figure 6(b), except for the case of n = 1, the other settings have comparable performance.

it can be seen that when n = 4, the average accuracy on the last task is the highest, which indicates that the model has the strongest ability to avoid catastrophic forgetting problem when n = 4.


The subproblems are separately solved using existing techniques.

existing unsupervised multilingual approaches (Chen and Cardie, 2018;Heyman et al., 2019;Alaux et al., 2019) solve the above subproblems jointly.


We also observe that UMWE fails at mapping Dutch language embeddings in the multilingual space even though Dutch is close to English.

in a separate bilingual experiment, UMWE learns an effective English-Dutch crosslingual space (obtaining an average en-nl and nl-en score of 75.2).


We observe that the proposed SL-GeoMM learns a highly effective multilingual space and obtains the best overall result, illustrating its robustness in this challenging setting.

other multilingual approaches fail to learn a reasonably good multilingual space.


In recent years, pre-trained language models, such as GPT (Radford et al., 2018), BERT (Devlin et al., 2018), XL-Net (Yang et al., 2019), have been proposed and applied to many NLP tasks, yielding state-of-the-art performances.

the promising results of the pre-trained language models come with the high costs of computation and memory in inference, which obstruct these pre-trained language models to be deployed on resource-constrained devices and real-time applications.


Concretely, both the embedding-layer distillation and the prediction-layer distillation employ the one-to-one layer mapping as in TinyBERT and BERT-PKD, where the two student layers are guided by the corresponding teacher layers, respectively.

different from the previous works, we propose to exploit the many-to-many layer mapping for Transformer (intermediate lay-ers) distillation (attention-based distillation and hidden states based distillation), where each student attention layer (resp. hidden layers).


In this paper, we compare our BERT-EMD with several state-of-the-art BERT compression approaches, including the original 4/6-layer BERT models (Devlin et al., 2018), DistilBERT (Tang et al., 2019), BERT-PKD , Tiny-BERT (Jiao et al., 2019), BERT-of-Theseus (Xu et al., 2020).

the original TinyBERT employs a data augmentation strategy in the training process, which is different from the other baseline models.


Previous works have shown that as the number of retrieved passages increases, so does the performance of the reader.

they assume all retrieved passages are of equal importance and allocate the same amount of computation to them, leading to a substantial increase in computational cost.


Open-Domain Question Answering (ODQA) requires a system to answer questions using a large collection of documents as the information source.

to context-based machine comprehension, where models are to extract answers from single paragraphs or documents, it poses a fundamental technical challenge in machine reading at scale (Chen et al., 2017) .


Models for reading comprehension (RC) commonly restrict their output space to the set of all single contiguous spans from the input, in order to alleviate the learning problem and avoid the need for a model that generates text explicitly.

forcing an answer to be a single span can be restrictive, and some recent datasets also include multi-span questions, i.e., questions whose answer is a set of non-contiguous spans in the text.


When the answer spans appear only once in the input, this is simple, since the ground-truth tagging is immediately available.

there are many cases where a given answer span appears multiple times in the input.


For query generation, prior research has focused mostly on extending standard Seq2Seq models where the input is a concatenation of earlier queries a user has submitted in a session (Sordoni et al.,Figure 1: An example search session where a user issues queries and optionally performs clicking at timestamps 1 to n. At time n+1, the user issues q n+1 following the previous search context of length n. 2015; Dehghani et al., 2017).

literature often leaves out the influence of clickthrough actions (i.e., red blocks in Figure 1), which we argue should be taken into account in the generative process as they could be surrogates of the user's implicit search intent (Yin et al., 2016).


One of the widely exercised steps to establish a semantic understanding of social media is Entity Linking (EL), i.e., the task of linking entities within a text to a suitable concept in a reference Knowledge Graph (KG) (Liu et al., 2013;Yang and Chang, 2015;Yang et al., 2016; Ran et al., 2018).

it is well-documented that poorly composed contexts, the ubiquitous presence of colloquialisms, shortened forms, typing/spelling mistakes, and out-of-vocabulary words introduce challenges for effective utilisation of social media text (Baldwin et al., 2013;Michel and Neubig, 2018).


As noted by Ethayarajh (2019) the deeper BERT goes, the more "contextualized" its representation becomes.

interpreting semantics of entities requires contextual knowledge in different degrees and always taking the last layer's output may not be the best solution.


Compared to similar corpora, COMETA has the largest scale.

from a learning perspective the lack of sufficient regularity in the data could still leave its toll at test phase.


Much of the recent progress in NLP is due to the transfer learning paradigm in which Transformerbased models first try to learn task-independent linguistic knowledge from large corpora, and then get fine-tuned on small datasets for specific tasks.

these models are overparametrized: we now know that most Transformer heads and even layers can be pruned without significant loss in performance (Voita et al., 2019;Kovaleva et al., 2019;Michel et al., 2019).


There are recent works (Nguyen et al., 2019; Agarwal et al., 2020) also applying the Transformer to model the interactions among many entities.

their models neglect the important early interaction of the answer entity and cannot naturally leverage the pretrained language representations from BERT like ours.


In the example, both the baseline model and the model with the sub-instruction module completes the task successfully.

unlike the baseline model which fails to follow the instruction and stops within 3 meters of the target by chance, our model correctly identifies the completeness of each sub-instruction, guides the agent to walk on the described path and eventually stops right at the target position.


Although replacing a real user with a user simulator could address the issue, the simulator only roughly approximates real user statistics, and its development process is costly (Su et al., 2016).

humans could independently reason potential responses based on past experiences from the true environment.


We propose a model-agnostic approach, COPT, that can be applied to any adversarial learning-based dialogue generation models.

to existing approaches, it learns on counterfactual responses inferred from the structural causal model, taking advantage of observed responses.


As an important research issue in the natural language processing community, multi-label emotion detection has been drawing more and more attention in the last few years.

almost all existing studies focus on one modality (e.g., textual modality).


This implies that the success of previous models may over-rely on the confounding non-target aspects, but not necessarily on the target aspect only.

no datasets can be used to analyze the aspect robustness more in depth.


These two ratios should ideally be both 400%, because there are three generation strategies, plus one original sentence.

this gap is because not every original test sentence can qualify for every generation strategy.


That is to say, whether one sentence could be selected depends on its salience and the redundancy with other selected sentences.

it is still difficult to model the dependency exactly.


Neural models have achieved remarkable success on relation extraction (RE) benchmarks.

there is no clear understanding which type of information affects existing RE models to make decisions and how to further improve the performance of these models.


From the observations in Section 2, we know that both context and entity type information is beneficial for RE models.

in some cases RE models cannot well understand the relational patterns in context and rely on the shallow cues of entity mentions for classification.


Alt et al. (2020) also point out that there may exist shallow cues in entity mentions.

there have not been systematical analyses about the topic and to the best of our knowledge, we are the first one to thoroughly carry out these studies.


Other models selected operands first before constructing expression trees with operators in the second step (Roy et al., 2015; Roy and Roth, 2015).

such two-step procedures in these early attempts can be performed via a single-step procedure with neural models.


It has received significant attention in question answering systems for structured data (Wang et al., 2015; Zhong et al., 2017; Yu et al., 2018b; Xu et al., 2020).

training a semantic parser with good accuracy requires a large amount of annotated data, which is expensive to acquire.


Neural networks have the merits of convenient end-to-end training and good generalization, however, they typically need a lot of training data and are not interpretable.

logicbased expert systems are interpretable and require less or no training data.


A unique feature of this operationalisation of lexical ambiguity is that it is language independent.

the quality of a possible approximation will vary from language to language, depending on the models and the data available in that language.


If p(m | w) is concentrated in a small region of the meaning space (corresponding to a word with nuanced implementations of the same sense), the bound in eq. (13) could be relatively tight.

a word with several unrelated homophones would correspond to a highly structured p(m | w) (e.g. with multiple modes in far distant regions of the space) for which this normal approximation would result in a very loose upper bound.


Deep learning has led to significant improvement in text summarization with various methods investigated and improved ROUGE scores reported over the years.

gaps still exist between summaries produced by automatic summarizers and human professionals.


In the research literature, human evaluation has been conducted as a complement (Narayan et al., 2018).

human evaluation reports that accompany ROUGE scores are limited in scope and coverage.


The above methods assign one score to each summarization output.

to these methods, our errorcount based metrics are motivated by MQM for human writing, and are more fine-grained and informative.


On PolyTope, as a representative of abstractive models, BART overwhelmingly outperforms the others (p < 0.01 using t-test).

excluding BART, extractive models take the following top three places.


With respect to Accuracy, extractive methods are notably stronger in terms of Inacc Intrinsic and Extrinsic, which reflects that through directly copying snippets from the source, extractive methods are guaranteed to produce a summary with fair grammaticality, rationality and loyalty.

extractive methods do not show stronger performances in Addition and Omission, which is because extracted sentences contain information not directly relevant to the main points.


There is a high proportion in the first five sentences and a smooth tail over all positions for reference summaries.

bertSumExt and SummaRuN-Ner extract sentences mostly from the beginning, thereby missing useful information towards the end.


Compared with Point-Generator, Point-Generator-with-Coverage reduces Duplication errors from 68 to 11 and Omission errors from 286 to 256, proving that coverage is useful for better content selection.

point-Generator-with-Coverage yields more Addition and Inacc Intrinsic errors than point-Generator.


As can be seen, abstractive models tend to neglect sentences in the middle and at the end of source documents (e.g.,  Bottom-Up, BertSumExtAbs), indicating that performance of abstractive summarizers is strongly affected by the leading bias of dataset.

bART can attend to sentences all around the whole document, slightly closer to the distribution of golden reference.


Their main goal is to verify the faithfulness and factuality in abstractive models.

we evaluate both rule-based baselines and extractive/abstractive summarizers on 8 error metrics, among which faithfulness and factuality are included.


It has been conjectured that multilingual information can help monolingual word sense disambiguation (WSD).

existing WSD systems rarely consider multilingual information, and no effective method has been proposed for improving WSD by generating translations.


Our first method extends the idea of Apidianaki and Gong (2015) to constrain S(e) based on sensetranslation mappings in BabelNet.

instead of relying on a single translation, we incorporate multiple languages by taking the intersection of the individual sets of senses; that is, we rule out senses if their corresponding BabelNet synsets do not contain translations from all target languages.


Supervised systems are trained on sense-annotated corpora and generally outperform knowledge-based systems.

knowledge-based systems usually apply graph-based algorithms to a semantic network and thus do not require any sense-annotated corpora.


The ability to fuse sentences is highly attractive for summarization systems because it is an essential step to produce succinct abstracts.

to date, summarizers can fail on fusing sentences.


Fusing two sentences together coherently requires connective phrases and sometimes requires rephrasing parts of sentences.

higher abstraction does not mean higher quality fusions, especially in neural models.


Recent innovations in Transformer-based ranking models have advanced the state-ofthe-art in information retrieval.

these Transformers are computationally expensive, and their opaque hidden states make it hard to understand the ranking process.


One could argue that this is a superficial problem, as we can always give the model more free bits and decrease the loss in intermediary positions.

this is not so simple because increasing capacity leads to a worse model fit, as was noted by Alemi et al. (2018).


While Yang et al. (2017) and Kim et al. (2018) both consider the use of pretrained LMs as encoders, the weights are not frozen such that it is hard to disentangle the impact of pretraining from subsequent training.

we freeze the weights so that the effect of pretraining can not be overridden.


Both baselines and variants have roughly similarly high agreement.

our variants produce more diverse beginnings, while still managing to reproduce the topic or sentiment of the original document.


Finally, verb-shuffling focuses on verbs as the salient element of an event, and should teach both principles of verb ordering and of verb suitability for context, and avoid artifacts from reordering arguments.

since verbs are shuffled naively, the task can in some cases be too easy due to differences in verb selectional preferences.


In the vision and machine learning community, unsupervised induction of structured image representations (aka scene graphs or world models) has been receiving increasing attention (Eslami et al., 2016; Burgess et al., 2019; Kipf et al., 2020).

they typically rely solely on visual signal.


Previous work has shown that incorporating natural language explanation into the classification training loop is effective in various settings (Andreas et al., 2018; Mu et al., 2020).

previous work neglects the fact that there is usually a limited time budget to interact with domain experts (e.g., medical experts, biologists)  and high-quality natural language explanations are expensive, by nature.


For example, an image with a Ringbilled gull has the description: "This is a white bird with a grey wing and orange eyes and beak."

this description also fits perfectly with a California gull (Figure 1).


Our results indicate that too many parameters can also harm multilinguality.

in practice it is difficult to create a model with so many parameters that it is overparameterized when being trained on 104 Wikipedias.


One might argue that our model 17 in Table 1 of the main paper is simply not trained enough and thus not multilingual.

table 10 shows that even when continuing to train this model for a long time no multilinguality arises.


This suggests that multilingual models can stimulate positive transfer for low-resource languages when monolingual models overfit.

when we compare bilingual models on English, models trained using different sizes of fr/ru data obtain similar performance, indicating that the training size of the source language has little impact on negative interference on the target language (English in this case).


Unlike language-specific adapters that can hinder transferability, shared adapters improve both within-language and cross-lingual performance with the extra capacity.

meta adapters still obtain better performance.


They observe that having more languages results in better zero-shot performance.

several artifacts arise, as described by Dabre et al. (2020); Zhang et al. (2020); Aharoni et al. (2019); Arivazhagan et al. (2019), like offtarget translation and insufficient modeling capacity of the MNMT models.


This might indicate that -at least during the adaptation -important information is captured in the encoder's adapter layer (in line with previous reports by Kudugunta et al., 2019) or that the decoder adaptation grows dependent on the encoder adapters, to the point where dropping the latter degrades the system.

further analysis would be needed to confirm either of these hypotheses.


Not only is beam search usually more accurate than greedy search, but it also outputs a diverse set of decodings, enabling reranking approaches to further improve accuracy (Yee et al., 2019; Ng et al., 2019; Charniak and Johnson, 2005; Ge and Mooney, 2006).

it is challenging to optimize the performance of beam search for modern neural architectures.


FIXED-OURS is slower than Fairseq's implementation.

while the two implementations achieve more similar BLEU on the development set, FIXED-OURS achieves higher BLEU on the test set (49.75 vs 49.57 on De-En and 39.19 vs 38.98 on Ru-En).


The produced meaning representations can then potentially be used to improve downstream NLP applications (e.g., Issa et al., 2018;Song et al., 2019;Mihaylov and Frank, 2019), though the introduction of large pretrained language models has shown that explicit formal meaning representations might not be a necessary component to achieve high accuracy.

it is now known that these models lack reasoning capabilities, often simply exploiting statistical artifacts in the data sets, instead of actually understanding language (Niven and Kao, 2019;McCoy et al., 2019).


A possible advantage of this model is that it might handle longer sentences and documents better.

it might be harder to tune (Popel and Bojar, 2018) 2 and its improved performance has mainly been shown for large data sets, as opposed to the generally smaller semantic parsing data sets (Section 3.3).


For both methods, it results in a clear and significant improvement over the BERT-only baseline, 87.6 versus 88.1.

another common method of improving performance is adding linguistic features to the tokenlevel representations.


This was done for efficiency and memory purposes, it did not make a difference in terms of F1-score.

for the Transformer model this improved F1-score by around 0.5.


Data efficiency can be improved by optimizing pre-training directly for future fine-tuning with few examples; this can be treated as a meta-learning problem.

standard meta-learning techniques require many training tasks in order to generalize; unfortunately, finding a diverse set of such supervised tasks is usually difficult.


Bansal et al. (2019) proposed an approach that applies to diverse tasks to enable practical meta-learning models and evaluate on generalization to new tasks.

they rely on supervised task data from multiple tasks and suffer from meta-overfitting as we show in our empirical results.


Owing to the warp layers, our training time per step and the GPU memory footprint is lower than LEOPARD (Bansal et al., 2019).

our training typically runs much longer as the model doesn't overfit unlike LEOPARD (see learning rate trajectory in main paper).


As Wiki itself is a collaborative knowledge repository, editors are likely to attack others due to disputes on specific domain knowledge.

the users are the general public who post comments and tweets more casually for Yahoo and Twitter.


Multilingual contextual embeddings have demonstrated state-of-the-art performance in zero-shot cross-lingual transfer learning, where multilingual BERT is fine-tuned on one source language and evaluated on a different target language.

published results for mBERT zero-shot accuracy vary as much as 17 points on the MLDoc classification task across four papers.


Many models inspired by BERT have since surpassed its performance.

in contrast to the original BERT paper, many obtained better results by excluding the NSP task.


The decoupled biLSTM extended with ELMo inputs is able to outperform the transformer model initialised with RoBERTa pretraining.

the best performance is achieved by using the transformer model with BART-large pretraining, with the decoupled model fine-tuned jointly on top of it (Lewis et al., 2019).


The concept of Dialogue Act (DA) is universal across different task-oriented dialogue domains-the act of "request" carries the same speaker intention whether it is for restaurant reservation or flight booking.

dA taggers trained on one domain do not generalize well to other domains, which leaves us with the expensive need for a large amount of annotated data in the target domain.


It is often challenging and costly to obtain a large amount of in-domain dialogues with annotations.

unlabeled dialogue corpora in target domain can easily be curated from past conversation logs or collected via crowd-sourcing (Byrne et al., 2019;Budzianowski et al., 2018) at a more reasonable cost.


In prior work (Xie et al., 2019;Wei and Zou, 2019), unsupervised data augmentation methods including word replacement and backtranslation have been shown useful for short written text classification.

such augmentation methods are shown to be less effective (Shleifer, 2019) when used with pre-trained models.


We find that for both tense and mood in the Indo-Aryan family, our model identifies required-agreement primarily for conjoined verbs, which mostly need to agree only if they share the same subject.

subsequent analysis revealed that in the treebanks nearly 50% of the agreeing verbs do not share the same subject but do agree by chance.


Professor Tanja Kallio and doctoral candidate Sami Tuomi consider the realisation of this goal entirely possible.

"scientifically we are in the dark about the consequences of rewilding, and we worry about the general lack of critical thinking surrounding these often very expensive attempts at conservation.


Such pairs show consistent improvement (+5 to +10), which suggests that the model learns to align the parallel knowledge from the source language to the target language.

we also must note that the effect is strongly dependent on the size of the overlapping sets.


Students need good textbooks to study before they can pass an exam, and the same holds for a good machine reading model.

finding the information needed to answer a question, especially for questions in such a narrow domain as the subjects studied in high schools, usually requires a collection of specialized texts.


However, little work has looked further up the pipeline and relied on the assumption that biases in data originate in human cognition.

this assumption motivates our work: an unsupervised approach to detecting implicit gender bias in text.


Psychology studies often examine human perceptions through word associations (Greenwald et al., 1998).

the implicit nature of bias suggests that human annotations for bias detection may not be reliable, which motivates an unsupervised approach.


is likely addressed towards a woman and identify it as biased.

we only want the model to learn that references to appearance are indicative of gender if they occur in unsolicited contexts.


For example, humans need the supervision of what is a noun before they do POS tagging, or what is a tiger in Wordnet before they classify an image of tiger in ImageNet.

for NLI, people are able to entail that a A man plays a piano contradicts b A man plays the clarinet for his family without any supervision from the NLI labels.


Both depGCN and kumaGCN can correctly classify the sentiment of "service" as negative.

depGCN cannot recognize the positive sentiment of "atmosphere" while kumaGCN can.


For the target "atmosphere", depGCN assigns the highest weight to the word "terrible", which is an irrelevant sentiment word to this target, leading to an incorrect prediction.

our model assigns the largest weight to the key sentiment word "cozy", classifying it correctly.


Although collecting tags from users is timeconsuming and often suffers from coverage issues (Katakis et al., 2008), NLP techniques like those in Kar et al. (2018b) and Gorinski and Lapata (2018) can be employed to generate tags automatically from written narratives such as synopses.

existing supervised approaches suffer from two significant weaknesses.


August is selected to perform the rhapsody he's been composing at the same concert.

wizard, who found out about August's performance by Arthur, interrupts the rehearsal and claims to be his father, and manages to pull August out of the school.


Mozart would be an absolute imbecile compared to this little kid August Rush, and for those familiar with music, this aspect (the foundation, really) just kills the movie.It is impossible to play like Michael Hedges in your first few minutes with a guitar.

i just finished watching August Rush and i am in no way exaggerating when i say that it is by far the best movie i have ever seen.


Im not sure who would actually enjoy this movie, maybe if you're 70, or under 12 but for everyone else I'd save your time.The acting itself wasn't bad, though the more interesting characters were played by Terrence Howard and Robin Williams, and they were both severely under-developed as you wanted to know more about them and less about this kid with the stupid smile all the time..

while the movie has a modern setting, it shares many plot elements with OLIVER TWIST, ending even better.It begins with a young couple of musicians that meets and has a one-night stand, and when she becomes pregnant her dad does everything to make her believe that the child died at birth, although he just put the child for adoption.A decade later the boy, Evan, lives in a orphanage and is mocked by the other kids because of his talents in music, that makes him like a savant with powerful skills.


Although this film plays well to a broad audience, it is very mystical and based on simple, yet emotional themes that will play flat to some movie-goers.If you have strong parental feelings or enjoy movies centered on the power of human love and attraction, this story will move you like few films ever have.

if you are easily bored with themes that are lacking in danger and suspense or prefer gritty true-to-life movies, this one may come off as a disappointment.The screenplay seems written as a spiritual message intimating that there is an energy field that connects all of life, and music is one of the domains available to any who care to experience it.The plot is simple but deep in implication-an orphaned boy wants to reunite with his parents and feels that his inherited musical genius can somehow guarantee their return.


Graph convolutional networks (GCN) is demonstrated to be an effective approach to model such contextual information between words in many NLP tasks (Marcheggiani and Titov, 2017;Huang and Carley, 2019;De Cao et al., 2019); thus we want to determine whether this approach can also help CCG supertagging.

we cannot directly apply conventional GCN models to CCG supertagging because in most of the previous studies the GCN models are built over the edges in the dependency tree of an input sentence.


Having more training data can help reduce overfitting and improve model robustness.

preparing a large amount of annotated data is usually costly, labor intensive and time-consuming.


In this work, we will focus on denoising recurrent neural network autoencoders (Vincent et al., 2010;Shen et al., 2020; see Appendix A).

any advancement in this research direction will directly benefit our framework.


This observation may seem counter to the widely seen success of finetuning across other NLP scenarios, in particular with pretrained transformer models like BERT (Devlin et al., 2019).

finetuning does not always lead to better performance.


Recent work starts to use gradient (Michel et al., 2019;Ebrahimi et al., 2017) to guide the search for universal trigger (Wallace et al., 2019) that are applicable to arbitrary sentences to fool the learner, though the reported attack success rate is rather low or they suffer from inefficiency when applied to other NLP tasks.

our proposed T3 framework is able to effectively generate syntactically correct adversarial text, achieving high targeted attack success rates across different models on multiple tasks.


There are also systems (Luo et al., 2018; Kumar et al., 2019) that incorporate sense definitions into language models and achieve state-of-the-art performance.

most of the systems are implemented in a supervised manner using a widely exploited sense-annotated corpus, SemCor (Miller et al., 1994), and merging knowledge from the sense inventory as a supplement.


By using regex-based extractors and a list of comprehensive dictionaries that capture crucial domain vocabularies, LUSTRE can generate rules that achieve SoTA results.

for more complex and realistic scenarios, dictionaries may not be available and regex-based extractors alone are not expressive enough.


EMR and LwF can achieve competitive performance at the beginning.

the gap between the two baselines and our method KCN becomes wider as more new classes arrive.


Very few models exist that can predict either open vocab (Rashkin et al., 2018), or variable size output .

no existing task has both open vocabulary and variable-size low specificity-placing OPENPI in a novel space.


The SCoNE dataset (Long et al., 2016) contains paragraphs describing a changing world state in three synthetic, deterministic domains.

approaches developed using synthetic data often fail to handle the inherent complexity in language when applied to organic, real-world data (Hermann et al., 2015;Winograd, 1972).


For informativeness, we notice that all models perform well on the seen domains.

on unseen domains, the Naive approach fares poorly.


Very similar to our task, Kang et al. (2019) developed language models informed by discourse relations on the bridging task; given the first and last sentences, predicting the intermediate sentences (bidirectional flow).

they did not explicitly predict content words given context nor use them as a self-supervision signal in training.


As an alternative metric of attention explainablity, (Jain and Wallace, 2019) considers the relationship between attention weights and gradient-based feature importance score of each word.

prior research suggests using word as a unit of importance feature is rather artificial, as word is contextualized by, and interacts with other words: (Wiegreffe and Pinter, 2019) observes such limitation, and Shapley (Chen et al., 2018) measures interaction between features for capturing dependency of arbitrary subsets.


The GNN-based models are particularly strong in this setting (see Appendix C), and this suggests that transferring knowledge about the relevancy of facts from structured to unstructured models may be a promising direction.

at the same time, the improvements for generalization were less substantial, indicating that some reasoning capacities are difficult to distill in this manner.


This is not surprising as the generalization ability is a known issue in modern NLP models and is an ongoing research topic (Bahdanau et al., 2019; Andreas, 2019).

the generalization is in parallel with our contribution that is to improve the reasoning ability of NLP models.


Predictive methods such as probing are flexible: Any task with data can be assessed.

they only track predictability of pre-defined categories, limiting their descriptive power.


Since nPMI is information-theoretic and chance-corrected, it is a reliable indicator of the degree of information about gold labels contained in a set of predicted clusters.

it is relatively insensitive to cluster granularity (e.g., the total number of predicted categories, or whether a single gold category is split into many different predicted clusters), which is better understood through our other metrics.


On one hand, this reflects surface patterns: primary core arguments are usually close to the verb, with ARG0 on the left and ARG1 on the right; trailing arguments and modifiers tend to be prepositional phrases or subordinate clauses; and modals and negation are identified by lexical and positional cues.

this also reflects error patterns in state-of-the-art systems, where label errors can sometimes be traced to ontological choices in PropBank, which distinguish between arguments and adjuncts that have very similar meaning Kingsbury et al., 2002).


Most of the existing work has adopted static sentiment lexicons as linguistic resource (Qian et al., 2017; Chen et al., 2019), and equipped each word with a fixed sentiment polarity across different contexts.

the same word may play different sentiment roles in different contexts due to the variety of part-ofspeech tags and word senses.


We observe that our proposed MT-H-LSTM-CRF consistently outperforms the baseline models.

it performs slightly worse on RR-submission than on RR-passage, plausibly because there is no context information (i.e., background knowledge from original submissions) shared between different passage pairs.


Sentence [1] is non-hyperbolic because the fact that "her one step equals my two steps" is not anything that would be surprising to anyone.

if one changes the number from "two" to "100", then the resulting sentence becomes hyperbolic because in reality it is not possible that one person's step would equal another person's 100 steps.


Their proposal is similar to ours; they exclude attention weights that do not affect the output owing to the application of transformation f and input x in the analysis.

our proposal differs from theirs in some aspects.


The target word in (1 a) is associated 3 with the gold gloss (1 b) from WordNet (Fellbaum, 1998), the most used sense inventory in WSD.

generationary arguably provides a better gloss (1 c).


It is not surprising that machine learning methods can easily surpass human performance if sufficient data is available (Wang et al., 2018).

data acquisition is a challenging task for some special domains.


The core idea of our method is finding a different entity for intervening on an entity in the observational example.

finding a new entity set in a specific domain needs human efforts to collect entities, which has no difference from annotating more data.


Second, the training and test data in these benchmarks are sampled from the same corpus, and therefore the training data usually have high mention coverage on the test data, i.e., a large proportion of mentions in the test set have been observed in the training set.

it is obvious that this high coverage is inconsistent with the primary goal of NER models, which is expected to identify unseen mentions from new data by capturing the generalization knowledge about names and contexts.


This is because they only annotate named mentions but ignore nominal and pronominal mentions.

the context of named and nominal/pronominal mentions is generally identical, and therefore the models will be unable to distinguish between them once name regularity is removed.


Temporal KGs often exhibit multiple simultaneous non-Euclidean structures, such as hierarchical and cyclic structures.

existing embedding approaches for temporal KGs typically learn entity representations and their dynamic evolution in the Euclidean space, which might not capture such intrinsic structures very well.


More recently,  generalized manifolds of constant curvature to a product manifold combining hyperbolic, spherical, and Euclidean components.

these methods consider graph data as static models and lack the ability to capture temporally evolving dynamics.


Building upon recent NLI systems, our approach leverages representations from unsupervised pretraining, and finetunes a multiclass classifier over the BERT model (Devlin et al., 2019).

we first consider other models for related tasks.


Such results are problematic for entailment (since it is defined to depend on the truth of the premise).

our problem is primarily about the meaning of answers.


The approach achieves consistently better performance compared to the first two rows on the inter-domain structure prediction task (For both, original Parseval and RST-Parseval), as we have previously shown in Huber and Carenini (2019).

only considering two out of three nuclearity classes (N-S and S-N), the system performs rather poorly on the nuclearity classification task.


Our proposed new method (BERT-BASE-LWAN) that employs LWAN on top of BERT-BASE has the best results among all methods on EURLEX57K and AMAZON13K, when all and frequent labels are considered.

in both datasets, the results are comparable to BERT-BASE, indicating that the multi-head attention mechanism of BERT can effectively handle the large number of labels.


To identify whether an example is biased, they employ a shallow model f b , a simple model trained to directly compute p(y|b(x)), where the features b(x) are hand-crafted based on the task-specific knowledge of the biases.

obtaining the prior information to design b(x) requires a dataset-specific analysis (Sharma et al., 2018).


Training a shallow model The analysis suggests that we can obtain a substitute f b by taking a checkpoint of the main model early in the training, i.e., when the model has only seen a small portion of the training data.

we observe that the resulting model makes predictions with rather low confidence, i.e., assigns a low probability to the predicted label.


This indicates that unregularized training optimizes faster on certain examples, possibly due to the presence of biases.

self-debiased training maintains relatively less variability of losses throughout the training.


Closely related to our work, Singh et al. (2019) showed that replacing segments of the training data with their translation during fine-tuning is helpful.

they attribute this behavior to a data augmentation effect, which we believe should be reconsidered given the new evidence we provide.


For example, by framing the immigration issue using the morality frame or using the security frame, the reader is primed to accept the liberal or conservative perspectives, respectively.

as shown in Example 1, in some cases this analysis is too coarse grained, as both articles frame the issue using the economic frame, suggesting that a finer grained analysis is needed to capture the differences in perspective.


In Example 1, both texts use the Economic frame using same unigram indicator ('wage').

other words in the text can help identify the nuanced talking points (e.g., 'minimum wage' in case of left and 'stagnant wages' in case of right).


Availability of large-scale datasets has enabled the use of statistical machine learning in vision and language understanding, and has lead to significant advances.

the commonly used evaluation criterion is the performance of models on test-samples drawn from the same distribution as the training dataset, which cannot be a measure of generalization.


The goal of OOD generalization is to mitigate negative bias while learning to perform the task.

existing methods such as LMH (Clark et al., 2019) try to remove all biases between question-answer pairs, by penalizing examples that can be answered without looking at the image; we believe this to be counterproductive.


Notice that both mutations do not significantly change the input, most of the pixels in the image and words in the question are unchanged, and the type of reasoning required to answer the question is unchanged.

the mutation significantly changes the answer.


As expected, the batch-aware strategies, DAL and Core-Set, which were designed to increase diversity, are characterized by the most diverse batches, with DAL achieving the highest diversity values, demonstrating the success of using mini-queries (Gissin and Shalev-Shwartz, 2019) to reduce redundancy of the selected examples.

the other strategies tend to select less diverse batches, i.e., they are prone to choose redundant examples, especially in the imbalanced-practical scenario.


This roughly translates to increasing the throughput of the training process.

when performing inference on a single data point, the latency of making predictions seems to dominate the runtime (Jouppi et al., 2017).


Recent work has shown that contextualised embeddings pre-trained on large written corpora can be fine-tuned on smaller spoken language corpora to learn structures of spoken language (Tran et al., 2019).

for NLP tasks, fillers and all disfluencies are typically removed in pre-processing, as NLP models achieve highest accuracy on syntactically correct utterances.


An assumption one could make based on the work by Radford et al. (2019), is that with this model, the results for any further downstream task would be improved by the presence of fillers.

we observe that to predict the persuasiveness of the speaker (using the high level attribute of persuasiveness annotated in the dataset (Park et al., 2014)), following the same procedure as outlined in subsubsection 2.1.2, that fillers, in fact, are not a discriminative feature.


Stehwien and Vu (2017) and Stehwien et al. (2018) (henceforth, SVS18) showed that neural methods can perform comparably to traditional methods using a relatively small amount of speech context— just a single word on either side of the target word.

since pitch accents are deviations from a speaker’s average pitch, intensity, and duration, we hypothesize that, as in some non-neural models (e.g. Levow 2005; Rosenberg and Hirschberg 2009), a wider input context will allow the model to better determine the speaker’s baseline for these features an


Recent advances in deep learning present a promising prospect in multimodal stock forecasting by analyzing online news , and social media (Guo et al., 2018) to learn latent patterns affecting stock prices (Jiang, 2020).

the challenging aspect in stock forecasting is that most existing work treats stock movements to be independent of each other, contrary to true market function (Diebold and Yılmaz, 2014).


The news interview setting revolves around sets of questions and answers-naively, one may assume the interviewer to be the sole questioner.

media dialog has steadily deviated from this rigid structure, tending toward the broadly conversational (Fairclough, 1988).


Natural language inference (NLI) data has proven useful in benchmarking and, especially, as pretraining data for tasks requiring language understanding.

the crowdsourcing protocol that was used to collect this data has known issues and was not explicitly optimized for either of these purposes, so it is likely far from ideal.


Longer texts offer the potential for discourse-level inferences, the addition of which should yield a dataset that is more difficult, more diverse, and less likely to contain trivial artifacts.

one might expect that asking annotators to read full paragraphs should increase the time required to create a single example; time which could potentially be better spent creating more examples.


Giving annotators difficult and varying constraints could encourage creativity and prevent annotators from falling into patterns in their writing that lead to easier or more repetitive data.

as with the use of longer contexts in PARAGRAPH, this protocol risks substantially slowing the annotation process.


Our chief results on transfer learning are conclusively negative: All four interventions yield substantially worse transfer performance than our base MNLI data collection protocol.

we also observe promising signs that all four of our interventions help to reduce the prevalence of artifacts in the generated hypotheses that reveal the label.


We bring up textual entailment as a unified solver for such NLP problems.

current research of textual entailment has not spilled much ink on the following questions: (i) How well does a pretrained textual entailment system generalize across domains with only a handful of domainspecific examples?


Thus, various stress-testing datasets have been proposed that probe NLI models for simple lexical inferences (Glockner et al., 2018), quantifiers (Geiger et al., 2018), numerical reasoning, antonymy and negation (Naik et al., 2018).

despite the heavy usage of conjunctions in English, there is no specific NLI dataset that tests their understanding in detail.


We presented some initial solutions via adversarial training and a predicate-aware RoBERTa model, and achieved some reasonable performance gains on CONJNLI.

we also show limitations of our proposed methods, thereby encouraging future work on CONJNLI for better understanding of conjunctive semantics.


This method is later extended to a hierarchical setting with a pre-defined hierarchy (Meng et al., 2019); ConWea (Mekala and Shang, 2020) leverages contextualized representation techniques to provide contextualized weak supervision for text classification.

all these techniques consider only the text data and don't leverage metadata information for classification.


Note that the same hypothesis can also be made at the paragraph level.

a major limitation of this approach is that paragraph sizes vary widely, ranging from a single word to a considerably huge block of text.


We use bidirectional contextual representation (Devlin et al., 2018) for encoding article text.

contrary to document representation using BERT (Adhikari et al., 2019), which is not adequate for large text documents, we first segment articles organically based on sections.


Finally, there has been some work on directly training a model to extract entities and associated negation constraints (Bhatia et al., 2019).

these works usually assume the availability of good quality annotated negated entities.


This behavior unique to S C+R is safe for  the noisy data filtering task since it can successfully detect lower-quality pairs with high precision.

improperly underestimating some acceptable pairs (i.e., low recall) is one downside of S C+R , and we discuss its influences in Section 6.3.


TXtract (Karamanolakis et al., 2020) incorporated the categorical structure into the value tagging system.

these methods suffer from irrelevant articles and is not able to filter out noisy answers.


Similar to prior work (Lee et al., 2017), our training objective is to maximize the probability of the correct antecedent (cluster) for each mention span.

rather than considering all correct antecedents, we are only interested in the cluster for the most recent one.


These plots show that models have relatively modest memory usage during inference.

their usage grows in training, due to gradients and optimizer parameters.


This would "correct" the training objective to match prior work.

this did not have a noticeable effect on performance.


Likewise, we were able to train a competitive model for which only the SpanBERT encoder from Joshi et al. (2019) was retained and the span scorer and pairwise scorer were randomly initialized.

we opted not to use that for the full experiments because training was more expensive in time.


Neural generation models based on different strategies like softtemplate (Wiseman et al., 2018; Ye et al., 2020), copy-mechanism (See et al., 2017), content planning (Reed et al., 2018; Moryossef et al., 2019), and structure awareness Colin and Gardent, 2019) have achieved impressive results.

existing studies are primarily focused on fully supervised setting requiring substantial labeled annotated data for each subtask, which restricts their adoption in real-world applications.


The work closest to our concept is Switch-GPT-2 (Chen et al., 2020b), which fits the pre-trained GPT-2 model as the decoder part to perform table-to-text generation.

their knowledge encoder is still trained from scratch, which compromises the performance.


GPT (Radford, 2018) and GPT-2 (Radford et al., 2019) use a leftto-right Transformer decoder to generate a text sequence token-by-token, which lacks an encoder to condition generation on context.

MASS (Song et al., 2019) and BART (Lewis et al., 2019) both employ a Transformer-based encoderdecoder framework, with a bidirectional encoder over corrupted (masked) text and a left-to-right decoder reconstructing the original text.


However, these autoencoding methods are not applicable to text generation where bidirectional contexts are not available.

an autoregressive model, such as GPT (Radford, 2018; Radford et al., 2019), is only trained to encode unidirectional context (either forward or backward).


A difference between UniLMs and PALM is that UniLMs are not fully autoregressive in the pre-training process.

pALM reduces the mismatch between pre-training and context-conditioned generation tasks by forcing the decoder to predict the continuation of text input on an unlabeled corpus.


For instance in Figure 1, players might prefer the command "move rug" over "knock on door" since the door is nailed shut.

even the state-of-the-art game-playing agents do not incorporate such priors, and instead rely on rule-based heuristics (Hausknecht et al., 2019a) or handicaps provided by the learning environment (Hausknecht et al., 2019a;Ammanabrolu and Hausknecht, 2020) to circumvent these issues.


In a slightly different setting, Urbanek et al. (2019) trained BERT (Devlin et al., 2018) to generate contextually relevant dialogue utterances and actions in fantasy settings.

these approaches are game-specific and do not use any reinforcement learning to optimize gameplay.


When k is small, CALM (n-gram) benefits from its strong action assumption of one verb plus one object.

this assumption also restricts CALM (n-gram) from generating more complex actions (e.g.  ‘open case with key’) that CALM (GPT2) can produce.


It is really the complex actions captured when k > 10 that makes GPT-2 much better than ngram.

though k = 20, 30, 40 achieve similar overall performance, they achieve different results for different games.


It is interesting that CALM (w/ Jericho) is significantly better than CALM (GPT-2) on the games of Temple and Deephome (non-trivial scores achieved), which are not the games with ClubFloyd scripts added.

games like 905 and moonlit have scripts added, but do not get improved.


Natural Language Generation (NLG) is a challenging problem in Natural Language Processing (NLP)-the complex nature of NLG tasks arise particularly in the output space.

to text classification or regression problems with finite output space, generation could be seen as a combinatorial optimization problem, where we often have exponentially many options |V | (here |V | is the size of the vocabulary and is the sentence length).


It is possible to stop training the decomposition model based on downstream QA accuracy.

training a QA model on each decom-position model checkpoint (1) is computationally expensive and (2) ties decompositions to a specific, downstream QA model.


The similarity between BERT sentence embeddings can be reduced to the similarity between BERT context embeddings h T c h c 2 .

as shown in Equation 1, the pretraining of BERT does not explicitly involve the computation of h T c h c .


Note that BERT sentence embeddings are produced by averaging the context embeddings, which is a convexitypreserving operation.

the holes violate the convexity of the embedding space.


We argue that NATSV can help eliminate anisotropy but it may also discard some useful information contained in the nulled vectors.

our method directly learns an invertible mapping to isotropic latent space without discarding any information.


The transferred model both increased the quality and diversity of the generation.

the transferred model exhibits narrower vocabulary usage.


Statistic-based automatic metrics, such as BLEU (Papineni et al., 2002), mostly rely on the degree of word overlap between a dialogue response and its corresponding gold response.

due to the ignorance of the underlying semantic of a response, they are biased and correlate poorly with human judgements in terms of response coherence (Liu et al., 2016).


For example, BLEU computes the geometric average of the n-gram precisions.

they can not cope with the one-to-many problem and have weak correlations with human judgements (Liu et al., 2016).


ADEM proposed by Lowe et al. (2017) achieves higher correlations with human judgements than the statistic-based metrics, which is trained with human-annotated data in a supervised manner.

it is time-consuming and expensive to obtain large amounts of annotated data.


From the example in the first row, we can see that the score given by our metric is closer to the human score than the other two baseline metrics.

in the second-row example, our metric performs poorly.


In this hard case, the topics of the model response are relevant to the dialogue context so that both our GRADE and BERT-RUBER, as learning-based metrics, deem that the response greatly matches the context.

the truth is that the model response is more likely a response for the previous utterance U1 rather than U2, which is hard for metrics to recognize.


Such tasks include probing syntax (Hewitt and Manning, 2019; Lin et al., 2019; Tenney et al., 2019a), semantics (Yaghoobzadeh et al., 2019), discourse features (Chen et al., 2019; Liu et al., 2019; Tenney et al., 2019b), and commonsense knowledge (Petroni et al., 2019; Poerner et al., 2019).

appropriate criteria for selecting a good probe is under debate.


BLEURT (Sellam et al., 2020) applies fine tuning of BERT, including training on prior human judgements.

our work exploits parallel bitext and doesn't require training on human judgements.


We find that a copy of the input is almost as probable as beam search output for the Prism model.

the model trained on ParaBank 2 prefers its own beam search output to a copy of the input.


We find that the probability of sys as estimated by an LM, as well as and the cosine distance between LASER embeddings of sys and ref, both have decent correlation with human judgments and are complementary.

cosine distance between LASER embeddings of sys and src have only weak correlation.


Its creation is a manifestation of creativity, and, as such, hard to automate.

since the development of creative machines is a crucial step towards real artificial intelligence, automatic poem generation is an important task at the intersection of computational creativity and natural language generation, and earliest attempts date back several decades; see Goncalo Oliveira (2017) for an overview.


Since those differ significantly in style from the poems in KnownTopicPoems and Unknown-TopicPoems, we do not train our language model directly on them.

we make use of the fact that sonnets follow a known rhyming scheme, and leverage them to train a neural model to produce rhymes, which will be explained in detail in Subsection 3.2.


In particular, the generated poems seem to be more fluent and coherent than the alternatives.

they do not relate to any specific topic, which probably causes the drop in quality for poeticness, where this model always performs worse than NeuralPoet.


The increase of K continues to bring benefits until K = 4.


Neural Graph Encoding Graph Attention Networks (GAT) (Velickovic et al., 2018) incorporates attention mechanism in feature aggregation, RGCN (Schlichtkrull et al., 2018) proposes relational message passing which makes it applicable to multi-relational graphs.

they only perform single-hop message passing and cannot be interpreted at path level.


RGCNs (Schlichtkrull et al., 2018) generalize GCNs by performing relationspecific aggregation, making it applicable to multirelational graphs.

these models do not distinguish the importance of different neighbors or relation types and thus cannot provide explicit relational paths for model behavior interpretation.


This problem is clearly related a continual learning (CL) (Chen and Liu, 2018; Parisi et al., 2019; Li and Hoiem, 2017; Wu et al., 2018; Schwarz et al., 2018; Hu et al., 2019; Ahn et al., 2019), which also aims to learn a sequence of tasks incrementally.

the main objective of the current CL techniques is to solve the catastrophic forgetting (CF) problem (McCloskey and Cohen, 1989).


That is, in learning each new task, the network parameters need to be modified in order to learn the new task.

this modification can result in accuracy degradation for the previously learned tasks.


For sentiment classification, recent deep learning models have been shown to outperform traditional methods (Kim, 2014;Devlin et al., 2018;Shen et al., 2018;Qin et al., 2020).

these models don't retain or transfer the knowledge to new tasks.


MTL is often considered the upper bound of continual learning because it trains all the tasks together.

its loss is the sum of the losses of all tasks, which does not mean it optimizes for every individual task.


Automated radiology report generation has the potential to reduce the time clinicians spend manually reviewing radiographs and streamline clinical care.

past work has shown that typical abstractive methods tend to produce fluent, but clinically incorrect radiology reports.


In this work we focused on developing abstractive techniques as was done by past work on the MIMIC-CXR dataset (Liu et al., 2019; Boag et al., 2019).

in the future we intend to combine the abstractive methods developed in this work with retrieval methods to further improve upon our framework.


Sentence fusion has the lowest TER, indicating that obtaining the fused targets requires only a limited number of local edits.

these edits require modeling the discourse relation between the two input sentences, since a common edit type is predicting the correct discourse connective (Geva et al., 2019).


Meng and Rumshisky (2018) propose a global context layer (GCL) to store/read the solved TLINK history upon a pre-trained pair-wise classifier.

they find slow converge when training the GCL and pair-wise classifier simultaneously.


Meanwhile, a two-way deliberation decoder (Xia et al., 2017) was used for response generation.

the relationship between the dialogue history and the last utterance is not well studied.


As a result, capturing the incongruity between modalities is significant for multi-modal sarcasm detection.

the existing models for multi-modal sarcasm detection either concatenate the features from multi modalities (Schifanella et al., 2016) or fuse the information from different modalities in a designed manner (Cai et al., 2019).


Thus, an effective sarcasm detector is beneficial to applications like sentiment analysis, opinion mining (Pang and Lee, 2007), and other tasks that require people's real sentiment.

the figurative nature of sarcasm makes it a challenging task (Liu, 2010).


For the store owner, the task of correctly identifying the buying-intent utterances is paramount.

the number of utterances related to searching for products is expected to be significantly higher, thus biasing the classifier toward this intent.


Other approaches for data balancing can include weak-labeling of available unlabeled data (Ratner et al., 2020), or even active learning (Settles, 2009).

both of these approaches require additional domain data which is not always available.


We focused our evaluation on the Semantic Utterance Classification (SUC) domain which is characterized by highly imbalanced data.

it is desirable to validate the applicability of our general balancing approach on other textual domains.


Knowledge+BERT turns out to be the strongest baseline, outperforming the other three baselines, which also shows the importance of leveraging external knowledge for the OAC2 task.

our model achieves superior performance over Knowledge+BERT which indicates leveraging domain-specific knowledge indeed helps.


This is expected as these tweets include less standard words, such as insults.

except for perhaps emotion detection and offensive language identification, the difference is not significant, considering that the original RoBERTa tokenizer was not trained on Twitter text.


In normal attention, all n-grams are weighted globally and short n-grams may dominate the attention because they occur much more frequently than long ones and are intensively updated.

there are cases that long n-grams can play an important role in parsing when they carry useful context and boundary information.


These results could be explained by that frequent short n-grams dominate the general attentions so that the long ones containing more contextual information fail to function well in filling the missing information in the span representation, and thus harm the understanding of long spans, which results in inferior results in complete match score.

the categorical span attention is able to weight n-grams in different length separately, so that the attentions are not dominated by high-frequency short n-grams and thus reasonable weights can be assigned to long n-grams.


This is not surprising because short n-grams occur more frequently and are thus updated more times than long ones.

the models with CATSA show a different weight distribution (the blue bars) among n-grams with different lengths, which indicates that the CATSA module could balance the weights distribution and thus enable the model to learn from infrequent long n-grams.


Since the distances between the boundary positions of the wrongly predicted spans (highlighted in red) are relatively long, the baseline system, which simply represents the span as subtraction of the hidden vectors at the boundary positions, may fail to capture the important context information within the text span.

the span representations used in our model are enhanced by weighted n-gram information and thus contain more context information.


Because training with soft targets provides smoother output distribution, T/S learning could outperform the single model training (Li et al., 2014;Hinton et al., 2015;Meng et al., 2018).

does a teacher always outperform a student?


All models benefit from the increasing of D as expected.

it is clear that Recurrence Online is the best performing model when D is small.


We observe that the use of Static Rebalancing (Equation 6) instead, which is an extreme version of AISLe, is better than not resampling at all.

it is unable to reach the performance of AISLe on coverage metrics.


Kreutzer and Sokolov (2018) proposed to jointly learn to segment and translate by using hierarchical RNN (Graves, 2016), but the method is not model-agnostic and slow due to the increased sequence length of characterlevel inputs.

our method is model-agnostic and operates on the word-level.


Kudo (2018) also report scores using n-best decoding, which averages scores from n-best segmentation results.

n-best decoding is n-times time consuming compared to the standard decoding method.


Starting from machine translation, it has been shown that subword regularization can improve the robustness of NLP models in various tasks (Kim, 2019;Provilkov et al., 2019;Drexler and Glass, 2019;Müller et al., 2019).

subword regularization relies on the unigram language models to sample candidates, where the language models are optimized based on the corpus-level statistics from training data with no regard to the translation task objective.


SurfCon (Wang et al., 2019b) discovered synonyms on privacy-aware clinical data by utilizing the surface form information and the global context information.

they suffer from either low precision or low recall.


In this example, the word victim in the first English sentence is identified by our tagger as a human entity.

its French translation victime is feminine by definition, and cannot be assigned another gender regardless of the context, causing a false positive result.


Furthermore, they explain IBT as a way to better approximate the true posterior distribution with the target-to-source model.

it is unclear how their heuristic objective relates to the ideal objective of maximizing the model's marginal likelihood of the target language monolingual data.


Deep neural models have demonstrated promising results in text classification tasks (Kim, 2014;Zhang et al., 2015;Howard and Ruder, 2018), owing to their strong expressive power and less requirement for feature engineering.

the deeper and more complex the neural model, the more it is essential for them to be trained on substantial amount of training data.


Several natural language processing methods, including deep neural network (DNN) models, have been applied to address this problem.

these methods were trained with hard-labeled data, which tend to become over-confident, leading to degradation of the model reliability.


In the training step, the model is trained to maximize the output probability of the correct class.

some studies reported that the deep learning classifier trained with hard-labeled data (1 for correct class, 0 for else) tends to become over-confident (Nixon et al., 2019;Thulasidasan et al., 2019).


These deep learning methods can effectively generate abstractive document summaries by directly optimizing pre-defined goals.

the meeting summarization task inherently bears a number of challenges that make it more difficult for end-to-end training than document summarization.


As shown in Figure 1, the medical report generation system should generate correct and concise reports for the input images.

data imbalance may reduce the quality of automatically generated reports.


To improve the clinical correctness of the generated reports, Liu et al. (2019a) and Irvin et al. (2019) adopted clinically coherent rewards for RL with CheXpert Labeler (Irvin et al., 2019), a rule-based finding mention annotator

in the medical domain, no such annotator is available in most cases other than English chest X-ray reports.


The input data, which comprise a set of finding labels, can be augmented easily by adding or removing a finding label automatically.

the augmentation cost is higher for the target reports than the input data because the target reports are written in natural language.


In addition to (c) above, we apply the modification process to the finding labels predicted by the image diagnosis module.

it is too expensive to evaluate the model in this condition because the cost of radiologist services is too high.


For comparison with the previous image captioning approaches, we used BLEU-1, BLEU-2, BLEU-3, and BLEU-4 metrics calculated by the nlg-eval 10 library.

word-overlap based metrics, such as BLEU, fail to assume the factual correctness of generated reports.


In ERNIE, entity embeddings are learned by TransE (Bordes et al., 2013), which is a popular transitionbased method for knowledge representation learning (KRL).

transE cannot deal with the modeling of complex relations , such as 1-to-n, n-to-1 and n-to-n relations.


An intuitive way for vanilla GCN to exploit these labels is to encode different types of dependency relation with different convolutional filters, which is similar to RGCN (Kipf and Welling, 2017).

rGCN suffers from over-parameterization, where the number of parameters grows rapidly with the number of relations.


We considered each sentence as a single claim to keep our experimental setting clean and avoid noise from an automatic claim extractor.

some generations contain multiple claims that could be independently assessed.


By thorough error analysis, we realize that for the order h-t-r (t-h-r follows the same logic), the model has to predict all t with regard to h in the second time step, without constraints from the r, and this makes every possible entity to be a prediction candidate.

the model is unable to eliminate no-relation entity pairs at the third time step, thus the model is prone to feed entity pairs to the classification layer with an low odds (low recall) but high confidence (high precision).


Over the past two decades, significant progress has been made in the development of word embedding techniques (Lund and Burgess, 1996; Bengio et al., 2003; Bullinaria and Levy, 2007; Mikolov et al., 2013b; Pennington et al., 2014).

existing word embedding methods do not handle numerals adequately and cannot directly encode the numeracy and magnitude of a numeral Naik et al. (2019).


These studies conclude that when certain nouns are dropped from the dominant language modality, multimodal models are capable of properly using the semantics provided by the image.

unlike this work, their explorations are limited to nouns and not expanded to other types of words.


They demonstrate high compression rate with little loss of performance.

they compress only the input embedding and not the softmax layer for language modeling and machine translation.


Specifically, they regard each segmentation criterion as a single  task under the framework of multi-task learning, where a shared layer is used to extract the criteriainvariant features, and a private layer is used to extract the criteria-specific features.

it is unnecessary to use a specific private layer for each criterion.


As shown in several previous research (Pang et al., 2016;Yang et al., 2016;Mitra et al., 2017;Xiong et al., 2017;Devlin et al., 2018), interaction-focused models usually achieve better performances for text pair tasks.

it is difficult to serve these types of models for applications involving large inference sets in practice.


However, it is difficult to serve these types of models for applications involving large inference sets in practice.

text embeddings from dual encoder models can be learned independently and thus pre-computed, leading to faster inference efficiency but at the cost of reduced quality.


Recently the PreTTR model (MacAvaney et al., 2020) aimed to reduce the query-time latency of deep transformer networks by pre-computing part of the document term representations.

their model still required modeling the full document/query input length in the head, thus limiting inference speedup.


One of the most effective ways to reduce the running time is to reduce the input sequence length.

as Table 4 reveals, blindly truncating the input to a BERT model will lead to a quick performance drop.


When the head is transformerbased, the two-stage training plays an important role: the AUC ROC improves from 0.891 to 0.930.

the gain introduced by using two-stage training is less significant in other approaches such as DE-FFNN and DIPAIRFFNN.


Recall that, in our framework, each encoder outputs its first few token embeddings as the input to the head, and we end to end to train the model to force the encoder to push the information of the input text into those outputted embeddings.

it is unclear to us what those outputted embeddings actually learn.


We formalize word reordering as a combinatorial optimization problem to find the permutation with the highest probability estimated by a POS-based language model.

it is computationally difficult to obtain the optimal word order.


With some classifiers, we reached the same F1-score as when training on the original dataset, which is 20x larger.

performance varied markedly between classifiers.


This demonstrates that GPT-2 significantly increased the vocabulary range of the training set, specifically with offensive words likely to be relevant for toxic language classification.

there is a risk that human annotators might not label GPT-2-generated documents as toxic.


TABLE-BERT is a BERT-base model that similar to our approach directly predicts the truth value of the statement.

the model does not use special embeddings to encode the table structure but relies on a template approach to format the table as natural language.


In language tasks, adversarial training brings wordlevel robustness by adding input noise, which is beneficial for text classification.

it lacks sufficient contextual information enhancement and thus is less useful for sequence labelling tasks such as chunking and named entity recognition (NER).


Masked language model (Devlin et al., 2018) smooths this inconsistency by applying replacement of tokens for some data while masking the rest (equivalent to word dropout).

the replacement in masked language model is randomly chosen from the full vocabulary, but the substitution in real scenarios follows some distribution (e.g. replacing “Massachusetts” with a location name is more likely than an animal name), which is not considered in masked language model.


Zhao et al. (2018) proposed the learning scheme to generate a gender-neutral version of Glove, called GN-Glove, which forces preserving the gender information in pre-specified embedding dimensions while other embedding dimensions are inferred to be gender-neutral.

learning new word embeddings for large-scale corpus can be difficult and expensive.


Table 4 shows that there are constant performance degradation effects for all debiasing methods from the original embedding.

our methods minimized the degradation of performances across the baseline models.


A clear-cut solution to this problem is to focus more on samples that are more relevant to the target task during pretraining.

this requires a task-specific pretraining, which in most cases is computational or time prohibitive.


Further, Moreo et al. (2019) concatenates label embedding with word embeddings.

this approach cannot be directly implemented into PLMs since the new (concatenated) embedding is not compatible with the pretrained parameters.


The growing interest in argument mining and computational argumentation brings with it a plethora of Natural Language Understanding (NLU) tasks and corresponding datasets.

as with many other NLU tasks, the dominant language is English, with resources in other languages being few and far between.


The ZS and TT baselines are almost always outperformed by the best translate-train model.

when a large-scale English corpus is available (Figure 2b), the TT baseline becomes comparable to the best translate-train models.


Dialogue policy learning for Task-oriented Dialogue Systems (TDSs) has enjoyed great progress recently mostly through employing Reinforcement Learning (RL) methods.

these approaches have become very sophisticated.


With respect to success rate, DiaAdv manages to achieve the highest performance by 6% compared to the second highest method GDPL.

diaAdv is not able to beat GdPL in terms of average turns.


As to DiaSeq, it can achieve almost the same  performance as GDPL from different perspectives while GDPL has a slightly higher F1 score.

the potential cost benefits of DiaSeq are huge since it does not require a user simulator in the training loop.


Beyond this, DiaMultiClass does not benefit from the increase in expert dialogues and starts to fluctuate between 55% and 59%.

di-aSeq can achieve higher performance when there are only 10% expert dialogue pairs and the success rate increases with the number of available expert dialogues.


The proposed methods can achieve state-of-the-art performance suggested by existing approaches based on Reinforcement Learning (RL) and adversarial learning.

we have demonstrated that our methods require fewer training efforts, namely the domain knowledge needed to design a user simulator and the intractable parameter tuning for RL or adversarial learning.


Our evaluation settings hiding one of the three inputs to the MCQA models -are similar to Kaushik and Lipton 2018's partial input settings which were designed to point out the existence of dataset artifacts in reading comprehension datasets.

we argue that our results additionally point to a need for more robust training methodologies and propose an improved training approach.


Among these, hyperedge replacement grammar (HRG) has been explored for parsing into semantic graphs (Habel, 1992;Chiang et al., 2013).

parsing with HRGs is not practical due to its complexity and large number of possible derivations per graph (Groschwitz et al., 2015).


Drawing on this result, a recent work by Fancellu et al. (2019) introduces recurrent neural network RDGs, a sequential decoder that models graph generation as a rewriting process with an underlying RDG.

despite the promising framework the approach in FA19 2 falls short in several aspects.